You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Amin Mohammed-Coleman <am...@gmail.com> on 2009/02/16 21:37:47 UTC
Faceted Search using Lucene
Hi
I am looking at building a faceted search using Lucene. I know that Solr
comes with this built in, however I would like to try this by myself
(something to add to my CV!). I have been looking around and I found that
you can use the IndexReader and use TermVectors. This looks ok but I'm not
sure how to filter the results so that a particular user can only see a
subset of results. The next option I was looking at was something like
Term term1 = new Term("brand", "ford");
Term term2 = new Term("brand", "vw");
Term[] termsArray = new Term[] { term1, term2 };un
int[] docFreqs = indexSearcher.docFreqs(termsArray);
The only problem here is that I have to provide the brand type each time a
new brand is created. Again I'm not sure how I can filter the results here.
It may be that I'm using the wrong api methods to do this.
I would be grateful if I could get some advice on this.
Cheers
Amin
P.S. I am basically trying to do something that displays the following
Personal Contact (23) Business Contact (45) and so on..
Re: Faceted Search using Lucene
Posted by Michael McCandless <lu...@mikemccandless.com>.
So then all is good.
We were only pursuing this to explain it. Now that we know your
directories are empty, that explains it.
So you should call maybeReopen() inside get(), as long as it does not
slow queries down.
Mike
Amin Mohammed-Coleman wrote:
> I think that is the case. When my SearchManager is initialised the
> directories are empty so when I do a get() nothing is present.
> Subsequent
> calls seem to work. Is there something I can do? or do I accept
> this or
> just do a maybeReopen and do a get(). As you mentioned it depends on
> timiing but I would be keen to know what the best practice would be
> in this
> situation...
>
> Cheers
>
> On Mon, Mar 2, 2009 at 8:43 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>>
>> Well the code looks fine.
>>
>> I can't explain why you see no search results if you don't call
>> maybeReopen() in get, unless at the time you first create
>> SearcherManager
>> the Directories each have an empty index in them.
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> Hi
>>> Here is the code that I am using, I've modified the get() method to
>>> include
>>> the maybeReopen() call. Again I'm not sure if this is a good idea.
>>>
>>> public Summary[] search(final SearchRequest searchRequest)
>>> throwsSearchExecutionException {
>>>
>>> final String searchTerm = searchRequest.getSearchTerm();
>>>
>>> if (StringUtils.isBlank(searchTerm)) {
>>>
>>> throw new SearchExecutionException("Search string cannot be empty.
>>> There
>>> will be too many results to process.");
>>>
>>> }
>>>
>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>
>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>
>>> stopWatch.start();
>>>
>>> MultiSearcher multiSearcher = get();
>>>
>>> try {
>>>
>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>
>>> Query query = queryParser.parse(searchTerm);
>>>
>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query
>>> '" +
>>> query.toString() +"'");
>>>
>>> Sort sort = null;
>>>
>>> sort = applySortIfApplicable(searchRequest);
>>>
>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>
>>> ChainedFilter chainedFilter = null;
>>>
>>> if (filters != null) {
>>>
>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>
>>> }
>>>
>>> TopDocs topDocs = multiSearcher.search(query,chainedFilter ,
>>> 100,sort);
>>>
>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>
>>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>>> "+topDocs.
>>> totalHits);
>>>
>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>
>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>
>>> float score = scoreDoc.score;
>>>
>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>
>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>
>>> summaryList.add(documentSummary);
>>>
>>> }
>>>
>>> } catch (Exception e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> } finally {
>>>
>>> if (multiSearcher != null) {
>>>
>>> release(multiSearcher);
>>>
>>> }
>>>
>>> }
>>>
>>> stopWatch.stop();
>>>
>>> LOGGER.debug("total time taken for document seach: " +
>>> stopWatch.getTotalTimeMillis() + " ms");
>>>
>>> return summaryList.toArray(new Summary[] {});
>>>
>>> }
>>>
>>>
>>> @Autowired
>>>
>>> public void setDirectories(@Qualifier("directories")ListFactoryBean
>>> listFactoryBean) throws Exception {
>>>
>>> this.directories = (List<Directory>) listFactoryBean.getObject();
>>>
>>> }
>>>
>>> @PostConstruct
>>>
>>> public void initialiseDocumentSearcher() {
>>>
>>> StopWatch stopWatch = new StopWatch("document-search-initialiser");
>>>
>>> stopWatch.start();
>>>
>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>> PerFieldAnalyzerWrapper(
>>> analyzer);
>>>
>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>> newKeywordAnalyzer());
>>>
>>> queryParser =
>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>> analyzerWrapper);
>>>
>>> try {
>>>
>>> LOGGER.debug("Initialising document searcher ....");
>>>
>>> documentSearcherManagers = new
>>> DocumentSearcherManager[directories.size()];
>>>
>>> for (int i = 0; i < directories.size() ;i++) {
>>>
>>> Directory directory = directories.get(i);
>>>
>>> DocumentSearcherManager documentSearcherManager =
>>> newDocumentSearcherManager(directory);
>>>
>>> documentSearcherManagers[i]=documentSearcherManager;
>>>
>>> }
>>>
>>> LOGGER.debug("Document searcher initialised");
>>>
>>> } catch (IOException e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> }
>>>
>>> stopWatch.stop();
>>>
>>> LOGGER.debug("Total time taken to initialise DocumentSearcher '" +
>>> stopWatch.getTotalTimeMillis() +"' ms.");
>>>
>>> }
>>>
>>> private void maybeReopen() throws SearchExecutionException {
>>>
>>> LOGGER.debug("Initiating reopening of index readers...");
>>>
>>> for (DocumentSearcherManager documentSearcherManager :
>>> documentSearcherManagers) {
>>>
>>> try {
>>>
>>> documentSearcherManager.maybeReopen();
>>>
>>> } catch (InterruptedException e) {
>>>
>>> throw new SearchExecutionException(e);
>>>
>>> } catch (IOException e) {
>>>
>>> throw new SearchExecutionException(e);
>>>
>>> }
>>>
>>> }
>>>
>>> LOGGER.debug("reopening of index readers complete.");
>>>
>>> }
>>>
>>>
>>>
>>> private void release(MultiSearcher multiSeacher) {
>>>
>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>> multiSeacher.getSearchables();
>>>
>>> for(int i =0 ; i < indexSearchers.length;i++) {
>>>
>>> try {
>>>
>>> documentSearcherManagers[i].release(indexSearchers[i]);
>>>
>>> } catch (IOException e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> }
>>>
>>> }
>>>
>>> }
>>>
>>>
>>> private MultiSearcher get() throws SearchExecutionException {
>>>
>>> maybeReopen();
>>>
>>> MultiSearcher multiSearcher = null;
>>>
>>> List<IndexSearcher> listOfIndexSeachers = new
>>> ArrayList<IndexSearcher>();
>>>
>>> for (DocumentSearcherManager documentSearcherManager :
>>> documentSearcherManagers) {
>>>
>>> listOfIndexSeachers.add(documentSearcherManager.get());
>>>
>>> }
>>>
>>> try {
>>>
>>> multiSearcher = new
>>> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[] {}));
>>>
>>> } catch (IOException e) {
>>>
>>> throw new SearchExecutionException(e);
>>>
>>> }
>>>
>>> return multiSearcher;
>>>
>>> }
>>>
>>>
>>> Hope there is enough information.
>>>
>>>
>>> Cheers
>>>
>>> Amin
>>>
>>>
>>> P.S. I will continue to debug.
>>>
>>>
>>>
>>>
>>> On Mon, Mar 2, 2009 at 6:55 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>>> It makes perfect sense to call maybeReopen() followed by get(),
>>>> as long
>>>> as
>>>> maybeReopen() is never slow enough to be noticeable to an end user
>>>> (because
>>>> you are making random queries pay the reopen/warming cost).
>>>>
>>>> If you call maybeReopen() after get(), then that search will not
>>>> see the
>>>> newly opened readers, but the next search will.
>>>>
>>>> I'm just thinking that since you see no results with get() alone,
>>>> debug
>>>> that case first. Then put back the maybeReopen().
>>>>
>>>> Can you post your full code at this point?
>>>>
>>>>
>>>> Mike
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> Hi
>>>>
>>>>>
>>>>> Just out of curiosity does it not make sense to call maybeReopen
>>>>> and
>>>>> then
>>>>> call get()? If I call get() then I have a new mulitsearcher, so
>>>>> a call
>>>>> to
>>>>> maybeopen won't reinitialise the multi searcher. Unless I pass
>>>>> the
>>>>> multi
>>>>> searcher into the maybereopen method. But somehow that doesn't
>>>>> make
>>>>> sense. I
>>>>> maybe missing something here.
>>>>>
>>>>>
>>>>> Cheers
>>>>>
>>>>> Amin
>>>>>
>>>>> On 2 Mar 2009, at 15:48, Amin Mohammed-Coleman <am...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> I'm seeing some interesting behviour when i do get() first
>>>>> followed by
>>>>>
>>>>>> maybeReopen then there are no documents in the directory
>>>>>> (directory
>>>>>> that i
>>>>>> am interested in. When i do the maybeReopen and then get()
>>>>>> then the
>>>>>> doc
>>>>>> count is correct. I can post stats later.
>>>>>>
>>>>>> Weird...
>>>>>>
>>>>>> On Mon, Mar 2, 2009 at 2:17 PM, Amin Mohammed-Coleman <
>>>>>> aminmc@gmail.com>
>>>>>> wrote:
>>>>>> oh dear...i think i may cry...i'll debug.
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 2, 2009 at 2:15 PM, Michael McCandless <
>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>
>>>>>> Or even just get() with no call to maybeReopen(). That should
>>>>>> work
>>>>>> fine
>>>>>> as well.
>>>>>>
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>> In my test case I have a set up method that should populate the
>>>>>> indexes
>>>>>> before I start using the document searcher. I will start
>>>>>> adding some
>>>>>> more
>>>>>> debug statements. So basically I should be able to do: get()
>>>>>> followed
>>>>>> by
>>>>>> maybeReopen.
>>>>>>
>>>>>> I will let you know what the outcome is.
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>> Amin
>>>>>>
>>>>>> On Mon, Mar 2, 2009 at 1:39 PM, Michael McCandless <
>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>
>>>>>>
>>>>>> Is it possible that when you first create the SearcherManager,
>>>>>> there is
>>>>>> no
>>>>>> index in each Directory?
>>>>>>
>>>>>> If not... you better start adding diagnostics. EG inside your
>>>>>> get(),
>>>>>> print
>>>>>> out the numDocs() of each IndexReader you get from the
>>>>>> SearcherManager?
>>>>>>
>>>>>> Something is wrong and it's best to explain it...
>>>>>>
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>> Nope. If i remove the maybeReopen the search doesn't work. It
>>>>>> only
>>>>>> works
>>>>>> when i cal maybeReopen followed by get().
>>>>>>
>>>>>> Cheers
>>>>>> Amin
>>>>>>
>>>>>> On Mon, Mar 2, 2009 at 12:56 PM, Michael McCandless <
>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>
>>>>>>
>>>>>> That's not right; something must be wrong.
>>>>>>
>>>>>> get() before maybeReopen() should simply let you search based
>>>>>> on the
>>>>>> searcher before reopening.
>>>>>>
>>>>>> If you just do get() and don't call maybeReopen() does it work?
>>>>>>
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>> I noticed that if i do the get() before the maybeReopen then I
>>>>>> get no
>>>>>>
>>>>>> results. But otherwise I can change it further.
>>>>>>
>>>>>> On Mon, Mar 2, 2009 at 11:46 AM, Michael McCandless <
>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>
>>>>>>
>>>>>> There is no such thing as final code -- code is alive and is
>>>>>> always
>>>>>> changing ;)
>>>>>>
>>>>>> It looks good to me.
>>>>>>
>>>>>> Though one trivial thing is: I would move the code in the try
>>>>>> clause up
>>>>>> to
>>>>>> and including the multiSearcher=get() out above the try. I
>>>>>> always
>>>>>> attempt
>>>>>> to "shrink wrap" what's inside a try clause to the minimum that
>>>>>> needs
>>>>>> to
>>>>>> be
>>>>>> there. Ie, your code that creates a query, finds the right
>>>>>> sort &
>>>>>> filter
>>>>>> to
>>>>>> use, etc, can all happen outside the try, because you have not
>>>>>> yet
>>>>>> acquired
>>>>>> the multiSearcher.
>>>>>>
>>>>>> If you do that, you also don't need the null check in the finally
>>>>>> clause,
>>>>>> because multiSearcher must be non-null on entering the try.
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>> Hi there
>>>>>>
>>>>>> Good morning! Here is the final search code:
>>>>>>
>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>> throwsSearchExecutionException {
>>>>>>
>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>
>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>
>>>>>> throw new SearchExecutionException("Search string cannot be
>>>>>> empty.
>>>>>> There
>>>>>> will be too many results to process.");
>>>>>>
>>>>>> }
>>>>>>
>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>
>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>
>>>>>> stopWatch.start();
>>>>>>
>>>>>> MultiSearcher multiSearcher = null;
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>
>>>>>> maybeReopen();
>>>>>>
>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>
>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>>>>>> Query '" +
>>>>>> query.toString() +"'");
>>>>>>
>>>>>> Sort sort = null;
>>>>>>
>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>
>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>
>>>>>> ChainedFilter chainedFilter = null;
>>>>>>
>>>>>> if (filters != null) {
>>>>>>
>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> multiSearcher = get();
>>>>>>
>>>>>> TopDocs topDocs = multiSearcher.search(query,chainedFilter ,
>>>>>> 100,sort);
>>>>>>
>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>
>>>>>> LOGGER.debug("total number of hits for [" + query.toString() +
>>>>>> " ] =
>>>>>> "+topDocs.
>>>>>> totalHits);
>>>>>>
>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>
>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>
>>>>>> float score = scoreDoc.score;
>>>>>>
>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>
>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>
>>>>>> summaryList.add(documentSummary);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> } catch (Exception e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> } finally {
>>>>>>
>>>>>> if (multiSearcher != null) {
>>>>>>
>>>>>> release(multiSearcher);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> stopWatch.stop();
>>>>>>
>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>
>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> I hope this makes sense...thanks again!
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> Amin
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Mar 1, 2009 at 8:09 PM, Michael McCandless <
>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>
>>>>>>
>>>>>> You're calling get() too many times. For every call to get()
>>>>>> you must
>>>>>>
>>>>>> match with a call to release().
>>>>>>
>>>>>> So, once at the front of your search method you should:
>>>>>>
>>>>>> MultiSearcher searcher = get();
>>>>>>
>>>>>> then use that searcher to do searching, retrieve docs, etc.
>>>>>>
>>>>>> Then in the finally clause, pass that searcher to release.
>>>>>>
>>>>>> So, only one call to get() and one matching call to release().
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> The searchers are injected into the class via Spring. So when a
>>>>>>
>>>>>> client
>>>>>> calls the class it is fully configured with a list of index
>>>>>> searchers.
>>>>>> However I have removed this list and instead injecting a list of
>>>>>> directories which are passed to the DocumentSearchManager.
>>>>>> DocumentSearchManager is SearchManager (should've mentioned that
>>>>>> earlier).
>>>>>> So finally I have modified by release code to do the following:
>>>>>>
>>>>>> private void release(MultiSearcher multiSeacher) throws
>>>>>> Exception {
>>>>>>
>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>>>> multiSeacher.getSearchables();
>>>>>>
>>>>>> for(int i =0 ; i < indexSearchers.length;i++) {
>>>>>>
>>>>>> documentSearcherManagers[i].release(indexSearchers[i]);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> and it's use looks like this:
>>>>>>
>>>>>>
>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>> throwsSearchExecutionException {
>>>>>>
>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>
>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>
>>>>>> throw new SearchExecutionException("Search string cannot be
>>>>>> empty.
>>>>>> There
>>>>>> will be too many results to process.");
>>>>>>
>>>>>> }
>>>>>>
>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>
>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>
>>>>>> stopWatch.start();
>>>>>>
>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>> ArrayList<IndexSearcher>();
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>
>>>>>> maybeReopen();
>>>>>>
>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>> searchers
>>>>>> '"
>>>>>> +
>>>>>> indexSearchers.size() +"'");
>>>>>>
>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>
>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>>>>>> Query '"
>>>>>> +
>>>>>> query.toString() +"'");
>>>>>>
>>>>>> Sort sort = null;
>>>>>>
>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>
>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>
>>>>>> ChainedFilter chainedFilter = null;
>>>>>>
>>>>>> if (filters != null) {
>>>>>>
>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>
>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>
>>>>>> LOGGER.debug("total number of hits for [" + query.toString() +
>>>>>> " ] =
>>>>>> "+topDocs.
>>>>>> totalHits);
>>>>>>
>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>
>>>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>>>
>>>>>> float score = scoreDoc.score;
>>>>>>
>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>
>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>
>>>>>> summaryList.add(documentSummary);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> } catch (Exception e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> } finally {
>>>>>>
>>>>>> release(get());
>>>>>>
>>>>>> }
>>>>>>
>>>>>> stopWatch.stop();
>>>>>>
>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>
>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> So the final post construct constructs the DocumentSearchMangers
>>>>>> with
>>>>>> the
>>>>>> list of directories..looking like this
>>>>>>
>>>>>>
>>>>>> @PostConstruct
>>>>>>
>>>>>> public void initialiseDocumentSearcher() {
>>>>>>
>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>> PerFieldAnalyzerWrapper(
>>>>>> analyzer);
>>>>>>
>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>> newKeywordAnalyzer());
>>>>>>
>>>>>> queryParser =
>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>> analyzerWrapper);
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>
>>>>>> documentSearcherManagers = new
>>>>>> DocumentSearcherManager[directories.size()];
>>>>>>
>>>>>> for (int i = 0; i < directories.size() ;i++) {
>>>>>>
>>>>>> Directory directory = directories.get(i);
>>>>>>
>>>>>> DocumentSearcherManager documentSearcherManager =
>>>>>> newDocumentSearcherManager(directory);
>>>>>>
>>>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>>>
>>>>>> }
>>>>>>
>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>
>>>>>> } catch (IOException e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> Amin
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Mar 1, 2009 at 6:15 PM, Michael McCandless <
>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>
>>>>>>
>>>>>> I don't understand where searchers comes from, prior to
>>>>>>
>>>>>> initializeDocumentSearcher? You should, instead, simply create
>>>>>> the
>>>>>> SearcherManager (from your Directory instances). You don't need
>>>>>> any
>>>>>> searchers during initialize.
>>>>>>
>>>>>> Is DocumentSearcherManager the same as SearcherManager (just
>>>>>> renamed)?
>>>>>>
>>>>>> The release method is wrong -- you're calling .get() and then
>>>>>> immediately release. Instead, you should step through the
>>>>>> searchers
>>>>>> from your MultiSearcher and release them to each SearcherManager.
>>>>>>
>>>>>> You should call your release() in a finally clause.
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>> Sorry...i'm getting slightly confused.
>>>>>>
>>>>>> I have a PostConstruct which is where I should create an array of
>>>>>>
>>>>>> SearchManagers (per indexSeacher). From there I initialise the
>>>>>> multisearcher using the get(). After which I need to call
>>>>>> maybeReopen
>>>>>> for
>>>>>> each IndexSearcher. So I'll do the following:
>>>>>>
>>>>>> @PostConstruct
>>>>>>
>>>>>> public void initialiseDocumentSearcher() {
>>>>>>
>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>> PerFieldAnalyzerWrapper(
>>>>>> analyzer);
>>>>>>
>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>> newKeywordAnalyzer());
>>>>>>
>>>>>> queryParser =
>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>> analyzerWrapper);
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>
>>>>>> documentSearcherManagers = new
>>>>>> DocumentSearcherManager[searchers.size()];
>>>>>>
>>>>>> for (int i = 0; i < searchers.size() ;i++) {
>>>>>>
>>>>>> IndexSearcher indexSearcher = searchers.get(i);
>>>>>>
>>>>>> Directory directory = indexSearcher.getIndexReader().directory();
>>>>>>
>>>>>> DocumentSearcherManager documentSearcherManager =
>>>>>> newDocumentSearcherManager(directory);
>>>>>>
>>>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>>>
>>>>>> }
>>>>>>
>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>
>>>>>> } catch (IOException e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> This initialises search managers. I then have methods:
>>>>>>
>>>>>>
>>>>>> private void maybeReopen() throws Exception {
>>>>>>
>>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>>
>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>> documentSearcherManagers) {
>>>>>>
>>>>>> documentSearcherManager.maybeReopen();
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> private void release() throws Exception {
>>>>>>
>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>> documentSearcherManagers) {
>>>>>>
>>>>>> documentSearcherManager.release(documentSearcherManager.get());
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> private MultiSearcher get() {
>>>>>>
>>>>>> List<IndexSearcher> listOfIndexSeachers = new
>>>>>> ArrayList<IndexSearcher>();
>>>>>>
>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>> documentSearcherManagers) {
>>>>>>
>>>>>> listOfIndexSeachers.add(documentSearcherManager.get());
>>>>>>
>>>>>> }
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> multiSearcher = new
>>>>>> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[]
>>>>>> {}));
>>>>>>
>>>>>> } catch (IOException e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> return multiSearcher;
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> These methods are used in the following manner in the search
>>>>>> code:
>>>>>>
>>>>>>
>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>> throwsSearchExecutionException {
>>>>>>
>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>
>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>
>>>>>> throw new SearchExecutionException("Search string cannot be
>>>>>> empty.
>>>>>> There
>>>>>> will be too many results to process.");
>>>>>>
>>>>>> }
>>>>>>
>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>
>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>
>>>>>> stopWatch.start();
>>>>>>
>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>> ArrayList<IndexSearcher>();
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>
>>>>>> maybeReopen();
>>>>>>
>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>> searchers
>>>>>> '"
>>>>>> +
>>>>>> indexSearchers.size() +"'");
>>>>>>
>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>
>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query
>>>>>> '"
>>>>>> +
>>>>>> query.toString() +"'");
>>>>>>
>>>>>> Sort sort = null;
>>>>>>
>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>
>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>
>>>>>> ChainedFilter chainedFilter = null;
>>>>>>
>>>>>> if (filters != null) {
>>>>>>
>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>
>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>
>>>>>> LOGGER.debug("total number of hits for [" + query.toString() +
>>>>>> " ]
>>>>>> =
>>>>>> "+topDocs.
>>>>>> totalHits);
>>>>>>
>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>
>>>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>>>
>>>>>> float score = scoreDoc.score;
>>>>>>
>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>
>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>
>>>>>> summaryList.add(documentSummary);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> release();
>>>>>>
>>>>>> } catch (Exception e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> stopWatch.stop();
>>>>>>
>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>
>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> Does this look better? Again..I really really appreciate your
>>>>>> help!
>>>>>>
>>>>>>
>>>>>> On Sun, Mar 1, 2009 at 4:18 PM, Michael McCandless <
>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>
>>>>>>
>>>>>> This is not quite right -- you should only create SearcherManager
>>>>>> once
>>>>>>
>>>>>> (per Direcotry) at startup/app load, not with every search
>>>>>> request.
>>>>>>
>>>>>>
>>>>>> And I don't see release -- it must call SearcherManager.release
>>>>>> of
>>>>>> each of the IndexSearchers previously returned from get().
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> Thanks again for helping on a Sunday!
>>>>>>
>>>>>>
>>>>>> I have now modified my maybeOpen() to do the following:
>>>>>>
>>>>>> private void maybeReopen() throws Exception {
>>>>>>
>>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>>
>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>>>>> .getSearchables();
>>>>>>
>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>
>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>
>>>>>> SearcherManager documentSearcherManager = new
>>>>>> SearcherManager(indexReader.directory());
>>>>>>
>>>>>> documentSearcherManager.maybeReopen();
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> And get() to:
>>>>>>
>>>>>>
>>>>>> private synchronized MultiSearcher get() {
>>>>>>
>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>>>>> .getSearchables();
>>>>>>
>>>>>> List<IndexSearcher> indexSearchersList = new
>>>>>> ArrayList<IndexSearcher>();
>>>>>>
>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>
>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>
>>>>>> SearcherManager documentSearcherManager = null;
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> documentSearcherManager = new
>>>>>> SearcherManager(indexReader.directory());
>>>>>>
>>>>>> } catch (IOException e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> indexSearchersList.add(documentSearcherManager.get());
>>>>>>
>>>>>> }
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> multiSearcher = new
>>>>>> MultiSearcher(indexSearchersList.toArray(newIndexSearcher[]
>>>>>> {}));
>>>>>>
>>>>>> } catch (IOException e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> return multiSearcher;
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> This makes all my test pass. I am using the SearchManager that
>>>>>> you
>>>>>> recommended. Does this look ok?
>>>>>>
>>>>>>
>>>>>> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>
>>>>>> Your maybeReopen has an excess incRef().
>>>>>>
>>>>>>
>>>>>> I'm not sure how you open the searchers in the first place? The
>>>>>>
>>>>>> list
>>>>>> starts as empty, and nothing populates it?
>>>>>>
>>>>>> When you do the initial population, you need an incRef.
>>>>>>
>>>>>> I think you're hitting IllegalStateException because
>>>>>> maybeReopen
>>>>>> is
>>>>>> closing a reader before get() can get it (since they
>>>>>> synchronize
>>>>>> on
>>>>>> different objects).
>>>>>>
>>>>>> I'd recommend switching to the SearcherManager class.
>>>>>> Instantiate
>>>>>> one
>>>>>> for each of your searchers. On each search request, go through
>>>>>> them
>>>>>> and call maybeReopen(), and then call get() and gather each
>>>>>> IndexSearcher instance into a new array. Then, make a new
>>>>>> MultiSearcher (opposite of what I said before): while that
>>>>>> creates
>>>>>> a
>>>>>> small amount of garbage, it'll keep your code simpler (good
>>>>>> tradeoff).
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>> sorrry I added
>>>>>>
>>>>>>
>>>>>> release(multiSearcher);
>>>>>>
>>>>>>
>>>>>>
>>>>>> instead of multiSearcher.close();
>>>>>>
>>>>>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <
>>>>>> aminmc@gmail.com
>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>>
>>>>>> I've now done the following:
>>>>>>
>>>>>>
>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>
>>>>>> throwsSearchExecutionException {
>>>>>>
>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>
>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>
>>>>>> throw new SearchExecutionException("Search string cannot be
>>>>>> empty.
>>>>>> There
>>>>>> will be too many results to process.");
>>>>>>
>>>>>> }
>>>>>>
>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>
>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>
>>>>>> stopWatch.start();
>>>>>>
>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>> ArrayList<IndexSearcher>();
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>
>>>>>> maybeReopen();
>>>>>>
>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>> searchers
>>>>>> '"+ indexSearchers.size() +
>>>>>> "'");
>>>>>>
>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>
>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>>>>>> Query
>>>>>> '"
>>>>>> +
>>>>>> query.toString() +"'");
>>>>>>
>>>>>> Sort sort = null;
>>>>>>
>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>
>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>
>>>>>> ChainedFilter chainedFilter = null;
>>>>>>
>>>>>> if (filters != null) {
>>>>>>
>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> TopDocs topDocs = get().search(query,chainedFilter
>>>>>> ,100,sort);
>>>>>>
>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>
>>>>>> LOGGER.debug("total number of hits for [" + query.toString()
>>>>>> +
>>>>>> "
>>>>>> ]
>>>>>> =
>>>>>> "+topDocs.
>>>>>> totalHits);
>>>>>>
>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>
>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>
>>>>>> float score = scoreDoc.score;
>>>>>>
>>>>>> final BaseDocument baseDocument = new BaseDocument(doc,
>>>>>> score);
>>>>>>
>>>>>> Summary documentSummary = new
>>>>>> DocumentSummaryImpl(baseDocument);
>>>>>>
>>>>>> summaryList.add(documentSummary);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> multiSearcher.close();
>>>>>>
>>>>>> } catch (Exception e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> stopWatch.stop();
>>>>>>
>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>
>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> And have the following methods:
>>>>>>
>>>>>> @PostConstruct
>>>>>>
>>>>>> public void initialiseQueryParser() {
>>>>>>
>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>> PerFieldAnalyzerWrapper(
>>>>>> analyzer);
>>>>>>
>>>>>>
>>>>>>
>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>> newKeywordAnalyzer());
>>>>>>
>>>>>> queryParser =
>>>>>>
>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>
>>>>>> analyzerWrapper);
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>
>>>>>> this.multiSearcher = new
>>>>>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>>>>>
>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>
>>>>>> } catch (IOException e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> Initialises mutltisearcher when this class is creared by
>>>>>> spring.
>>>>>>
>>>>>>
>>>>>> private synchronized void swapMultiSearcher(MultiSearcher
>>>>>> newMultiSearcher) {
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> release(multiSearcher);
>>>>>>
>>>>>> } catch (IOException e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> multiSearcher = newMultiSearcher;
>>>>>>
>>>>>> }
>>>>>>
>>>>>> public void maybeReopen() throws IOException {
>>>>>>
>>>>>> MultiSearcher newMultiSeacher = null;
>>>>>>
>>>>>> boolean refreshMultiSeacher = false;
>>>>>>
>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>> ArrayList<IndexSearcher>();
>>>>>>
>>>>>> synchronized (searchers) {
>>>>>>
>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>
>>>>>> IndexReader reader = indexSearcher.getIndexReader();
>>>>>>
>>>>>> reader.incRef();
>>>>>>
>>>>>> Directory directory = reader.directory();
>>>>>>
>>>>>> long currentVersion = reader.getVersion();
>>>>>>
>>>>>> if (IndexReader.getCurrentVersion(directory) !=
>>>>>> currentVersion)
>>>>>> {
>>>>>>
>>>>>> IndexReader newReader =
>>>>>> indexSearcher.getIndexReader().reopen();
>>>>>>
>>>>>> if (newReader != reader) {
>>>>>>
>>>>>> reader.decRef();
>>>>>>
>>>>>> refreshMultiSeacher = true;
>>>>>>
>>>>>> }
>>>>>>
>>>>>> reader = newReader;
>>>>>>
>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>
>>>>>> indexSearchers.add(newSearcher);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> if (refreshMultiSeacher) {
>>>>>>
>>>>>> newMultiSeacher = new
>>>>>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
>>>>>>
>>>>>> warm(newMultiSeacher);
>>>>>>
>>>>>> swapMultiSearcher(newMultiSeacher);
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> private void warm(MultiSearcher newMultiSeacher) {
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> private synchronized MultiSearcher get() {
>>>>>>
>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>
>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>
>>>>>> }
>>>>>>
>>>>>> return multiSearcher;
>>>>>>
>>>>>> }
>>>>>>
>>>>>> private synchronized void release(MultiSearcher
>>>>>> multiSearcher)
>>>>>> throwsIOException {
>>>>>>
>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>
>>>>>> indexSearcher.getIndexReader().decRef();
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> However I am now getting
>>>>>>
>>>>>>
>>>>>> java.lang.IllegalStateException:
>>>>>> org.apache.lucene.store.AlreadyClosedException: this
>>>>>> IndexReader
>>>>>> is
>>>>>> closed
>>>>>>
>>>>>>
>>>>>> on the call:
>>>>>>
>>>>>>
>>>>>> private synchronized MultiSearcher get() {
>>>>>>
>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>
>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>
>>>>>> }
>>>>>>
>>>>>> return multiSearcher;
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> I'm doing something wrong ..obviously..not sure where
>>>>>> though..
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>>>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>
>>>>>>
>>>>>> I was wondering the same thing ;)
>>>>>>
>>>>>>
>>>>>> It's best to call this method from a single BG "warming"
>>>>>> thread,
>>>>>>
>>>>>> in
>>>>>> which
>>>>>> case it would not need its own synchronization.
>>>>>>
>>>>>> But, to be safe, I'll add internal synchronization to it.
>>>>>> You
>>>>>> can't
>>>>>> simply put synchronized in front of the method, since you
>>>>>> don't
>>>>>> want
>>>>>> this to
>>>>>> block searching.
>>>>>>
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>> just a quick point:
>>>>>>
>>>>>> public void maybeReopen() throws IOException {
>>>>>> //D
>>>>>>
>>>>>> long currentVersion =
>>>>>>
>>>>>> currentSearcher.getIndexReader().getVersion();
>>>>>>
>>>>>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>>>>>> IndexReader newReader =
>>>>>> currentSearcher.getIndexReader().reopen();
>>>>>> assert newReader != currentSearcher.getIndexReader();
>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>> warm(newSearcher);
>>>>>> swapSearcher(newSearcher);
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> should the above be synchronised?
>>>>>>
>>>>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
>>>>>> aminmc@gmail.com
>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> thanks. i will rewrite..in between giving my baby her feed
>>>>>>
>>>>>> and
>>>>>>
>>>>>> playing
>>>>>>
>>>>>>
>>>>>> with the other child and my wife who wants me to do several
>>>>>> other
>>>>>>
>>>>>> things!
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>>
>>>>>> Thanks for your input. I would like to have a go at
>>>>>> doing
>>>>>> this
>>>>>>
>>>>>> myself
>>>>>>
>>>>>> first, Solr may be an option.
>>>>>>
>>>>>>
>>>>>> * You are creating a new Analyzer & QueryParser every
>>>>>> time,
>>>>>> also
>>>>>> creating unnecessary garbage; instead, they should be
>>>>>> created
>>>>>> once
>>>>>> & reused.
>>>>>>
>>>>>> -- I can moved the code out so that it is only created
>>>>>> once
>>>>>> and
>>>>>> reused.
>>>>>>
>>>>>>
>>>>>> * You always make a new IndexSearcher and a new
>>>>>> MultiSearcher
>>>>>> even
>>>>>> when nothing has changed. This just generates
>>>>>> unnecessary
>>>>>> garbage
>>>>>> which GC then must sweep up.
>>>>>>
>>>>>> -- This was something I thought about. I could move it
>>>>>> out
>>>>>> so
>>>>>> that
>>>>>> it's
>>>>>> created once. However I presume inside my code i need
>>>>>> to
>>>>>> check
>>>>>> whether
>>>>>> the
>>>>>> indexreaders are update to date. This needs to be
>>>>>> synchronized
>>>>>> as
>>>>>> well I
>>>>>> guess(?)
>>>>>>
>>>>>>
>>>>>> Yes you should synchronize the check for whether the
>>>>>> IndexReader
>>>>>> is
>>>>>>
>>>>>> current.
>>>>>>
>>>>>>
>>>>>> * I don't see any synchronization -- it looks like two
>>>>>>
>>>>>> search
>>>>>>
>>>>>> requests are allowed into this method at the same time?
>>>>>> Which
>>>>>> is
>>>>>>
>>>>>> dangerous... eg both (or, more) will wastefully reopen
>>>>>> the
>>>>>>
>>>>>> readers.
>>>>>>
>>>>>> -- So i need to extract the logic for reopening and
>>>>>> provide
>>>>>> a
>>>>>> synchronisation mechanism.
>>>>>>
>>>>>>
>>>>>> Yes.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Ok. So I have some work to do. I'll refactor the code
>>>>>> and
>>>>>>
>>>>>> see
>>>>>> if
>>>>>> I
>>>>>> can
>>>>>>
>>>>>> get
>>>>>>
>>>>>> inline to your recommendations.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>
>>>>>>
>>>>>> On a quick look, I think there are a few problems with
>>>>>> the
>>>>>> code:
>>>>>>
>>>>>>
>>>>>> * I don't see any synchronization -- it looks like two
>>>>>> search
>>>>>>
>>>>>> requests are allowed into this method at the same time?
>>>>>>
>>>>>> Which
>>>>>> is
>>>>>> dangerous... eg both (or, more) will wastefully reopen
>>>>>> the
>>>>>> readers.
>>>>>>
>>>>>> * You are over-incRef'ing (the reader.incRef inside the
>>>>>> loop)
>>>>>> --
>>>>>> I
>>>>>> don't see a corresponding decRef.
>>>>>>
>>>>>> * You reopen and warm your searchers "live" (vs with BG
>>>>>> thread);
>>>>>> meaning the unlucky search request that hits a reopen
>>>>>> pays
>>>>>> the
>>>>>> cost. This might be OK if the index is small enough
>>>>>> that
>>>>>> reopening & warming takes very little time. But if
>>>>>> index
>>>>>> gets
>>>>>> large, making a random search pay that warming cost is
>>>>>> not
>>>>>> nice
>>>>>> to
>>>>>> the end user. It erodes their trust in you.
>>>>>>
>>>>>> * You always make a new IndexSearcher and a new
>>>>>> MultiSearcher
>>>>>> even
>>>>>> when nothing has changed. This just generates
>>>>>> unnecessary
>>>>>> garbage
>>>>>> which GC then must sweep up.
>>>>>>
>>>>>> * You are creating a new Analyzer & QueryParser every
>>>>>> time,
>>>>>> also
>>>>>> creating unnecessary garbage; instead, they should be
>>>>>> created
>>>>>> once
>>>>>> & reused.
>>>>>>
>>>>>> You should consider simply using Solr -- it handles all
>>>>>> this
>>>>>> logic
>>>>>> for
>>>>>> you and has been well debugged with time...
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>> The reason for the indexreader.reopen is because I have
>>>>>> a
>>>>>> webapp
>>>>>> which
>>>>>>
>>>>>> enables users to upload files and then search for the
>>>>>> documents.
>>>>>> If
>>>>>>
>>>>>> I
>>>>>>
>>>>>> don't
>>>>>>
>>>>>> reopen i'm concerned that the facet hit counter won't
>>>>>> be
>>>>>>
>>>>>> updated.
>>>>>>
>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman
>>>>>> <
>>>>>> aminmc@gmail.com
>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have been able to get the code working for my
>>>>>> scenario,
>>>>>>
>>>>>> however
>>>>>>
>>>>>> I
>>>>>>
>>>>>> have
>>>>>>
>>>>>> a
>>>>>>
>>>>>> question and I was wondering if I could get some help.
>>>>>>
>>>>>> I
>>>>>> have
>>>>>> a
>>>>>> list
>>>>>> of
>>>>>> IndexSearchers which are used in a MultiSearcher
>>>>>> class.
>>>>>> I
>>>>>> use
>>>>>> the
>>>>>> indexsearchers to get each indexreader and put them
>>>>>> into
>>>>>> a
>>>>>> MultiIndexReader.
>>>>>>
>>>>>> IndexReader[] readers = new
>>>>>> IndexReader[searchables.length];
>>>>>>
>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>
>>>>>> IndexSearcher indexSearcher =
>>>>>> (IndexSearcher)searchables[i];
>>>>>>
>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>
>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>
>>>>>> if (newReader != readers[i]) {
>>>>>>
>>>>>> readers[i].close();
>>>>>>
>>>>>> }
>>>>>>
>>>>>> readers[i] = newReader;
>>>>>>
>>>>>>
>>>>>>
>>>>>> }
>>>>>>
>>>>>> multiReader = new MultiReader(readers);
>>>>>>
>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>
>>>>>> IndexSearcher indexSearcher = new
>>>>>> IndexSearcher(multiReader);
>>>>>>
>>>>>>
>>>>>> I then use the indexseacher to do the facet stuff. I
>>>>>> end
>>>>>> the
>>>>>> code
>>>>>> with
>>>>>> closing the multireader. This is causing problems in
>>>>>> another
>>>>>> method
>>>>>> where I
>>>>>> do some other search as the indexreaders are closed.
>>>>>> Is
>>>>>> it
>>>>>> ok
>>>>>> to
>>>>>> not
>>>>>> close
>>>>>> the multiindexreader or should I do some additional
>>>>>> checks
>>>>>> in
>>>>>> the
>>>>>> other
>>>>>> method to see if the indexreader is closed?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>>>> P.S. Hope that made sense...!
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin
>>>>>> Mohammed-Coleman
>>>>>> <
>>>>>> aminmc@gmail.com
>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks just what I needed!
>>>>>>
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> Amin
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>>>>>> marcelo.ochoa@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Amin:
>>>>>>
>>>>>> Please take a look a this blog post:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>> Best regards, Marcelo.
>>>>>>
>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin
>>>>>> Mohammed-Coleman
>>>>>> <
>>>>>> aminmc@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>>
>>>>>> Sorry to re send this email but I was wondering if
>>>>>> I
>>>>>> could
>>>>>> get
>>>>>>
>>>>>> some
>>>>>>
>>>>>> advice
>>>>>>
>>>>>> on this.
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> Amin
>>>>>>
>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>>>>>> aminmc@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>>
>>>>>> I am looking at building a faceted search using
>>>>>> Lucene.
>>>>>> I
>>>>>> know
>>>>>>
>>>>>> that
>>>>>>
>>>>>> Solr
>>>>>>
>>>>>> comes with this built in, however I would like to
>>>>>>
>>>>>> try
>>>>>> this
>>>>>> by
>>>>>> myself
>>>>>> (something to add to my CV!). I have been
>>>>>> looking
>>>>>> around
>>>>>> and
>>>>>> I
>>>>>> found
>>>>>> that
>>>>>> you can use the IndexReader and use TermVectors.
>>>>>> This
>>>>>> looks
>>>>>> ok
>>>>>> but
>>>>>> I'm
>>>>>> not
>>>>>> sure how to filter the results so that a
>>>>>> particular
>>>>>> user
>>>>>> can
>>>>>> only
>>>>>> see
>>>>>> a
>>>>>> subset of results. The next option I was looking
>>>>>> at
>>>>>> was
>>>>>> something
>>>>>> like
>>>>>>
>>>>>> Term term1 = new Term("brand", "ford");
>>>>>> Term term2 = new Term("brand", "vw");
>>>>>> Term[] termsArray = new Term[] { term1, term2
>>>>>> };un
>>>>>> int[] docFreqs =
>>>>>> indexSearcher.docFreqs(termsArray);
>>>>>>
>>>>>> The only problem here is that I have to provide
>>>>>> the
>>>>>> brand
>>>>>> type
>>>>>> each
>>>>>> time a
>>>>>> new brand is created. Again I'm not sure how I
>>>>>> can
>>>>>> filter
>>>>>> the
>>>>>> results
>>>>>> here.
>>>>>> It may be that I'm using the wrong api methods to
>>>>>> do
>>>>>> this.
>>>>>>
>>>>>> I would be grateful if I could get some advice on
>>>>>> this.
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>> Amin
>>>>>>
>>>>>> P.S. I am basically trying to do something that
>>>>>> displays
>>>>>> the
>>>>>> following
>>>>>>
>>>>>> Personal Contact (23) Business Contact (45) and
>>>>>> so
>>>>>> on..
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>>
>>>>>> Marcelo F. Ochoa
>>>>>>
>>>>>>
>>>>>> http://marceloochoa.blogspot.com/
>>>>>>
>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>
>>>>>> ______________
>>>>>> Want to integrate Lucene and Oracle?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>> Is Oracle 11g REST ready?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail:
>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail:
>>>>>> java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>>
>>>>>>
>>>>>> To unsubscribe, e-mail:
>>>>>>
>>>>>>
>>>>>>
>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>> For additional commands, e-mail:
>>>>>>
>>>>>> java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>>
>>>>>> To unsubscribe, e-mail:
>>>>>>
>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>
>>>>>>
>>>>>> For additional commands, e-mail:
>>>>>>
>>>>>> java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>>
>>>>>>
>>>>>> To unsubscribe, e-mail:
>>>>>>
>>>>>>
>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>
>>>>>>
>>>>>> For additional commands, e-mail:
>>>>>>
>>>>>> java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>>
>>>>>>
>>>>>> To unsubscribe, e-mail:
>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>
>>>>>>
>>>>>> For additional commands, e-mail:
>>>>>>
>>>>>> java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>>
>>>>>> To unsubscribe, e-mail:
>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>
>>>>>>
>>>>>> For additional commands, e-mail:
>>>>>> java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>>
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>>
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>> rg" target="_blank">unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>> div>
>>>>>>
>>>>>>
>>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
I think that is the case. When my SearchManager is initialised the
directories are empty so when I do a get() nothing is present. Subsequent
calls seem to work. Is there something I can do? or do I accept this or
just do a maybeReopen and do a get(). As you mentioned it depends on
timiing but I would be keen to know what the best practice would be in this
situation...
Cheers
On Mon, Mar 2, 2009 at 8:43 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
>
> Well the code looks fine.
>
> I can't explain why you see no search results if you don't call
> maybeReopen() in get, unless at the time you first create SearcherManager
> the Directories each have an empty index in them.
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> Hi
>> Here is the code that I am using, I've modified the get() method to
>> include
>> the maybeReopen() call. Again I'm not sure if this is a good idea.
>>
>> public Summary[] search(final SearchRequest searchRequest)
>> throwsSearchExecutionException {
>>
>> final String searchTerm = searchRequest.getSearchTerm();
>>
>> if (StringUtils.isBlank(searchTerm)) {
>>
>> throw new SearchExecutionException("Search string cannot be empty. There
>> will be too many results to process.");
>>
>> }
>>
>> List<Summary> summaryList = new ArrayList<Summary>();
>>
>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>
>> stopWatch.start();
>>
>> MultiSearcher multiSearcher = get();
>>
>> try {
>>
>> LOGGER.debug("Ensuring all index readers are up to date...");
>>
>> Query query = queryParser.parse(searchTerm);
>>
>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
>> query.toString() +"'");
>>
>> Sort sort = null;
>>
>> sort = applySortIfApplicable(searchRequest);
>>
>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>
>> ChainedFilter chainedFilter = null;
>>
>> if (filters != null) {
>>
>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>
>> }
>>
>> TopDocs topDocs = multiSearcher.search(query,chainedFilter ,100,sort);
>>
>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>
>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>> "+topDocs.
>> totalHits);
>>
>> for (ScoreDoc scoreDoc : scoreDocs) {
>>
>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>
>> float score = scoreDoc.score;
>>
>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>
>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>
>> summaryList.add(documentSummary);
>>
>> }
>>
>> } catch (Exception e) {
>>
>> throw new IllegalStateException(e);
>>
>> } finally {
>>
>> if (multiSearcher != null) {
>>
>> release(multiSearcher);
>>
>> }
>>
>> }
>>
>> stopWatch.stop();
>>
>> LOGGER.debug("total time taken for document seach: " +
>> stopWatch.getTotalTimeMillis() + " ms");
>>
>> return summaryList.toArray(new Summary[] {});
>>
>> }
>>
>>
>> @Autowired
>>
>> public void setDirectories(@Qualifier("directories")ListFactoryBean
>> listFactoryBean) throws Exception {
>>
>> this.directories = (List<Directory>) listFactoryBean.getObject();
>>
>> }
>>
>> @PostConstruct
>>
>> public void initialiseDocumentSearcher() {
>>
>> StopWatch stopWatch = new StopWatch("document-search-initialiser");
>>
>> stopWatch.start();
>>
>> PerFieldAnalyzerWrapper analyzerWrapper = new PerFieldAnalyzerWrapper(
>> analyzer);
>>
>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>> newKeywordAnalyzer());
>>
>> queryParser =
>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>> analyzerWrapper);
>>
>> try {
>>
>> LOGGER.debug("Initialising document searcher ....");
>>
>> documentSearcherManagers = new
>> DocumentSearcherManager[directories.size()];
>>
>> for (int i = 0; i < directories.size() ;i++) {
>>
>> Directory directory = directories.get(i);
>>
>> DocumentSearcherManager documentSearcherManager =
>> newDocumentSearcherManager(directory);
>>
>> documentSearcherManagers[i]=documentSearcherManager;
>>
>> }
>>
>> LOGGER.debug("Document searcher initialised");
>>
>> } catch (IOException e) {
>>
>> throw new IllegalStateException(e);
>>
>> }
>>
>> stopWatch.stop();
>>
>> LOGGER.debug("Total time taken to initialise DocumentSearcher '" +
>> stopWatch.getTotalTimeMillis() +"' ms.");
>>
>> }
>>
>> private void maybeReopen() throws SearchExecutionException {
>>
>> LOGGER.debug("Initiating reopening of index readers...");
>>
>> for (DocumentSearcherManager documentSearcherManager :
>> documentSearcherManagers) {
>>
>> try {
>>
>> documentSearcherManager.maybeReopen();
>>
>> } catch (InterruptedException e) {
>>
>> throw new SearchExecutionException(e);
>>
>> } catch (IOException e) {
>>
>> throw new SearchExecutionException(e);
>>
>> }
>>
>> }
>>
>> LOGGER.debug("reopening of index readers complete.");
>>
>> }
>>
>>
>>
>> private void release(MultiSearcher multiSeacher) {
>>
>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>> multiSeacher.getSearchables();
>>
>> for(int i =0 ; i < indexSearchers.length;i++) {
>>
>> try {
>>
>> documentSearcherManagers[i].release(indexSearchers[i]);
>>
>> } catch (IOException e) {
>>
>> throw new IllegalStateException(e);
>>
>> }
>>
>> }
>>
>> }
>>
>>
>> private MultiSearcher get() throws SearchExecutionException {
>>
>> maybeReopen();
>>
>> MultiSearcher multiSearcher = null;
>>
>> List<IndexSearcher> listOfIndexSeachers = new ArrayList<IndexSearcher>();
>>
>> for (DocumentSearcherManager documentSearcherManager :
>> documentSearcherManagers) {
>>
>> listOfIndexSeachers.add(documentSearcherManager.get());
>>
>> }
>>
>> try {
>>
>> multiSearcher = new
>> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[] {}));
>>
>> } catch (IOException e) {
>>
>> throw new SearchExecutionException(e);
>>
>> }
>>
>> return multiSearcher;
>>
>> }
>>
>>
>> Hope there is enough information.
>>
>>
>> Cheers
>>
>> Amin
>>
>>
>> P.S. I will continue to debug.
>>
>>
>>
>>
>> On Mon, Mar 2, 2009 at 6:55 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>
>>> It makes perfect sense to call maybeReopen() followed by get(), as long
>>> as
>>> maybeReopen() is never slow enough to be noticeable to an end user
>>> (because
>>> you are making random queries pay the reopen/warming cost).
>>>
>>> If you call maybeReopen() after get(), then that search will not see the
>>> newly opened readers, but the next search will.
>>>
>>> I'm just thinking that since you see no results with get() alone, debug
>>> that case first. Then put back the maybeReopen().
>>>
>>> Can you post your full code at this point?
>>>
>>>
>>> Mike
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> Hi
>>>
>>>>
>>>> Just out of curiosity does it not make sense to call maybeReopen and
>>>> then
>>>> call get()? If I call get() then I have a new mulitsearcher, so a call
>>>> to
>>>> maybeopen won't reinitialise the multi searcher. Unless I pass the
>>>> multi
>>>> searcher into the maybereopen method. But somehow that doesn't make
>>>> sense. I
>>>> maybe missing something here.
>>>>
>>>>
>>>> Cheers
>>>>
>>>> Amin
>>>>
>>>> On 2 Mar 2009, at 15:48, Amin Mohammed-Coleman <am...@gmail.com>
>>>> wrote:
>>>>
>>>> I'm seeing some interesting behviour when i do get() first followed by
>>>>
>>>>> maybeReopen then there are no documents in the directory (directory
>>>>> that i
>>>>> am interested in. When i do the maybeReopen and then get() then the
>>>>> doc
>>>>> count is correct. I can post stats later.
>>>>>
>>>>> Weird...
>>>>>
>>>>> On Mon, Mar 2, 2009 at 2:17 PM, Amin Mohammed-Coleman <
>>>>> aminmc@gmail.com>
>>>>> wrote:
>>>>> oh dear...i think i may cry...i'll debug.
>>>>>
>>>>>
>>>>> On Mon, Mar 2, 2009 at 2:15 PM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>> Or even just get() with no call to maybeReopen(). That should work
>>>>> fine
>>>>> as well.
>>>>>
>>>>>
>>>>> Mike
>>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>> In my test case I have a set up method that should populate the indexes
>>>>> before I start using the document searcher. I will start adding some
>>>>> more
>>>>> debug statements. So basically I should be able to do: get() followed
>>>>> by
>>>>> maybeReopen.
>>>>>
>>>>> I will let you know what the outcome is.
>>>>>
>>>>>
>>>>> Cheers
>>>>> Amin
>>>>>
>>>>> On Mon, Mar 2, 2009 at 1:39 PM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>>
>>>>> Is it possible that when you first create the SearcherManager, there is
>>>>> no
>>>>> index in each Directory?
>>>>>
>>>>> If not... you better start adding diagnostics. EG inside your get(),
>>>>> print
>>>>> out the numDocs() of each IndexReader you get from the SearcherManager?
>>>>>
>>>>> Something is wrong and it's best to explain it...
>>>>>
>>>>>
>>>>> Mike
>>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>> Nope. If i remove the maybeReopen the search doesn't work. It only
>>>>> works
>>>>> when i cal maybeReopen followed by get().
>>>>>
>>>>> Cheers
>>>>> Amin
>>>>>
>>>>> On Mon, Mar 2, 2009 at 12:56 PM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>>
>>>>> That's not right; something must be wrong.
>>>>>
>>>>> get() before maybeReopen() should simply let you search based on the
>>>>> searcher before reopening.
>>>>>
>>>>> If you just do get() and don't call maybeReopen() does it work?
>>>>>
>>>>>
>>>>> Mike
>>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>> I noticed that if i do the get() before the maybeReopen then I get no
>>>>>
>>>>> results. But otherwise I can change it further.
>>>>>
>>>>> On Mon, Mar 2, 2009 at 11:46 AM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>>
>>>>> There is no such thing as final code -- code is alive and is always
>>>>> changing ;)
>>>>>
>>>>> It looks good to me.
>>>>>
>>>>> Though one trivial thing is: I would move the code in the try clause up
>>>>> to
>>>>> and including the multiSearcher=get() out above the try. I always
>>>>> attempt
>>>>> to "shrink wrap" what's inside a try clause to the minimum that needs
>>>>> to
>>>>> be
>>>>> there. Ie, your code that creates a query, finds the right sort &
>>>>> filter
>>>>> to
>>>>> use, etc, can all happen outside the try, because you have not yet
>>>>> acquired
>>>>> the multiSearcher.
>>>>>
>>>>> If you do that, you also don't need the null check in the finally
>>>>> clause,
>>>>> because multiSearcher must be non-null on entering the try.
>>>>>
>>>>> Mike
>>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>> Hi there
>>>>>
>>>>> Good morning! Here is the final search code:
>>>>>
>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>> throwsSearchExecutionException {
>>>>>
>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>
>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>
>>>>> throw new SearchExecutionException("Search string cannot be empty.
>>>>> There
>>>>> will be too many results to process.");
>>>>>
>>>>> }
>>>>>
>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>
>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>
>>>>> stopWatch.start();
>>>>>
>>>>> MultiSearcher multiSearcher = null;
>>>>>
>>>>> try {
>>>>>
>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>
>>>>> maybeReopen();
>>>>>
>>>>> Query query = queryParser.parse(searchTerm);
>>>>>
>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
>>>>> query.toString() +"'");
>>>>>
>>>>> Sort sort = null;
>>>>>
>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>
>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>
>>>>> ChainedFilter chainedFilter = null;
>>>>>
>>>>> if (filters != null) {
>>>>>
>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>
>>>>> }
>>>>>
>>>>> multiSearcher = get();
>>>>>
>>>>> TopDocs topDocs = multiSearcher.search(query,chainedFilter ,100,sort);
>>>>>
>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>
>>>>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>>>>> "+topDocs.
>>>>> totalHits);
>>>>>
>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>
>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>
>>>>> float score = scoreDoc.score;
>>>>>
>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>
>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>
>>>>> summaryList.add(documentSummary);
>>>>>
>>>>> }
>>>>>
>>>>> } catch (Exception e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> } finally {
>>>>>
>>>>> if (multiSearcher != null) {
>>>>>
>>>>> release(multiSearcher);
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> stopWatch.stop();
>>>>>
>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>
>>>>> return summaryList.toArray(new Summary[] {});
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> I hope this makes sense...thanks again!
>>>>>
>>>>>
>>>>> Cheers
>>>>>
>>>>> Amin
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Mar 1, 2009 at 8:09 PM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>>
>>>>> You're calling get() too many times. For every call to get() you must
>>>>>
>>>>> match with a call to release().
>>>>>
>>>>> So, once at the front of your search method you should:
>>>>>
>>>>> MultiSearcher searcher = get();
>>>>>
>>>>> then use that searcher to do searching, retrieve docs, etc.
>>>>>
>>>>> Then in the finally clause, pass that searcher to release.
>>>>>
>>>>> So, only one call to get() and one matching call to release().
>>>>>
>>>>> Mike
>>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> The searchers are injected into the class via Spring. So when a
>>>>>
>>>>> client
>>>>> calls the class it is fully configured with a list of index
>>>>> searchers.
>>>>> However I have removed this list and instead injecting a list of
>>>>> directories which are passed to the DocumentSearchManager.
>>>>> DocumentSearchManager is SearchManager (should've mentioned that
>>>>> earlier).
>>>>> So finally I have modified by release code to do the following:
>>>>>
>>>>> private void release(MultiSearcher multiSeacher) throws Exception {
>>>>>
>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>>> multiSeacher.getSearchables();
>>>>>
>>>>> for(int i =0 ; i < indexSearchers.length;i++) {
>>>>>
>>>>> documentSearcherManagers[i].release(indexSearchers[i]);
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> and it's use looks like this:
>>>>>
>>>>>
>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>> throwsSearchExecutionException {
>>>>>
>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>
>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>
>>>>> throw new SearchExecutionException("Search string cannot be empty.
>>>>> There
>>>>> will be too many results to process.");
>>>>>
>>>>> }
>>>>>
>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>
>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>
>>>>> stopWatch.start();
>>>>>
>>>>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>>>>
>>>>> try {
>>>>>
>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>
>>>>> maybeReopen();
>>>>>
>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>> searchers
>>>>> '"
>>>>> +
>>>>> indexSearchers.size() +"'");
>>>>>
>>>>> Query query = queryParser.parse(searchTerm);
>>>>>
>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '"
>>>>> +
>>>>> query.toString() +"'");
>>>>>
>>>>> Sort sort = null;
>>>>>
>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>
>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>
>>>>> ChainedFilter chainedFilter = null;
>>>>>
>>>>> if (filters != null) {
>>>>>
>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>
>>>>> }
>>>>>
>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>
>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>
>>>>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>>>>> "+topDocs.
>>>>> totalHits);
>>>>>
>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>
>>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>>
>>>>> float score = scoreDoc.score;
>>>>>
>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>
>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>
>>>>> summaryList.add(documentSummary);
>>>>>
>>>>> }
>>>>>
>>>>> } catch (Exception e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> } finally {
>>>>>
>>>>> release(get());
>>>>>
>>>>> }
>>>>>
>>>>> stopWatch.stop();
>>>>>
>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>
>>>>> return summaryList.toArray(new Summary[] {});
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> So the final post construct constructs the DocumentSearchMangers
>>>>> with
>>>>> the
>>>>> list of directories..looking like this
>>>>>
>>>>>
>>>>> @PostConstruct
>>>>>
>>>>> public void initialiseDocumentSearcher() {
>>>>>
>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>> PerFieldAnalyzerWrapper(
>>>>> analyzer);
>>>>>
>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>> newKeywordAnalyzer());
>>>>>
>>>>> queryParser =
>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>> analyzerWrapper);
>>>>>
>>>>> try {
>>>>>
>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>
>>>>> documentSearcherManagers = new
>>>>> DocumentSearcherManager[directories.size()];
>>>>>
>>>>> for (int i = 0; i < directories.size() ;i++) {
>>>>>
>>>>> Directory directory = directories.get(i);
>>>>>
>>>>> DocumentSearcherManager documentSearcherManager =
>>>>> newDocumentSearcherManager(directory);
>>>>>
>>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>>
>>>>> }
>>>>>
>>>>> LOGGER.debug("multi searcher initialised");
>>>>>
>>>>> } catch (IOException e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> Cheers
>>>>>
>>>>> Amin
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Mar 1, 2009 at 6:15 PM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>>
>>>>> I don't understand where searchers comes from, prior to
>>>>>
>>>>> initializeDocumentSearcher? You should, instead, simply create the
>>>>> SearcherManager (from your Directory instances). You don't need
>>>>> any
>>>>> searchers during initialize.
>>>>>
>>>>> Is DocumentSearcherManager the same as SearcherManager (just
>>>>> renamed)?
>>>>>
>>>>> The release method is wrong -- you're calling .get() and then
>>>>> immediately release. Instead, you should step through the
>>>>> searchers
>>>>> from your MultiSearcher and release them to each SearcherManager.
>>>>>
>>>>> You should call your release() in a finally clause.
>>>>>
>>>>> Mike
>>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>> Sorry...i'm getting slightly confused.
>>>>>
>>>>> I have a PostConstruct which is where I should create an array of
>>>>>
>>>>> SearchManagers (per indexSeacher). From there I initialise the
>>>>> multisearcher using the get(). After which I need to call
>>>>> maybeReopen
>>>>> for
>>>>> each IndexSearcher. So I'll do the following:
>>>>>
>>>>> @PostConstruct
>>>>>
>>>>> public void initialiseDocumentSearcher() {
>>>>>
>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>> PerFieldAnalyzerWrapper(
>>>>> analyzer);
>>>>>
>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>> newKeywordAnalyzer());
>>>>>
>>>>> queryParser =
>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>> analyzerWrapper);
>>>>>
>>>>> try {
>>>>>
>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>
>>>>> documentSearcherManagers = new
>>>>> DocumentSearcherManager[searchers.size()];
>>>>>
>>>>> for (int i = 0; i < searchers.size() ;i++) {
>>>>>
>>>>> IndexSearcher indexSearcher = searchers.get(i);
>>>>>
>>>>> Directory directory = indexSearcher.getIndexReader().directory();
>>>>>
>>>>> DocumentSearcherManager documentSearcherManager =
>>>>> newDocumentSearcherManager(directory);
>>>>>
>>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>>
>>>>> }
>>>>>
>>>>> LOGGER.debug("multi searcher initialised");
>>>>>
>>>>> } catch (IOException e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> This initialises search managers. I then have methods:
>>>>>
>>>>>
>>>>> private void maybeReopen() throws Exception {
>>>>>
>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>
>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>> documentSearcherManagers) {
>>>>>
>>>>> documentSearcherManager.maybeReopen();
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> private void release() throws Exception {
>>>>>
>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>> documentSearcherManagers) {
>>>>>
>>>>> documentSearcherManager.release(documentSearcherManager.get());
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> private MultiSearcher get() {
>>>>>
>>>>> List<IndexSearcher> listOfIndexSeachers = new
>>>>> ArrayList<IndexSearcher>();
>>>>>
>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>> documentSearcherManagers) {
>>>>>
>>>>> listOfIndexSeachers.add(documentSearcherManager.get());
>>>>>
>>>>> }
>>>>>
>>>>> try {
>>>>>
>>>>> multiSearcher = new
>>>>> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[] {}));
>>>>>
>>>>> } catch (IOException e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> }
>>>>>
>>>>> return multiSearcher;
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> These methods are used in the following manner in the search code:
>>>>>
>>>>>
>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>> throwsSearchExecutionException {
>>>>>
>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>
>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>
>>>>> throw new SearchExecutionException("Search string cannot be empty.
>>>>> There
>>>>> will be too many results to process.");
>>>>>
>>>>> }
>>>>>
>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>
>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>
>>>>> stopWatch.start();
>>>>>
>>>>> List<IndexSearcher> indexSearchers = new
>>>>> ArrayList<IndexSearcher>();
>>>>>
>>>>> try {
>>>>>
>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>
>>>>> maybeReopen();
>>>>>
>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>> searchers
>>>>> '"
>>>>> +
>>>>> indexSearchers.size() +"'");
>>>>>
>>>>> Query query = queryParser.parse(searchTerm);
>>>>>
>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query
>>>>> '"
>>>>> +
>>>>> query.toString() +"'");
>>>>>
>>>>> Sort sort = null;
>>>>>
>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>
>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>
>>>>> ChainedFilter chainedFilter = null;
>>>>>
>>>>> if (filters != null) {
>>>>>
>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>
>>>>> }
>>>>>
>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>
>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>
>>>>> LOGGER.debug("total number of hits for [" + query.toString() + " ]
>>>>> =
>>>>> "+topDocs.
>>>>> totalHits);
>>>>>
>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>
>>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>>
>>>>> float score = scoreDoc.score;
>>>>>
>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>
>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>
>>>>> summaryList.add(documentSummary);
>>>>>
>>>>> }
>>>>>
>>>>> release();
>>>>>
>>>>> } catch (Exception e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> }
>>>>>
>>>>> stopWatch.stop();
>>>>>
>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>
>>>>> return summaryList.toArray(new Summary[] {});
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> Does this look better? Again..I really really appreciate your
>>>>> help!
>>>>>
>>>>>
>>>>> On Sun, Mar 1, 2009 at 4:18 PM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>>
>>>>> This is not quite right -- you should only create SearcherManager
>>>>> once
>>>>>
>>>>> (per Direcotry) at startup/app load, not with every search
>>>>> request.
>>>>>
>>>>>
>>>>> And I don't see release -- it must call SearcherManager.release
>>>>> of
>>>>> each of the IndexSearchers previously returned from get().
>>>>>
>>>>> Mike
>>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> Thanks again for helping on a Sunday!
>>>>>
>>>>>
>>>>> I have now modified my maybeOpen() to do the following:
>>>>>
>>>>> private void maybeReopen() throws Exception {
>>>>>
>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>
>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>>>> .getSearchables();
>>>>>
>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>
>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>
>>>>> SearcherManager documentSearcherManager = new
>>>>> SearcherManager(indexReader.directory());
>>>>>
>>>>> documentSearcherManager.maybeReopen();
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> And get() to:
>>>>>
>>>>>
>>>>> private synchronized MultiSearcher get() {
>>>>>
>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>>>> .getSearchables();
>>>>>
>>>>> List<IndexSearcher> indexSearchersList = new
>>>>> ArrayList<IndexSearcher>();
>>>>>
>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>
>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>
>>>>> SearcherManager documentSearcherManager = null;
>>>>>
>>>>> try {
>>>>>
>>>>> documentSearcherManager = new
>>>>> SearcherManager(indexReader.directory());
>>>>>
>>>>> } catch (IOException e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> }
>>>>>
>>>>> indexSearchersList.add(documentSearcherManager.get());
>>>>>
>>>>> }
>>>>>
>>>>> try {
>>>>>
>>>>> multiSearcher = new
>>>>> MultiSearcher(indexSearchersList.toArray(newIndexSearcher[]
>>>>> {}));
>>>>>
>>>>> } catch (IOException e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> }
>>>>>
>>>>> return multiSearcher;
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> This makes all my test pass. I am using the SearchManager that
>>>>> you
>>>>> recommended. Does this look ok?
>>>>>
>>>>>
>>>>> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>> Your maybeReopen has an excess incRef().
>>>>>
>>>>>
>>>>> I'm not sure how you open the searchers in the first place? The
>>>>>
>>>>> list
>>>>> starts as empty, and nothing populates it?
>>>>>
>>>>> When you do the initial population, you need an incRef.
>>>>>
>>>>> I think you're hitting IllegalStateException because
>>>>> maybeReopen
>>>>> is
>>>>> closing a reader before get() can get it (since they
>>>>> synchronize
>>>>> on
>>>>> different objects).
>>>>>
>>>>> I'd recommend switching to the SearcherManager class.
>>>>> Instantiate
>>>>> one
>>>>> for each of your searchers. On each search request, go through
>>>>> them
>>>>> and call maybeReopen(), and then call get() and gather each
>>>>> IndexSearcher instance into a new array. Then, make a new
>>>>> MultiSearcher (opposite of what I said before): while that
>>>>> creates
>>>>> a
>>>>> small amount of garbage, it'll keep your code simpler (good
>>>>> tradeoff).
>>>>>
>>>>> Mike
>>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>> sorrry I added
>>>>>
>>>>>
>>>>> release(multiSearcher);
>>>>>
>>>>>
>>>>>
>>>>> instead of multiSearcher.close();
>>>>>
>>>>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <
>>>>> aminmc@gmail.com
>>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> Hi
>>>>>
>>>>>
>>>>> I've now done the following:
>>>>>
>>>>>
>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>
>>>>> throwsSearchExecutionException {
>>>>>
>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>
>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>
>>>>> throw new SearchExecutionException("Search string cannot be
>>>>> empty.
>>>>> There
>>>>> will be too many results to process.");
>>>>>
>>>>> }
>>>>>
>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>
>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>
>>>>> stopWatch.start();
>>>>>
>>>>> List<IndexSearcher> indexSearchers = new
>>>>> ArrayList<IndexSearcher>();
>>>>>
>>>>> try {
>>>>>
>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>
>>>>> maybeReopen();
>>>>>
>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>> searchers
>>>>> '"+ indexSearchers.size() +
>>>>> "'");
>>>>>
>>>>> Query query = queryParser.parse(searchTerm);
>>>>>
>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>>>>> Query
>>>>> '"
>>>>> +
>>>>> query.toString() +"'");
>>>>>
>>>>> Sort sort = null;
>>>>>
>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>
>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>
>>>>> ChainedFilter chainedFilter = null;
>>>>>
>>>>> if (filters != null) {
>>>>>
>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>
>>>>> }
>>>>>
>>>>> TopDocs topDocs = get().search(query,chainedFilter
>>>>> ,100,sort);
>>>>>
>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>
>>>>> LOGGER.debug("total number of hits for [" + query.toString()
>>>>> +
>>>>> "
>>>>> ]
>>>>> =
>>>>> "+topDocs.
>>>>> totalHits);
>>>>>
>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>
>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>
>>>>> float score = scoreDoc.score;
>>>>>
>>>>> final BaseDocument baseDocument = new BaseDocument(doc,
>>>>> score);
>>>>>
>>>>> Summary documentSummary = new
>>>>> DocumentSummaryImpl(baseDocument);
>>>>>
>>>>> summaryList.add(documentSummary);
>>>>>
>>>>> }
>>>>>
>>>>> multiSearcher.close();
>>>>>
>>>>> } catch (Exception e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> }
>>>>>
>>>>> stopWatch.stop();
>>>>>
>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>
>>>>> return summaryList.toArray(new Summary[] {});
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> And have the following methods:
>>>>>
>>>>> @PostConstruct
>>>>>
>>>>> public void initialiseQueryParser() {
>>>>>
>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>> PerFieldAnalyzerWrapper(
>>>>> analyzer);
>>>>>
>>>>>
>>>>>
>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>> newKeywordAnalyzer());
>>>>>
>>>>> queryParser =
>>>>>
>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>
>>>>> analyzerWrapper);
>>>>>
>>>>> try {
>>>>>
>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>
>>>>> this.multiSearcher = new
>>>>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>>>>
>>>>> LOGGER.debug("multi searcher initialised");
>>>>>
>>>>> } catch (IOException e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> Initialises mutltisearcher when this class is creared by
>>>>> spring.
>>>>>
>>>>>
>>>>> private synchronized void swapMultiSearcher(MultiSearcher
>>>>> newMultiSearcher) {
>>>>>
>>>>> try {
>>>>>
>>>>> release(multiSearcher);
>>>>>
>>>>> } catch (IOException e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> }
>>>>>
>>>>> multiSearcher = newMultiSearcher;
>>>>>
>>>>> }
>>>>>
>>>>> public void maybeReopen() throws IOException {
>>>>>
>>>>> MultiSearcher newMultiSeacher = null;
>>>>>
>>>>> boolean refreshMultiSeacher = false;
>>>>>
>>>>> List<IndexSearcher> indexSearchers = new
>>>>> ArrayList<IndexSearcher>();
>>>>>
>>>>> synchronized (searchers) {
>>>>>
>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>
>>>>> IndexReader reader = indexSearcher.getIndexReader();
>>>>>
>>>>> reader.incRef();
>>>>>
>>>>> Directory directory = reader.directory();
>>>>>
>>>>> long currentVersion = reader.getVersion();
>>>>>
>>>>> if (IndexReader.getCurrentVersion(directory) !=
>>>>> currentVersion)
>>>>> {
>>>>>
>>>>> IndexReader newReader =
>>>>> indexSearcher.getIndexReader().reopen();
>>>>>
>>>>> if (newReader != reader) {
>>>>>
>>>>> reader.decRef();
>>>>>
>>>>> refreshMultiSeacher = true;
>>>>>
>>>>> }
>>>>>
>>>>> reader = newReader;
>>>>>
>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>
>>>>> indexSearchers.add(newSearcher);
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> if (refreshMultiSeacher) {
>>>>>
>>>>> newMultiSeacher = new
>>>>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
>>>>>
>>>>> warm(newMultiSeacher);
>>>>>
>>>>> swapMultiSearcher(newMultiSeacher);
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> private void warm(MultiSearcher newMultiSeacher) {
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> private synchronized MultiSearcher get() {
>>>>>
>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>
>>>>> indexSearcher.getIndexReader().incRef();
>>>>>
>>>>> }
>>>>>
>>>>> return multiSearcher;
>>>>>
>>>>> }
>>>>>
>>>>> private synchronized void release(MultiSearcher
>>>>> multiSearcher)
>>>>> throwsIOException {
>>>>>
>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>
>>>>> indexSearcher.getIndexReader().decRef();
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> However I am now getting
>>>>>
>>>>>
>>>>> java.lang.IllegalStateException:
>>>>> org.apache.lucene.store.AlreadyClosedException: this
>>>>> IndexReader
>>>>> is
>>>>> closed
>>>>>
>>>>>
>>>>> on the call:
>>>>>
>>>>>
>>>>> private synchronized MultiSearcher get() {
>>>>>
>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>
>>>>> indexSearcher.getIndexReader().incRef();
>>>>>
>>>>> }
>>>>>
>>>>> return multiSearcher;
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> I'm doing something wrong ..obviously..not sure where
>>>>> though..
>>>>>
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>>
>>>>> I was wondering the same thing ;)
>>>>>
>>>>>
>>>>> It's best to call this method from a single BG "warming"
>>>>> thread,
>>>>>
>>>>> in
>>>>> which
>>>>> case it would not need its own synchronization.
>>>>>
>>>>> But, to be safe, I'll add internal synchronization to it.
>>>>> You
>>>>> can't
>>>>> simply put synchronized in front of the method, since you
>>>>> don't
>>>>> want
>>>>> this to
>>>>> block searching.
>>>>>
>>>>>
>>>>> Mike
>>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>> just a quick point:
>>>>>
>>>>> public void maybeReopen() throws IOException {
>>>>> //D
>>>>>
>>>>> long currentVersion =
>>>>>
>>>>> currentSearcher.getIndexReader().getVersion();
>>>>>
>>>>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>>>>> IndexReader newReader =
>>>>> currentSearcher.getIndexReader().reopen();
>>>>> assert newReader != currentSearcher.getIndexReader();
>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>> warm(newSearcher);
>>>>> swapSearcher(newSearcher);
>>>>> }
>>>>> }
>>>>>
>>>>> should the above be synchronised?
>>>>>
>>>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
>>>>> aminmc@gmail.com
>>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> thanks. i will rewrite..in between giving my baby her feed
>>>>>
>>>>> and
>>>>>
>>>>> playing
>>>>>
>>>>>
>>>>> with the other child and my wife who wants me to do several
>>>>> other
>>>>>
>>>>> things!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>>
>>>>> Hi
>>>>>
>>>>>
>>>>> Thanks for your input. I would like to have a go at
>>>>> doing
>>>>> this
>>>>>
>>>>> myself
>>>>>
>>>>> first, Solr may be an option.
>>>>>
>>>>>
>>>>> * You are creating a new Analyzer & QueryParser every
>>>>> time,
>>>>> also
>>>>> creating unnecessary garbage; instead, they should be
>>>>> created
>>>>> once
>>>>> & reused.
>>>>>
>>>>> -- I can moved the code out so that it is only created
>>>>> once
>>>>> and
>>>>> reused.
>>>>>
>>>>>
>>>>> * You always make a new IndexSearcher and a new
>>>>> MultiSearcher
>>>>> even
>>>>> when nothing has changed. This just generates
>>>>> unnecessary
>>>>> garbage
>>>>> which GC then must sweep up.
>>>>>
>>>>> -- This was something I thought about. I could move it
>>>>> out
>>>>> so
>>>>> that
>>>>> it's
>>>>> created once. However I presume inside my code i need
>>>>> to
>>>>> check
>>>>> whether
>>>>> the
>>>>> indexreaders are update to date. This needs to be
>>>>> synchronized
>>>>> as
>>>>> well I
>>>>> guess(?)
>>>>>
>>>>>
>>>>> Yes you should synchronize the check for whether the
>>>>> IndexReader
>>>>> is
>>>>>
>>>>> current.
>>>>>
>>>>>
>>>>> * I don't see any synchronization -- it looks like two
>>>>>
>>>>> search
>>>>>
>>>>> requests are allowed into this method at the same time?
>>>>> Which
>>>>> is
>>>>>
>>>>> dangerous... eg both (or, more) will wastefully reopen
>>>>> the
>>>>>
>>>>> readers.
>>>>>
>>>>> -- So i need to extract the logic for reopening and
>>>>> provide
>>>>> a
>>>>> synchronisation mechanism.
>>>>>
>>>>>
>>>>> Yes.
>>>>>
>>>>>
>>>>>
>>>>> Ok. So I have some work to do. I'll refactor the code
>>>>> and
>>>>>
>>>>> see
>>>>> if
>>>>> I
>>>>> can
>>>>>
>>>>> get
>>>>>
>>>>> inline to your recommendations.
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>>
>>>>> On a quick look, I think there are a few problems with
>>>>> the
>>>>> code:
>>>>>
>>>>>
>>>>> * I don't see any synchronization -- it looks like two
>>>>> search
>>>>>
>>>>> requests are allowed into this method at the same time?
>>>>>
>>>>> Which
>>>>> is
>>>>> dangerous... eg both (or, more) will wastefully reopen
>>>>> the
>>>>> readers.
>>>>>
>>>>> * You are over-incRef'ing (the reader.incRef inside the
>>>>> loop)
>>>>> --
>>>>> I
>>>>> don't see a corresponding decRef.
>>>>>
>>>>> * You reopen and warm your searchers "live" (vs with BG
>>>>> thread);
>>>>> meaning the unlucky search request that hits a reopen
>>>>> pays
>>>>> the
>>>>> cost. This might be OK if the index is small enough
>>>>> that
>>>>> reopening & warming takes very little time. But if
>>>>> index
>>>>> gets
>>>>> large, making a random search pay that warming cost is
>>>>> not
>>>>> nice
>>>>> to
>>>>> the end user. It erodes their trust in you.
>>>>>
>>>>> * You always make a new IndexSearcher and a new
>>>>> MultiSearcher
>>>>> even
>>>>> when nothing has changed. This just generates
>>>>> unnecessary
>>>>> garbage
>>>>> which GC then must sweep up.
>>>>>
>>>>> * You are creating a new Analyzer & QueryParser every
>>>>> time,
>>>>> also
>>>>> creating unnecessary garbage; instead, they should be
>>>>> created
>>>>> once
>>>>> & reused.
>>>>>
>>>>> You should consider simply using Solr -- it handles all
>>>>> this
>>>>> logic
>>>>> for
>>>>> you and has been well debugged with time...
>>>>>
>>>>> Mike
>>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>> The reason for the indexreader.reopen is because I have
>>>>> a
>>>>> webapp
>>>>> which
>>>>>
>>>>> enables users to upload files and then search for the
>>>>> documents.
>>>>> If
>>>>>
>>>>> I
>>>>>
>>>>> don't
>>>>>
>>>>> reopen i'm concerned that the facet hit counter won't
>>>>> be
>>>>>
>>>>> updated.
>>>>>
>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman
>>>>> <
>>>>> aminmc@gmail.com
>>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> Hi
>>>>>
>>>>>
>>>>>
>>>>> I have been able to get the code working for my
>>>>> scenario,
>>>>>
>>>>> however
>>>>>
>>>>> I
>>>>>
>>>>> have
>>>>>
>>>>> a
>>>>>
>>>>> question and I was wondering if I could get some help.
>>>>>
>>>>> I
>>>>> have
>>>>> a
>>>>> list
>>>>> of
>>>>> IndexSearchers which are used in a MultiSearcher
>>>>> class.
>>>>> I
>>>>> use
>>>>> the
>>>>> indexsearchers to get each indexreader and put them
>>>>> into
>>>>> a
>>>>> MultiIndexReader.
>>>>>
>>>>> IndexReader[] readers = new
>>>>> IndexReader[searchables.length];
>>>>>
>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>
>>>>> IndexSearcher indexSearcher =
>>>>> (IndexSearcher)searchables[i];
>>>>>
>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>
>>>>> IndexReader newReader = readers[i].reopen();
>>>>>
>>>>> if (newReader != readers[i]) {
>>>>>
>>>>> readers[i].close();
>>>>>
>>>>> }
>>>>>
>>>>> readers[i] = newReader;
>>>>>
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>> multiReader = new MultiReader(readers);
>>>>>
>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>> newOpenBitSetFacetHitCounter();
>>>>>
>>>>> IndexSearcher indexSearcher = new
>>>>> IndexSearcher(multiReader);
>>>>>
>>>>>
>>>>> I then use the indexseacher to do the facet stuff. I
>>>>> end
>>>>> the
>>>>> code
>>>>> with
>>>>> closing the multireader. This is causing problems in
>>>>> another
>>>>> method
>>>>> where I
>>>>> do some other search as the indexreaders are closed.
>>>>> Is
>>>>> it
>>>>> ok
>>>>> to
>>>>> not
>>>>> close
>>>>> the multiindexreader or should I do some additional
>>>>> checks
>>>>> in
>>>>> the
>>>>> other
>>>>> method to see if the indexreader is closed?
>>>>>
>>>>>
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> P.S. Hope that made sense...!
>>>>>
>>>>>
>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin
>>>>> Mohammed-Coleman
>>>>> <
>>>>> aminmc@gmail.com
>>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> Hi
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks just what I needed!
>>>>>
>>>>>
>>>>>
>>>>> Cheers
>>>>>
>>>>> Amin
>>>>>
>>>>>
>>>>>
>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>>>>> marcelo.ochoa@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi Amin:
>>>>>
>>>>> Please take a look a this blog post:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>> Best regards, Marcelo.
>>>>>
>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin
>>>>> Mohammed-Coleman
>>>>> <
>>>>> aminmc@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>>
>>>>> Sorry to re send this email but I was wondering if
>>>>> I
>>>>> could
>>>>> get
>>>>>
>>>>> some
>>>>>
>>>>> advice
>>>>>
>>>>> on this.
>>>>>
>>>>> Cheers
>>>>>
>>>>> Amin
>>>>>
>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>>>>> aminmc@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>>
>>>>> I am looking at building a faceted search using
>>>>> Lucene.
>>>>> I
>>>>> know
>>>>>
>>>>> that
>>>>>
>>>>> Solr
>>>>>
>>>>> comes with this built in, however I would like to
>>>>>
>>>>> try
>>>>> this
>>>>> by
>>>>> myself
>>>>> (something to add to my CV!). I have been
>>>>> looking
>>>>> around
>>>>> and
>>>>> I
>>>>> found
>>>>> that
>>>>> you can use the IndexReader and use TermVectors.
>>>>> This
>>>>> looks
>>>>> ok
>>>>> but
>>>>> I'm
>>>>> not
>>>>> sure how to filter the results so that a
>>>>> particular
>>>>> user
>>>>> can
>>>>> only
>>>>> see
>>>>> a
>>>>> subset of results. The next option I was looking
>>>>> at
>>>>> was
>>>>> something
>>>>> like
>>>>>
>>>>> Term term1 = new Term("brand", "ford");
>>>>> Term term2 = new Term("brand", "vw");
>>>>> Term[] termsArray = new Term[] { term1, term2
>>>>> };un
>>>>> int[] docFreqs =
>>>>> indexSearcher.docFreqs(termsArray);
>>>>>
>>>>> The only problem here is that I have to provide
>>>>> the
>>>>> brand
>>>>> type
>>>>> each
>>>>> time a
>>>>> new brand is created. Again I'm not sure how I
>>>>> can
>>>>> filter
>>>>> the
>>>>> results
>>>>> here.
>>>>> It may be that I'm using the wrong api methods to
>>>>> do
>>>>> this.
>>>>>
>>>>> I would be grateful if I could get some advice on
>>>>> this.
>>>>>
>>>>>
>>>>> Cheers
>>>>> Amin
>>>>>
>>>>> P.S. I am basically trying to do something that
>>>>> displays
>>>>> the
>>>>> following
>>>>>
>>>>> Personal Contact (23) Business Contact (45) and
>>>>> so
>>>>> on..
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>> Marcelo F. Ochoa
>>>>>
>>>>>
>>>>> http://marceloochoa.blogspot.com/
>>>>>
>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>
>>>>> ______________
>>>>> Want to integrate Lucene and Oracle?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>> Is Oracle 11g REST ready?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail:
>>>>> java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail:
>>>>> java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>>
>>>>>
>>>>> To unsubscribe, e-mail:
>>>>>
>>>>>
>>>>>
>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>> For additional commands, e-mail:
>>>>>
>>>>> java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>>
>>>>> To unsubscribe, e-mail:
>>>>>
>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>
>>>>>
>>>>> For additional commands, e-mail:
>>>>>
>>>>> java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>>
>>>>>
>>>>> To unsubscribe, e-mail:
>>>>>
>>>>>
>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>
>>>>>
>>>>> For additional commands, e-mail:
>>>>>
>>>>> java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>>
>>>>>
>>>>> To unsubscribe, e-mail:
>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>
>>>>>
>>>>> For additional commands, e-mail:
>>>>>
>>>>> java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>>
>>>>> To unsubscribe, e-mail:
>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>
>>>>>
>>>>> For additional commands, e-mail:
>>>>> java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>>
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>>
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>> rg" target="_blank">unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>> div>
>>>>>
>>>>>
>>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Faceted Search using Lucene
Posted by Michael McCandless <lu...@mikemccandless.com>.
Well the code looks fine.
I can't explain why you see no search results if you don't call
maybeReopen() in get, unless at the time you first create
SearcherManager the Directories each have an empty index in them.
Mike
Amin Mohammed-Coleman wrote:
> Hi
> Here is the code that I am using, I've modified the get() method to
> include
> the maybeReopen() call. Again I'm not sure if this is a good idea.
>
> public Summary[] search(final SearchRequest searchRequest)
> throwsSearchExecutionException {
>
> final String searchTerm = searchRequest.getSearchTerm();
>
> if (StringUtils.isBlank(searchTerm)) {
>
> throw new SearchExecutionException("Search string cannot be empty.
> There
> will be too many results to process.");
>
> }
>
> List<Summary> summaryList = new ArrayList<Summary>();
>
> StopWatch stopWatch = new StopWatch("searchStopWatch");
>
> stopWatch.start();
>
> MultiSearcher multiSearcher = get();
>
> try {
>
> LOGGER.debug("Ensuring all index readers are up to date...");
>
> Query query = queryParser.parse(searchTerm);
>
> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
> query.toString() +"'");
>
> Sort sort = null;
>
> sort = applySortIfApplicable(searchRequest);
>
> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>
> ChainedFilter chainedFilter = null;
>
> if (filters != null) {
>
> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>
> }
>
> TopDocs topDocs = multiSearcher.search(query,chainedFilter ,100,sort);
>
> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>
> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
> "+topDocs.
> totalHits);
>
> for (ScoreDoc scoreDoc : scoreDocs) {
>
> final Document doc = multiSearcher.doc(scoreDoc.doc);
>
> float score = scoreDoc.score;
>
> final BaseDocument baseDocument = new BaseDocument(doc, score);
>
> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>
> summaryList.add(documentSummary);
>
> }
>
> } catch (Exception e) {
>
> throw new IllegalStateException(e);
>
> } finally {
>
> if (multiSearcher != null) {
>
> release(multiSearcher);
>
> }
>
> }
>
> stopWatch.stop();
>
> LOGGER.debug("total time taken for document seach: " +
> stopWatch.getTotalTimeMillis() + " ms");
>
> return summaryList.toArray(new Summary[] {});
>
> }
>
>
> @Autowired
>
> public void setDirectories(@Qualifier("directories")ListFactoryBean
> listFactoryBean) throws Exception {
>
> this.directories = (List<Directory>) listFactoryBean.getObject();
>
> }
>
> @PostConstruct
>
> public void initialiseDocumentSearcher() {
>
> StopWatch stopWatch = new StopWatch("document-search-initialiser");
>
> stopWatch.start();
>
> PerFieldAnalyzerWrapper analyzerWrapper = new PerFieldAnalyzerWrapper(
> analyzer);
>
> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
> newKeywordAnalyzer());
>
> queryParser =
> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
> analyzerWrapper);
>
> try {
>
> LOGGER.debug("Initialising document searcher ....");
>
> documentSearcherManagers = new
> DocumentSearcherManager[directories.size()];
>
> for (int i = 0; i < directories.size() ;i++) {
>
> Directory directory = directories.get(i);
>
> DocumentSearcherManager documentSearcherManager =
> newDocumentSearcherManager(directory);
>
> documentSearcherManagers[i]=documentSearcherManager;
>
> }
>
> LOGGER.debug("Document searcher initialised");
>
> } catch (IOException e) {
>
> throw new IllegalStateException(e);
>
> }
>
> stopWatch.stop();
>
> LOGGER.debug("Total time taken to initialise DocumentSearcher '" +
> stopWatch.getTotalTimeMillis() +"' ms.");
>
> }
>
> private void maybeReopen() throws SearchExecutionException {
>
> LOGGER.debug("Initiating reopening of index readers...");
>
> for (DocumentSearcherManager documentSearcherManager :
> documentSearcherManagers) {
>
> try {
>
> documentSearcherManager.maybeReopen();
>
> } catch (InterruptedException e) {
>
> throw new SearchExecutionException(e);
>
> } catch (IOException e) {
>
> throw new SearchExecutionException(e);
>
> }
>
> }
>
> LOGGER.debug("reopening of index readers complete.");
>
> }
>
>
>
> private void release(MultiSearcher multiSeacher) {
>
> IndexSearcher[] indexSearchers = (IndexSearcher[])
> multiSeacher.getSearchables();
>
> for(int i =0 ; i < indexSearchers.length;i++) {
>
> try {
>
> documentSearcherManagers[i].release(indexSearchers[i]);
>
> } catch (IOException e) {
>
> throw new IllegalStateException(e);
>
> }
>
> }
>
> }
>
>
> private MultiSearcher get() throws SearchExecutionException {
>
> maybeReopen();
>
> MultiSearcher multiSearcher = null;
>
> List<IndexSearcher> listOfIndexSeachers = new
> ArrayList<IndexSearcher>();
>
> for (DocumentSearcherManager documentSearcherManager :
> documentSearcherManagers) {
>
> listOfIndexSeachers.add(documentSearcherManager.get());
>
> }
>
> try {
>
> multiSearcher = new
> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[] {}));
>
> } catch (IOException e) {
>
> throw new SearchExecutionException(e);
>
> }
>
> return multiSearcher;
>
> }
>
>
> Hope there is enough information.
>
>
> Cheers
>
> Amin
>
>
> P.S. I will continue to debug.
>
>
>
>
> On Mon, Mar 2, 2009 at 6:55 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>>
>> It makes perfect sense to call maybeReopen() followed by get(), as
>> long as
>> maybeReopen() is never slow enough to be noticeable to an end user
>> (because
>> you are making random queries pay the reopen/warming cost).
>>
>> If you call maybeReopen() after get(), then that search will not
>> see the
>> newly opened readers, but the next search will.
>>
>> I'm just thinking that since you see no results with get() alone,
>> debug
>> that case first. Then put back the maybeReopen().
>>
>> Can you post your full code at this point?
>>
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> Hi
>>>
>>> Just out of curiosity does it not make sense to call maybeReopen
>>> and then
>>> call get()? If I call get() then I have a new mulitsearcher, so a
>>> call to
>>> maybeopen won't reinitialise the multi searcher. Unless I pass
>>> the multi
>>> searcher into the maybereopen method. But somehow that doesn't
>>> make sense. I
>>> maybe missing something here.
>>>
>>>
>>> Cheers
>>>
>>> Amin
>>>
>>> On 2 Mar 2009, at 15:48, Amin Mohammed-Coleman <am...@gmail.com>
>>> wrote:
>>>
>>> I'm seeing some interesting behviour when i do get() first
>>> followed by
>>>> maybeReopen then there are no documents in the directory
>>>> (directory that i
>>>> am interested in. When i do the maybeReopen and then get() then
>>>> the doc
>>>> count is correct. I can post stats later.
>>>>
>>>> Weird...
>>>>
>>>> On Mon, Mar 2, 2009 at 2:17 PM, Amin Mohammed-Coleman <aminmc@gmail.com
>>>> >
>>>> wrote:
>>>> oh dear...i think i may cry...i'll debug.
>>>>
>>>>
>>>> On Mon, Mar 2, 2009 at 2:15 PM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>> Or even just get() with no call to maybeReopen(). That should
>>>> work fine
>>>> as well.
>>>>
>>>>
>>>> Mike
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> In my test case I have a set up method that should populate the
>>>> indexes
>>>> before I start using the document searcher. I will start adding
>>>> some
>>>> more
>>>> debug statements. So basically I should be able to do: get()
>>>> followed by
>>>> maybeReopen.
>>>>
>>>> I will let you know what the outcome is.
>>>>
>>>>
>>>> Cheers
>>>> Amin
>>>>
>>>> On Mon, Mar 2, 2009 at 1:39 PM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>>
>>>> Is it possible that when you first create the SearcherManager,
>>>> there is
>>>> no
>>>> index in each Directory?
>>>>
>>>> If not... you better start adding diagnostics. EG inside your
>>>> get(),
>>>> print
>>>> out the numDocs() of each IndexReader you get from the
>>>> SearcherManager?
>>>>
>>>> Something is wrong and it's best to explain it...
>>>>
>>>>
>>>> Mike
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> Nope. If i remove the maybeReopen the search doesn't work. It
>>>> only works
>>>> when i cal maybeReopen followed by get().
>>>>
>>>> Cheers
>>>> Amin
>>>>
>>>> On Mon, Mar 2, 2009 at 12:56 PM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>>
>>>> That's not right; something must be wrong.
>>>>
>>>> get() before maybeReopen() should simply let you search based on
>>>> the
>>>> searcher before reopening.
>>>>
>>>> If you just do get() and don't call maybeReopen() does it work?
>>>>
>>>>
>>>> Mike
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> I noticed that if i do the get() before the maybeReopen then I
>>>> get no
>>>>
>>>> results. But otherwise I can change it further.
>>>>
>>>> On Mon, Mar 2, 2009 at 11:46 AM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>>
>>>> There is no such thing as final code -- code is alive and is always
>>>> changing ;)
>>>>
>>>> It looks good to me.
>>>>
>>>> Though one trivial thing is: I would move the code in the try
>>>> clause up
>>>> to
>>>> and including the multiSearcher=get() out above the try. I always
>>>> attempt
>>>> to "shrink wrap" what's inside a try clause to the minimum that
>>>> needs
>>>> to
>>>> be
>>>> there. Ie, your code that creates a query, finds the right sort &
>>>> filter
>>>> to
>>>> use, etc, can all happen outside the try, because you have not yet
>>>> acquired
>>>> the multiSearcher.
>>>>
>>>> If you do that, you also don't need the null check in the finally
>>>> clause,
>>>> because multiSearcher must be non-null on entering the try.
>>>>
>>>> Mike
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> Hi there
>>>>
>>>> Good morning! Here is the final search code:
>>>>
>>>> public Summary[] search(final SearchRequest searchRequest)
>>>> throwsSearchExecutionException {
>>>>
>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>
>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>
>>>> throw new SearchExecutionException("Search string cannot be empty.
>>>> There
>>>> will be too many results to process.");
>>>>
>>>> }
>>>>
>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>
>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>
>>>> stopWatch.start();
>>>>
>>>> MultiSearcher multiSearcher = null;
>>>>
>>>> try {
>>>>
>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>
>>>> maybeReopen();
>>>>
>>>> Query query = queryParser.parse(searchTerm);
>>>>
>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query
>>>> '" +
>>>> query.toString() +"'");
>>>>
>>>> Sort sort = null;
>>>>
>>>> sort = applySortIfApplicable(searchRequest);
>>>>
>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>
>>>> ChainedFilter chainedFilter = null;
>>>>
>>>> if (filters != null) {
>>>>
>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>
>>>> }
>>>>
>>>> multiSearcher = get();
>>>>
>>>> TopDocs topDocs = multiSearcher.search(query,chainedFilter ,
>>>> 100,sort);
>>>>
>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>
>>>> LOGGER.debug("total number of hits for [" + query.toString() +
>>>> " ] =
>>>> "+topDocs.
>>>> totalHits);
>>>>
>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>
>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>
>>>> float score = scoreDoc.score;
>>>>
>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>
>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>
>>>> summaryList.add(documentSummary);
>>>>
>>>> }
>>>>
>>>> } catch (Exception e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> } finally {
>>>>
>>>> if (multiSearcher != null) {
>>>>
>>>> release(multiSearcher);
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> stopWatch.stop();
>>>>
>>>> LOGGER.debug("total time taken for document seach: " +
>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>
>>>> return summaryList.toArray(new Summary[] {});
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> I hope this makes sense...thanks again!
>>>>
>>>>
>>>> Cheers
>>>>
>>>> Amin
>>>>
>>>>
>>>>
>>>> On Sun, Mar 1, 2009 at 8:09 PM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>>
>>>> You're calling get() too many times. For every call to get() you
>>>> must
>>>>
>>>> match with a call to release().
>>>>
>>>> So, once at the front of your search method you should:
>>>>
>>>> MultiSearcher searcher = get();
>>>>
>>>> then use that searcher to do searching, retrieve docs, etc.
>>>>
>>>> Then in the finally clause, pass that searcher to release.
>>>>
>>>> So, only one call to get() and one matching call to release().
>>>>
>>>> Mike
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> Hi
>>>>
>>>> The searchers are injected into the class via Spring. So when a
>>>>
>>>> client
>>>> calls the class it is fully configured with a list of index
>>>> searchers.
>>>> However I have removed this list and instead injecting a list of
>>>> directories which are passed to the DocumentSearchManager.
>>>> DocumentSearchManager is SearchManager (should've mentioned that
>>>> earlier).
>>>> So finally I have modified by release code to do the following:
>>>>
>>>> private void release(MultiSearcher multiSeacher) throws Exception {
>>>>
>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>> multiSeacher.getSearchables();
>>>>
>>>> for(int i =0 ; i < indexSearchers.length;i++) {
>>>>
>>>> documentSearcherManagers[i].release(indexSearchers[i]);
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> and it's use looks like this:
>>>>
>>>>
>>>> public Summary[] search(final SearchRequest searchRequest)
>>>> throwsSearchExecutionException {
>>>>
>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>
>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>
>>>> throw new SearchExecutionException("Search string cannot be empty.
>>>> There
>>>> will be too many results to process.");
>>>>
>>>> }
>>>>
>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>
>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>
>>>> stopWatch.start();
>>>>
>>>> List<IndexSearcher> indexSearchers = new
>>>> ArrayList<IndexSearcher>();
>>>>
>>>> try {
>>>>
>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>
>>>> maybeReopen();
>>>>
>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>> searchers
>>>> '"
>>>> +
>>>> indexSearchers.size() +"'");
>>>>
>>>> Query query = queryParser.parse(searchTerm);
>>>>
>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '"
>>>> +
>>>> query.toString() +"'");
>>>>
>>>> Sort sort = null;
>>>>
>>>> sort = applySortIfApplicable(searchRequest);
>>>>
>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>
>>>> ChainedFilter chainedFilter = null;
>>>>
>>>> if (filters != null) {
>>>>
>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>
>>>> }
>>>>
>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>
>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>
>>>> LOGGER.debug("total number of hits for [" + query.toString() +
>>>> " ] =
>>>> "+topDocs.
>>>> totalHits);
>>>>
>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>
>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>
>>>> float score = scoreDoc.score;
>>>>
>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>
>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>
>>>> summaryList.add(documentSummary);
>>>>
>>>> }
>>>>
>>>> } catch (Exception e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> } finally {
>>>>
>>>> release(get());
>>>>
>>>> }
>>>>
>>>> stopWatch.stop();
>>>>
>>>> LOGGER.debug("total time taken for document seach: " +
>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>
>>>> return summaryList.toArray(new Summary[] {});
>>>>
>>>> }
>>>>
>>>>
>>>> So the final post construct constructs the DocumentSearchMangers
>>>> with
>>>> the
>>>> list of directories..looking like this
>>>>
>>>>
>>>> @PostConstruct
>>>>
>>>> public void initialiseDocumentSearcher() {
>>>>
>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>> PerFieldAnalyzerWrapper(
>>>> analyzer);
>>>>
>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>> newKeywordAnalyzer());
>>>>
>>>> queryParser =
>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>> analyzerWrapper);
>>>>
>>>> try {
>>>>
>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>
>>>> documentSearcherManagers = new
>>>> DocumentSearcherManager[directories.size()];
>>>>
>>>> for (int i = 0; i < directories.size() ;i++) {
>>>>
>>>> Directory directory = directories.get(i);
>>>>
>>>> DocumentSearcherManager documentSearcherManager =
>>>> newDocumentSearcherManager(directory);
>>>>
>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>
>>>> }
>>>>
>>>> LOGGER.debug("multi searcher initialised");
>>>>
>>>> } catch (IOException e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> Cheers
>>>>
>>>> Amin
>>>>
>>>>
>>>>
>>>> On Sun, Mar 1, 2009 at 6:15 PM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>>
>>>> I don't understand where searchers comes from, prior to
>>>>
>>>> initializeDocumentSearcher? You should, instead, simply create the
>>>> SearcherManager (from your Directory instances). You don't need
>>>> any
>>>> searchers during initialize.
>>>>
>>>> Is DocumentSearcherManager the same as SearcherManager (just
>>>> renamed)?
>>>>
>>>> The release method is wrong -- you're calling .get() and then
>>>> immediately release. Instead, you should step through the
>>>> searchers
>>>> from your MultiSearcher and release them to each SearcherManager.
>>>>
>>>> You should call your release() in a finally clause.
>>>>
>>>> Mike
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> Sorry...i'm getting slightly confused.
>>>>
>>>> I have a PostConstruct which is where I should create an array of
>>>>
>>>> SearchManagers (per indexSeacher). From there I initialise the
>>>> multisearcher using the get(). After which I need to call
>>>> maybeReopen
>>>> for
>>>> each IndexSearcher. So I'll do the following:
>>>>
>>>> @PostConstruct
>>>>
>>>> public void initialiseDocumentSearcher() {
>>>>
>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>> PerFieldAnalyzerWrapper(
>>>> analyzer);
>>>>
>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>> newKeywordAnalyzer());
>>>>
>>>> queryParser =
>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>> analyzerWrapper);
>>>>
>>>> try {
>>>>
>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>
>>>> documentSearcherManagers = new
>>>> DocumentSearcherManager[searchers.size()];
>>>>
>>>> for (int i = 0; i < searchers.size() ;i++) {
>>>>
>>>> IndexSearcher indexSearcher = searchers.get(i);
>>>>
>>>> Directory directory = indexSearcher.getIndexReader().directory();
>>>>
>>>> DocumentSearcherManager documentSearcherManager =
>>>> newDocumentSearcherManager(directory);
>>>>
>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>
>>>> }
>>>>
>>>> LOGGER.debug("multi searcher initialised");
>>>>
>>>> } catch (IOException e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> This initialises search managers. I then have methods:
>>>>
>>>>
>>>> private void maybeReopen() throws Exception {
>>>>
>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>
>>>> for (DocumentSearcherManager documentSearcherManager :
>>>> documentSearcherManagers) {
>>>>
>>>> documentSearcherManager.maybeReopen();
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> private void release() throws Exception {
>>>>
>>>> for (DocumentSearcherManager documentSearcherManager :
>>>> documentSearcherManagers) {
>>>>
>>>> documentSearcherManager.release(documentSearcherManager.get());
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> private MultiSearcher get() {
>>>>
>>>> List<IndexSearcher> listOfIndexSeachers = new
>>>> ArrayList<IndexSearcher>();
>>>>
>>>> for (DocumentSearcherManager documentSearcherManager :
>>>> documentSearcherManagers) {
>>>>
>>>> listOfIndexSeachers.add(documentSearcherManager.get());
>>>>
>>>> }
>>>>
>>>> try {
>>>>
>>>> multiSearcher = new
>>>> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[] {}));
>>>>
>>>> } catch (IOException e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> }
>>>>
>>>> return multiSearcher;
>>>>
>>>> }
>>>>
>>>>
>>>> These methods are used in the following manner in the search code:
>>>>
>>>>
>>>> public Summary[] search(final SearchRequest searchRequest)
>>>> throwsSearchExecutionException {
>>>>
>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>
>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>
>>>> throw new SearchExecutionException("Search string cannot be empty.
>>>> There
>>>> will be too many results to process.");
>>>>
>>>> }
>>>>
>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>
>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>
>>>> stopWatch.start();
>>>>
>>>> List<IndexSearcher> indexSearchers = new
>>>> ArrayList<IndexSearcher>();
>>>>
>>>> try {
>>>>
>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>
>>>> maybeReopen();
>>>>
>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>> searchers
>>>> '"
>>>> +
>>>> indexSearchers.size() +"'");
>>>>
>>>> Query query = queryParser.parse(searchTerm);
>>>>
>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query
>>>> '"
>>>> +
>>>> query.toString() +"'");
>>>>
>>>> Sort sort = null;
>>>>
>>>> sort = applySortIfApplicable(searchRequest);
>>>>
>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>
>>>> ChainedFilter chainedFilter = null;
>>>>
>>>> if (filters != null) {
>>>>
>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>
>>>> }
>>>>
>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>
>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>
>>>> LOGGER.debug("total number of hits for [" + query.toString() + " ]
>>>> =
>>>> "+topDocs.
>>>> totalHits);
>>>>
>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>
>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>
>>>> float score = scoreDoc.score;
>>>>
>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>
>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>
>>>> summaryList.add(documentSummary);
>>>>
>>>> }
>>>>
>>>> release();
>>>>
>>>> } catch (Exception e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> }
>>>>
>>>> stopWatch.stop();
>>>>
>>>> LOGGER.debug("total time taken for document seach: " +
>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>
>>>> return summaryList.toArray(new Summary[] {});
>>>>
>>>> }
>>>>
>>>>
>>>> Does this look better? Again..I really really appreciate your
>>>> help!
>>>>
>>>>
>>>> On Sun, Mar 1, 2009 at 4:18 PM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>>
>>>> This is not quite right -- you should only create SearcherManager
>>>> once
>>>>
>>>> (per Direcotry) at startup/app load, not with every search
>>>> request.
>>>>
>>>>
>>>> And I don't see release -- it must call SearcherManager.release
>>>> of
>>>> each of the IndexSearchers previously returned from get().
>>>>
>>>> Mike
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> Hi
>>>>
>>>> Thanks again for helping on a Sunday!
>>>>
>>>>
>>>> I have now modified my maybeOpen() to do the following:
>>>>
>>>> private void maybeReopen() throws Exception {
>>>>
>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>
>>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>>> .getSearchables();
>>>>
>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>
>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>
>>>> SearcherManager documentSearcherManager = new
>>>> SearcherManager(indexReader.directory());
>>>>
>>>> documentSearcherManager.maybeReopen();
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> And get() to:
>>>>
>>>>
>>>> private synchronized MultiSearcher get() {
>>>>
>>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>>> .getSearchables();
>>>>
>>>> List<IndexSearcher> indexSearchersList = new
>>>> ArrayList<IndexSearcher>();
>>>>
>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>
>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>
>>>> SearcherManager documentSearcherManager = null;
>>>>
>>>> try {
>>>>
>>>> documentSearcherManager = new
>>>> SearcherManager(indexReader.directory());
>>>>
>>>> } catch (IOException e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> }
>>>>
>>>> indexSearchersList.add(documentSearcherManager.get());
>>>>
>>>> }
>>>>
>>>> try {
>>>>
>>>> multiSearcher = new
>>>> MultiSearcher(indexSearchersList.toArray(newIndexSearcher[]
>>>> {}));
>>>>
>>>> } catch (IOException e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> }
>>>>
>>>> return multiSearcher;
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> This makes all my test pass. I am using the SearchManager that
>>>> you
>>>> recommended. Does this look ok?
>>>>
>>>>
>>>> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>> Your maybeReopen has an excess incRef().
>>>>
>>>>
>>>> I'm not sure how you open the searchers in the first place? The
>>>>
>>>> list
>>>> starts as empty, and nothing populates it?
>>>>
>>>> When you do the initial population, you need an incRef.
>>>>
>>>> I think you're hitting IllegalStateException because
>>>> maybeReopen
>>>> is
>>>> closing a reader before get() can get it (since they
>>>> synchronize
>>>> on
>>>> different objects).
>>>>
>>>> I'd recommend switching to the SearcherManager class.
>>>> Instantiate
>>>> one
>>>> for each of your searchers. On each search request, go through
>>>> them
>>>> and call maybeReopen(), and then call get() and gather each
>>>> IndexSearcher instance into a new array. Then, make a new
>>>> MultiSearcher (opposite of what I said before): while that
>>>> creates
>>>> a
>>>> small amount of garbage, it'll keep your code simpler (good
>>>> tradeoff).
>>>>
>>>> Mike
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> sorrry I added
>>>>
>>>>
>>>> release(multiSearcher);
>>>>
>>>>
>>>>
>>>> instead of multiSearcher.close();
>>>>
>>>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <
>>>> aminmc@gmail.com
>>>>
>>>> wrote:
>>>>
>>>>
>>>>
>>>> Hi
>>>>
>>>>
>>>> I've now done the following:
>>>>
>>>>
>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>
>>>> throwsSearchExecutionException {
>>>>
>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>
>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>
>>>> throw new SearchExecutionException("Search string cannot be
>>>> empty.
>>>> There
>>>> will be too many results to process.");
>>>>
>>>> }
>>>>
>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>
>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>
>>>> stopWatch.start();
>>>>
>>>> List<IndexSearcher> indexSearchers = new
>>>> ArrayList<IndexSearcher>();
>>>>
>>>> try {
>>>>
>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>
>>>> maybeReopen();
>>>>
>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>> searchers
>>>> '"+ indexSearchers.size() +
>>>> "'");
>>>>
>>>> Query query = queryParser.parse(searchTerm);
>>>>
>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>>>> Query
>>>> '"
>>>> +
>>>> query.toString() +"'");
>>>>
>>>> Sort sort = null;
>>>>
>>>> sort = applySortIfApplicable(searchRequest);
>>>>
>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>
>>>> ChainedFilter chainedFilter = null;
>>>>
>>>> if (filters != null) {
>>>>
>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>
>>>> }
>>>>
>>>> TopDocs topDocs = get().search(query,chainedFilter
>>>> ,100,sort);
>>>>
>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>
>>>> LOGGER.debug("total number of hits for [" + query.toString()
>>>> +
>>>> "
>>>> ]
>>>> =
>>>> "+topDocs.
>>>> totalHits);
>>>>
>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>
>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>
>>>> float score = scoreDoc.score;
>>>>
>>>> final BaseDocument baseDocument = new BaseDocument(doc,
>>>> score);
>>>>
>>>> Summary documentSummary = new
>>>> DocumentSummaryImpl(baseDocument);
>>>>
>>>> summaryList.add(documentSummary);
>>>>
>>>> }
>>>>
>>>> multiSearcher.close();
>>>>
>>>> } catch (Exception e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> }
>>>>
>>>> stopWatch.stop();
>>>>
>>>> LOGGER.debug("total time taken for document seach: " +
>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>
>>>> return summaryList.toArray(new Summary[] {});
>>>>
>>>> }
>>>>
>>>>
>>>> And have the following methods:
>>>>
>>>> @PostConstruct
>>>>
>>>> public void initialiseQueryParser() {
>>>>
>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>> PerFieldAnalyzerWrapper(
>>>> analyzer);
>>>>
>>>>
>>>>
>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>> newKeywordAnalyzer());
>>>>
>>>> queryParser =
>>>>
>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>
>>>> analyzerWrapper);
>>>>
>>>> try {
>>>>
>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>
>>>> this.multiSearcher = new
>>>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>>>
>>>> LOGGER.debug("multi searcher initialised");
>>>>
>>>> } catch (IOException e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> Initialises mutltisearcher when this class is creared by
>>>> spring.
>>>>
>>>>
>>>> private synchronized void swapMultiSearcher(MultiSearcher
>>>> newMultiSearcher) {
>>>>
>>>> try {
>>>>
>>>> release(multiSearcher);
>>>>
>>>> } catch (IOException e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> }
>>>>
>>>> multiSearcher = newMultiSearcher;
>>>>
>>>> }
>>>>
>>>> public void maybeReopen() throws IOException {
>>>>
>>>> MultiSearcher newMultiSeacher = null;
>>>>
>>>> boolean refreshMultiSeacher = false;
>>>>
>>>> List<IndexSearcher> indexSearchers = new
>>>> ArrayList<IndexSearcher>();
>>>>
>>>> synchronized (searchers) {
>>>>
>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>
>>>> IndexReader reader = indexSearcher.getIndexReader();
>>>>
>>>> reader.incRef();
>>>>
>>>> Directory directory = reader.directory();
>>>>
>>>> long currentVersion = reader.getVersion();
>>>>
>>>> if (IndexReader.getCurrentVersion(directory) !=
>>>> currentVersion)
>>>> {
>>>>
>>>> IndexReader newReader =
>>>> indexSearcher.getIndexReader().reopen();
>>>>
>>>> if (newReader != reader) {
>>>>
>>>> reader.decRef();
>>>>
>>>> refreshMultiSeacher = true;
>>>>
>>>> }
>>>>
>>>> reader = newReader;
>>>>
>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>
>>>> indexSearchers.add(newSearcher);
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> if (refreshMultiSeacher) {
>>>>
>>>> newMultiSeacher = new
>>>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
>>>>
>>>> warm(newMultiSeacher);
>>>>
>>>> swapMultiSearcher(newMultiSeacher);
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> }
>>>>
>>>>
>>>> private void warm(MultiSearcher newMultiSeacher) {
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> private synchronized MultiSearcher get() {
>>>>
>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>
>>>> indexSearcher.getIndexReader().incRef();
>>>>
>>>> }
>>>>
>>>> return multiSearcher;
>>>>
>>>> }
>>>>
>>>> private synchronized void release(MultiSearcher
>>>> multiSearcher)
>>>> throwsIOException {
>>>>
>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>
>>>> indexSearcher.getIndexReader().decRef();
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> However I am now getting
>>>>
>>>>
>>>> java.lang.IllegalStateException:
>>>> org.apache.lucene.store.AlreadyClosedException: this
>>>> IndexReader
>>>> is
>>>> closed
>>>>
>>>>
>>>> on the call:
>>>>
>>>>
>>>> private synchronized MultiSearcher get() {
>>>>
>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>
>>>> indexSearcher.getIndexReader().incRef();
>>>>
>>>> }
>>>>
>>>> return multiSearcher;
>>>>
>>>> }
>>>>
>>>>
>>>> I'm doing something wrong ..obviously..not sure where
>>>> though..
>>>>
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>>
>>>> I was wondering the same thing ;)
>>>>
>>>>
>>>> It's best to call this method from a single BG "warming"
>>>> thread,
>>>>
>>>> in
>>>> which
>>>> case it would not need its own synchronization.
>>>>
>>>> But, to be safe, I'll add internal synchronization to it.
>>>> You
>>>> can't
>>>> simply put synchronized in front of the method, since you
>>>> don't
>>>> want
>>>> this to
>>>> block searching.
>>>>
>>>>
>>>> Mike
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> just a quick point:
>>>>
>>>> public void maybeReopen() throws IOException {
>>>> //D
>>>>
>>>> long currentVersion =
>>>>
>>>> currentSearcher.getIndexReader().getVersion();
>>>>
>>>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>>>> IndexReader newReader =
>>>> currentSearcher.getIndexReader().reopen();
>>>> assert newReader != currentSearcher.getIndexReader();
>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>> warm(newSearcher);
>>>> swapSearcher(newSearcher);
>>>> }
>>>> }
>>>>
>>>> should the above be synchronised?
>>>>
>>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
>>>> aminmc@gmail.com
>>>>
>>>> wrote:
>>>>
>>>>
>>>>
>>>> thanks. i will rewrite..in between giving my baby her feed
>>>>
>>>> and
>>>>
>>>> playing
>>>>
>>>>
>>>> with the other child and my wife who wants me to do several
>>>> other
>>>>
>>>> things!
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>>
>>>> Hi
>>>>
>>>>
>>>> Thanks for your input. I would like to have a go at
>>>> doing
>>>> this
>>>>
>>>> myself
>>>>
>>>> first, Solr may be an option.
>>>>
>>>>
>>>> * You are creating a new Analyzer & QueryParser every
>>>> time,
>>>> also
>>>> creating unnecessary garbage; instead, they should be
>>>> created
>>>> once
>>>> & reused.
>>>>
>>>> -- I can moved the code out so that it is only created
>>>> once
>>>> and
>>>> reused.
>>>>
>>>>
>>>> * You always make a new IndexSearcher and a new
>>>> MultiSearcher
>>>> even
>>>> when nothing has changed. This just generates
>>>> unnecessary
>>>> garbage
>>>> which GC then must sweep up.
>>>>
>>>> -- This was something I thought about. I could move it
>>>> out
>>>> so
>>>> that
>>>> it's
>>>> created once. However I presume inside my code i need
>>>> to
>>>> check
>>>> whether
>>>> the
>>>> indexreaders are update to date. This needs to be
>>>> synchronized
>>>> as
>>>> well I
>>>> guess(?)
>>>>
>>>>
>>>> Yes you should synchronize the check for whether the
>>>> IndexReader
>>>> is
>>>>
>>>> current.
>>>>
>>>>
>>>> * I don't see any synchronization -- it looks like two
>>>>
>>>> search
>>>>
>>>> requests are allowed into this method at the same time?
>>>> Which
>>>> is
>>>>
>>>> dangerous... eg both (or, more) will wastefully reopen
>>>> the
>>>>
>>>> readers.
>>>>
>>>> -- So i need to extract the logic for reopening and
>>>> provide
>>>> a
>>>> synchronisation mechanism.
>>>>
>>>>
>>>> Yes.
>>>>
>>>>
>>>>
>>>> Ok. So I have some work to do. I'll refactor the code
>>>> and
>>>>
>>>> see
>>>> if
>>>> I
>>>> can
>>>>
>>>> get
>>>>
>>>> inline to your recommendations.
>>>>
>>>>
>>>>
>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>>
>>>> On a quick look, I think there are a few problems with
>>>> the
>>>> code:
>>>>
>>>>
>>>> * I don't see any synchronization -- it looks like two
>>>> search
>>>>
>>>> requests are allowed into this method at the same time?
>>>>
>>>> Which
>>>> is
>>>> dangerous... eg both (or, more) will wastefully reopen
>>>> the
>>>> readers.
>>>>
>>>> * You are over-incRef'ing (the reader.incRef inside the
>>>> loop)
>>>> --
>>>> I
>>>> don't see a corresponding decRef.
>>>>
>>>> * You reopen and warm your searchers "live" (vs with BG
>>>> thread);
>>>> meaning the unlucky search request that hits a reopen
>>>> pays
>>>> the
>>>> cost. This might be OK if the index is small enough
>>>> that
>>>> reopening & warming takes very little time. But if
>>>> index
>>>> gets
>>>> large, making a random search pay that warming cost is
>>>> not
>>>> nice
>>>> to
>>>> the end user. It erodes their trust in you.
>>>>
>>>> * You always make a new IndexSearcher and a new
>>>> MultiSearcher
>>>> even
>>>> when nothing has changed. This just generates
>>>> unnecessary
>>>> garbage
>>>> which GC then must sweep up.
>>>>
>>>> * You are creating a new Analyzer & QueryParser every
>>>> time,
>>>> also
>>>> creating unnecessary garbage; instead, they should be
>>>> created
>>>> once
>>>> & reused.
>>>>
>>>> You should consider simply using Solr -- it handles all
>>>> this
>>>> logic
>>>> for
>>>> you and has been well debugged with time...
>>>>
>>>> Mike
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> The reason for the indexreader.reopen is because I have
>>>> a
>>>> webapp
>>>> which
>>>>
>>>> enables users to upload files and then search for the
>>>> documents.
>>>> If
>>>>
>>>> I
>>>>
>>>> don't
>>>>
>>>> reopen i'm concerned that the facet hit counter won't
>>>> be
>>>>
>>>> updated.
>>>>
>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman
>>>> <
>>>> aminmc@gmail.com
>>>>
>>>> wrote:
>>>>
>>>>
>>>>
>>>> Hi
>>>>
>>>>
>>>>
>>>> I have been able to get the code working for my
>>>> scenario,
>>>>
>>>> however
>>>>
>>>> I
>>>>
>>>> have
>>>>
>>>> a
>>>>
>>>> question and I was wondering if I could get some help.
>>>>
>>>> I
>>>> have
>>>> a
>>>> list
>>>> of
>>>> IndexSearchers which are used in a MultiSearcher
>>>> class.
>>>> I
>>>> use
>>>> the
>>>> indexsearchers to get each indexreader and put them
>>>> into
>>>> a
>>>> MultiIndexReader.
>>>>
>>>> IndexReader[] readers = new
>>>> IndexReader[searchables.length];
>>>>
>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>
>>>> IndexSearcher indexSearcher =
>>>> (IndexSearcher)searchables[i];
>>>>
>>>> readers[i] = indexSearcher.getIndexReader();
>>>>
>>>> IndexReader newReader = readers[i].reopen();
>>>>
>>>> if (newReader != readers[i]) {
>>>>
>>>> readers[i].close();
>>>>
>>>> }
>>>>
>>>> readers[i] = newReader;
>>>>
>>>>
>>>>
>>>> }
>>>>
>>>> multiReader = new MultiReader(readers);
>>>>
>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>> newOpenBitSetFacetHitCounter();
>>>>
>>>> IndexSearcher indexSearcher = new
>>>> IndexSearcher(multiReader);
>>>>
>>>>
>>>> I then use the indexseacher to do the facet stuff. I
>>>> end
>>>> the
>>>> code
>>>> with
>>>> closing the multireader. This is causing problems in
>>>> another
>>>> method
>>>> where I
>>>> do some other search as the indexreaders are closed.
>>>> Is
>>>> it
>>>> ok
>>>> to
>>>> not
>>>> close
>>>> the multiindexreader or should I do some additional
>>>> checks
>>>> in
>>>> the
>>>> other
>>>> method to see if the indexreader is closed?
>>>>
>>>>
>>>>
>>>> Cheers
>>>>
>>>>
>>>> P.S. Hope that made sense...!
>>>>
>>>>
>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin
>>>> Mohammed-Coleman
>>>> <
>>>> aminmc@gmail.com
>>>>
>>>> wrote:
>>>>
>>>>
>>>>
>>>> Hi
>>>>
>>>>
>>>>
>>>>
>>>> Thanks just what I needed!
>>>>
>>>>
>>>>
>>>> Cheers
>>>>
>>>> Amin
>>>>
>>>>
>>>>
>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>>>> marcelo.ochoa@gmail.com>
>>>> wrote:
>>>>
>>>> Hi Amin:
>>>>
>>>> Please take a look a this blog post:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>> Best regards, Marcelo.
>>>>
>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin
>>>> Mohammed-Coleman
>>>> <
>>>> aminmc@gmail.com>
>>>> wrote:
>>>>
>>>> Hi
>>>>
>>>>
>>>> Sorry to re send this email but I was wondering if
>>>> I
>>>> could
>>>> get
>>>>
>>>> some
>>>>
>>>> advice
>>>>
>>>> on this.
>>>>
>>>> Cheers
>>>>
>>>> Amin
>>>>
>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>>>> aminmc@gmail.com>
>>>> wrote:
>>>>
>>>> Hi
>>>>
>>>>
>>>> I am looking at building a faceted search using
>>>> Lucene.
>>>> I
>>>> know
>>>>
>>>> that
>>>>
>>>> Solr
>>>>
>>>> comes with this built in, however I would like to
>>>>
>>>> try
>>>> this
>>>> by
>>>> myself
>>>> (something to add to my CV!). I have been
>>>> looking
>>>> around
>>>> and
>>>> I
>>>> found
>>>> that
>>>> you can use the IndexReader and use TermVectors.
>>>> This
>>>> looks
>>>> ok
>>>> but
>>>> I'm
>>>> not
>>>> sure how to filter the results so that a
>>>> particular
>>>> user
>>>> can
>>>> only
>>>> see
>>>> a
>>>> subset of results. The next option I was looking
>>>> at
>>>> was
>>>> something
>>>> like
>>>>
>>>> Term term1 = new Term("brand", "ford");
>>>> Term term2 = new Term("brand", "vw");
>>>> Term[] termsArray = new Term[] { term1, term2
>>>> };un
>>>> int[] docFreqs =
>>>> indexSearcher.docFreqs(termsArray);
>>>>
>>>> The only problem here is that I have to provide
>>>> the
>>>> brand
>>>> type
>>>> each
>>>> time a
>>>> new brand is created. Again I'm not sure how I
>>>> can
>>>> filter
>>>> the
>>>> results
>>>> here.
>>>> It may be that I'm using the wrong api methods to
>>>> do
>>>> this.
>>>>
>>>> I would be grateful if I could get some advice on
>>>> this.
>>>>
>>>>
>>>> Cheers
>>>> Amin
>>>>
>>>> P.S. I am basically trying to do something that
>>>> displays
>>>> the
>>>> following
>>>>
>>>> Personal Contact (23) Business Contact (45) and
>>>> so
>>>> on..
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>>
>>>> Marcelo F. Ochoa
>>>>
>>>>
>>>> http://marceloochoa.blogspot.com/
>>>>
>>>> http://marcelo.ochoa.googlepages.com/home
>>>>
>>>> ______________
>>>> Want to integrate Lucene and Oracle?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>> Is Oracle 11g REST ready?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail:
>>>> java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail:
>>>> java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>>
>>>>
>>>> To unsubscribe, e-mail:
>>>>
>>>>
>>>>
>>>> java-user-unsubscribe@lucene.apache.org
>>>>
>>>>
>>>>
>>>> For additional commands, e-mail:
>>>>
>>>> java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>>
>>>> To unsubscribe, e-mail:
>>>>
>>>> java-user-unsubscribe@lucene.apache.org
>>>>
>>>>
>>>> For additional commands, e-mail:
>>>>
>>>> java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>>
>>>>
>>>> To unsubscribe, e-mail:
>>>>
>>>>
>>>> java-user-unsubscribe@lucene.apache.org
>>>>
>>>>
>>>> For additional commands, e-mail:
>>>>
>>>> java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>>
>>>>
>>>> To unsubscribe, e-mail:
>>>> java-user-unsubscribe@lucene.apache.org
>>>>
>>>>
>>>> For additional commands, e-mail:
>>>>
>>>> java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>>
>>>> To unsubscribe, e-mail:
>>>> java-user-unsubscribe@lucene.apache.org
>>>>
>>>>
>>>> For additional commands, e-mail:
>>>> java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>>
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>>
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>> rg" target="_blank">unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>> div>
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
Hi
Here is the code that I am using, I've modified the get() method to include
the maybeReopen() call. Again I'm not sure if this is a good idea.
public Summary[] search(final SearchRequest searchRequest)
throwsSearchExecutionException {
final String searchTerm = searchRequest.getSearchTerm();
if (StringUtils.isBlank(searchTerm)) {
throw new SearchExecutionException("Search string cannot be empty. There
will be too many results to process.");
}
List<Summary> summaryList = new ArrayList<Summary>();
StopWatch stopWatch = new StopWatch("searchStopWatch");
stopWatch.start();
MultiSearcher multiSearcher = get();
try {
LOGGER.debug("Ensuring all index readers are up to date...");
Query query = queryParser.parse(searchTerm);
LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
query.toString() +"'");
Sort sort = null;
sort = applySortIfApplicable(searchRequest);
Filter[] filters =applyFiltersIfApplicable(searchRequest);
ChainedFilter chainedFilter = null;
if (filters != null) {
chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
}
TopDocs topDocs = multiSearcher.search(query,chainedFilter ,100,sort);
ScoreDoc[] scoreDocs = topDocs.scoreDocs;
LOGGER.debug("total number of hits for [" + query.toString() + " ] = "+topDocs.
totalHits);
for (ScoreDoc scoreDoc : scoreDocs) {
final Document doc = multiSearcher.doc(scoreDoc.doc);
float score = scoreDoc.score;
final BaseDocument baseDocument = new BaseDocument(doc, score);
Summary documentSummary = new DocumentSummaryImpl(baseDocument);
summaryList.add(documentSummary);
}
} catch (Exception e) {
throw new IllegalStateException(e);
} finally {
if (multiSearcher != null) {
release(multiSearcher);
}
}
stopWatch.stop();
LOGGER.debug("total time taken for document seach: " +
stopWatch.getTotalTimeMillis() + " ms");
return summaryList.toArray(new Summary[] {});
}
@Autowired
public void setDirectories(@Qualifier("directories")ListFactoryBean
listFactoryBean) throws Exception {
this.directories = (List<Directory>) listFactoryBean.getObject();
}
@PostConstruct
public void initialiseDocumentSearcher() {
StopWatch stopWatch = new StopWatch("document-search-initialiser");
stopWatch.start();
PerFieldAnalyzerWrapper analyzerWrapper = new PerFieldAnalyzerWrapper(
analyzer);
analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
newKeywordAnalyzer());
queryParser = newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
analyzerWrapper);
try {
LOGGER.debug("Initialising document searcher ....");
documentSearcherManagers = new DocumentSearcherManager[directories.size()];
for (int i = 0; i < directories.size() ;i++) {
Directory directory = directories.get(i);
DocumentSearcherManager documentSearcherManager =
newDocumentSearcherManager(directory);
documentSearcherManagers[i]=documentSearcherManager;
}
LOGGER.debug("Document searcher initialised");
} catch (IOException e) {
throw new IllegalStateException(e);
}
stopWatch.stop();
LOGGER.debug("Total time taken to initialise DocumentSearcher '" +
stopWatch.getTotalTimeMillis() +"' ms.");
}
private void maybeReopen() throws SearchExecutionException {
LOGGER.debug("Initiating reopening of index readers...");
for (DocumentSearcherManager documentSearcherManager :
documentSearcherManagers) {
try {
documentSearcherManager.maybeReopen();
} catch (InterruptedException e) {
throw new SearchExecutionException(e);
} catch (IOException e) {
throw new SearchExecutionException(e);
}
}
LOGGER.debug("reopening of index readers complete.");
}
private void release(MultiSearcher multiSeacher) {
IndexSearcher[] indexSearchers = (IndexSearcher[])
multiSeacher.getSearchables();
for(int i =0 ; i < indexSearchers.length;i++) {
try {
documentSearcherManagers[i].release(indexSearchers[i]);
} catch (IOException e) {
throw new IllegalStateException(e);
}
}
}
private MultiSearcher get() throws SearchExecutionException {
maybeReopen();
MultiSearcher multiSearcher = null;
List<IndexSearcher> listOfIndexSeachers = new ArrayList<IndexSearcher>();
for (DocumentSearcherManager documentSearcherManager :
documentSearcherManagers) {
listOfIndexSeachers.add(documentSearcherManager.get());
}
try {
multiSearcher = new
MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[] {}));
} catch (IOException e) {
throw new SearchExecutionException(e);
}
return multiSearcher;
}
Hope there is enough information.
Cheers
Amin
P.S. I will continue to debug.
On Mon, Mar 2, 2009 at 6:55 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
>
> It makes perfect sense to call maybeReopen() followed by get(), as long as
> maybeReopen() is never slow enough to be noticeable to an end user (because
> you are making random queries pay the reopen/warming cost).
>
> If you call maybeReopen() after get(), then that search will not see the
> newly opened readers, but the next search will.
>
> I'm just thinking that since you see no results with get() alone, debug
> that case first. Then put back the maybeReopen().
>
> Can you post your full code at this point?
>
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> Hi
>>
>> Just out of curiosity does it not make sense to call maybeReopen and then
>> call get()? If I call get() then I have a new mulitsearcher, so a call to
>> maybeopen won't reinitialise the multi searcher. Unless I pass the multi
>> searcher into the maybereopen method. But somehow that doesn't make sense. I
>> maybe missing something here.
>>
>>
>> Cheers
>>
>> Amin
>>
>> On 2 Mar 2009, at 15:48, Amin Mohammed-Coleman <am...@gmail.com> wrote:
>>
>> I'm seeing some interesting behviour when i do get() first followed by
>>> maybeReopen then there are no documents in the directory (directory that i
>>> am interested in. When i do the maybeReopen and then get() then the doc
>>> count is correct. I can post stats later.
>>>
>>> Weird...
>>>
>>> On Mon, Mar 2, 2009 at 2:17 PM, Amin Mohammed-Coleman <am...@gmail.com>
>>> wrote:
>>> oh dear...i think i may cry...i'll debug.
>>>
>>>
>>> On Mon, Mar 2, 2009 at 2:15 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>> Or even just get() with no call to maybeReopen(). That should work fine
>>> as well.
>>>
>>>
>>> Mike
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> In my test case I have a set up method that should populate the indexes
>>> before I start using the document searcher. I will start adding some
>>> more
>>> debug statements. So basically I should be able to do: get() followed by
>>> maybeReopen.
>>>
>>> I will let you know what the outcome is.
>>>
>>>
>>> Cheers
>>> Amin
>>>
>>> On Mon, Mar 2, 2009 at 1:39 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>> Is it possible that when you first create the SearcherManager, there is
>>> no
>>> index in each Directory?
>>>
>>> If not... you better start adding diagnostics. EG inside your get(),
>>> print
>>> out the numDocs() of each IndexReader you get from the SearcherManager?
>>>
>>> Something is wrong and it's best to explain it...
>>>
>>>
>>> Mike
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> Nope. If i remove the maybeReopen the search doesn't work. It only works
>>> when i cal maybeReopen followed by get().
>>>
>>> Cheers
>>> Amin
>>>
>>> On Mon, Mar 2, 2009 at 12:56 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>> That's not right; something must be wrong.
>>>
>>> get() before maybeReopen() should simply let you search based on the
>>> searcher before reopening.
>>>
>>> If you just do get() and don't call maybeReopen() does it work?
>>>
>>>
>>> Mike
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> I noticed that if i do the get() before the maybeReopen then I get no
>>>
>>> results. But otherwise I can change it further.
>>>
>>> On Mon, Mar 2, 2009 at 11:46 AM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>> There is no such thing as final code -- code is alive and is always
>>> changing ;)
>>>
>>> It looks good to me.
>>>
>>> Though one trivial thing is: I would move the code in the try clause up
>>> to
>>> and including the multiSearcher=get() out above the try. I always
>>> attempt
>>> to "shrink wrap" what's inside a try clause to the minimum that needs
>>> to
>>> be
>>> there. Ie, your code that creates a query, finds the right sort &
>>> filter
>>> to
>>> use, etc, can all happen outside the try, because you have not yet
>>> acquired
>>> the multiSearcher.
>>>
>>> If you do that, you also don't need the null check in the finally
>>> clause,
>>> because multiSearcher must be non-null on entering the try.
>>>
>>> Mike
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> Hi there
>>>
>>> Good morning! Here is the final search code:
>>>
>>> public Summary[] search(final SearchRequest searchRequest)
>>> throwsSearchExecutionException {
>>>
>>> final String searchTerm = searchRequest.getSearchTerm();
>>>
>>> if (StringUtils.isBlank(searchTerm)) {
>>>
>>> throw new SearchExecutionException("Search string cannot be empty.
>>> There
>>> will be too many results to process.");
>>>
>>> }
>>>
>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>
>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>
>>> stopWatch.start();
>>>
>>> MultiSearcher multiSearcher = null;
>>>
>>> try {
>>>
>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>
>>> maybeReopen();
>>>
>>> Query query = queryParser.parse(searchTerm);
>>>
>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
>>> query.toString() +"'");
>>>
>>> Sort sort = null;
>>>
>>> sort = applySortIfApplicable(searchRequest);
>>>
>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>
>>> ChainedFilter chainedFilter = null;
>>>
>>> if (filters != null) {
>>>
>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>
>>> }
>>>
>>> multiSearcher = get();
>>>
>>> TopDocs topDocs = multiSearcher.search(query,chainedFilter ,100,sort);
>>>
>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>
>>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>>> "+topDocs.
>>> totalHits);
>>>
>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>
>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>
>>> float score = scoreDoc.score;
>>>
>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>
>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>
>>> summaryList.add(documentSummary);
>>>
>>> }
>>>
>>> } catch (Exception e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> } finally {
>>>
>>> if (multiSearcher != null) {
>>>
>>> release(multiSearcher);
>>>
>>> }
>>>
>>> }
>>>
>>> stopWatch.stop();
>>>
>>> LOGGER.debug("total time taken for document seach: " +
>>> stopWatch.getTotalTimeMillis() + " ms");
>>>
>>> return summaryList.toArray(new Summary[] {});
>>>
>>> }
>>>
>>>
>>>
>>> I hope this makes sense...thanks again!
>>>
>>>
>>> Cheers
>>>
>>> Amin
>>>
>>>
>>>
>>> On Sun, Mar 1, 2009 at 8:09 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>> You're calling get() too many times. For every call to get() you must
>>>
>>> match with a call to release().
>>>
>>> So, once at the front of your search method you should:
>>>
>>> MultiSearcher searcher = get();
>>>
>>> then use that searcher to do searching, retrieve docs, etc.
>>>
>>> Then in the finally clause, pass that searcher to release.
>>>
>>> So, only one call to get() and one matching call to release().
>>>
>>> Mike
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> Hi
>>>
>>> The searchers are injected into the class via Spring. So when a
>>>
>>> client
>>> calls the class it is fully configured with a list of index
>>> searchers.
>>> However I have removed this list and instead injecting a list of
>>> directories which are passed to the DocumentSearchManager.
>>> DocumentSearchManager is SearchManager (should've mentioned that
>>> earlier).
>>> So finally I have modified by release code to do the following:
>>>
>>> private void release(MultiSearcher multiSeacher) throws Exception {
>>>
>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>> multiSeacher.getSearchables();
>>>
>>> for(int i =0 ; i < indexSearchers.length;i++) {
>>>
>>> documentSearcherManagers[i].release(indexSearchers[i]);
>>>
>>> }
>>>
>>> }
>>>
>>>
>>> and it's use looks like this:
>>>
>>>
>>> public Summary[] search(final SearchRequest searchRequest)
>>> throwsSearchExecutionException {
>>>
>>> final String searchTerm = searchRequest.getSearchTerm();
>>>
>>> if (StringUtils.isBlank(searchTerm)) {
>>>
>>> throw new SearchExecutionException("Search string cannot be empty.
>>> There
>>> will be too many results to process.");
>>>
>>> }
>>>
>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>
>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>
>>> stopWatch.start();
>>>
>>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>>
>>> try {
>>>
>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>
>>> maybeReopen();
>>>
>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>> searchers
>>> '"
>>> +
>>> indexSearchers.size() +"'");
>>>
>>> Query query = queryParser.parse(searchTerm);
>>>
>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '"
>>> +
>>> query.toString() +"'");
>>>
>>> Sort sort = null;
>>>
>>> sort = applySortIfApplicable(searchRequest);
>>>
>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>
>>> ChainedFilter chainedFilter = null;
>>>
>>> if (filters != null) {
>>>
>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>
>>> }
>>>
>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>
>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>
>>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>>> "+topDocs.
>>> totalHits);
>>>
>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>
>>> final Document doc = get().doc(scoreDoc.doc);
>>>
>>> float score = scoreDoc.score;
>>>
>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>
>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>
>>> summaryList.add(documentSummary);
>>>
>>> }
>>>
>>> } catch (Exception e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> } finally {
>>>
>>> release(get());
>>>
>>> }
>>>
>>> stopWatch.stop();
>>>
>>> LOGGER.debug("total time taken for document seach: " +
>>> stopWatch.getTotalTimeMillis() + " ms");
>>>
>>> return summaryList.toArray(new Summary[] {});
>>>
>>> }
>>>
>>>
>>> So the final post construct constructs the DocumentSearchMangers
>>> with
>>> the
>>> list of directories..looking like this
>>>
>>>
>>> @PostConstruct
>>>
>>> public void initialiseDocumentSearcher() {
>>>
>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>> PerFieldAnalyzerWrapper(
>>> analyzer);
>>>
>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>> newKeywordAnalyzer());
>>>
>>> queryParser =
>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>> analyzerWrapper);
>>>
>>> try {
>>>
>>> LOGGER.debug("Initialising multi searcher ....");
>>>
>>> documentSearcherManagers = new
>>> DocumentSearcherManager[directories.size()];
>>>
>>> for (int i = 0; i < directories.size() ;i++) {
>>>
>>> Directory directory = directories.get(i);
>>>
>>> DocumentSearcherManager documentSearcherManager =
>>> newDocumentSearcherManager(directory);
>>>
>>> documentSearcherManagers[i]=documentSearcherManager;
>>>
>>> }
>>>
>>> LOGGER.debug("multi searcher initialised");
>>>
>>> } catch (IOException e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> }
>>>
>>> }
>>>
>>>
>>>
>>> Cheers
>>>
>>> Amin
>>>
>>>
>>>
>>> On Sun, Mar 1, 2009 at 6:15 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>> I don't understand where searchers comes from, prior to
>>>
>>> initializeDocumentSearcher? You should, instead, simply create the
>>> SearcherManager (from your Directory instances). You don't need
>>> any
>>> searchers during initialize.
>>>
>>> Is DocumentSearcherManager the same as SearcherManager (just
>>> renamed)?
>>>
>>> The release method is wrong -- you're calling .get() and then
>>> immediately release. Instead, you should step through the
>>> searchers
>>> from your MultiSearcher and release them to each SearcherManager.
>>>
>>> You should call your release() in a finally clause.
>>>
>>> Mike
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> Sorry...i'm getting slightly confused.
>>>
>>> I have a PostConstruct which is where I should create an array of
>>>
>>> SearchManagers (per indexSeacher). From there I initialise the
>>> multisearcher using the get(). After which I need to call
>>> maybeReopen
>>> for
>>> each IndexSearcher. So I'll do the following:
>>>
>>> @PostConstruct
>>>
>>> public void initialiseDocumentSearcher() {
>>>
>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>> PerFieldAnalyzerWrapper(
>>> analyzer);
>>>
>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>> newKeywordAnalyzer());
>>>
>>> queryParser =
>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>> analyzerWrapper);
>>>
>>> try {
>>>
>>> LOGGER.debug("Initialising multi searcher ....");
>>>
>>> documentSearcherManagers = new
>>> DocumentSearcherManager[searchers.size()];
>>>
>>> for (int i = 0; i < searchers.size() ;i++) {
>>>
>>> IndexSearcher indexSearcher = searchers.get(i);
>>>
>>> Directory directory = indexSearcher.getIndexReader().directory();
>>>
>>> DocumentSearcherManager documentSearcherManager =
>>> newDocumentSearcherManager(directory);
>>>
>>> documentSearcherManagers[i]=documentSearcherManager;
>>>
>>> }
>>>
>>> LOGGER.debug("multi searcher initialised");
>>>
>>> } catch (IOException e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> }
>>>
>>> }
>>>
>>>
>>> This initialises search managers. I then have methods:
>>>
>>>
>>> private void maybeReopen() throws Exception {
>>>
>>> LOGGER.debug("Initiating reopening of index readers...");
>>>
>>> for (DocumentSearcherManager documentSearcherManager :
>>> documentSearcherManagers) {
>>>
>>> documentSearcherManager.maybeReopen();
>>>
>>> }
>>>
>>> }
>>>
>>>
>>>
>>> private void release() throws Exception {
>>>
>>> for (DocumentSearcherManager documentSearcherManager :
>>> documentSearcherManagers) {
>>>
>>> documentSearcherManager.release(documentSearcherManager.get());
>>>
>>> }
>>>
>>> }
>>>
>>>
>>> private MultiSearcher get() {
>>>
>>> List<IndexSearcher> listOfIndexSeachers = new
>>> ArrayList<IndexSearcher>();
>>>
>>> for (DocumentSearcherManager documentSearcherManager :
>>> documentSearcherManagers) {
>>>
>>> listOfIndexSeachers.add(documentSearcherManager.get());
>>>
>>> }
>>>
>>> try {
>>>
>>> multiSearcher = new
>>> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[] {}));
>>>
>>> } catch (IOException e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> }
>>>
>>> return multiSearcher;
>>>
>>> }
>>>
>>>
>>> These methods are used in the following manner in the search code:
>>>
>>>
>>> public Summary[] search(final SearchRequest searchRequest)
>>> throwsSearchExecutionException {
>>>
>>> final String searchTerm = searchRequest.getSearchTerm();
>>>
>>> if (StringUtils.isBlank(searchTerm)) {
>>>
>>> throw new SearchExecutionException("Search string cannot be empty.
>>> There
>>> will be too many results to process.");
>>>
>>> }
>>>
>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>
>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>
>>> stopWatch.start();
>>>
>>> List<IndexSearcher> indexSearchers = new
>>> ArrayList<IndexSearcher>();
>>>
>>> try {
>>>
>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>
>>> maybeReopen();
>>>
>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>> searchers
>>> '"
>>> +
>>> indexSearchers.size() +"'");
>>>
>>> Query query = queryParser.parse(searchTerm);
>>>
>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query
>>> '"
>>> +
>>> query.toString() +"'");
>>>
>>> Sort sort = null;
>>>
>>> sort = applySortIfApplicable(searchRequest);
>>>
>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>
>>> ChainedFilter chainedFilter = null;
>>>
>>> if (filters != null) {
>>>
>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>
>>> }
>>>
>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>
>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>
>>> LOGGER.debug("total number of hits for [" + query.toString() + " ]
>>> =
>>> "+topDocs.
>>> totalHits);
>>>
>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>
>>> final Document doc = get().doc(scoreDoc.doc);
>>>
>>> float score = scoreDoc.score;
>>>
>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>
>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>
>>> summaryList.add(documentSummary);
>>>
>>> }
>>>
>>> release();
>>>
>>> } catch (Exception e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> }
>>>
>>> stopWatch.stop();
>>>
>>> LOGGER.debug("total time taken for document seach: " +
>>> stopWatch.getTotalTimeMillis() + " ms");
>>>
>>> return summaryList.toArray(new Summary[] {});
>>>
>>> }
>>>
>>>
>>> Does this look better? Again..I really really appreciate your
>>> help!
>>>
>>>
>>> On Sun, Mar 1, 2009 at 4:18 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>> This is not quite right -- you should only create SearcherManager
>>> once
>>>
>>> (per Direcotry) at startup/app load, not with every search
>>> request.
>>>
>>>
>>> And I don't see release -- it must call SearcherManager.release
>>> of
>>> each of the IndexSearchers previously returned from get().
>>>
>>> Mike
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> Hi
>>>
>>> Thanks again for helping on a Sunday!
>>>
>>>
>>> I have now modified my maybeOpen() to do the following:
>>>
>>> private void maybeReopen() throws Exception {
>>>
>>> LOGGER.debug("Initiating reopening of index readers...");
>>>
>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>> .getSearchables();
>>>
>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>
>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>
>>> SearcherManager documentSearcherManager = new
>>> SearcherManager(indexReader.directory());
>>>
>>> documentSearcherManager.maybeReopen();
>>>
>>> }
>>>
>>> }
>>>
>>>
>>> And get() to:
>>>
>>>
>>> private synchronized MultiSearcher get() {
>>>
>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>> .getSearchables();
>>>
>>> List<IndexSearcher> indexSearchersList = new
>>> ArrayList<IndexSearcher>();
>>>
>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>
>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>
>>> SearcherManager documentSearcherManager = null;
>>>
>>> try {
>>>
>>> documentSearcherManager = new
>>> SearcherManager(indexReader.directory());
>>>
>>> } catch (IOException e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> }
>>>
>>> indexSearchersList.add(documentSearcherManager.get());
>>>
>>> }
>>>
>>> try {
>>>
>>> multiSearcher = new
>>> MultiSearcher(indexSearchersList.toArray(newIndexSearcher[]
>>> {}));
>>>
>>> } catch (IOException e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> }
>>>
>>> return multiSearcher;
>>>
>>> }
>>>
>>>
>>>
>>> This makes all my test pass. I am using the SearchManager that
>>> you
>>> recommended. Does this look ok?
>>>
>>>
>>> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>> Your maybeReopen has an excess incRef().
>>>
>>>
>>> I'm not sure how you open the searchers in the first place? The
>>>
>>> list
>>> starts as empty, and nothing populates it?
>>>
>>> When you do the initial population, you need an incRef.
>>>
>>> I think you're hitting IllegalStateException because
>>> maybeReopen
>>> is
>>> closing a reader before get() can get it (since they
>>> synchronize
>>> on
>>> different objects).
>>>
>>> I'd recommend switching to the SearcherManager class.
>>> Instantiate
>>> one
>>> for each of your searchers. On each search request, go through
>>> them
>>> and call maybeReopen(), and then call get() and gather each
>>> IndexSearcher instance into a new array. Then, make a new
>>> MultiSearcher (opposite of what I said before): while that
>>> creates
>>> a
>>> small amount of garbage, it'll keep your code simpler (good
>>> tradeoff).
>>>
>>> Mike
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> sorrry I added
>>>
>>>
>>> release(multiSearcher);
>>>
>>>
>>>
>>> instead of multiSearcher.close();
>>>
>>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <
>>> aminmc@gmail.com
>>>
>>> wrote:
>>>
>>>
>>>
>>> Hi
>>>
>>>
>>> I've now done the following:
>>>
>>>
>>> public Summary[] search(final SearchRequest searchRequest)
>>>
>>> throwsSearchExecutionException {
>>>
>>> final String searchTerm = searchRequest.getSearchTerm();
>>>
>>> if (StringUtils.isBlank(searchTerm)) {
>>>
>>> throw new SearchExecutionException("Search string cannot be
>>> empty.
>>> There
>>> will be too many results to process.");
>>>
>>> }
>>>
>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>
>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>
>>> stopWatch.start();
>>>
>>> List<IndexSearcher> indexSearchers = new
>>> ArrayList<IndexSearcher>();
>>>
>>> try {
>>>
>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>
>>> maybeReopen();
>>>
>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>> searchers
>>> '"+ indexSearchers.size() +
>>> "'");
>>>
>>> Query query = queryParser.parse(searchTerm);
>>>
>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>>> Query
>>> '"
>>> +
>>> query.toString() +"'");
>>>
>>> Sort sort = null;
>>>
>>> sort = applySortIfApplicable(searchRequest);
>>>
>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>
>>> ChainedFilter chainedFilter = null;
>>>
>>> if (filters != null) {
>>>
>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>
>>> }
>>>
>>> TopDocs topDocs = get().search(query,chainedFilter
>>> ,100,sort);
>>>
>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>
>>> LOGGER.debug("total number of hits for [" + query.toString()
>>> +
>>> "
>>> ]
>>> =
>>> "+topDocs.
>>> totalHits);
>>>
>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>
>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>
>>> float score = scoreDoc.score;
>>>
>>> final BaseDocument baseDocument = new BaseDocument(doc,
>>> score);
>>>
>>> Summary documentSummary = new
>>> DocumentSummaryImpl(baseDocument);
>>>
>>> summaryList.add(documentSummary);
>>>
>>> }
>>>
>>> multiSearcher.close();
>>>
>>> } catch (Exception e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> }
>>>
>>> stopWatch.stop();
>>>
>>> LOGGER.debug("total time taken for document seach: " +
>>> stopWatch.getTotalTimeMillis() + " ms");
>>>
>>> return summaryList.toArray(new Summary[] {});
>>>
>>> }
>>>
>>>
>>> And have the following methods:
>>>
>>> @PostConstruct
>>>
>>> public void initialiseQueryParser() {
>>>
>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>> PerFieldAnalyzerWrapper(
>>> analyzer);
>>>
>>>
>>>
>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>> newKeywordAnalyzer());
>>>
>>> queryParser =
>>>
>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>
>>> analyzerWrapper);
>>>
>>> try {
>>>
>>> LOGGER.debug("Initialising multi searcher ....");
>>>
>>> this.multiSearcher = new
>>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>>
>>> LOGGER.debug("multi searcher initialised");
>>>
>>> } catch (IOException e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> }
>>>
>>> }
>>>
>>>
>>> Initialises mutltisearcher when this class is creared by
>>> spring.
>>>
>>>
>>> private synchronized void swapMultiSearcher(MultiSearcher
>>> newMultiSearcher) {
>>>
>>> try {
>>>
>>> release(multiSearcher);
>>>
>>> } catch (IOException e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> }
>>>
>>> multiSearcher = newMultiSearcher;
>>>
>>> }
>>>
>>> public void maybeReopen() throws IOException {
>>>
>>> MultiSearcher newMultiSeacher = null;
>>>
>>> boolean refreshMultiSeacher = false;
>>>
>>> List<IndexSearcher> indexSearchers = new
>>> ArrayList<IndexSearcher>();
>>>
>>> synchronized (searchers) {
>>>
>>> for (IndexSearcher indexSearcher: searchers) {
>>>
>>> IndexReader reader = indexSearcher.getIndexReader();
>>>
>>> reader.incRef();
>>>
>>> Directory directory = reader.directory();
>>>
>>> long currentVersion = reader.getVersion();
>>>
>>> if (IndexReader.getCurrentVersion(directory) !=
>>> currentVersion)
>>> {
>>>
>>> IndexReader newReader =
>>> indexSearcher.getIndexReader().reopen();
>>>
>>> if (newReader != reader) {
>>>
>>> reader.decRef();
>>>
>>> refreshMultiSeacher = true;
>>>
>>> }
>>>
>>> reader = newReader;
>>>
>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>
>>> indexSearchers.add(newSearcher);
>>>
>>> }
>>>
>>> }
>>>
>>> }
>>>
>>>
>>>
>>> if (refreshMultiSeacher) {
>>>
>>> newMultiSeacher = new
>>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
>>>
>>> warm(newMultiSeacher);
>>>
>>> swapMultiSearcher(newMultiSeacher);
>>>
>>> }
>>>
>>>
>>>
>>> }
>>>
>>>
>>> private void warm(MultiSearcher newMultiSeacher) {
>>>
>>> }
>>>
>>>
>>>
>>> private synchronized MultiSearcher get() {
>>>
>>> for (IndexSearcher indexSearcher: searchers) {
>>>
>>> indexSearcher.getIndexReader().incRef();
>>>
>>> }
>>>
>>> return multiSearcher;
>>>
>>> }
>>>
>>> private synchronized void release(MultiSearcher
>>> multiSearcher)
>>> throwsIOException {
>>>
>>> for (IndexSearcher indexSearcher: searchers) {
>>>
>>> indexSearcher.getIndexReader().decRef();
>>>
>>> }
>>>
>>> }
>>>
>>>
>>> However I am now getting
>>>
>>>
>>> java.lang.IllegalStateException:
>>> org.apache.lucene.store.AlreadyClosedException: this
>>> IndexReader
>>> is
>>> closed
>>>
>>>
>>> on the call:
>>>
>>>
>>> private synchronized MultiSearcher get() {
>>>
>>> for (IndexSearcher indexSearcher: searchers) {
>>>
>>> indexSearcher.getIndexReader().incRef();
>>>
>>> }
>>>
>>> return multiSearcher;
>>>
>>> }
>>>
>>>
>>> I'm doing something wrong ..obviously..not sure where
>>> though..
>>>
>>>
>>> Cheers
>>>
>>>
>>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>> I was wondering the same thing ;)
>>>
>>>
>>> It's best to call this method from a single BG "warming"
>>> thread,
>>>
>>> in
>>> which
>>> case it would not need its own synchronization.
>>>
>>> But, to be safe, I'll add internal synchronization to it.
>>> You
>>> can't
>>> simply put synchronized in front of the method, since you
>>> don't
>>> want
>>> this to
>>> block searching.
>>>
>>>
>>> Mike
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> just a quick point:
>>>
>>> public void maybeReopen() throws IOException {
>>> //D
>>>
>>> long currentVersion =
>>>
>>> currentSearcher.getIndexReader().getVersion();
>>>
>>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>>> IndexReader newReader =
>>> currentSearcher.getIndexReader().reopen();
>>> assert newReader != currentSearcher.getIndexReader();
>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>> warm(newSearcher);
>>> swapSearcher(newSearcher);
>>> }
>>> }
>>>
>>> should the above be synchronised?
>>>
>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
>>> aminmc@gmail.com
>>>
>>> wrote:
>>>
>>>
>>>
>>> thanks. i will rewrite..in between giving my baby her feed
>>>
>>> and
>>>
>>> playing
>>>
>>>
>>> with the other child and my wife who wants me to do several
>>> other
>>>
>>> things!
>>>
>>>
>>>
>>>
>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>>
>>> Hi
>>>
>>>
>>> Thanks for your input. I would like to have a go at
>>> doing
>>> this
>>>
>>> myself
>>>
>>> first, Solr may be an option.
>>>
>>>
>>> * You are creating a new Analyzer & QueryParser every
>>> time,
>>> also
>>> creating unnecessary garbage; instead, they should be
>>> created
>>> once
>>> & reused.
>>>
>>> -- I can moved the code out so that it is only created
>>> once
>>> and
>>> reused.
>>>
>>>
>>> * You always make a new IndexSearcher and a new
>>> MultiSearcher
>>> even
>>> when nothing has changed. This just generates
>>> unnecessary
>>> garbage
>>> which GC then must sweep up.
>>>
>>> -- This was something I thought about. I could move it
>>> out
>>> so
>>> that
>>> it's
>>> created once. However I presume inside my code i need
>>> to
>>> check
>>> whether
>>> the
>>> indexreaders are update to date. This needs to be
>>> synchronized
>>> as
>>> well I
>>> guess(?)
>>>
>>>
>>> Yes you should synchronize the check for whether the
>>> IndexReader
>>> is
>>>
>>> current.
>>>
>>>
>>> * I don't see any synchronization -- it looks like two
>>>
>>> search
>>>
>>> requests are allowed into this method at the same time?
>>> Which
>>> is
>>>
>>> dangerous... eg both (or, more) will wastefully reopen
>>> the
>>>
>>> readers.
>>>
>>> -- So i need to extract the logic for reopening and
>>> provide
>>> a
>>> synchronisation mechanism.
>>>
>>>
>>> Yes.
>>>
>>>
>>>
>>> Ok. So I have some work to do. I'll refactor the code
>>> and
>>>
>>> see
>>> if
>>> I
>>> can
>>>
>>> get
>>>
>>> inline to your recommendations.
>>>
>>>
>>>
>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>> On a quick look, I think there are a few problems with
>>> the
>>> code:
>>>
>>>
>>> * I don't see any synchronization -- it looks like two
>>> search
>>>
>>> requests are allowed into this method at the same time?
>>>
>>> Which
>>> is
>>> dangerous... eg both (or, more) will wastefully reopen
>>> the
>>> readers.
>>>
>>> * You are over-incRef'ing (the reader.incRef inside the
>>> loop)
>>> --
>>> I
>>> don't see a corresponding decRef.
>>>
>>> * You reopen and warm your searchers "live" (vs with BG
>>> thread);
>>> meaning the unlucky search request that hits a reopen
>>> pays
>>> the
>>> cost. This might be OK if the index is small enough
>>> that
>>> reopening & warming takes very little time. But if
>>> index
>>> gets
>>> large, making a random search pay that warming cost is
>>> not
>>> nice
>>> to
>>> the end user. It erodes their trust in you.
>>>
>>> * You always make a new IndexSearcher and a new
>>> MultiSearcher
>>> even
>>> when nothing has changed. This just generates
>>> unnecessary
>>> garbage
>>> which GC then must sweep up.
>>>
>>> * You are creating a new Analyzer & QueryParser every
>>> time,
>>> also
>>> creating unnecessary garbage; instead, they should be
>>> created
>>> once
>>> & reused.
>>>
>>> You should consider simply using Solr -- it handles all
>>> this
>>> logic
>>> for
>>> you and has been well debugged with time...
>>>
>>> Mike
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> The reason for the indexreader.reopen is because I have
>>> a
>>> webapp
>>> which
>>>
>>> enables users to upload files and then search for the
>>> documents.
>>> If
>>>
>>> I
>>>
>>> don't
>>>
>>> reopen i'm concerned that the facet hit counter won't
>>> be
>>>
>>> updated.
>>>
>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman
>>> <
>>> aminmc@gmail.com
>>>
>>> wrote:
>>>
>>>
>>>
>>> Hi
>>>
>>>
>>>
>>> I have been able to get the code working for my
>>> scenario,
>>>
>>> however
>>>
>>> I
>>>
>>> have
>>>
>>> a
>>>
>>> question and I was wondering if I could get some help.
>>>
>>> I
>>> have
>>> a
>>> list
>>> of
>>> IndexSearchers which are used in a MultiSearcher
>>> class.
>>> I
>>> use
>>> the
>>> indexsearchers to get each indexreader and put them
>>> into
>>> a
>>> MultiIndexReader.
>>>
>>> IndexReader[] readers = new
>>> IndexReader[searchables.length];
>>>
>>> for (int i =0 ; i < searchables.length;i++) {
>>>
>>> IndexSearcher indexSearcher =
>>> (IndexSearcher)searchables[i];
>>>
>>> readers[i] = indexSearcher.getIndexReader();
>>>
>>> IndexReader newReader = readers[i].reopen();
>>>
>>> if (newReader != readers[i]) {
>>>
>>> readers[i].close();
>>>
>>> }
>>>
>>> readers[i] = newReader;
>>>
>>>
>>>
>>> }
>>>
>>> multiReader = new MultiReader(readers);
>>>
>>> OpenBitSetFacetHitCounter facetHitCounter =
>>> newOpenBitSetFacetHitCounter();
>>>
>>> IndexSearcher indexSearcher = new
>>> IndexSearcher(multiReader);
>>>
>>>
>>> I then use the indexseacher to do the facet stuff. I
>>> end
>>> the
>>> code
>>> with
>>> closing the multireader. This is causing problems in
>>> another
>>> method
>>> where I
>>> do some other search as the indexreaders are closed.
>>> Is
>>> it
>>> ok
>>> to
>>> not
>>> close
>>> the multiindexreader or should I do some additional
>>> checks
>>> in
>>> the
>>> other
>>> method to see if the indexreader is closed?
>>>
>>>
>>>
>>> Cheers
>>>
>>>
>>> P.S. Hope that made sense...!
>>>
>>>
>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin
>>> Mohammed-Coleman
>>> <
>>> aminmc@gmail.com
>>>
>>> wrote:
>>>
>>>
>>>
>>> Hi
>>>
>>>
>>>
>>>
>>> Thanks just what I needed!
>>>
>>>
>>>
>>> Cheers
>>>
>>> Amin
>>>
>>>
>>>
>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>>> marcelo.ochoa@gmail.com>
>>> wrote:
>>>
>>> Hi Amin:
>>>
>>> Please take a look a this blog post:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>> Best regards, Marcelo.
>>>
>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin
>>> Mohammed-Coleman
>>> <
>>> aminmc@gmail.com>
>>> wrote:
>>>
>>> Hi
>>>
>>>
>>> Sorry to re send this email but I was wondering if
>>> I
>>> could
>>> get
>>>
>>> some
>>>
>>> advice
>>>
>>> on this.
>>>
>>> Cheers
>>>
>>> Amin
>>>
>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>>> aminmc@gmail.com>
>>> wrote:
>>>
>>> Hi
>>>
>>>
>>> I am looking at building a faceted search using
>>> Lucene.
>>> I
>>> know
>>>
>>> that
>>>
>>> Solr
>>>
>>> comes with this built in, however I would like to
>>>
>>> try
>>> this
>>> by
>>> myself
>>> (something to add to my CV!). I have been
>>> looking
>>> around
>>> and
>>> I
>>> found
>>> that
>>> you can use the IndexReader and use TermVectors.
>>> This
>>> looks
>>> ok
>>> but
>>> I'm
>>> not
>>> sure how to filter the results so that a
>>> particular
>>> user
>>> can
>>> only
>>> see
>>> a
>>> subset of results. The next option I was looking
>>> at
>>> was
>>> something
>>> like
>>>
>>> Term term1 = new Term("brand", "ford");
>>> Term term2 = new Term("brand", "vw");
>>> Term[] termsArray = new Term[] { term1, term2
>>> };un
>>> int[] docFreqs =
>>> indexSearcher.docFreqs(termsArray);
>>>
>>> The only problem here is that I have to provide
>>> the
>>> brand
>>> type
>>> each
>>> time a
>>> new brand is created. Again I'm not sure how I
>>> can
>>> filter
>>> the
>>> results
>>> here.
>>> It may be that I'm using the wrong api methods to
>>> do
>>> this.
>>>
>>> I would be grateful if I could get some advice on
>>> this.
>>>
>>>
>>> Cheers
>>> Amin
>>>
>>> P.S. I am basically trying to do something that
>>> displays
>>> the
>>> following
>>>
>>> Personal Contact (23) Business Contact (45) and
>>> so
>>> on..
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>> Marcelo F. Ochoa
>>>
>>>
>>> http://marceloochoa.blogspot.com/
>>>
>>> http://marcelo.ochoa.googlepages.com/home
>>>
>>> ______________
>>> Want to integrate Lucene and Oracle?
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>> Is Oracle 11g REST ready?
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail:
>>> java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail:
>>> java-user-help@lucene.apache.org
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>>
>>>
>>> To unsubscribe, e-mail:
>>>
>>>
>>>
>>> java-user-unsubscribe@lucene.apache.org
>>>
>>>
>>>
>>> For additional commands, e-mail:
>>>
>>> java-user-help@lucene.apache.org
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>>
>>> To unsubscribe, e-mail:
>>>
>>> java-user-unsubscribe@lucene.apache.org
>>>
>>>
>>> For additional commands, e-mail:
>>>
>>> java-user-help@lucene.apache.org
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>>
>>>
>>> To unsubscribe, e-mail:
>>>
>>>
>>> java-user-unsubscribe@lucene.apache.org
>>>
>>>
>>> For additional commands, e-mail:
>>>
>>> java-user-help@lucene.apache.org
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>>
>>>
>>> To unsubscribe, e-mail:
>>> java-user-unsubscribe@lucene.apache.org
>>>
>>>
>>> For additional commands, e-mail:
>>>
>>> java-user-help@lucene.apache.org
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>>
>>> To unsubscribe, e-mail:
>>> java-user-unsubscribe@lucene.apache.org
>>>
>>>
>>> For additional commands, e-mail:
>>> java-user-help@lucene.apache.org
>>>
>>>
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>>
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>>
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>> rg" target="_blank">unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>> div>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Faceted Search using Lucene
Posted by Michael McCandless <lu...@mikemccandless.com>.
It makes perfect sense to call maybeReopen() followed by get(), as
long as maybeReopen() is never slow enough to be noticeable to an end
user (because you are making random queries pay the reopen/warming
cost).
If you call maybeReopen() after get(), then that search will not see
the newly opened readers, but the next search will.
I'm just thinking that since you see no results with get() alone,
debug that case first. Then put back the maybeReopen().
Can you post your full code at this point?
Mike
Amin Mohammed-Coleman wrote:
> Hi
>
> Just out of curiosity does it not make sense to call maybeReopen and
> then call get()? If I call get() then I have a new mulitsearcher, so
> a call to maybeopen won't reinitialise the multi searcher. Unless I
> pass the multi searcher into the maybereopen method. But somehow
> that doesn't make sense. I maybe missing something here.
>
>
> Cheers
>
> Amin
>
> On 2 Mar 2009, at 15:48, Amin Mohammed-Coleman <am...@gmail.com>
> wrote:
>
>> I'm seeing some interesting behviour when i do get() first followed
>> by maybeReopen then there are no documents in the directory
>> (directory that i am interested in. When i do the maybeReopen and
>> then get() then the doc count is correct. I can post stats later.
>>
>> Weird...
>>
>> On Mon, Mar 2, 2009 at 2:17 PM, Amin Mohammed-Coleman <aminmc@gmail.com
>> > wrote:
>> oh dear...i think i may cry...i'll debug.
>>
>>
>> On Mon, Mar 2, 2009 at 2:15 PM, Michael McCandless <lucene@mikemccandless.com
>> > wrote:
>>
>> Or even just get() with no call to maybeReopen(). That should work
>> fine as well.
>>
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> In my test case I have a set up method that should populate the
>> indexes
>> before I start using the document searcher. I will start adding
>> some more
>> debug statements. So basically I should be able to do: get()
>> followed by
>> maybeReopen.
>>
>> I will let you know what the outcome is.
>>
>>
>> Cheers
>> Amin
>>
>> On Mon, Mar 2, 2009 at 1:39 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>
>> Is it possible that when you first create the SearcherManager,
>> there is no
>> index in each Directory?
>>
>> If not... you better start adding diagnostics. EG inside your
>> get(), print
>> out the numDocs() of each IndexReader you get from the
>> SearcherManager?
>>
>> Something is wrong and it's best to explain it...
>>
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> Nope. If i remove the maybeReopen the search doesn't work. It only
>> works
>> when i cal maybeReopen followed by get().
>>
>> Cheers
>> Amin
>>
>> On Mon, Mar 2, 2009 at 12:56 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>
>> That's not right; something must be wrong.
>>
>> get() before maybeReopen() should simply let you search based on the
>> searcher before reopening.
>>
>> If you just do get() and don't call maybeReopen() does it work?
>>
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> I noticed that if i do the get() before the maybeReopen then I get no
>>
>> results. But otherwise I can change it further.
>>
>> On Mon, Mar 2, 2009 at 11:46 AM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>
>> There is no such thing as final code -- code is alive and is always
>> changing ;)
>>
>> It looks good to me.
>>
>> Though one trivial thing is: I would move the code in the try
>> clause up
>> to
>> and including the multiSearcher=get() out above the try. I always
>> attempt
>> to "shrink wrap" what's inside a try clause to the minimum that needs
>> to
>> be
>> there. Ie, your code that creates a query, finds the right sort &
>> filter
>> to
>> use, etc, can all happen outside the try, because you have not yet
>> acquired
>> the multiSearcher.
>>
>> If you do that, you also don't need the null check in the finally
>> clause,
>> because multiSearcher must be non-null on entering the try.
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> Hi there
>>
>> Good morning! Here is the final search code:
>>
>> public Summary[] search(final SearchRequest searchRequest)
>> throwsSearchExecutionException {
>>
>> final String searchTerm = searchRequest.getSearchTerm();
>>
>> if (StringUtils.isBlank(searchTerm)) {
>>
>> throw new SearchExecutionException("Search string cannot be empty.
>> There
>> will be too many results to process.");
>>
>> }
>>
>> List<Summary> summaryList = new ArrayList<Summary>();
>>
>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>
>> stopWatch.start();
>>
>> MultiSearcher multiSearcher = null;
>>
>> try {
>>
>> LOGGER.debug("Ensuring all index readers are up to date...");
>>
>> maybeReopen();
>>
>> Query query = queryParser.parse(searchTerm);
>>
>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
>> query.toString() +"'");
>>
>> Sort sort = null;
>>
>> sort = applySortIfApplicable(searchRequest);
>>
>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>
>> ChainedFilter chainedFilter = null;
>>
>> if (filters != null) {
>>
>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>
>> }
>>
>> multiSearcher = get();
>>
>> TopDocs topDocs = multiSearcher.search(query,chainedFilter ,
>> 100,sort);
>>
>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>
>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>> "+topDocs.
>> totalHits);
>>
>> for (ScoreDoc scoreDoc : scoreDocs) {
>>
>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>
>> float score = scoreDoc.score;
>>
>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>
>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>
>> summaryList.add(documentSummary);
>>
>> }
>>
>> } catch (Exception e) {
>>
>> throw new IllegalStateException(e);
>>
>> } finally {
>>
>> if (multiSearcher != null) {
>>
>> release(multiSearcher);
>>
>> }
>>
>> }
>>
>> stopWatch.stop();
>>
>> LOGGER.debug("total time taken for document seach: " +
>> stopWatch.getTotalTimeMillis() + " ms");
>>
>> return summaryList.toArray(new Summary[] {});
>>
>> }
>>
>>
>>
>> I hope this makes sense...thanks again!
>>
>>
>> Cheers
>>
>> Amin
>>
>>
>>
>> On Sun, Mar 1, 2009 at 8:09 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>
>> You're calling get() too many times. For every call to get() you
>> must
>>
>> match with a call to release().
>>
>> So, once at the front of your search method you should:
>>
>> MultiSearcher searcher = get();
>>
>> then use that searcher to do searching, retrieve docs, etc.
>>
>> Then in the finally clause, pass that searcher to release.
>>
>> So, only one call to get() and one matching call to release().
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> Hi
>>
>> The searchers are injected into the class via Spring. So when a
>>
>> client
>> calls the class it is fully configured with a list of index
>> searchers.
>> However I have removed this list and instead injecting a list of
>> directories which are passed to the DocumentSearchManager.
>> DocumentSearchManager is SearchManager (should've mentioned that
>> earlier).
>> So finally I have modified by release code to do the following:
>>
>> private void release(MultiSearcher multiSeacher) throws Exception {
>>
>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>> multiSeacher.getSearchables();
>>
>> for(int i =0 ; i < indexSearchers.length;i++) {
>>
>> documentSearcherManagers[i].release(indexSearchers[i]);
>>
>> }
>>
>> }
>>
>>
>> and it's use looks like this:
>>
>>
>> public Summary[] search(final SearchRequest searchRequest)
>> throwsSearchExecutionException {
>>
>> final String searchTerm = searchRequest.getSearchTerm();
>>
>> if (StringUtils.isBlank(searchTerm)) {
>>
>> throw new SearchExecutionException("Search string cannot be empty.
>> There
>> will be too many results to process.");
>>
>> }
>>
>> List<Summary> summaryList = new ArrayList<Summary>();
>>
>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>
>> stopWatch.start();
>>
>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>
>> try {
>>
>> LOGGER.debug("Ensuring all index readers are up to date...");
>>
>> maybeReopen();
>>
>> LOGGER.debug("All Index Searchers are up to date. No of index
>> searchers
>> '"
>> +
>> indexSearchers.size() +"'");
>>
>> Query query = queryParser.parse(searchTerm);
>>
>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '"
>> +
>> query.toString() +"'");
>>
>> Sort sort = null;
>>
>> sort = applySortIfApplicable(searchRequest);
>>
>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>
>> ChainedFilter chainedFilter = null;
>>
>> if (filters != null) {
>>
>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>
>> }
>>
>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>
>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>
>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>> "+topDocs.
>> totalHits);
>>
>> for (ScoreDoc scoreDoc : scoreDocs) {
>>
>> final Document doc = get().doc(scoreDoc.doc);
>>
>> float score = scoreDoc.score;
>>
>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>
>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>
>> summaryList.add(documentSummary);
>>
>> }
>>
>> } catch (Exception e) {
>>
>> throw new IllegalStateException(e);
>>
>> } finally {
>>
>> release(get());
>>
>> }
>>
>> stopWatch.stop();
>>
>> LOGGER.debug("total time taken for document seach: " +
>> stopWatch.getTotalTimeMillis() + " ms");
>>
>> return summaryList.toArray(new Summary[] {});
>>
>> }
>>
>>
>> So the final post construct constructs the DocumentSearchMangers
>> with
>> the
>> list of directories..looking like this
>>
>>
>> @PostConstruct
>>
>> public void initialiseDocumentSearcher() {
>>
>> PerFieldAnalyzerWrapper analyzerWrapper = new
>> PerFieldAnalyzerWrapper(
>> analyzer);
>>
>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>> newKeywordAnalyzer());
>>
>> queryParser =
>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>> analyzerWrapper);
>>
>> try {
>>
>> LOGGER.debug("Initialising multi searcher ....");
>>
>> documentSearcherManagers = new
>> DocumentSearcherManager[directories.size()];
>>
>> for (int i = 0; i < directories.size() ;i++) {
>>
>> Directory directory = directories.get(i);
>>
>> DocumentSearcherManager documentSearcherManager =
>> newDocumentSearcherManager(directory);
>>
>> documentSearcherManagers[i]=documentSearcherManager;
>>
>> }
>>
>> LOGGER.debug("multi searcher initialised");
>>
>> } catch (IOException e) {
>>
>> throw new IllegalStateException(e);
>>
>> }
>>
>> }
>>
>>
>>
>> Cheers
>>
>> Amin
>>
>>
>>
>> On Sun, Mar 1, 2009 at 6:15 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>
>> I don't understand where searchers comes from, prior to
>>
>> initializeDocumentSearcher? You should, instead, simply create the
>> SearcherManager (from your Directory instances). You don't need
>> any
>> searchers during initialize.
>>
>> Is DocumentSearcherManager the same as SearcherManager (just
>> renamed)?
>>
>> The release method is wrong -- you're calling .get() and then
>> immediately release. Instead, you should step through the
>> searchers
>> from your MultiSearcher and release them to each SearcherManager.
>>
>> You should call your release() in a finally clause.
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> Sorry...i'm getting slightly confused.
>>
>> I have a PostConstruct which is where I should create an array of
>>
>> SearchManagers (per indexSeacher). From there I initialise the
>> multisearcher using the get(). After which I need to call
>> maybeReopen
>> for
>> each IndexSearcher. So I'll do the following:
>>
>> @PostConstruct
>>
>> public void initialiseDocumentSearcher() {
>>
>> PerFieldAnalyzerWrapper analyzerWrapper = new
>> PerFieldAnalyzerWrapper(
>> analyzer);
>>
>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>> newKeywordAnalyzer());
>>
>> queryParser =
>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>> analyzerWrapper);
>>
>> try {
>>
>> LOGGER.debug("Initialising multi searcher ....");
>>
>> documentSearcherManagers = new
>> DocumentSearcherManager[searchers.size()];
>>
>> for (int i = 0; i < searchers.size() ;i++) {
>>
>> IndexSearcher indexSearcher = searchers.get(i);
>>
>> Directory directory = indexSearcher.getIndexReader().directory();
>>
>> DocumentSearcherManager documentSearcherManager =
>> newDocumentSearcherManager(directory);
>>
>> documentSearcherManagers[i]=documentSearcherManager;
>>
>> }
>>
>> LOGGER.debug("multi searcher initialised");
>>
>> } catch (IOException e) {
>>
>> throw new IllegalStateException(e);
>>
>> }
>>
>> }
>>
>>
>> This initialises search managers. I then have methods:
>>
>>
>> private void maybeReopen() throws Exception {
>>
>> LOGGER.debug("Initiating reopening of index readers...");
>>
>> for (DocumentSearcherManager documentSearcherManager :
>> documentSearcherManagers) {
>>
>> documentSearcherManager.maybeReopen();
>>
>> }
>>
>> }
>>
>>
>>
>> private void release() throws Exception {
>>
>> for (DocumentSearcherManager documentSearcherManager :
>> documentSearcherManagers) {
>>
>> documentSearcherManager.release(documentSearcherManager.get());
>>
>> }
>>
>> }
>>
>>
>> private MultiSearcher get() {
>>
>> List<IndexSearcher> listOfIndexSeachers = new
>> ArrayList<IndexSearcher>();
>>
>> for (DocumentSearcherManager documentSearcherManager :
>> documentSearcherManagers) {
>>
>> listOfIndexSeachers.add(documentSearcherManager.get());
>>
>> }
>>
>> try {
>>
>> multiSearcher = new
>> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[] {}));
>>
>> } catch (IOException e) {
>>
>> throw new IllegalStateException(e);
>>
>> }
>>
>> return multiSearcher;
>>
>> }
>>
>>
>> These methods are used in the following manner in the search code:
>>
>>
>> public Summary[] search(final SearchRequest searchRequest)
>> throwsSearchExecutionException {
>>
>> final String searchTerm = searchRequest.getSearchTerm();
>>
>> if (StringUtils.isBlank(searchTerm)) {
>>
>> throw new SearchExecutionException("Search string cannot be empty.
>> There
>> will be too many results to process.");
>>
>> }
>>
>> List<Summary> summaryList = new ArrayList<Summary>();
>>
>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>
>> stopWatch.start();
>>
>> List<IndexSearcher> indexSearchers = new
>> ArrayList<IndexSearcher>();
>>
>> try {
>>
>> LOGGER.debug("Ensuring all index readers are up to date...");
>>
>> maybeReopen();
>>
>> LOGGER.debug("All Index Searchers are up to date. No of index
>> searchers
>> '"
>> +
>> indexSearchers.size() +"'");
>>
>> Query query = queryParser.parse(searchTerm);
>>
>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query
>> '"
>> +
>> query.toString() +"'");
>>
>> Sort sort = null;
>>
>> sort = applySortIfApplicable(searchRequest);
>>
>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>
>> ChainedFilter chainedFilter = null;
>>
>> if (filters != null) {
>>
>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>
>> }
>>
>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>
>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>
>> LOGGER.debug("total number of hits for [" + query.toString() + " ]
>> =
>> "+topDocs.
>> totalHits);
>>
>> for (ScoreDoc scoreDoc : scoreDocs) {
>>
>> final Document doc = get().doc(scoreDoc.doc);
>>
>> float score = scoreDoc.score;
>>
>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>
>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>
>> summaryList.add(documentSummary);
>>
>> }
>>
>> release();
>>
>> } catch (Exception e) {
>>
>> throw new IllegalStateException(e);
>>
>> }
>>
>> stopWatch.stop();
>>
>> LOGGER.debug("total time taken for document seach: " +
>> stopWatch.getTotalTimeMillis() + " ms");
>>
>> return summaryList.toArray(new Summary[] {});
>>
>> }
>>
>>
>> Does this look better? Again..I really really appreciate your
>> help!
>>
>>
>> On Sun, Mar 1, 2009 at 4:18 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>
>> This is not quite right -- you should only create SearcherManager
>> once
>>
>> (per Direcotry) at startup/app load, not with every search
>> request.
>>
>>
>> And I don't see release -- it must call SearcherManager.release
>> of
>> each of the IndexSearchers previously returned from get().
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> Hi
>>
>> Thanks again for helping on a Sunday!
>>
>>
>> I have now modified my maybeOpen() to do the following:
>>
>> private void maybeReopen() throws Exception {
>>
>> LOGGER.debug("Initiating reopening of index readers...");
>>
>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>> .getSearchables();
>>
>> for (IndexSearcher indexSearcher : indexSearchers) {
>>
>> IndexReader indexReader = indexSearcher.getIndexReader();
>>
>> SearcherManager documentSearcherManager = new
>> SearcherManager(indexReader.directory());
>>
>> documentSearcherManager.maybeReopen();
>>
>> }
>>
>> }
>>
>>
>> And get() to:
>>
>>
>> private synchronized MultiSearcher get() {
>>
>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>> .getSearchables();
>>
>> List<IndexSearcher> indexSearchersList = new
>> ArrayList<IndexSearcher>();
>>
>> for (IndexSearcher indexSearcher : indexSearchers) {
>>
>> IndexReader indexReader = indexSearcher.getIndexReader();
>>
>> SearcherManager documentSearcherManager = null;
>>
>> try {
>>
>> documentSearcherManager = new
>> SearcherManager(indexReader.directory());
>>
>> } catch (IOException e) {
>>
>> throw new IllegalStateException(e);
>>
>> }
>>
>> indexSearchersList.add(documentSearcherManager.get());
>>
>> }
>>
>> try {
>>
>> multiSearcher = new
>> MultiSearcher(indexSearchersList.toArray(newIndexSearcher[]
>> {}));
>>
>> } catch (IOException e) {
>>
>> throw new IllegalStateException(e);
>>
>> }
>>
>> return multiSearcher;
>>
>> }
>>
>>
>>
>> This makes all my test pass. I am using the SearchManager that
>> you
>> recommended. Does this look ok?
>>
>>
>> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>> Your maybeReopen has an excess incRef().
>>
>>
>> I'm not sure how you open the searchers in the first place? The
>>
>> list
>> starts as empty, and nothing populates it?
>>
>> When you do the initial population, you need an incRef.
>>
>> I think you're hitting IllegalStateException because
>> maybeReopen
>> is
>> closing a reader before get() can get it (since they
>> synchronize
>> on
>> different objects).
>>
>> I'd recommend switching to the SearcherManager class.
>> Instantiate
>> one
>> for each of your searchers. On each search request, go through
>> them
>> and call maybeReopen(), and then call get() and gather each
>> IndexSearcher instance into a new array. Then, make a new
>> MultiSearcher (opposite of what I said before): while that
>> creates
>> a
>> small amount of garbage, it'll keep your code simpler (good
>> tradeoff).
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> sorrry I added
>>
>>
>> release(multiSearcher);
>>
>>
>>
>> instead of multiSearcher.close();
>>
>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <
>> aminmc@gmail.com
>>
>> wrote:
>>
>>
>>
>> Hi
>>
>>
>> I've now done the following:
>>
>>
>> public Summary[] search(final SearchRequest searchRequest)
>>
>> throwsSearchExecutionException {
>>
>> final String searchTerm = searchRequest.getSearchTerm();
>>
>> if (StringUtils.isBlank(searchTerm)) {
>>
>> throw new SearchExecutionException("Search string cannot be
>> empty.
>> There
>> will be too many results to process.");
>>
>> }
>>
>> List<Summary> summaryList = new ArrayList<Summary>();
>>
>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>
>> stopWatch.start();
>>
>> List<IndexSearcher> indexSearchers = new
>> ArrayList<IndexSearcher>();
>>
>> try {
>>
>> LOGGER.debug("Ensuring all index readers are up to date...");
>>
>> maybeReopen();
>>
>> LOGGER.debug("All Index Searchers are up to date. No of index
>> searchers
>> '"+ indexSearchers.size() +
>> "'");
>>
>> Query query = queryParser.parse(searchTerm);
>>
>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>> Query
>> '"
>> +
>> query.toString() +"'");
>>
>> Sort sort = null;
>>
>> sort = applySortIfApplicable(searchRequest);
>>
>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>
>> ChainedFilter chainedFilter = null;
>>
>> if (filters != null) {
>>
>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>
>> }
>>
>> TopDocs topDocs = get().search(query,chainedFilter
>> ,100,sort);
>>
>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>
>> LOGGER.debug("total number of hits for [" + query.toString()
>> +
>> "
>> ]
>> =
>> "+topDocs.
>> totalHits);
>>
>> for (ScoreDoc scoreDoc : scoreDocs) {
>>
>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>
>> float score = scoreDoc.score;
>>
>> final BaseDocument baseDocument = new BaseDocument(doc,
>> score);
>>
>> Summary documentSummary = new
>> DocumentSummaryImpl(baseDocument);
>>
>> summaryList.add(documentSummary);
>>
>> }
>>
>> multiSearcher.close();
>>
>> } catch (Exception e) {
>>
>> throw new IllegalStateException(e);
>>
>> }
>>
>> stopWatch.stop();
>>
>> LOGGER.debug("total time taken for document seach: " +
>> stopWatch.getTotalTimeMillis() + " ms");
>>
>> return summaryList.toArray(new Summary[] {});
>>
>> }
>>
>>
>> And have the following methods:
>>
>> @PostConstruct
>>
>> public void initialiseQueryParser() {
>>
>> PerFieldAnalyzerWrapper analyzerWrapper = new
>> PerFieldAnalyzerWrapper(
>> analyzer);
>>
>>
>>
>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>> newKeywordAnalyzer());
>>
>> queryParser =
>>
>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>
>> analyzerWrapper);
>>
>> try {
>>
>> LOGGER.debug("Initialising multi searcher ....");
>>
>> this.multiSearcher = new
>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>
>> LOGGER.debug("multi searcher initialised");
>>
>> } catch (IOException e) {
>>
>> throw new IllegalStateException(e);
>>
>> }
>>
>> }
>>
>>
>> Initialises mutltisearcher when this class is creared by
>> spring.
>>
>>
>> private synchronized void swapMultiSearcher(MultiSearcher
>> newMultiSearcher) {
>>
>> try {
>>
>> release(multiSearcher);
>>
>> } catch (IOException e) {
>>
>> throw new IllegalStateException(e);
>>
>> }
>>
>> multiSearcher = newMultiSearcher;
>>
>> }
>>
>> public void maybeReopen() throws IOException {
>>
>> MultiSearcher newMultiSeacher = null;
>>
>> boolean refreshMultiSeacher = false;
>>
>> List<IndexSearcher> indexSearchers = new
>> ArrayList<IndexSearcher>();
>>
>> synchronized (searchers) {
>>
>> for (IndexSearcher indexSearcher: searchers) {
>>
>> IndexReader reader = indexSearcher.getIndexReader();
>>
>> reader.incRef();
>>
>> Directory directory = reader.directory();
>>
>> long currentVersion = reader.getVersion();
>>
>> if (IndexReader.getCurrentVersion(directory) !=
>> currentVersion)
>> {
>>
>> IndexReader newReader =
>> indexSearcher.getIndexReader().reopen();
>>
>> if (newReader != reader) {
>>
>> reader.decRef();
>>
>> refreshMultiSeacher = true;
>>
>> }
>>
>> reader = newReader;
>>
>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>
>> indexSearchers.add(newSearcher);
>>
>> }
>>
>> }
>>
>> }
>>
>>
>>
>> if (refreshMultiSeacher) {
>>
>> newMultiSeacher = new
>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
>>
>> warm(newMultiSeacher);
>>
>> swapMultiSearcher(newMultiSeacher);
>>
>> }
>>
>>
>>
>> }
>>
>>
>> private void warm(MultiSearcher newMultiSeacher) {
>>
>> }
>>
>>
>>
>> private synchronized MultiSearcher get() {
>>
>> for (IndexSearcher indexSearcher: searchers) {
>>
>> indexSearcher.getIndexReader().incRef();
>>
>> }
>>
>> return multiSearcher;
>>
>> }
>>
>> private synchronized void release(MultiSearcher
>> multiSearcher)
>> throwsIOException {
>>
>> for (IndexSearcher indexSearcher: searchers) {
>>
>> indexSearcher.getIndexReader().decRef();
>>
>> }
>>
>> }
>>
>>
>> However I am now getting
>>
>>
>> java.lang.IllegalStateException:
>> org.apache.lucene.store.AlreadyClosedException: this
>> IndexReader
>> is
>> closed
>>
>>
>> on the call:
>>
>>
>> private synchronized MultiSearcher get() {
>>
>> for (IndexSearcher indexSearcher: searchers) {
>>
>> indexSearcher.getIndexReader().incRef();
>>
>> }
>>
>> return multiSearcher;
>>
>> }
>>
>>
>> I'm doing something wrong ..obviously..not sure where
>> though..
>>
>>
>> Cheers
>>
>>
>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>
>> I was wondering the same thing ;)
>>
>>
>> It's best to call this method from a single BG "warming"
>> thread,
>>
>> in
>> which
>> case it would not need its own synchronization.
>>
>> But, to be safe, I'll add internal synchronization to it.
>> You
>> can't
>> simply put synchronized in front of the method, since you
>> don't
>> want
>> this to
>> block searching.
>>
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> just a quick point:
>>
>> public void maybeReopen() throws IOException {
>> //D
>>
>> long currentVersion =
>>
>> currentSearcher.getIndexReader().getVersion();
>>
>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>> IndexReader newReader =
>> currentSearcher.getIndexReader().reopen();
>> assert newReader != currentSearcher.getIndexReader();
>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>> warm(newSearcher);
>> swapSearcher(newSearcher);
>> }
>> }
>>
>> should the above be synchronised?
>>
>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
>> aminmc@gmail.com
>>
>> wrote:
>>
>>
>>
>> thanks. i will rewrite..in between giving my baby her feed
>>
>> and
>>
>> playing
>>
>>
>> with the other child and my wife who wants me to do several
>> other
>>
>> things!
>>
>>
>>
>>
>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>
>> Amin Mohammed-Coleman wrote:
>>
>>
>> Hi
>>
>>
>> Thanks for your input. I would like to have a go at
>> doing
>> this
>>
>> myself
>>
>> first, Solr may be an option.
>>
>>
>> * You are creating a new Analyzer & QueryParser every
>> time,
>> also
>> creating unnecessary garbage; instead, they should be
>> created
>> once
>> & reused.
>>
>> -- I can moved the code out so that it is only created
>> once
>> and
>> reused.
>>
>>
>> * You always make a new IndexSearcher and a new
>> MultiSearcher
>> even
>> when nothing has changed. This just generates
>> unnecessary
>> garbage
>> which GC then must sweep up.
>>
>> -- This was something I thought about. I could move it
>> out
>> so
>> that
>> it's
>> created once. However I presume inside my code i need
>> to
>> check
>> whether
>> the
>> indexreaders are update to date. This needs to be
>> synchronized
>> as
>> well I
>> guess(?)
>>
>>
>> Yes you should synchronize the check for whether the
>> IndexReader
>> is
>>
>> current.
>>
>>
>> * I don't see any synchronization -- it looks like two
>>
>> search
>>
>> requests are allowed into this method at the same time?
>> Which
>> is
>>
>> dangerous... eg both (or, more) will wastefully reopen
>> the
>>
>> readers.
>>
>> -- So i need to extract the logic for reopening and
>> provide
>> a
>> synchronisation mechanism.
>>
>>
>> Yes.
>>
>>
>>
>> Ok. So I have some work to do. I'll refactor the code
>> and
>>
>> see
>> if
>> I
>> can
>>
>> get
>>
>> inline to your recommendations.
>>
>>
>>
>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>
>> On a quick look, I think there are a few problems with
>> the
>> code:
>>
>>
>> * I don't see any synchronization -- it looks like two
>> search
>>
>> requests are allowed into this method at the same time?
>>
>> Which
>> is
>> dangerous... eg both (or, more) will wastefully reopen
>> the
>> readers.
>>
>> * You are over-incRef'ing (the reader.incRef inside the
>> loop)
>> --
>> I
>> don't see a corresponding decRef.
>>
>> * You reopen and warm your searchers "live" (vs with BG
>> thread);
>> meaning the unlucky search request that hits a reopen
>> pays
>> the
>> cost. This might be OK if the index is small enough
>> that
>> reopening & warming takes very little time. But if
>> index
>> gets
>> large, making a random search pay that warming cost is
>> not
>> nice
>> to
>> the end user. It erodes their trust in you.
>>
>> * You always make a new IndexSearcher and a new
>> MultiSearcher
>> even
>> when nothing has changed. This just generates
>> unnecessary
>> garbage
>> which GC then must sweep up.
>>
>> * You are creating a new Analyzer & QueryParser every
>> time,
>> also
>> creating unnecessary garbage; instead, they should be
>> created
>> once
>> & reused.
>>
>> You should consider simply using Solr -- it handles all
>> this
>> logic
>> for
>> you and has been well debugged with time...
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> The reason for the indexreader.reopen is because I have
>> a
>> webapp
>> which
>>
>> enables users to upload files and then search for the
>> documents.
>> If
>>
>> I
>>
>> don't
>>
>> reopen i'm concerned that the facet hit counter won't
>> be
>>
>> updated.
>>
>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman
>> <
>> aminmc@gmail.com
>>
>> wrote:
>>
>>
>>
>> Hi
>>
>>
>>
>> I have been able to get the code working for my
>> scenario,
>>
>> however
>>
>> I
>>
>> have
>>
>> a
>>
>> question and I was wondering if I could get some help.
>>
>> I
>> have
>> a
>> list
>> of
>> IndexSearchers which are used in a MultiSearcher
>> class.
>> I
>> use
>> the
>> indexsearchers to get each indexreader and put them
>> into
>> a
>> MultiIndexReader.
>>
>> IndexReader[] readers = new
>> IndexReader[searchables.length];
>>
>> for (int i =0 ; i < searchables.length;i++) {
>>
>> IndexSearcher indexSearcher =
>> (IndexSearcher)searchables[i];
>>
>> readers[i] = indexSearcher.getIndexReader();
>>
>> IndexReader newReader = readers[i].reopen();
>>
>> if (newReader != readers[i]) {
>>
>> readers[i].close();
>>
>> }
>>
>> readers[i] = newReader;
>>
>>
>>
>> }
>>
>> multiReader = new MultiReader(readers);
>>
>> OpenBitSetFacetHitCounter facetHitCounter =
>> newOpenBitSetFacetHitCounter();
>>
>> IndexSearcher indexSearcher = new
>> IndexSearcher(multiReader);
>>
>>
>> I then use the indexseacher to do the facet stuff. I
>> end
>> the
>> code
>> with
>> closing the multireader. This is causing problems in
>> another
>> method
>> where I
>> do some other search as the indexreaders are closed.
>> Is
>> it
>> ok
>> to
>> not
>> close
>> the multiindexreader or should I do some additional
>> checks
>> in
>> the
>> other
>> method to see if the indexreader is closed?
>>
>>
>>
>> Cheers
>>
>>
>> P.S. Hope that made sense...!
>>
>>
>> On Mon, Feb 23, 2009 at 7:20 AM, Amin
>> Mohammed-Coleman
>> <
>> aminmc@gmail.com
>>
>> wrote:
>>
>>
>>
>> Hi
>>
>>
>>
>>
>> Thanks just what I needed!
>>
>>
>>
>> Cheers
>>
>> Amin
>>
>>
>>
>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>> marcelo.ochoa@gmail.com>
>> wrote:
>>
>> Hi Amin:
>>
>> Please take a look a this blog post:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>> Best regards, Marcelo.
>>
>> On Sun, Feb 22, 2009 at 1:18 PM, Amin
>> Mohammed-Coleman
>> <
>> aminmc@gmail.com>
>> wrote:
>>
>> Hi
>>
>>
>> Sorry to re send this email but I was wondering if
>> I
>> could
>> get
>>
>> some
>>
>> advice
>>
>> on this.
>>
>> Cheers
>>
>> Amin
>>
>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>> aminmc@gmail.com>
>> wrote:
>>
>> Hi
>>
>>
>> I am looking at building a faceted search using
>> Lucene.
>> I
>> know
>>
>> that
>>
>> Solr
>>
>> comes with this built in, however I would like to
>>
>> try
>> this
>> by
>> myself
>> (something to add to my CV!). I have been
>> looking
>> around
>> and
>> I
>> found
>> that
>> you can use the IndexReader and use TermVectors.
>> This
>> looks
>> ok
>> but
>> I'm
>> not
>> sure how to filter the results so that a
>> particular
>> user
>> can
>> only
>> see
>> a
>> subset of results. The next option I was looking
>> at
>> was
>> something
>> like
>>
>> Term term1 = new Term("brand", "ford");
>> Term term2 = new Term("brand", "vw");
>> Term[] termsArray = new Term[] { term1, term2
>> };un
>> int[] docFreqs =
>> indexSearcher.docFreqs(termsArray);
>>
>> The only problem here is that I have to provide
>> the
>> brand
>> type
>> each
>> time a
>> new brand is created. Again I'm not sure how I
>> can
>> filter
>> the
>> results
>> here.
>> It may be that I'm using the wrong api methods to
>> do
>> this.
>>
>> I would be grateful if I could get some advice on
>> this.
>>
>>
>> Cheers
>> Amin
>>
>> P.S. I am basically trying to do something that
>> displays
>> the
>> following
>>
>> Personal Contact (23) Business Contact (45) and
>> so
>> on..
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>>
>> Marcelo F. Ochoa
>>
>>
>> http://marceloochoa.blogspot.com/
>>
>> http://marcelo.ochoa.googlepages.com/home
>>
>> ______________
>> Want to integrate Lucene and Oracle?
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>> Is Oracle 11g REST ready?
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-
>> ready.html
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail:
>> java-user-help@lucene.apache.org
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>>
>>
>> To unsubscribe, e-mail:
>>
>>
>>
>> java-user-unsubscribe@lucene.apache.org
>>
>>
>>
>> For additional commands, e-mail:
>>
>> java-user-help@lucene.apache.org
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>>
>> To unsubscribe, e-mail:
>>
>> java-user-unsubscribe@lucene.apache.org
>>
>>
>> For additional commands, e-mail:
>>
>> java-user-help@lucene.apache.org
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>>
>>
>> To unsubscribe, e-mail:
>>
>>
>> java-user-unsubscribe@lucene.apache.org
>>
>>
>> For additional commands, e-mail:
>>
>> java-user-help@lucene.apache.org
>>
>>
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>>
>>
>> To unsubscribe, e-mail:
>> java-user-unsubscribe@lucene.apache.org
>>
>>
>> For additional commands, e-mail:
>>
>> java-user-help@lucene.apache.org
>>
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>>
>> To unsubscribe, e-mail:
>> java-user-unsubscribe@lucene.apache.org
>>
>>
>> For additional commands, e-mail:
>> java-user-help@lucene.apache.org
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>>
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>>
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>> rg" target="_blank">unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>
>> div>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
Hi
Just out of curiosity does it not make sense to call maybeReopen and
then call get()? If I call get() then I have a new mulitsearcher, so a
call to maybeopen won't reinitialise the multi searcher. Unless I
pass the multi searcher into the maybereopen method. But somehow that
doesn't make sense. I maybe missing something here.
Cheers
Amin
On 2 Mar 2009, at 15:48, Amin Mohammed-Coleman <am...@gmail.com> wrote:
> I'm seeing some interesting behviour when i do get() first followed
> by maybeReopen then there are no documents in the directory
> (directory that i am interested in. When i do the maybeReopen and
> then get() then the doc count is correct. I can post stats later.
>
> Weird...
>
> On Mon, Mar 2, 2009 at 2:17 PM, Amin Mohammed-Coleman <aminmc@gmail.com
> > wrote:
> oh dear...i think i may cry...i'll debug.
>
>
> On Mon, Mar 2, 2009 at 2:15 PM, Michael McCandless <lucene@mikemccandless.com
> > wrote:
>
> Or even just get() with no call to maybeReopen(). That should work
> fine as well.
>
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> In my test case I have a set up method that should populate the
> indexes
> before I start using the document searcher. I will start adding
> some more
> debug statements. So basically I should be able to do: get()
> followed by
> maybeReopen.
>
> I will let you know what the outcome is.
>
>
> Cheers
> Amin
>
> On Mon, Mar 2, 2009 at 1:39 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>
> Is it possible that when you first create the SearcherManager, there
> is no
> index in each Directory?
>
> If not... you better start adding diagnostics. EG inside your
> get(), print
> out the numDocs() of each IndexReader you get from the
> SearcherManager?
>
> Something is wrong and it's best to explain it...
>
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> Nope. If i remove the maybeReopen the search doesn't work. It only
> works
> when i cal maybeReopen followed by get().
>
> Cheers
> Amin
>
> On Mon, Mar 2, 2009 at 12:56 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>
> That's not right; something must be wrong.
>
> get() before maybeReopen() should simply let you search based on the
> searcher before reopening.
>
> If you just do get() and don't call maybeReopen() does it work?
>
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> I noticed that if i do the get() before the maybeReopen then I get no
>
> results. But otherwise I can change it further.
>
> On Mon, Mar 2, 2009 at 11:46 AM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>
> There is no such thing as final code -- code is alive and is always
> changing ;)
>
> It looks good to me.
>
> Though one trivial thing is: I would move the code in the try clause
> up
> to
> and including the multiSearcher=get() out above the try. I always
> attempt
> to "shrink wrap" what's inside a try clause to the minimum that needs
> to
> be
> there. Ie, your code that creates a query, finds the right sort &
> filter
> to
> use, etc, can all happen outside the try, because you have not yet
> acquired
> the multiSearcher.
>
> If you do that, you also don't need the null check in the finally
> clause,
> because multiSearcher must be non-null on entering the try.
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> Hi there
>
> Good morning! Here is the final search code:
>
> public Summary[] search(final SearchRequest searchRequest)
> throwsSearchExecutionException {
>
> final String searchTerm = searchRequest.getSearchTerm();
>
> if (StringUtils.isBlank(searchTerm)) {
>
> throw new SearchExecutionException("Search string cannot be empty.
> There
> will be too many results to process.");
>
> }
>
> List<Summary> summaryList = new ArrayList<Summary>();
>
> StopWatch stopWatch = new StopWatch("searchStopWatch");
>
> stopWatch.start();
>
> MultiSearcher multiSearcher = null;
>
> try {
>
> LOGGER.debug("Ensuring all index readers are up to date...");
>
> maybeReopen();
>
> Query query = queryParser.parse(searchTerm);
>
> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
> query.toString() +"'");
>
> Sort sort = null;
>
> sort = applySortIfApplicable(searchRequest);
>
> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>
> ChainedFilter chainedFilter = null;
>
> if (filters != null) {
>
> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>
> }
>
> multiSearcher = get();
>
> TopDocs topDocs = multiSearcher.search(query,chainedFilter ,100,sort);
>
> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>
> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
> "+topDocs.
> totalHits);
>
> for (ScoreDoc scoreDoc : scoreDocs) {
>
> final Document doc = multiSearcher.doc(scoreDoc.doc);
>
> float score = scoreDoc.score;
>
> final BaseDocument baseDocument = new BaseDocument(doc, score);
>
> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>
> summaryList.add(documentSummary);
>
> }
>
> } catch (Exception e) {
>
> throw new IllegalStateException(e);
>
> } finally {
>
> if (multiSearcher != null) {
>
> release(multiSearcher);
>
> }
>
> }
>
> stopWatch.stop();
>
> LOGGER.debug("total time taken for document seach: " +
> stopWatch.getTotalTimeMillis() + " ms");
>
> return summaryList.toArray(new Summary[] {});
>
> }
>
>
>
> I hope this makes sense...thanks again!
>
>
> Cheers
>
> Amin
>
>
>
> On Sun, Mar 1, 2009 at 8:09 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>
> You're calling get() too many times. For every call to get() you must
>
> match with a call to release().
>
> So, once at the front of your search method you should:
>
> MultiSearcher searcher = get();
>
> then use that searcher to do searching, retrieve docs, etc.
>
> Then in the finally clause, pass that searcher to release.
>
> So, only one call to get() and one matching call to release().
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> Hi
>
> The searchers are injected into the class via Spring. So when a
>
> client
> calls the class it is fully configured with a list of index
> searchers.
> However I have removed this list and instead injecting a list of
> directories which are passed to the DocumentSearchManager.
> DocumentSearchManager is SearchManager (should've mentioned that
> earlier).
> So finally I have modified by release code to do the following:
>
> private void release(MultiSearcher multiSeacher) throws Exception {
>
> IndexSearcher[] indexSearchers = (IndexSearcher[])
> multiSeacher.getSearchables();
>
> for(int i =0 ; i < indexSearchers.length;i++) {
>
> documentSearcherManagers[i].release(indexSearchers[i]);
>
> }
>
> }
>
>
> and it's use looks like this:
>
>
> public Summary[] search(final SearchRequest searchRequest)
> throwsSearchExecutionException {
>
> final String searchTerm = searchRequest.getSearchTerm();
>
> if (StringUtils.isBlank(searchTerm)) {
>
> throw new SearchExecutionException("Search string cannot be empty.
> There
> will be too many results to process.");
>
> }
>
> List<Summary> summaryList = new ArrayList<Summary>();
>
> StopWatch stopWatch = new StopWatch("searchStopWatch");
>
> stopWatch.start();
>
> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>
> try {
>
> LOGGER.debug("Ensuring all index readers are up to date...");
>
> maybeReopen();
>
> LOGGER.debug("All Index Searchers are up to date. No of index
> searchers
> '"
> +
> indexSearchers.size() +"'");
>
> Query query = queryParser.parse(searchTerm);
>
> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '"
> +
> query.toString() +"'");
>
> Sort sort = null;
>
> sort = applySortIfApplicable(searchRequest);
>
> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>
> ChainedFilter chainedFilter = null;
>
> if (filters != null) {
>
> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>
> }
>
> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>
> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>
> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
> "+topDocs.
> totalHits);
>
> for (ScoreDoc scoreDoc : scoreDocs) {
>
> final Document doc = get().doc(scoreDoc.doc);
>
> float score = scoreDoc.score;
>
> final BaseDocument baseDocument = new BaseDocument(doc, score);
>
> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>
> summaryList.add(documentSummary);
>
> }
>
> } catch (Exception e) {
>
> throw new IllegalStateException(e);
>
> } finally {
>
> release(get());
>
> }
>
> stopWatch.stop();
>
> LOGGER.debug("total time taken for document seach: " +
> stopWatch.getTotalTimeMillis() + " ms");
>
> return summaryList.toArray(new Summary[] {});
>
> }
>
>
> So the final post construct constructs the DocumentSearchMangers
> with
> the
> list of directories..looking like this
>
>
> @PostConstruct
>
> public void initialiseDocumentSearcher() {
>
> PerFieldAnalyzerWrapper analyzerWrapper = new
> PerFieldAnalyzerWrapper(
> analyzer);
>
> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
> newKeywordAnalyzer());
>
> queryParser =
> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
> analyzerWrapper);
>
> try {
>
> LOGGER.debug("Initialising multi searcher ....");
>
> documentSearcherManagers = new
> DocumentSearcherManager[directories.size()];
>
> for (int i = 0; i < directories.size() ;i++) {
>
> Directory directory = directories.get(i);
>
> DocumentSearcherManager documentSearcherManager =
> newDocumentSearcherManager(directory);
>
> documentSearcherManagers[i]=documentSearcherManager;
>
> }
>
> LOGGER.debug("multi searcher initialised");
>
> } catch (IOException e) {
>
> throw new IllegalStateException(e);
>
> }
>
> }
>
>
>
> Cheers
>
> Amin
>
>
>
> On Sun, Mar 1, 2009 at 6:15 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>
> I don't understand where searchers comes from, prior to
>
> initializeDocumentSearcher? You should, instead, simply create the
> SearcherManager (from your Directory instances). You don't need
> any
> searchers during initialize.
>
> Is DocumentSearcherManager the same as SearcherManager (just
> renamed)?
>
> The release method is wrong -- you're calling .get() and then
> immediately release. Instead, you should step through the
> searchers
> from your MultiSearcher and release them to each SearcherManager.
>
> You should call your release() in a finally clause.
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> Sorry...i'm getting slightly confused.
>
> I have a PostConstruct which is where I should create an array of
>
> SearchManagers (per indexSeacher). From there I initialise the
> multisearcher using the get(). After which I need to call
> maybeReopen
> for
> each IndexSearcher. So I'll do the following:
>
> @PostConstruct
>
> public void initialiseDocumentSearcher() {
>
> PerFieldAnalyzerWrapper analyzerWrapper = new
> PerFieldAnalyzerWrapper(
> analyzer);
>
> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
> newKeywordAnalyzer());
>
> queryParser =
> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
> analyzerWrapper);
>
> try {
>
> LOGGER.debug("Initialising multi searcher ....");
>
> documentSearcherManagers = new
> DocumentSearcherManager[searchers.size()];
>
> for (int i = 0; i < searchers.size() ;i++) {
>
> IndexSearcher indexSearcher = searchers.get(i);
>
> Directory directory = indexSearcher.getIndexReader().directory();
>
> DocumentSearcherManager documentSearcherManager =
> newDocumentSearcherManager(directory);
>
> documentSearcherManagers[i]=documentSearcherManager;
>
> }
>
> LOGGER.debug("multi searcher initialised");
>
> } catch (IOException e) {
>
> throw new IllegalStateException(e);
>
> }
>
> }
>
>
> This initialises search managers. I then have methods:
>
>
> private void maybeReopen() throws Exception {
>
> LOGGER.debug("Initiating reopening of index readers...");
>
> for (DocumentSearcherManager documentSearcherManager :
> documentSearcherManagers) {
>
> documentSearcherManager.maybeReopen();
>
> }
>
> }
>
>
>
> private void release() throws Exception {
>
> for (DocumentSearcherManager documentSearcherManager :
> documentSearcherManagers) {
>
> documentSearcherManager.release(documentSearcherManager.get());
>
> }
>
> }
>
>
> private MultiSearcher get() {
>
> List<IndexSearcher> listOfIndexSeachers = new
> ArrayList<IndexSearcher>();
>
> for (DocumentSearcherManager documentSearcherManager :
> documentSearcherManagers) {
>
> listOfIndexSeachers.add(documentSearcherManager.get());
>
> }
>
> try {
>
> multiSearcher = new
> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[] {}));
>
> } catch (IOException e) {
>
> throw new IllegalStateException(e);
>
> }
>
> return multiSearcher;
>
> }
>
>
> These methods are used in the following manner in the search code:
>
>
> public Summary[] search(final SearchRequest searchRequest)
> throwsSearchExecutionException {
>
> final String searchTerm = searchRequest.getSearchTerm();
>
> if (StringUtils.isBlank(searchTerm)) {
>
> throw new SearchExecutionException("Search string cannot be empty.
> There
> will be too many results to process.");
>
> }
>
> List<Summary> summaryList = new ArrayList<Summary>();
>
> StopWatch stopWatch = new StopWatch("searchStopWatch");
>
> stopWatch.start();
>
> List<IndexSearcher> indexSearchers = new
> ArrayList<IndexSearcher>();
>
> try {
>
> LOGGER.debug("Ensuring all index readers are up to date...");
>
> maybeReopen();
>
> LOGGER.debug("All Index Searchers are up to date. No of index
> searchers
> '"
> +
> indexSearchers.size() +"'");
>
> Query query = queryParser.parse(searchTerm);
>
> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query
> '"
> +
> query.toString() +"'");
>
> Sort sort = null;
>
> sort = applySortIfApplicable(searchRequest);
>
> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>
> ChainedFilter chainedFilter = null;
>
> if (filters != null) {
>
> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>
> }
>
> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>
> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>
> LOGGER.debug("total number of hits for [" + query.toString() + " ]
> =
> "+topDocs.
> totalHits);
>
> for (ScoreDoc scoreDoc : scoreDocs) {
>
> final Document doc = get().doc(scoreDoc.doc);
>
> float score = scoreDoc.score;
>
> final BaseDocument baseDocument = new BaseDocument(doc, score);
>
> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>
> summaryList.add(documentSummary);
>
> }
>
> release();
>
> } catch (Exception e) {
>
> throw new IllegalStateException(e);
>
> }
>
> stopWatch.stop();
>
> LOGGER.debug("total time taken for document seach: " +
> stopWatch.getTotalTimeMillis() + " ms");
>
> return summaryList.toArray(new Summary[] {});
>
> }
>
>
> Does this look better? Again..I really really appreciate your
> help!
>
>
> On Sun, Mar 1, 2009 at 4:18 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>
> This is not quite right -- you should only create SearcherManager
> once
>
> (per Direcotry) at startup/app load, not with every search
> request.
>
>
> And I don't see release -- it must call SearcherManager.release
> of
> each of the IndexSearchers previously returned from get().
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> Hi
>
> Thanks again for helping on a Sunday!
>
>
> I have now modified my maybeOpen() to do the following:
>
> private void maybeReopen() throws Exception {
>
> LOGGER.debug("Initiating reopening of index readers...");
>
> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
> .getSearchables();
>
> for (IndexSearcher indexSearcher : indexSearchers) {
>
> IndexReader indexReader = indexSearcher.getIndexReader();
>
> SearcherManager documentSearcherManager = new
> SearcherManager(indexReader.directory());
>
> documentSearcherManager.maybeReopen();
>
> }
>
> }
>
>
> And get() to:
>
>
> private synchronized MultiSearcher get() {
>
> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
> .getSearchables();
>
> List<IndexSearcher> indexSearchersList = new
> ArrayList<IndexSearcher>();
>
> for (IndexSearcher indexSearcher : indexSearchers) {
>
> IndexReader indexReader = indexSearcher.getIndexReader();
>
> SearcherManager documentSearcherManager = null;
>
> try {
>
> documentSearcherManager = new
> SearcherManager(indexReader.directory());
>
> } catch (IOException e) {
>
> throw new IllegalStateException(e);
>
> }
>
> indexSearchersList.add(documentSearcherManager.get());
>
> }
>
> try {
>
> multiSearcher = new
> MultiSearcher(indexSearchersList.toArray(newIndexSearcher[]
> {}));
>
> } catch (IOException e) {
>
> throw new IllegalStateException(e);
>
> }
>
> return multiSearcher;
>
> }
>
>
>
> This makes all my test pass. I am using the SearchManager that
> you
> recommended. Does this look ok?
>
>
> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
> Your maybeReopen has an excess incRef().
>
>
> I'm not sure how you open the searchers in the first place? The
>
> list
> starts as empty, and nothing populates it?
>
> When you do the initial population, you need an incRef.
>
> I think you're hitting IllegalStateException because
> maybeReopen
> is
> closing a reader before get() can get it (since they
> synchronize
> on
> different objects).
>
> I'd recommend switching to the SearcherManager class.
> Instantiate
> one
> for each of your searchers. On each search request, go through
> them
> and call maybeReopen(), and then call get() and gather each
> IndexSearcher instance into a new array. Then, make a new
> MultiSearcher (opposite of what I said before): while that
> creates
> a
> small amount of garbage, it'll keep your code simpler (good
> tradeoff).
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> sorrry I added
>
>
> release(multiSearcher);
>
>
>
> instead of multiSearcher.close();
>
> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <
> aminmc@gmail.com
>
> wrote:
>
>
>
> Hi
>
>
> I've now done the following:
>
>
> public Summary[] search(final SearchRequest searchRequest)
>
> throwsSearchExecutionException {
>
> final String searchTerm = searchRequest.getSearchTerm();
>
> if (StringUtils.isBlank(searchTerm)) {
>
> throw new SearchExecutionException("Search string cannot be
> empty.
> There
> will be too many results to process.");
>
> }
>
> List<Summary> summaryList = new ArrayList<Summary>();
>
> StopWatch stopWatch = new StopWatch("searchStopWatch");
>
> stopWatch.start();
>
> List<IndexSearcher> indexSearchers = new
> ArrayList<IndexSearcher>();
>
> try {
>
> LOGGER.debug("Ensuring all index readers are up to date...");
>
> maybeReopen();
>
> LOGGER.debug("All Index Searchers are up to date. No of index
> searchers
> '"+ indexSearchers.size() +
> "'");
>
> Query query = queryParser.parse(searchTerm);
>
> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
> Query
> '"
> +
> query.toString() +"'");
>
> Sort sort = null;
>
> sort = applySortIfApplicable(searchRequest);
>
> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>
> ChainedFilter chainedFilter = null;
>
> if (filters != null) {
>
> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>
> }
>
> TopDocs topDocs = get().search(query,chainedFilter
> ,100,sort);
>
> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>
> LOGGER.debug("total number of hits for [" + query.toString()
> +
> "
> ]
> =
> "+topDocs.
> totalHits);
>
> for (ScoreDoc scoreDoc : scoreDocs) {
>
> final Document doc = multiSearcher.doc(scoreDoc.doc);
>
> float score = scoreDoc.score;
>
> final BaseDocument baseDocument = new BaseDocument(doc,
> score);
>
> Summary documentSummary = new
> DocumentSummaryImpl(baseDocument);
>
> summaryList.add(documentSummary);
>
> }
>
> multiSearcher.close();
>
> } catch (Exception e) {
>
> throw new IllegalStateException(e);
>
> }
>
> stopWatch.stop();
>
> LOGGER.debug("total time taken for document seach: " +
> stopWatch.getTotalTimeMillis() + " ms");
>
> return summaryList.toArray(new Summary[] {});
>
> }
>
>
> And have the following methods:
>
> @PostConstruct
>
> public void initialiseQueryParser() {
>
> PerFieldAnalyzerWrapper analyzerWrapper = new
> PerFieldAnalyzerWrapper(
> analyzer);
>
>
>
> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
> newKeywordAnalyzer());
>
> queryParser =
>
> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>
> analyzerWrapper);
>
> try {
>
> LOGGER.debug("Initialising multi searcher ....");
>
> this.multiSearcher = new
> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>
> LOGGER.debug("multi searcher initialised");
>
> } catch (IOException e) {
>
> throw new IllegalStateException(e);
>
> }
>
> }
>
>
> Initialises mutltisearcher when this class is creared by
> spring.
>
>
> private synchronized void swapMultiSearcher(MultiSearcher
> newMultiSearcher) {
>
> try {
>
> release(multiSearcher);
>
> } catch (IOException e) {
>
> throw new IllegalStateException(e);
>
> }
>
> multiSearcher = newMultiSearcher;
>
> }
>
> public void maybeReopen() throws IOException {
>
> MultiSearcher newMultiSeacher = null;
>
> boolean refreshMultiSeacher = false;
>
> List<IndexSearcher> indexSearchers = new
> ArrayList<IndexSearcher>();
>
> synchronized (searchers) {
>
> for (IndexSearcher indexSearcher: searchers) {
>
> IndexReader reader = indexSearcher.getIndexReader();
>
> reader.incRef();
>
> Directory directory = reader.directory();
>
> long currentVersion = reader.getVersion();
>
> if (IndexReader.getCurrentVersion(directory) !=
> currentVersion)
> {
>
> IndexReader newReader =
> indexSearcher.getIndexReader().reopen();
>
> if (newReader != reader) {
>
> reader.decRef();
>
> refreshMultiSeacher = true;
>
> }
>
> reader = newReader;
>
> IndexSearcher newSearcher = new IndexSearcher(newReader);
>
> indexSearchers.add(newSearcher);
>
> }
>
> }
>
> }
>
>
>
> if (refreshMultiSeacher) {
>
> newMultiSeacher = new
> MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
>
> warm(newMultiSeacher);
>
> swapMultiSearcher(newMultiSeacher);
>
> }
>
>
>
> }
>
>
> private void warm(MultiSearcher newMultiSeacher) {
>
> }
>
>
>
> private synchronized MultiSearcher get() {
>
> for (IndexSearcher indexSearcher: searchers) {
>
> indexSearcher.getIndexReader().incRef();
>
> }
>
> return multiSearcher;
>
> }
>
> private synchronized void release(MultiSearcher
> multiSearcher)
> throwsIOException {
>
> for (IndexSearcher indexSearcher: searchers) {
>
> indexSearcher.getIndexReader().decRef();
>
> }
>
> }
>
>
> However I am now getting
>
>
> java.lang.IllegalStateException:
> org.apache.lucene.store.AlreadyClosedException: this
> IndexReader
> is
> closed
>
>
> on the call:
>
>
> private synchronized MultiSearcher get() {
>
> for (IndexSearcher indexSearcher: searchers) {
>
> indexSearcher.getIndexReader().incRef();
>
> }
>
> return multiSearcher;
>
> }
>
>
> I'm doing something wrong ..obviously..not sure where
> though..
>
>
> Cheers
>
>
> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>
> I was wondering the same thing ;)
>
>
> It's best to call this method from a single BG "warming"
> thread,
>
> in
> which
> case it would not need its own synchronization.
>
> But, to be safe, I'll add internal synchronization to it.
> You
> can't
> simply put synchronized in front of the method, since you
> don't
> want
> this to
> block searching.
>
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> just a quick point:
>
> public void maybeReopen() throws IOException {
> //D
>
> long currentVersion =
>
> currentSearcher.getIndexReader().getVersion();
>
> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
> IndexReader newReader =
> currentSearcher.getIndexReader().reopen();
> assert newReader != currentSearcher.getIndexReader();
> IndexSearcher newSearcher = new IndexSearcher(newReader);
> warm(newSearcher);
> swapSearcher(newSearcher);
> }
> }
>
> should the above be synchronised?
>
> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
> aminmc@gmail.com
>
> wrote:
>
>
>
> thanks. i will rewrite..in between giving my baby her feed
>
> and
>
> playing
>
>
> with the other child and my wife who wants me to do several
> other
>
> things!
>
>
>
>
> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>
> Amin Mohammed-Coleman wrote:
>
>
> Hi
>
>
> Thanks for your input. I would like to have a go at
> doing
> this
>
> myself
>
> first, Solr may be an option.
>
>
> * You are creating a new Analyzer & QueryParser every
> time,
> also
> creating unnecessary garbage; instead, they should be
> created
> once
> & reused.
>
> -- I can moved the code out so that it is only created
> once
> and
> reused.
>
>
> * You always make a new IndexSearcher and a new
> MultiSearcher
> even
> when nothing has changed. This just generates
> unnecessary
> garbage
> which GC then must sweep up.
>
> -- This was something I thought about. I could move it
> out
> so
> that
> it's
> created once. However I presume inside my code i need
> to
> check
> whether
> the
> indexreaders are update to date. This needs to be
> synchronized
> as
> well I
> guess(?)
>
>
> Yes you should synchronize the check for whether the
> IndexReader
> is
>
> current.
>
>
> * I don't see any synchronization -- it looks like two
>
> search
>
> requests are allowed into this method at the same time?
> Which
> is
>
> dangerous... eg both (or, more) will wastefully reopen
> the
>
> readers.
>
> -- So i need to extract the logic for reopening and
> provide
> a
> synchronisation mechanism.
>
>
> Yes.
>
>
>
> Ok. So I have some work to do. I'll refactor the code
> and
>
> see
> if
> I
> can
>
> get
>
> inline to your recommendations.
>
>
>
> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>
> On a quick look, I think there are a few problems with
> the
> code:
>
>
> * I don't see any synchronization -- it looks like two
> search
>
> requests are allowed into this method at the same time?
>
> Which
> is
> dangerous... eg both (or, more) will wastefully reopen
> the
> readers.
>
> * You are over-incRef'ing (the reader.incRef inside the
> loop)
> --
> I
> don't see a corresponding decRef.
>
> * You reopen and warm your searchers "live" (vs with BG
> thread);
> meaning the unlucky search request that hits a reopen
> pays
> the
> cost. This might be OK if the index is small enough
> that
> reopening & warming takes very little time. But if
> index
> gets
> large, making a random search pay that warming cost is
> not
> nice
> to
> the end user. It erodes their trust in you.
>
> * You always make a new IndexSearcher and a new
> MultiSearcher
> even
> when nothing has changed. This just generates
> unnecessary
> garbage
> which GC then must sweep up.
>
> * You are creating a new Analyzer & QueryParser every
> time,
> also
> creating unnecessary garbage; instead, they should be
> created
> once
> & reused.
>
> You should consider simply using Solr -- it handles all
> this
> logic
> for
> you and has been well debugged with time...
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> The reason for the indexreader.reopen is because I have
> a
> webapp
> which
>
> enables users to upload files and then search for the
> documents.
> If
>
> I
>
> don't
>
> reopen i'm concerned that the facet hit counter won't
> be
>
> updated.
>
> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman
> <
> aminmc@gmail.com
>
> wrote:
>
>
>
> Hi
>
>
>
> I have been able to get the code working for my
> scenario,
>
> however
>
> I
>
> have
>
> a
>
> question and I was wondering if I could get some help.
>
> I
> have
> a
> list
> of
> IndexSearchers which are used in a MultiSearcher
> class.
> I
> use
> the
> indexsearchers to get each indexreader and put them
> into
> a
> MultiIndexReader.
>
> IndexReader[] readers = new
> IndexReader[searchables.length];
>
> for (int i =0 ; i < searchables.length;i++) {
>
> IndexSearcher indexSearcher =
> (IndexSearcher)searchables[i];
>
> readers[i] = indexSearcher.getIndexReader();
>
> IndexReader newReader = readers[i].reopen();
>
> if (newReader != readers[i]) {
>
> readers[i].close();
>
> }
>
> readers[i] = newReader;
>
>
>
> }
>
> multiReader = new MultiReader(readers);
>
> OpenBitSetFacetHitCounter facetHitCounter =
> newOpenBitSetFacetHitCounter();
>
> IndexSearcher indexSearcher = new
> IndexSearcher(multiReader);
>
>
> I then use the indexseacher to do the facet stuff. I
> end
> the
> code
> with
> closing the multireader. This is causing problems in
> another
> method
> where I
> do some other search as the indexreaders are closed.
> Is
> it
> ok
> to
> not
> close
> the multiindexreader or should I do some additional
> checks
> in
> the
> other
> method to see if the indexreader is closed?
>
>
>
> Cheers
>
>
> P.S. Hope that made sense...!
>
>
> On Mon, Feb 23, 2009 at 7:20 AM, Amin
> Mohammed-Coleman
> <
> aminmc@gmail.com
>
> wrote:
>
>
>
> Hi
>
>
>
>
> Thanks just what I needed!
>
>
>
> Cheers
>
> Amin
>
>
>
> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
> marcelo.ochoa@gmail.com>
> wrote:
>
> Hi Amin:
>
> Please take a look a this blog post:
>
>
>
>
>
>
>
>
>
>
>
>
> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
> Best regards, Marcelo.
>
> On Sun, Feb 22, 2009 at 1:18 PM, Amin
> Mohammed-Coleman
> <
> aminmc@gmail.com>
> wrote:
>
> Hi
>
>
> Sorry to re send this email but I was wondering if
> I
> could
> get
>
> some
>
> advice
>
> on this.
>
> Cheers
>
> Amin
>
> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
> aminmc@gmail.com>
> wrote:
>
> Hi
>
>
> I am looking at building a faceted search using
> Lucene.
> I
> know
>
> that
>
> Solr
>
> comes with this built in, however I would like to
>
> try
> this
> by
> myself
> (something to add to my CV!). I have been
> looking
> around
> and
> I
> found
> that
> you can use the IndexReader and use TermVectors.
> This
> looks
> ok
> but
> I'm
> not
> sure how to filter the results so that a
> particular
> user
> can
> only
> see
> a
> subset of results. The next option I was looking
> at
> was
> something
> like
>
> Term term1 = new Term("brand", "ford");
> Term term2 = new Term("brand", "vw");
> Term[] termsArray = new Term[] { term1, term2
> };un
> int[] docFreqs =
> indexSearcher.docFreqs(termsArray);
>
> The only problem here is that I have to provide
> the
> brand
> type
> each
> time a
> new brand is created. Again I'm not sure how I
> can
> filter
> the
> results
> here.
> It may be that I'm using the wrong api methods to
> do
> this.
>
> I would be grateful if I could get some advice on
> this.
>
>
> Cheers
> Amin
>
> P.S. I am basically trying to do something that
> displays
> the
> following
>
> Personal Contact (23) Business Contact (45) and
> so
> on..
>
>
>
>
>
>
>
>
>
>
> --
>
>
> Marcelo F. Ochoa
>
>
> http://marceloochoa.blogspot.com/
>
> http://marcelo.ochoa.googlepages.com/home
>
> ______________
> Want to integrate Lucene and Oracle?
>
>
>
>
>
>
>
>
>
>
>
> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
> Is Oracle 11g REST ready?
>
>
>
>
>
>
>
>
>
> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>
>
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail:
> java-user-help@lucene.apache.org
>
>
>
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
>
>
> To unsubscribe, e-mail:
>
>
>
> java-user-unsubscribe@lucene.apache.org
>
>
>
> For additional commands, e-mail:
>
> java-user-help@lucene.apache.org
>
>
>
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
>
> To unsubscribe, e-mail:
>
> java-user-unsubscribe@lucene.apache.org
>
>
> For additional commands, e-mail:
>
> java-user-help@lucene.apache.org
>
>
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
>
>
> To unsubscribe, e-mail:
>
>
> java-user-unsubscribe@lucene.apache.org
>
>
> For additional commands, e-mail:
>
> java-user-help@lucene.apache.org
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
>
>
> To unsubscribe, e-mail:
> java-user-unsubscribe@lucene.apache.org
>
>
> For additional commands, e-mail:
>
> java-user-help@lucene.apache.org
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
>
> To unsubscribe, e-mail:
> java-user-unsubscribe@lucene.apache.org
>
>
> For additional commands, e-mail:
> java-user-help@lucene.apache.org
>
>
>
>
>
>
> ---------------------------------------------------------------------
>
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
>
> ---------------------------------------------------------------------
>
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
> rg" target="_blank">unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
> div>
>
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
I'm seeing some interesting behviour when i do get() first followed by
maybeReopen then there are no documents in the directory (directory that i
am interested in. When i do the maybeReopen and then get() then the doc
count is correct. I can post stats later.
Weird...
On Mon, Mar 2, 2009 at 2:17 PM, Amin Mohammed-Coleman <am...@gmail.com>wrote:
> oh dear...i think i may cry...i'll debug.
>
>
> On Mon, Mar 2, 2009 at 2:15 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>>
>> Or even just get() with no call to maybeReopen(). That should work fine
>> as well.
>>
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> In my test case I have a set up method that should populate the indexes
>>> before I start using the document searcher. I will start adding some
>>> more
>>> debug statements. So basically I should be able to do: get() followed by
>>> maybeReopen.
>>>
>>> I will let you know what the outcome is.
>>>
>>>
>>> Cheers
>>> Amin
>>>
>>> On Mon, Mar 2, 2009 at 1:39 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>>> Is it possible that when you first create the SearcherManager, there is
>>>> no
>>>> index in each Directory?
>>>>
>>>> If not... you better start adding diagnostics. EG inside your get(),
>>>> print
>>>> out the numDocs() of each IndexReader you get from the SearcherManager?
>>>>
>>>> Something is wrong and it's best to explain it...
>>>>
>>>>
>>>> Mike
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> Nope. If i remove the maybeReopen the search doesn't work. It only
>>>> works
>>>>
>>>>> when i cal maybeReopen followed by get().
>>>>>
>>>>> Cheers
>>>>> Amin
>>>>>
>>>>> On Mon, Mar 2, 2009 at 12:56 PM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>>
>>>>> That's not right; something must be wrong.
>>>>>>
>>>>>> get() before maybeReopen() should simply let you search based on the
>>>>>> searcher before reopening.
>>>>>>
>>>>>> If you just do get() and don't call maybeReopen() does it work?
>>>>>>
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>> I noticed that if i do the get() before the maybeReopen then I get no
>>>>>>
>>>>>> results. But otherwise I can change it further.
>>>>>>>
>>>>>>> On Mon, Mar 2, 2009 at 11:46 AM, Michael McCandless <
>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> There is no such thing as final code -- code is alive and is always
>>>>>>>
>>>>>>>> changing ;)
>>>>>>>>
>>>>>>>> It looks good to me.
>>>>>>>>
>>>>>>>> Though one trivial thing is: I would move the code in the try clause
>>>>>>>> up
>>>>>>>> to
>>>>>>>> and including the multiSearcher=get() out above the try. I always
>>>>>>>> attempt
>>>>>>>> to "shrink wrap" what's inside a try clause to the minimum that
>>>>>>>> needs
>>>>>>>> to
>>>>>>>> be
>>>>>>>> there. Ie, your code that creates a query, finds the right sort &
>>>>>>>> filter
>>>>>>>> to
>>>>>>>> use, etc, can all happen outside the try, because you have not yet
>>>>>>>> acquired
>>>>>>>> the multiSearcher.
>>>>>>>>
>>>>>>>> If you do that, you also don't need the null check in the finally
>>>>>>>> clause,
>>>>>>>> because multiSearcher must be non-null on entering the try.
>>>>>>>>
>>>>>>>> Mike
>>>>>>>>
>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>
>>>>>>>> Hi there
>>>>>>>>
>>>>>>>> Good morning! Here is the final search code:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>>> throwsSearchExecutionException {
>>>>>>>>>
>>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>>
>>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>>
>>>>>>>>> throw new SearchExecutionException("Search string cannot be empty.
>>>>>>>>> There
>>>>>>>>> will be too many results to process.");
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>
>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>
>>>>>>>>> stopWatch.start();
>>>>>>>>>
>>>>>>>>> MultiSearcher multiSearcher = null;
>>>>>>>>>
>>>>>>>>> try {
>>>>>>>>>
>>>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>>>
>>>>>>>>> maybeReopen();
>>>>>>>>>
>>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>>
>>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '"
>>>>>>>>> +
>>>>>>>>> query.toString() +"'");
>>>>>>>>>
>>>>>>>>> Sort sort = null;
>>>>>>>>>
>>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>>
>>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>>
>>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>>
>>>>>>>>> if (filters != null) {
>>>>>>>>>
>>>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> multiSearcher = get();
>>>>>>>>>
>>>>>>>>> TopDocs topDocs = multiSearcher.search(query,chainedFilter
>>>>>>>>> ,100,sort);
>>>>>>>>>
>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>
>>>>>>>>> LOGGER.debug("total number of hits for [" + query.toString() + " ]
>>>>>>>>> =
>>>>>>>>> "+topDocs.
>>>>>>>>> totalHits);
>>>>>>>>>
>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>
>>>>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>>>>
>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>
>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>>>>
>>>>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>>>>
>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> } catch (Exception e) {
>>>>>>>>>
>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>
>>>>>>>>> } finally {
>>>>>>>>>
>>>>>>>>> if (multiSearcher != null) {
>>>>>>>>>
>>>>>>>>> release(multiSearcher);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> stopWatch.stop();
>>>>>>>>>
>>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>
>>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I hope this makes sense...thanks again!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>>
>>>>>>>>> Amin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Mar 1, 2009 at 8:09 PM, Michael McCandless <
>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> You're calling get() too many times. For every call to get() you
>>>>>>>>> must
>>>>>>>>>
>>>>>>>>> match with a call to release().
>>>>>>>>>>
>>>>>>>>>> So, once at the front of your search method you should:
>>>>>>>>>>
>>>>>>>>>> MultiSearcher searcher = get();
>>>>>>>>>>
>>>>>>>>>> then use that searcher to do searching, retrieve docs, etc.
>>>>>>>>>>
>>>>>>>>>> Then in the finally clause, pass that searcher to release.
>>>>>>>>>>
>>>>>>>>>> So, only one call to get() and one matching call to release().
>>>>>>>>>>
>>>>>>>>>> Mike
>>>>>>>>>>
>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> The searchers are injected into the class via Spring. So when a
>>>>>>>>>>
>>>>>>>>>> client
>>>>>>>>>>> calls the class it is fully configured with a list of index
>>>>>>>>>>> searchers.
>>>>>>>>>>> However I have removed this list and instead injecting a list of
>>>>>>>>>>> directories which are passed to the DocumentSearchManager.
>>>>>>>>>>> DocumentSearchManager is SearchManager (should've mentioned that
>>>>>>>>>>> earlier).
>>>>>>>>>>> So finally I have modified by release code to do the following:
>>>>>>>>>>>
>>>>>>>>>>> private void release(MultiSearcher multiSeacher) throws Exception
>>>>>>>>>>> {
>>>>>>>>>>>
>>>>>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>>>>>>>>> multiSeacher.getSearchables();
>>>>>>>>>>>
>>>>>>>>>>> for(int i =0 ; i < indexSearchers.length;i++) {
>>>>>>>>>>>
>>>>>>>>>>> documentSearcherManagers[i].release(indexSearchers[i]);
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> and it's use looks like this:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>>>>> throwsSearchExecutionException {
>>>>>>>>>>>
>>>>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>>>>
>>>>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>>>>
>>>>>>>>>>> throw new SearchExecutionException("Search string cannot be
>>>>>>>>>>> empty.
>>>>>>>>>>> There
>>>>>>>>>>> will be too many results to process.");
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>>>
>>>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>>>
>>>>>>>>>>> stopWatch.start();
>>>>>>>>>>>
>>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>
>>>>>>>>>>> try {
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>>>>>
>>>>>>>>>>> maybeReopen();
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>>>>>>> searchers
>>>>>>>>>>> '"
>>>>>>>>>>> +
>>>>>>>>>>> indexSearchers.size() +"'");
>>>>>>>>>>>
>>>>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query
>>>>>>>>>>> '"
>>>>>>>>>>> +
>>>>>>>>>>> query.toString() +"'");
>>>>>>>>>>>
>>>>>>>>>>> Sort sort = null;
>>>>>>>>>>>
>>>>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>>>>
>>>>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>>>>
>>>>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>>>>
>>>>>>>>>>> if (filters != null) {
>>>>>>>>>>>
>>>>>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>>>>>>
>>>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("total number of hits for [" + query.toString() + "
>>>>>>>>>>> ] =
>>>>>>>>>>> "+topDocs.
>>>>>>>>>>> totalHits);
>>>>>>>>>>>
>>>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>>>
>>>>>>>>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>>>>>>>>
>>>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>>>
>>>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>>>>>>
>>>>>>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>>>>>>
>>>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>>
>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>
>>>>>>>>>>> } finally {
>>>>>>>>>>>
>>>>>>>>>>> release(get());
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> stopWatch.stop();
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>>>
>>>>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> So the final post construct constructs the DocumentSearchMangers
>>>>>>>>>>> with
>>>>>>>>>>> the
>>>>>>>>>>> list of directories..looking like this
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> @PostConstruct
>>>>>>>>>>>
>>>>>>>>>>> public void initialiseDocumentSearcher() {
>>>>>>>>>>>
>>>>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>>>>> analyzer);
>>>>>>>>>>>
>>>>>>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>>>>> newKeywordAnalyzer());
>>>>>>>>>>>
>>>>>>>>>>> queryParser =
>>>>>>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>>>> analyzerWrapper);
>>>>>>>>>>>
>>>>>>>>>>> try {
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>>>>
>>>>>>>>>>> documentSearcherManagers = new
>>>>>>>>>>> DocumentSearcherManager[directories.size()];
>>>>>>>>>>>
>>>>>>>>>>> for (int i = 0; i < directories.size() ;i++) {
>>>>>>>>>>>
>>>>>>>>>>> Directory directory = directories.get(i);
>>>>>>>>>>>
>>>>>>>>>>> DocumentSearcherManager documentSearcherManager =
>>>>>>>>>>> newDocumentSearcherManager(directory);
>>>>>>>>>>>
>>>>>>>>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>>>>
>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>
>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Cheers
>>>>>>>>>>>
>>>>>>>>>>> Amin
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Mar 1, 2009 at 6:15 PM, Michael McCandless <
>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I don't understand where searchers comes from, prior to
>>>>>>>>>>>
>>>>>>>>>>> initializeDocumentSearcher? You should, instead, simply create
>>>>>>>>>>> the
>>>>>>>>>>>
>>>>>>>>>>>> SearcherManager (from your Directory instances). You don't need
>>>>>>>>>>>> any
>>>>>>>>>>>> searchers during initialize.
>>>>>>>>>>>>
>>>>>>>>>>>> Is DocumentSearcherManager the same as SearcherManager (just
>>>>>>>>>>>> renamed)?
>>>>>>>>>>>>
>>>>>>>>>>>> The release method is wrong -- you're calling .get() and then
>>>>>>>>>>>> immediately release. Instead, you should step through the
>>>>>>>>>>>> searchers
>>>>>>>>>>>> from your MultiSearcher and release them to each
>>>>>>>>>>>> SearcherManager.
>>>>>>>>>>>>
>>>>>>>>>>>> You should call your release() in a finally clause.
>>>>>>>>>>>>
>>>>>>>>>>>> Mike
>>>>>>>>>>>>
>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Sorry...i'm getting slightly confused.
>>>>>>>>>>>>
>>>>>>>>>>>> I have a PostConstruct which is where I should create an array
>>>>>>>>>>>> of
>>>>>>>>>>>>
>>>>>>>>>>>> SearchManagers (per indexSeacher). From there I initialise the
>>>>>>>>>>>>
>>>>>>>>>>>>> multisearcher using the get(). After which I need to call
>>>>>>>>>>>>> maybeReopen
>>>>>>>>>>>>> for
>>>>>>>>>>>>> each IndexSearcher. So I'll do the following:
>>>>>>>>>>>>>
>>>>>>>>>>>>> @PostConstruct
>>>>>>>>>>>>>
>>>>>>>>>>>>> public void initialiseDocumentSearcher() {
>>>>>>>>>>>>>
>>>>>>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>>>>>>> analyzer);
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>>>>>>> newKeywordAnalyzer());
>>>>>>>>>>>>>
>>>>>>>>>>>>> queryParser =
>>>>>>>>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>>>>>> analyzerWrapper);
>>>>>>>>>>>>>
>>>>>>>>>>>>> try {
>>>>>>>>>>>>>
>>>>>>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>>>>>>
>>>>>>>>>>>>> documentSearcherManagers = new
>>>>>>>>>>>>> DocumentSearcherManager[searchers.size()];
>>>>>>>>>>>>>
>>>>>>>>>>>>> for (int i = 0; i < searchers.size() ;i++) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> IndexSearcher indexSearcher = searchers.get(i);
>>>>>>>>>>>>>
>>>>>>>>>>>>> Directory directory =
>>>>>>>>>>>>> indexSearcher.getIndexReader().directory();
>>>>>>>>>>>>>
>>>>>>>>>>>>> DocumentSearcherManager documentSearcherManager =
>>>>>>>>>>>>> newDocumentSearcherManager(directory);
>>>>>>>>>>>>>
>>>>>>>>>>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>>>>>>
>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This initialises search managers. I then have methods:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> private void maybeReopen() throws Exception {
>>>>>>>>>>>>>
>>>>>>>>>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>>>>>>>>>
>>>>>>>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>>>>>>>> documentSearcherManagers) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> documentSearcherManager.maybeReopen();
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> private void release() throws Exception {
>>>>>>>>>>>>>
>>>>>>>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>>>>>>>> documentSearcherManagers) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> documentSearcherManager.release(documentSearcherManager.get());
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> private MultiSearcher get() {
>>>>>>>>>>>>>
>>>>>>>>>>>>> List<IndexSearcher> listOfIndexSeachers = new
>>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>>
>>>>>>>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>>>>>>>> documentSearcherManagers) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> listOfIndexSeachers.add(documentSearcherManager.get());
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> try {
>>>>>>>>>>>>>
>>>>>>>>>>>>> multiSearcher = new
>>>>>>>>>>>>> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[]
>>>>>>>>>>>>> {}));
>>>>>>>>>>>>>
>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> These methods are used in the following manner in the search
>>>>>>>>>>>>> code:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>>>>>>> throwsSearchExecutionException {
>>>>>>>>>>>>>
>>>>>>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>>>>>>
>>>>>>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> throw new SearchExecutionException("Search string cannot be
>>>>>>>>>>>>> empty.
>>>>>>>>>>>>> There
>>>>>>>>>>>>> will be too many results to process.");
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>>>>>
>>>>>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>>>>>
>>>>>>>>>>>>> stopWatch.start();
>>>>>>>>>>>>>
>>>>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>>
>>>>>>>>>>>>> try {
>>>>>>>>>>>>>
>>>>>>>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>>>>>>>
>>>>>>>>>>>>> maybeReopen();
>>>>>>>>>>>>>
>>>>>>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>>>>>>>>> searchers
>>>>>>>>>>>>> '"
>>>>>>>>>>>>> +
>>>>>>>>>>>>> indexSearchers.size() +"'");
>>>>>>>>>>>>>
>>>>>>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>>>>>>
>>>>>>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>>>>>>>>>>>>> Query
>>>>>>>>>>>>> '"
>>>>>>>>>>>>> +
>>>>>>>>>>>>> query.toString() +"'");
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sort sort = null;
>>>>>>>>>>>>>
>>>>>>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>>>>>>
>>>>>>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>>>>>>
>>>>>>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>>>>>>
>>>>>>>>>>>>> if (filters != null) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>>>>>>>>
>>>>>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>>>>>
>>>>>>>>>>>>> LOGGER.debug("total number of hits for [" + query.toString() +
>>>>>>>>>>>>> " ]
>>>>>>>>>>>>> =
>>>>>>>>>>>>> "+topDocs.
>>>>>>>>>>>>> totalHits);
>>>>>>>>>>>>>
>>>>>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>>>>>>>>>>
>>>>>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>>>>>
>>>>>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>>>>>>>>
>>>>>>>>>>>>> Summary documentSummary = new
>>>>>>>>>>>>> DocumentSummaryImpl(baseDocument);
>>>>>>>>>>>>>
>>>>>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> release();
>>>>>>>>>>>>>
>>>>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> stopWatch.stop();
>>>>>>>>>>>>>
>>>>>>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>>>>>
>>>>>>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Does this look better? Again..I really really appreciate your
>>>>>>>>>>>>> help!
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 4:18 PM, Michael McCandless <
>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This is not quite right -- you should only create
>>>>>>>>>>>>> SearcherManager
>>>>>>>>>>>>> once
>>>>>>>>>>>>>
>>>>>>>>>>>>> (per Direcotry) at startup/app load, not with every search
>>>>>>>>>>>>> request.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> And I don't see release -- it must call
>>>>>>>>>>>>>> SearcherManager.release
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> each of the IndexSearchers previously returned from get().
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks again for helping on a Sunday!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have now modified my maybeOpen() to do the following:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> private void maybeReopen() throws Exception {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>>>>>>>>>>>>> multiSearcher
>>>>>>>>>>>>>>> .getSearchables();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> SearcherManager documentSearcherManager = new
>>>>>>>>>>>>>>> SearcherManager(indexReader.directory());
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> documentSearcherManager.maybeReopen();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> And get() to:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>>>>>>>>>>>>> multiSearcher
>>>>>>>>>>>>>>> .getSearchables();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> List<IndexSearcher> indexSearchersList = new
>>>>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> SearcherManager documentSearcherManager = null;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> documentSearcherManager = new
>>>>>>>>>>>>>>> SearcherManager(indexReader.directory());
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> indexSearchersList.add(documentSearcherManager.get());
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> multiSearcher = new
>>>>>>>>>>>>>>> MultiSearcher(indexSearchersList.toArray(newIndexSearcher[]
>>>>>>>>>>>>>>> {}));
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This makes all my test pass. I am using the SearchManager
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> recommended. Does this look ok?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Your maybeReopen has an excess incRef().
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm not sure how you open the searchers in the first place?
>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> starts as empty, and nothing populates it?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When you do the initial population, you need an incRef.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think you're hitting IllegalStateException because
>>>>>>>>>>>>>>>> maybeReopen
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> closing a reader before get() can get it (since they
>>>>>>>>>>>>>>>> synchronize
>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>> different objects).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'd recommend switching to the SearcherManager class.
>>>>>>>>>>>>>>>> Instantiate
>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>> for each of your searchers. On each search request, go
>>>>>>>>>>>>>>>> through
>>>>>>>>>>>>>>>> them
>>>>>>>>>>>>>>>> and call maybeReopen(), and then call get() and gather each
>>>>>>>>>>>>>>>> IndexSearcher instance into a new array. Then, make a new
>>>>>>>>>>>>>>>> MultiSearcher (opposite of what I said before): while that
>>>>>>>>>>>>>>>> creates
>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> small amount of garbage, it'll keep your code simpler (good
>>>>>>>>>>>>>>>> tradeoff).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> sorrry I added
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> release(multiSearcher);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> instead of multiSearcher.close();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I've now done the following:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> throwsSearchExecutionException {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> throw new SearchExecutionException("Search string cannot
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> empty.
>>>>>>>>>>>>>>>>>> There
>>>>>>>>>>>>>>>>>> will be too many results to process.");
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> stopWatch.start();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> LOGGER.debug("Ensuring all index readers are up to
>>>>>>>>>>>>>>>>>> date...");
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> maybeReopen();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of
>>>>>>>>>>>>>>>>>> index
>>>>>>>>>>>>>>>>>> searchers
>>>>>>>>>>>>>>>>>> '"+ indexSearchers.size() +
>>>>>>>>>>>>>>>>>> "'");
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>>>>>>>>>>>>>>>>>> Query
>>>>>>>>>>>>>>>>>> '"
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> query.toString() +"'");
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Sort sort = null;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> if (filters != null) {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> chainedFilter = new ChainedFilter(filters,
>>>>>>>>>>>>>>>>>> ChainedFilter.OR);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter
>>>>>>>>>>>>>>>>>> ,100,sort);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> LOGGER.debug("total number of hits for [" +
>>>>>>>>>>>>>>>>>> query.toString()
>>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>>> "
>>>>>>>>>>>>>>>>>> ]
>>>>>>>>>>>>>>>>>> =
>>>>>>>>>>>>>>>>>> "+topDocs.
>>>>>>>>>>>>>>>>>> totalHits);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc,
>>>>>>>>>>>>>>>>>> score);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Summary documentSummary = new
>>>>>>>>>>>>>>>>>> DocumentSummaryImpl(baseDocument);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> multiSearcher.close();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> stopWatch.stop();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>>>>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> And have the following methods:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> @PostConstruct
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> public void initialiseQueryParser() {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>>>>>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>>>>>>>>>>>> analyzer);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>>>>>>>>>>>> newKeywordAnalyzer());
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> queryParser =
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> analyzerWrapper);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> this.multiSearcher = new
>>>>>>>>>>>>>>>>>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Initialises mutltisearcher when this class is creared by
>>>>>>>>>>>>>>>>>> spring.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> private synchronized void swapMultiSearcher(MultiSearcher
>>>>>>>>>>>>>>>>>> newMultiSearcher) {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> release(multiSearcher);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> multiSearcher = newMultiSearcher;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> public void maybeReopen() throws IOException {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> MultiSearcher newMultiSeacher = null;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> boolean refreshMultiSeacher = false;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> synchronized (searchers) {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> IndexReader reader = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> reader.incRef();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Directory directory = reader.directory();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> long currentVersion = reader.getVersion();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> if (IndexReader.getCurrentVersion(directory) !=
>>>>>>>>>>>>>>>>>> currentVersion)
>>>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> IndexReader newReader =
>>>>>>>>>>>>>>>>>> indexSearcher.getIndexReader().reopen();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> if (newReader != reader) {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> reader.decRef();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> refreshMultiSeacher = true;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> reader = newReader;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> indexSearchers.add(newSearcher);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> if (refreshMultiSeacher) {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> newMultiSeacher = new
>>>>>>>>>>>>>>>>>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[]
>>>>>>>>>>>>>>>>>> {}));
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> warm(newMultiSeacher);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> swapMultiSearcher(newMultiSeacher);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> private void warm(MultiSearcher newMultiSeacher) {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> private synchronized void release(MultiSearcher
>>>>>>>>>>>>>>>>>> multiSearcher)
>>>>>>>>>>>>>>>>>> throwsIOException {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> indexSearcher.getIndexReader().decRef();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> However I am now getting
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> java.lang.IllegalStateException:
>>>>>>>>>>>>>>>>>> org.apache.lucene.store.AlreadyClosedException: this
>>>>>>>>>>>>>>>>>> IndexReader
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> closed
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> on the call:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'm doing something wrong ..obviously..not sure where
>>>>>>>>>>>>>>>>>> though..
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>>>>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I was wondering the same thing ;)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> It's best to call this method from a single BG "warming"
>>>>>>>>>>>>>>>>>> thread,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>> case it would not need its own synchronization.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> But, to be safe, I'll add internal synchronization to it.
>>>>>>>>>>>>>>>>>>> You
>>>>>>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>>>> simply put synchronized in front of the method, since you
>>>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>>>>>> this to
>>>>>>>>>>>>>>>>>>> block searching.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> just a quick point:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> public void maybeReopen() throws IOException {
>>>>>>>>>>>>>>>>>>> //D
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> long currentVersion =
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> currentSearcher.getIndexReader().getVersion();
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> if (IndexReader.getCurrentVersion(dir) !=
>>>>>>>>>>>>>>>>>>>> currentVersion) {
>>>>>>>>>>>>>>>>>>>> IndexReader newReader =
>>>>>>>>>>>>>>>>>>>> currentSearcher.getIndexReader().reopen();
>>>>>>>>>>>>>>>>>>>> assert newReader != currentSearcher.getIndexReader();
>>>>>>>>>>>>>>>>>>>> IndexSearcher newSearcher = new
>>>>>>>>>>>>>>>>>>>> IndexSearcher(newReader);
>>>>>>>>>>>>>>>>>>>> warm(newSearcher);
>>>>>>>>>>>>>>>>>>>> swapSearcher(newSearcher);
>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> should the above be synchronised?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> thanks. i will rewrite..in between giving my baby her
>>>>>>>>>>>>>>>>>>>> feed
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> playing
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> with the other child and my wife who wants me to do
>>>>>>>>>>>>>>>>>>>> several
>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> things!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>>>>>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for your input. I would like to have a go at
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> doing
>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> first, Solr may be an option.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser every
>>>>>>>>>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>> creating unnecessary garbage; instead, they should be
>>>>>>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> -- I can moved the code out so that it is only
>>>>>>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>> reused.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>>>>>>>>> when nothing has changed. This just generates
>>>>>>>>>>>>>>>>>>>>>>> unnecessary
>>>>>>>>>>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> -- This was something I thought about. I could move
>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>> out
>>>>>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>> created once. However I presume inside my code i
>>>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>>>>>>>> whether
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> indexreaders are update to date. This needs to be
>>>>>>>>>>>>>>>>>>>>>>> synchronized
>>>>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>>>> well I
>>>>>>>>>>>>>>>>>>>>>>> guess(?)
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Yes you should synchronize the check for whether the
>>>>>>>>>>>>>>>>>>>>>>> IndexReader
>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> current.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> * I don't see any synchronization -- it looks like
>>>>>>>>>>>>>>>>>>>>>>> two
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> search
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> requests are allowed into this method at the same
>>>>>>>>>>>>>>>>>>>>>> time?
>>>>>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> -- So i need to extract the logic for reopening and
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>> synchronisation mechanism.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Yes.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Ok. So I have some work to do. I'll refactor the
>>>>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> inline to your recommendations.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On a quick look, I think there are a few problems
>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> code:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> * I don't see any synchronization -- it looks like
>>>>>>>>>>>>>>>>>>>>>>> two
>>>>>>>>>>>>>>>>>>>>>>> search
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> requests are allowed into this method at the same
>>>>>>>>>>>>>>>>>>>>>>> time?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully
>>>>>>>>>>>>>>>>>>>>>>>> reopen
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> * You are over-incRef'ing (the reader.incRef inside
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> loop)
>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>> don't see a corresponding decRef.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> * You reopen and warm your searchers "live" (vs with
>>>>>>>>>>>>>>>>>>>>>>>> BG
>>>>>>>>>>>>>>>>>>>>>>>> thread);
>>>>>>>>>>>>>>>>>>>>>>>> meaning the unlucky search request that hits a
>>>>>>>>>>>>>>>>>>>>>>>> reopen
>>>>>>>>>>>>>>>>>>>>>>>> pays
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> cost. This might be OK if the index is small enough
>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>> reopening & warming takes very little time. But if
>>>>>>>>>>>>>>>>>>>>>>>> index
>>>>>>>>>>>>>>>>>>>>>>>> gets
>>>>>>>>>>>>>>>>>>>>>>>> large, making a random search pay that warming cost
>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>> nice
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> the end user. It erodes their trust in you.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>>>>>>>>>> when nothing has changed. This just generates
>>>>>>>>>>>>>>>>>>>>>>>> unnecessary
>>>>>>>>>>>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser
>>>>>>>>>>>>>>>>>>>>>>>> every
>>>>>>>>>>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>>> creating unnecessary garbage; instead, they should
>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> You should consider simply using Solr -- it handles
>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>> logic
>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>> you and has been well debugged with time...
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> The reason for the indexreader.reopen is because I
>>>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>> webapp
>>>>>>>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> enables users to upload files and then search for
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> documents.
>>>>>>>>>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> reopen i'm concerned that the facet hit counter
>>>>>>>>>>>>>>>>>>>>>>>> won't
>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> updated.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin
>>>>>>>>>>>>>>>>>>>>>>>>> Mohammed-Coleman
>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I have been able to get the code working for my
>>>>>>>>>>>>>>>>>>>>>>>>>> scenario,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> however
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> question and I was wondering if I could get some
>>>>>>>>>>>>>>>>>>>>>>>>> help.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>> IndexSearchers which are used in a MultiSearcher
>>>>>>>>>>>>>>>>>>>>>>>>>> class.
>>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> indexsearchers to get each indexreader and put
>>>>>>>>>>>>>>>>>>>>>>>>>> them
>>>>>>>>>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>> MultiIndexReader.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> IndexReader[] readers = new
>>>>>>>>>>>>>>>>>>>>>>>>>> IndexReader[searchables.length];
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> IndexSearcher indexSearcher =
>>>>>>>>>>>>>>>>>>>>>>>>>> (IndexSearcher)searchables[i];
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> readers[i].close();
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> readers[i] = newReader;
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>>>>>>>>>>>>>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> IndexSearcher indexSearcher = new
>>>>>>>>>>>>>>>>>>>>>>>>>> IndexSearcher(multiReader);
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I then use the indexseacher to do the facet stuff.
>>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>> end
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>> closing the multireader. This is causing problems
>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>> another
>>>>>>>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>>>>> where I
>>>>>>>>>>>>>>>>>>>>>>>>>> do some other search as the indexreaders are
>>>>>>>>>>>>>>>>>>>>>>>>>> closed.
>>>>>>>>>>>>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>> close
>>>>>>>>>>>>>>>>>>>>>>>>>> the multiindexreader or should I do some
>>>>>>>>>>>>>>>>>>>>>>>>>> additional
>>>>>>>>>>>>>>>>>>>>>>>>>> checks
>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin
>>>>>>>>>>>>>>>>>>>>>>>>>> Mohammed-Coleman
>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks just what I needed!
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>>>>>>>>>>>>>>>>>>>>>>>>>>> marcelo.ochoa@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Amin:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Mohammed-Coleman
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry to re send this email but I was wondering
>>>>>>>>>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> advice
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I am looking at building a faceted search using
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lucene.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Solr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comes with this built in, however I would like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> try
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (something to add to my CV!). I have been
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> looking
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you can use the IndexReader and use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> TermVectors.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sure how to filter the results so that a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> particular
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> subset of results. The next option I was
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> looking
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> };un
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> int[] docFreqs =
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The only problem here is that I have to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> brand
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new brand is created. Again I'm not sure how
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> filter
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> results
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It may be that I'm using the wrong api methods
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would be grateful if I could get some advice
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> P.S. I am basically trying to do something
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> displays
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Personal Contact (23) Business Contact (45)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on..
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> To unsubscribe, e-mail: java-user-
>>>>>>>>>>> unsubscribe@lucene.apache.org
>>>>>>>>>>>
>>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>
>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
oh dear...i think i may cry...i'll debug.
On Mon, Mar 2, 2009 at 2:15 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
>
> Or even just get() with no call to maybeReopen(). That should work fine as
> well.
>
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> In my test case I have a set up method that should populate the indexes
>> before I start using the document searcher. I will start adding some more
>> debug statements. So basically I should be able to do: get() followed by
>> maybeReopen.
>>
>> I will let you know what the outcome is.
>>
>>
>> Cheers
>> Amin
>>
>> On Mon, Mar 2, 2009 at 1:39 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>
>>> Is it possible that when you first create the SearcherManager, there is
>>> no
>>> index in each Directory?
>>>
>>> If not... you better start adding diagnostics. EG inside your get(),
>>> print
>>> out the numDocs() of each IndexReader you get from the SearcherManager?
>>>
>>> Something is wrong and it's best to explain it...
>>>
>>>
>>> Mike
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> Nope. If i remove the maybeReopen the search doesn't work. It only works
>>>
>>>> when i cal maybeReopen followed by get().
>>>>
>>>> Cheers
>>>> Amin
>>>>
>>>> On Mon, Mar 2, 2009 at 12:56 PM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>>
>>>> That's not right; something must be wrong.
>>>>>
>>>>> get() before maybeReopen() should simply let you search based on the
>>>>> searcher before reopening.
>>>>>
>>>>> If you just do get() and don't call maybeReopen() does it work?
>>>>>
>>>>>
>>>>> Mike
>>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>> I noticed that if i do the get() before the maybeReopen then I get no
>>>>>
>>>>> results. But otherwise I can change it further.
>>>>>>
>>>>>> On Mon, Mar 2, 2009 at 11:46 AM, Michael McCandless <
>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>
>>>>>>
>>>>>> There is no such thing as final code -- code is alive and is always
>>>>>>
>>>>>>> changing ;)
>>>>>>>
>>>>>>> It looks good to me.
>>>>>>>
>>>>>>> Though one trivial thing is: I would move the code in the try clause
>>>>>>> up
>>>>>>> to
>>>>>>> and including the multiSearcher=get() out above the try. I always
>>>>>>> attempt
>>>>>>> to "shrink wrap" what's inside a try clause to the minimum that needs
>>>>>>> to
>>>>>>> be
>>>>>>> there. Ie, your code that creates a query, finds the right sort &
>>>>>>> filter
>>>>>>> to
>>>>>>> use, etc, can all happen outside the try, because you have not yet
>>>>>>> acquired
>>>>>>> the multiSearcher.
>>>>>>>
>>>>>>> If you do that, you also don't need the null check in the finally
>>>>>>> clause,
>>>>>>> because multiSearcher must be non-null on entering the try.
>>>>>>>
>>>>>>> Mike
>>>>>>>
>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>
>>>>>>> Hi there
>>>>>>>
>>>>>>> Good morning! Here is the final search code:
>>>>>>>
>>>>>>>>
>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>> throwsSearchExecutionException {
>>>>>>>>
>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>
>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>
>>>>>>>> throw new SearchExecutionException("Search string cannot be empty.
>>>>>>>> There
>>>>>>>> will be too many results to process.");
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>
>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>
>>>>>>>> stopWatch.start();
>>>>>>>>
>>>>>>>> MultiSearcher multiSearcher = null;
>>>>>>>>
>>>>>>>> try {
>>>>>>>>
>>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>>
>>>>>>>> maybeReopen();
>>>>>>>>
>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>
>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '"
>>>>>>>> +
>>>>>>>> query.toString() +"'");
>>>>>>>>
>>>>>>>> Sort sort = null;
>>>>>>>>
>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>
>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>
>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>
>>>>>>>> if (filters != null) {
>>>>>>>>
>>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> multiSearcher = get();
>>>>>>>>
>>>>>>>> TopDocs topDocs = multiSearcher.search(query,chainedFilter
>>>>>>>> ,100,sort);
>>>>>>>>
>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>
>>>>>>>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>>>>>>>> "+topDocs.
>>>>>>>> totalHits);
>>>>>>>>
>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>
>>>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>>>
>>>>>>>> float score = scoreDoc.score;
>>>>>>>>
>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>>>
>>>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>>>
>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> } catch (Exception e) {
>>>>>>>>
>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>
>>>>>>>> } finally {
>>>>>>>>
>>>>>>>> if (multiSearcher != null) {
>>>>>>>>
>>>>>>>> release(multiSearcher);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> stopWatch.stop();
>>>>>>>>
>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>
>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I hope this makes sense...thanks again!
>>>>>>>>
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>>
>>>>>>>> Amin
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Mar 1, 2009 at 8:09 PM, Michael McCandless <
>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> You're calling get() too many times. For every call to get() you
>>>>>>>> must
>>>>>>>>
>>>>>>>> match with a call to release().
>>>>>>>>>
>>>>>>>>> So, once at the front of your search method you should:
>>>>>>>>>
>>>>>>>>> MultiSearcher searcher = get();
>>>>>>>>>
>>>>>>>>> then use that searcher to do searching, retrieve docs, etc.
>>>>>>>>>
>>>>>>>>> Then in the finally clause, pass that searcher to release.
>>>>>>>>>
>>>>>>>>> So, only one call to get() and one matching call to release().
>>>>>>>>>
>>>>>>>>> Mike
>>>>>>>>>
>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> The searchers are injected into the class via Spring. So when a
>>>>>>>>>
>>>>>>>>> client
>>>>>>>>>> calls the class it is fully configured with a list of index
>>>>>>>>>> searchers.
>>>>>>>>>> However I have removed this list and instead injecting a list of
>>>>>>>>>> directories which are passed to the DocumentSearchManager.
>>>>>>>>>> DocumentSearchManager is SearchManager (should've mentioned that
>>>>>>>>>> earlier).
>>>>>>>>>> So finally I have modified by release code to do the following:
>>>>>>>>>>
>>>>>>>>>> private void release(MultiSearcher multiSeacher) throws Exception
>>>>>>>>>> {
>>>>>>>>>>
>>>>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>>>>>>>> multiSeacher.getSearchables();
>>>>>>>>>>
>>>>>>>>>> for(int i =0 ; i < indexSearchers.length;i++) {
>>>>>>>>>>
>>>>>>>>>> documentSearcherManagers[i].release(indexSearchers[i]);
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> and it's use looks like this:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>>>> throwsSearchExecutionException {
>>>>>>>>>>
>>>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>>>
>>>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>>>
>>>>>>>>>> throw new SearchExecutionException("Search string cannot be empty.
>>>>>>>>>> There
>>>>>>>>>> will be too many results to process.");
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>>
>>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>>
>>>>>>>>>> stopWatch.start();
>>>>>>>>>>
>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>
>>>>>>>>>> try {
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>>>>
>>>>>>>>>> maybeReopen();
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>>>>>> searchers
>>>>>>>>>> '"
>>>>>>>>>> +
>>>>>>>>>> indexSearchers.size() +"'");
>>>>>>>>>>
>>>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query
>>>>>>>>>> '"
>>>>>>>>>> +
>>>>>>>>>> query.toString() +"'");
>>>>>>>>>>
>>>>>>>>>> Sort sort = null;
>>>>>>>>>>
>>>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>>>
>>>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>>>
>>>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>>>
>>>>>>>>>> if (filters != null) {
>>>>>>>>>>
>>>>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>>>>>
>>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("total number of hits for [" + query.toString() + " ]
>>>>>>>>>> =
>>>>>>>>>> "+topDocs.
>>>>>>>>>> totalHits);
>>>>>>>>>>
>>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>>
>>>>>>>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>>>>>>>
>>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>>
>>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>>>>>
>>>>>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>>>>>
>>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>
>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>
>>>>>>>>>> } finally {
>>>>>>>>>>
>>>>>>>>>> release(get());
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> stopWatch.stop();
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>>
>>>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> So the final post construct constructs the DocumentSearchMangers
>>>>>>>>>> with
>>>>>>>>>> the
>>>>>>>>>> list of directories..looking like this
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> @PostConstruct
>>>>>>>>>>
>>>>>>>>>> public void initialiseDocumentSearcher() {
>>>>>>>>>>
>>>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>>>> analyzer);
>>>>>>>>>>
>>>>>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>>>> newKeywordAnalyzer());
>>>>>>>>>>
>>>>>>>>>> queryParser =
>>>>>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>>> analyzerWrapper);
>>>>>>>>>>
>>>>>>>>>> try {
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>>>
>>>>>>>>>> documentSearcherManagers = new
>>>>>>>>>> DocumentSearcherManager[directories.size()];
>>>>>>>>>>
>>>>>>>>>> for (int i = 0; i < directories.size() ;i++) {
>>>>>>>>>>
>>>>>>>>>> Directory directory = directories.get(i);
>>>>>>>>>>
>>>>>>>>>> DocumentSearcherManager documentSearcherManager =
>>>>>>>>>> newDocumentSearcherManager(directory);
>>>>>>>>>>
>>>>>>>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>>>
>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>
>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>>
>>>>>>>>>> Amin
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Mar 1, 2009 at 6:15 PM, Michael McCandless <
>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I don't understand where searchers comes from, prior to
>>>>>>>>>>
>>>>>>>>>> initializeDocumentSearcher? You should, instead, simply create
>>>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>>> SearcherManager (from your Directory instances). You don't need
>>>>>>>>>>> any
>>>>>>>>>>> searchers during initialize.
>>>>>>>>>>>
>>>>>>>>>>> Is DocumentSearcherManager the same as SearcherManager (just
>>>>>>>>>>> renamed)?
>>>>>>>>>>>
>>>>>>>>>>> The release method is wrong -- you're calling .get() and then
>>>>>>>>>>> immediately release. Instead, you should step through the
>>>>>>>>>>> searchers
>>>>>>>>>>> from your MultiSearcher and release them to each SearcherManager.
>>>>>>>>>>>
>>>>>>>>>>> You should call your release() in a finally clause.
>>>>>>>>>>>
>>>>>>>>>>> Mike
>>>>>>>>>>>
>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>
>>>>>>>>>>> Sorry...i'm getting slightly confused.
>>>>>>>>>>>
>>>>>>>>>>> I have a PostConstruct which is where I should create an array of
>>>>>>>>>>>
>>>>>>>>>>> SearchManagers (per indexSeacher). From there I initialise the
>>>>>>>>>>>
>>>>>>>>>>>> multisearcher using the get(). After which I need to call
>>>>>>>>>>>> maybeReopen
>>>>>>>>>>>> for
>>>>>>>>>>>> each IndexSearcher. So I'll do the following:
>>>>>>>>>>>>
>>>>>>>>>>>> @PostConstruct
>>>>>>>>>>>>
>>>>>>>>>>>> public void initialiseDocumentSearcher() {
>>>>>>>>>>>>
>>>>>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>>>>>> analyzer);
>>>>>>>>>>>>
>>>>>>>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>>>>>> newKeywordAnalyzer());
>>>>>>>>>>>>
>>>>>>>>>>>> queryParser =
>>>>>>>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>>>>> analyzerWrapper);
>>>>>>>>>>>>
>>>>>>>>>>>> try {
>>>>>>>>>>>>
>>>>>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>>>>>
>>>>>>>>>>>> documentSearcherManagers = new
>>>>>>>>>>>> DocumentSearcherManager[searchers.size()];
>>>>>>>>>>>>
>>>>>>>>>>>> for (int i = 0; i < searchers.size() ;i++) {
>>>>>>>>>>>>
>>>>>>>>>>>> IndexSearcher indexSearcher = searchers.get(i);
>>>>>>>>>>>>
>>>>>>>>>>>> Directory directory =
>>>>>>>>>>>> indexSearcher.getIndexReader().directory();
>>>>>>>>>>>>
>>>>>>>>>>>> DocumentSearcherManager documentSearcherManager =
>>>>>>>>>>>> newDocumentSearcherManager(directory);
>>>>>>>>>>>>
>>>>>>>>>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>>>>>
>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>
>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This initialises search managers. I then have methods:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> private void maybeReopen() throws Exception {
>>>>>>>>>>>>
>>>>>>>>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>>>>>>>>
>>>>>>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>>>>>>> documentSearcherManagers) {
>>>>>>>>>>>>
>>>>>>>>>>>> documentSearcherManager.maybeReopen();
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> private void release() throws Exception {
>>>>>>>>>>>>
>>>>>>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>>>>>>> documentSearcherManagers) {
>>>>>>>>>>>>
>>>>>>>>>>>> documentSearcherManager.release(documentSearcherManager.get());
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> private MultiSearcher get() {
>>>>>>>>>>>>
>>>>>>>>>>>> List<IndexSearcher> listOfIndexSeachers = new
>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>
>>>>>>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>>>>>>> documentSearcherManagers) {
>>>>>>>>>>>>
>>>>>>>>>>>> listOfIndexSeachers.add(documentSearcherManager.get());
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> try {
>>>>>>>>>>>>
>>>>>>>>>>>> multiSearcher = new
>>>>>>>>>>>> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[]
>>>>>>>>>>>> {}));
>>>>>>>>>>>>
>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>
>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> These methods are used in the following manner in the search
>>>>>>>>>>>> code:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>>>>>> throwsSearchExecutionException {
>>>>>>>>>>>>
>>>>>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>>>>>
>>>>>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>>>>>
>>>>>>>>>>>> throw new SearchExecutionException("Search string cannot be
>>>>>>>>>>>> empty.
>>>>>>>>>>>> There
>>>>>>>>>>>> will be too many results to process.");
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>>>>
>>>>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>>>>
>>>>>>>>>>>> stopWatch.start();
>>>>>>>>>>>>
>>>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>
>>>>>>>>>>>> try {
>>>>>>>>>>>>
>>>>>>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>>>>>>
>>>>>>>>>>>> maybeReopen();
>>>>>>>>>>>>
>>>>>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>>>>>>>> searchers
>>>>>>>>>>>> '"
>>>>>>>>>>>> +
>>>>>>>>>>>> indexSearchers.size() +"'");
>>>>>>>>>>>>
>>>>>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>>>>>
>>>>>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query
>>>>>>>>>>>> '"
>>>>>>>>>>>> +
>>>>>>>>>>>> query.toString() +"'");
>>>>>>>>>>>>
>>>>>>>>>>>> Sort sort = null;
>>>>>>>>>>>>
>>>>>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>>>>>
>>>>>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>>>>>
>>>>>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>>>>>
>>>>>>>>>>>> if (filters != null) {
>>>>>>>>>>>>
>>>>>>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>>>>>>>
>>>>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>>>>
>>>>>>>>>>>> LOGGER.debug("total number of hits for [" + query.toString() + "
>>>>>>>>>>>> ]
>>>>>>>>>>>> =
>>>>>>>>>>>> "+topDocs.
>>>>>>>>>>>> totalHits);
>>>>>>>>>>>>
>>>>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>>>>
>>>>>>>>>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>>>>>>>>>
>>>>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>>>>
>>>>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>>>>>>>
>>>>>>>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>>>>>>>
>>>>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> release();
>>>>>>>>>>>>
>>>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>>>
>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> stopWatch.stop();
>>>>>>>>>>>>
>>>>>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>>>>
>>>>>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Does this look better? Again..I really really appreciate your
>>>>>>>>>>>> help!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Mar 1, 2009 at 4:18 PM, Michael McCandless <
>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This is not quite right -- you should only create
>>>>>>>>>>>> SearcherManager
>>>>>>>>>>>> once
>>>>>>>>>>>>
>>>>>>>>>>>> (per Direcotry) at startup/app load, not with every search
>>>>>>>>>>>> request.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> And I don't see release -- it must call SearcherManager.release
>>>>>>>>>>>>> of
>>>>>>>>>>>>> each of the IndexSearchers previously returned from get().
>>>>>>>>>>>>>
>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>
>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks again for helping on a Sunday!
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have now modified my maybeOpen() to do the following:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> private void maybeReopen() throws Exception {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>>>>>>>>>>>> multiSearcher
>>>>>>>>>>>>>> .getSearchables();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> SearcherManager documentSearcherManager = new
>>>>>>>>>>>>>> SearcherManager(indexReader.directory());
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> documentSearcherManager.maybeReopen();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And get() to:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>>>>>>>>>>>> multiSearcher
>>>>>>>>>>>>>> .getSearchables();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> List<IndexSearcher> indexSearchersList = new
>>>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> SearcherManager documentSearcherManager = null;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> documentSearcherManager = new
>>>>>>>>>>>>>> SearcherManager(indexReader.directory());
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> indexSearchersList.add(documentSearcherManager.get());
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> multiSearcher = new
>>>>>>>>>>>>>> MultiSearcher(indexSearchersList.toArray(newIndexSearcher[]
>>>>>>>>>>>>>> {}));
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This makes all my test pass. I am using the SearchManager
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>> recommended. Does this look ok?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Your maybeReopen has an excess incRef().
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm not sure how you open the searchers in the first place?
>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> starts as empty, and nothing populates it?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When you do the initial population, you need an incRef.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think you're hitting IllegalStateException because
>>>>>>>>>>>>>>> maybeReopen
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> closing a reader before get() can get it (since they
>>>>>>>>>>>>>>> synchronize
>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>> different objects).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'd recommend switching to the SearcherManager class.
>>>>>>>>>>>>>>> Instantiate
>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>> for each of your searchers. On each search request, go
>>>>>>>>>>>>>>> through
>>>>>>>>>>>>>>> them
>>>>>>>>>>>>>>> and call maybeReopen(), and then call get() and gather each
>>>>>>>>>>>>>>> IndexSearcher instance into a new array. Then, make a new
>>>>>>>>>>>>>>> MultiSearcher (opposite of what I said before): while that
>>>>>>>>>>>>>>> creates
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>> small amount of garbage, it'll keep your code simpler (good
>>>>>>>>>>>>>>> tradeoff).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> sorrry I added
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> release(multiSearcher);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> instead of multiSearcher.close();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've now done the following:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> throwsSearchExecutionException {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> throw new SearchExecutionException("Search string cannot be
>>>>>>>>>>>>>>>>> empty.
>>>>>>>>>>>>>>>>> There
>>>>>>>>>>>>>>>>> will be too many results to process.");
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> stopWatch.start();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> LOGGER.debug("Ensuring all index readers are up to
>>>>>>>>>>>>>>>>> date...");
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> maybeReopen();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of
>>>>>>>>>>>>>>>>> index
>>>>>>>>>>>>>>>>> searchers
>>>>>>>>>>>>>>>>> '"+ indexSearchers.size() +
>>>>>>>>>>>>>>>>> "'");
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>>>>>>>>>>>>>>>>> Query
>>>>>>>>>>>>>>>>> '"
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> query.toString() +"'");
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Sort sort = null;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> if (filters != null) {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> chainedFilter = new ChainedFilter(filters,
>>>>>>>>>>>>>>>>> ChainedFilter.OR);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter
>>>>>>>>>>>>>>>>> ,100,sort);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> LOGGER.debug("total number of hits for [" +
>>>>>>>>>>>>>>>>> query.toString()
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> "
>>>>>>>>>>>>>>>>> ]
>>>>>>>>>>>>>>>>> =
>>>>>>>>>>>>>>>>> "+topDocs.
>>>>>>>>>>>>>>>>> totalHits);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc,
>>>>>>>>>>>>>>>>> score);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Summary documentSummary = new
>>>>>>>>>>>>>>>>> DocumentSummaryImpl(baseDocument);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> multiSearcher.close();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> stopWatch.stop();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>>>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> And have the following methods:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> @PostConstruct
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> public void initialiseQueryParser() {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>>>>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>>>>>>>>>>> analyzer);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>>>>>>>>>>> newKeywordAnalyzer());
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> queryParser =
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> analyzerWrapper);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> this.multiSearcher = new
>>>>>>>>>>>>>>>>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Initialises mutltisearcher when this class is creared by
>>>>>>>>>>>>>>>>> spring.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> private synchronized void swapMultiSearcher(MultiSearcher
>>>>>>>>>>>>>>>>> newMultiSearcher) {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> release(multiSearcher);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> multiSearcher = newMultiSearcher;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> public void maybeReopen() throws IOException {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> MultiSearcher newMultiSeacher = null;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> boolean refreshMultiSeacher = false;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> synchronized (searchers) {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> IndexReader reader = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> reader.incRef();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Directory directory = reader.directory();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> long currentVersion = reader.getVersion();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> if (IndexReader.getCurrentVersion(directory) !=
>>>>>>>>>>>>>>>>> currentVersion)
>>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> IndexReader newReader =
>>>>>>>>>>>>>>>>> indexSearcher.getIndexReader().reopen();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> if (newReader != reader) {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> reader.decRef();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> refreshMultiSeacher = true;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> reader = newReader;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> indexSearchers.add(newSearcher);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> if (refreshMultiSeacher) {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> newMultiSeacher = new
>>>>>>>>>>>>>>>>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[]
>>>>>>>>>>>>>>>>> {}));
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> warm(newMultiSeacher);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> swapMultiSearcher(newMultiSeacher);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> private void warm(MultiSearcher newMultiSeacher) {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> private synchronized void release(MultiSearcher
>>>>>>>>>>>>>>>>> multiSearcher)
>>>>>>>>>>>>>>>>> throwsIOException {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> indexSearcher.getIndexReader().decRef();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> However I am now getting
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> java.lang.IllegalStateException:
>>>>>>>>>>>>>>>>> org.apache.lucene.store.AlreadyClosedException: this
>>>>>>>>>>>>>>>>> IndexReader
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> closed
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> on the call:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm doing something wrong ..obviously..not sure where
>>>>>>>>>>>>>>>>> though..
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>>>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I was wondering the same thing ;)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It's best to call this method from a single BG "warming"
>>>>>>>>>>>>>>>>> thread,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>> case it would not need its own synchronization.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> But, to be safe, I'll add internal synchronization to it.
>>>>>>>>>>>>>>>>>> You
>>>>>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>>> simply put synchronized in front of the method, since you
>>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>>>>> this to
>>>>>>>>>>>>>>>>>> block searching.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> just a quick point:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> public void maybeReopen() throws IOException {
>>>>>>>>>>>>>>>>>> //D
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> long currentVersion =
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> currentSearcher.getIndexReader().getVersion();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> if (IndexReader.getCurrentVersion(dir) != currentVersion)
>>>>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>>>> IndexReader newReader =
>>>>>>>>>>>>>>>>>>> currentSearcher.getIndexReader().reopen();
>>>>>>>>>>>>>>>>>>> assert newReader != currentSearcher.getIndexReader();
>>>>>>>>>>>>>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>>>>>>>>>>>>> warm(newSearcher);
>>>>>>>>>>>>>>>>>>> swapSearcher(newSearcher);
>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> should the above be synchronised?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> thanks. i will rewrite..in between giving my baby her
>>>>>>>>>>>>>>>>>>> feed
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> playing
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> with the other child and my wife who wants me to do
>>>>>>>>>>>>>>>>>>> several
>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> things!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>>>>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks for your input. I would like to have a go at
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> doing
>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> first, Solr may be an option.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser every
>>>>>>>>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>> creating unnecessary garbage; instead, they should be
>>>>>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> -- I can moved the code out so that it is only created
>>>>>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> reused.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>>>>>>>> when nothing has changed. This just generates
>>>>>>>>>>>>>>>>>>>>>> unnecessary
>>>>>>>>>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> -- This was something I thought about. I could move
>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>> out
>>>>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>> created once. However I presume inside my code i need
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>>>>>>> whether
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> indexreaders are update to date. This needs to be
>>>>>>>>>>>>>>>>>>>>>> synchronized
>>>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>>> well I
>>>>>>>>>>>>>>>>>>>>>> guess(?)
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Yes you should synchronize the check for whether the
>>>>>>>>>>>>>>>>>>>>>> IndexReader
>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> current.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * I don't see any synchronization -- it looks like two
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> search
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> requests are allowed into this method at the same time?
>>>>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -- So i need to extract the logic for reopening and
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>> synchronisation mechanism.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Yes.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Ok. So I have some work to do. I'll refactor the
>>>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> inline to your recommendations.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On a quick look, I think there are a few problems with
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> code:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * I don't see any synchronization -- it looks like two
>>>>>>>>>>>>>>>>>>>>>> search
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> requests are allowed into this method at the same
>>>>>>>>>>>>>>>>>>>>>> time?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully
>>>>>>>>>>>>>>>>>>>>>>> reopen
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> * You are over-incRef'ing (the reader.incRef inside
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> loop)
>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>> don't see a corresponding decRef.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> * You reopen and warm your searchers "live" (vs with
>>>>>>>>>>>>>>>>>>>>>>> BG
>>>>>>>>>>>>>>>>>>>>>>> thread);
>>>>>>>>>>>>>>>>>>>>>>> meaning the unlucky search request that hits a reopen
>>>>>>>>>>>>>>>>>>>>>>> pays
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> cost. This might be OK if the index is small enough
>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>> reopening & warming takes very little time. But if
>>>>>>>>>>>>>>>>>>>>>>> index
>>>>>>>>>>>>>>>>>>>>>>> gets
>>>>>>>>>>>>>>>>>>>>>>> large, making a random search pay that warming cost
>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>> nice
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> the end user. It erodes their trust in you.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>>>>>>>>> when nothing has changed. This just generates
>>>>>>>>>>>>>>>>>>>>>>> unnecessary
>>>>>>>>>>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser every
>>>>>>>>>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>> creating unnecessary garbage; instead, they should be
>>>>>>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> You should consider simply using Solr -- it handles
>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>> logic
>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> you and has been well debugged with time...
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> The reason for the indexreader.reopen is because I
>>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>> webapp
>>>>>>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> enables users to upload files and then search for the
>>>>>>>>>>>>>>>>>>>>>>> documents.
>>>>>>>>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> reopen i'm concerned that the facet hit counter won't
>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> updated.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin
>>>>>>>>>>>>>>>>>>>>>>>> Mohammed-Coleman
>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I have been able to get the code working for my
>>>>>>>>>>>>>>>>>>>>>>>>> scenario,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> however
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> question and I was wondering if I could get some
>>>>>>>>>>>>>>>>>>>>>>>> help.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>> IndexSearchers which are used in a MultiSearcher
>>>>>>>>>>>>>>>>>>>>>>>>> class.
>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> indexsearchers to get each indexreader and put them
>>>>>>>>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>> MultiIndexReader.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> IndexReader[] readers = new
>>>>>>>>>>>>>>>>>>>>>>>>> IndexReader[searchables.length];
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> IndexSearcher indexSearcher =
>>>>>>>>>>>>>>>>>>>>>>>>> (IndexSearcher)searchables[i];
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> readers[i].close();
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> readers[i] = newReader;
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>>>>>>>>>>>>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> IndexSearcher indexSearcher = new
>>>>>>>>>>>>>>>>>>>>>>>>> IndexSearcher(multiReader);
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I then use the indexseacher to do the facet stuff.
>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>> end
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>> closing the multireader. This is causing problems
>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>> another
>>>>>>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>>>> where I
>>>>>>>>>>>>>>>>>>>>>>>>> do some other search as the indexreaders are
>>>>>>>>>>>>>>>>>>>>>>>>> closed.
>>>>>>>>>>>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>> close
>>>>>>>>>>>>>>>>>>>>>>>>> the multiindexreader or should I do some additional
>>>>>>>>>>>>>>>>>>>>>>>>> checks
>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin
>>>>>>>>>>>>>>>>>>>>>>>>> Mohammed-Coleman
>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks just what I needed!
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>>>>>>>>>>>>>>>>>>>>>>>>>> marcelo.ochoa@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Amin:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin
>>>>>>>>>>>>>>>>>>>>>>>>>>> Mohammed-Coleman
>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry to re send this email but I was wondering
>>>>>>>>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> advice
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I am looking at building a faceted search using
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Lucene.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Solr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> comes with this built in, however I would like
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> try
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (something to add to my CV!). I have been
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> looking
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you can use the IndexReader and use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> TermVectors.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sure how to filter the results so that a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> particular
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> subset of results. The next option I was
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> looking
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> };un
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> int[] docFreqs =
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The only problem here is that I have to provide
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> brand
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> time a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new brand is created. Again I'm not sure how I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> filter
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> results
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It may be that I'm using the wrong api methods
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would be grateful if I could get some advice
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> P.S. I am basically trying to do something
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> displays
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Personal Contact (23) Business Contact (45) and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on..
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ______________
>>>>>>>>>>>>>>>>>>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>>>>>>>>>>>>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>
>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>
>>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>
>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>
>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Faceted Search using Lucene
Posted by Michael McCandless <lu...@mikemccandless.com>.
Or even just get() with no call to maybeReopen(). That should work
fine as well.
Mike
Amin Mohammed-Coleman wrote:
> In my test case I have a set up method that should populate the
> indexes
> before I start using the document searcher. I will start adding
> some more
> debug statements. So basically I should be able to do: get()
> followed by
> maybeReopen.
>
> I will let you know what the outcome is.
>
>
> Cheers
> Amin
>
> On Mon, Mar 2, 2009 at 1:39 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>>
>> Is it possible that when you first create the SearcherManager,
>> there is no
>> index in each Directory?
>>
>> If not... you better start adding diagnostics. EG inside your
>> get(), print
>> out the numDocs() of each IndexReader you get from the
>> SearcherManager?
>>
>> Something is wrong and it's best to explain it...
>>
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> Nope. If i remove the maybeReopen the search doesn't work. It only
>> works
>>> when i cal maybeReopen followed by get().
>>>
>>> Cheers
>>> Amin
>>>
>>> On Mon, Mar 2, 2009 at 12:56 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>>> That's not right; something must be wrong.
>>>>
>>>> get() before maybeReopen() should simply let you search based on
>>>> the
>>>> searcher before reopening.
>>>>
>>>> If you just do get() and don't call maybeReopen() does it work?
>>>>
>>>>
>>>> Mike
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> I noticed that if i do the get() before the maybeReopen then I
>>>> get no
>>>>
>>>>> results. But otherwise I can change it further.
>>>>>
>>>>> On Mon, Mar 2, 2009 at 11:46 AM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>>
>>>>> There is no such thing as final code -- code is alive and is
>>>>> always
>>>>>> changing ;)
>>>>>>
>>>>>> It looks good to me.
>>>>>>
>>>>>> Though one trivial thing is: I would move the code in the try
>>>>>> clause up
>>>>>> to
>>>>>> and including the multiSearcher=get() out above the try. I
>>>>>> always
>>>>>> attempt
>>>>>> to "shrink wrap" what's inside a try clause to the minimum that
>>>>>> needs
>>>>>> to
>>>>>> be
>>>>>> there. Ie, your code that creates a query, finds the right
>>>>>> sort &
>>>>>> filter
>>>>>> to
>>>>>> use, etc, can all happen outside the try, because you have not
>>>>>> yet
>>>>>> acquired
>>>>>> the multiSearcher.
>>>>>>
>>>>>> If you do that, you also don't need the null check in the finally
>>>>>> clause,
>>>>>> because multiSearcher must be non-null on entering the try.
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>> Hi there
>>>>>>
>>>>>> Good morning! Here is the final search code:
>>>>>>>
>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>> throwsSearchExecutionException {
>>>>>>>
>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>
>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>
>>>>>>> throw new SearchExecutionException("Search string cannot be
>>>>>>> empty.
>>>>>>> There
>>>>>>> will be too many results to process.");
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>
>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>
>>>>>>> stopWatch.start();
>>>>>>>
>>>>>>> MultiSearcher multiSearcher = null;
>>>>>>>
>>>>>>> try {
>>>>>>>
>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>
>>>>>>> maybeReopen();
>>>>>>>
>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>
>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>>>>>>> Query '" +
>>>>>>> query.toString() +"'");
>>>>>>>
>>>>>>> Sort sort = null;
>>>>>>>
>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>
>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>
>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>
>>>>>>> if (filters != null) {
>>>>>>>
>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> multiSearcher = get();
>>>>>>>
>>>>>>> TopDocs topDocs = multiSearcher.search(query,chainedFilter ,
>>>>>>> 100,sort);
>>>>>>>
>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>
>>>>>>> LOGGER.debug("total number of hits for [" + query.toString() +
>>>>>>> " ] =
>>>>>>> "+topDocs.
>>>>>>> totalHits);
>>>>>>>
>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>
>>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>>
>>>>>>> float score = scoreDoc.score;
>>>>>>>
>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>>
>>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>>
>>>>>>> summaryList.add(documentSummary);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> } catch (Exception e) {
>>>>>>>
>>>>>>> throw new IllegalStateException(e);
>>>>>>>
>>>>>>> } finally {
>>>>>>>
>>>>>>> if (multiSearcher != null) {
>>>>>>>
>>>>>>> release(multiSearcher);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> stopWatch.stop();
>>>>>>>
>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>
>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I hope this makes sense...thanks again!
>>>>>>>
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>> Amin
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Mar 1, 2009 at 8:09 PM, Michael McCandless <
>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> You're calling get() too many times. For every call to get()
>>>>>>> you must
>>>>>>>
>>>>>>>> match with a call to release().
>>>>>>>>
>>>>>>>> So, once at the front of your search method you should:
>>>>>>>>
>>>>>>>> MultiSearcher searcher = get();
>>>>>>>>
>>>>>>>> then use that searcher to do searching, retrieve docs, etc.
>>>>>>>>
>>>>>>>> Then in the finally clause, pass that searcher to release.
>>>>>>>>
>>>>>>>> So, only one call to get() and one matching call to release().
>>>>>>>>
>>>>>>>> Mike
>>>>>>>>
>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> The searchers are injected into the class via Spring. So
>>>>>>>> when a
>>>>>>>>
>>>>>>>>> client
>>>>>>>>> calls the class it is fully configured with a list of index
>>>>>>>>> searchers.
>>>>>>>>> However I have removed this list and instead injecting a
>>>>>>>>> list of
>>>>>>>>> directories which are passed to the DocumentSearchManager.
>>>>>>>>> DocumentSearchManager is SearchManager (should've mentioned
>>>>>>>>> that
>>>>>>>>> earlier).
>>>>>>>>> So finally I have modified by release code to do the
>>>>>>>>> following:
>>>>>>>>>
>>>>>>>>> private void release(MultiSearcher multiSeacher) throws
>>>>>>>>> Exception {
>>>>>>>>>
>>>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>>>>>>> multiSeacher.getSearchables();
>>>>>>>>>
>>>>>>>>> for(int i =0 ; i < indexSearchers.length;i++) {
>>>>>>>>>
>>>>>>>>> documentSearcherManagers[i].release(indexSearchers[i]);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> and it's use looks like this:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>>> throwsSearchExecutionException {
>>>>>>>>>
>>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>>
>>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>>
>>>>>>>>> throw new SearchExecutionException("Search string cannot be
>>>>>>>>> empty.
>>>>>>>>> There
>>>>>>>>> will be too many results to process.");
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>
>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>
>>>>>>>>> stopWatch.start();
>>>>>>>>>
>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>
>>>>>>>>> try {
>>>>>>>>>
>>>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>>>
>>>>>>>>> maybeReopen();
>>>>>>>>>
>>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>>>>> searchers
>>>>>>>>> '"
>>>>>>>>> +
>>>>>>>>> indexSearchers.size() +"'");
>>>>>>>>>
>>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>>
>>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>>>>>>>>> Query '"
>>>>>>>>> +
>>>>>>>>> query.toString() +"'");
>>>>>>>>>
>>>>>>>>> Sort sort = null;
>>>>>>>>>
>>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>>
>>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>>
>>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>>
>>>>>>>>> if (filters != null) {
>>>>>>>>>
>>>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>>>>
>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>
>>>>>>>>> LOGGER.debug("total number of hits for [" + query.toString()
>>>>>>>>> + " ] =
>>>>>>>>> "+topDocs.
>>>>>>>>> totalHits);
>>>>>>>>>
>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>
>>>>>>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>>>>>>
>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>
>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc,
>>>>>>>>> score);
>>>>>>>>>
>>>>>>>>> Summary documentSummary = new
>>>>>>>>> DocumentSummaryImpl(baseDocument);
>>>>>>>>>
>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> } catch (Exception e) {
>>>>>>>>>
>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>
>>>>>>>>> } finally {
>>>>>>>>>
>>>>>>>>> release(get());
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> stopWatch.stop();
>>>>>>>>>
>>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>
>>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> So the final post construct constructs the
>>>>>>>>> DocumentSearchMangers
>>>>>>>>> with
>>>>>>>>> the
>>>>>>>>> list of directories..looking like this
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> @PostConstruct
>>>>>>>>>
>>>>>>>>> public void initialiseDocumentSearcher() {
>>>>>>>>>
>>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>>> analyzer);
>>>>>>>>>
>>>>>>>>> analyzerWrapper
>>>>>>>>> .addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>>> newKeywordAnalyzer());
>>>>>>>>>
>>>>>>>>> queryParser =
>>>>>>>>> newMultiFieldQueryParser
>>>>>>>>> (FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>> analyzerWrapper);
>>>>>>>>>
>>>>>>>>> try {
>>>>>>>>>
>>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>>
>>>>>>>>> documentSearcherManagers = new
>>>>>>>>> DocumentSearcherManager[directories.size()];
>>>>>>>>>
>>>>>>>>> for (int i = 0; i < directories.size() ;i++) {
>>>>>>>>>
>>>>>>>>> Directory directory = directories.get(i);
>>>>>>>>>
>>>>>>>>> DocumentSearcherManager documentSearcherManager =
>>>>>>>>> newDocumentSearcherManager(directory);
>>>>>>>>>
>>>>>>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>>
>>>>>>>>> } catch (IOException e) {
>>>>>>>>>
>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>>
>>>>>>>>> Amin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Mar 1, 2009 at 6:15 PM, Michael McCandless <
>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I don't understand where searchers comes from, prior to
>>>>>>>>>
>>>>>>>>> initializeDocumentSearcher? You should, instead, simply
>>>>>>>>> create the
>>>>>>>>>> SearcherManager (from your Directory instances). You don't
>>>>>>>>>> need
>>>>>>>>>> any
>>>>>>>>>> searchers during initialize.
>>>>>>>>>>
>>>>>>>>>> Is DocumentSearcherManager the same as SearcherManager (just
>>>>>>>>>> renamed)?
>>>>>>>>>>
>>>>>>>>>> The release method is wrong -- you're calling .get() and then
>>>>>>>>>> immediately release. Instead, you should step through the
>>>>>>>>>> searchers
>>>>>>>>>> from your MultiSearcher and release them to each
>>>>>>>>>> SearcherManager.
>>>>>>>>>>
>>>>>>>>>> You should call your release() in a finally clause.
>>>>>>>>>>
>>>>>>>>>> Mike
>>>>>>>>>>
>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>
>>>>>>>>>> Sorry...i'm getting slightly confused.
>>>>>>>>>>
>>>>>>>>>> I have a PostConstruct which is where I should create an
>>>>>>>>>> array of
>>>>>>>>>>
>>>>>>>>>> SearchManagers (per indexSeacher). From there I initialise
>>>>>>>>>> the
>>>>>>>>>>> multisearcher using the get(). After which I need to call
>>>>>>>>>>> maybeReopen
>>>>>>>>>>> for
>>>>>>>>>>> each IndexSearcher. So I'll do the following:
>>>>>>>>>>>
>>>>>>>>>>> @PostConstruct
>>>>>>>>>>>
>>>>>>>>>>> public void initialiseDocumentSearcher() {
>>>>>>>>>>>
>>>>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>>>>> analyzer);
>>>>>>>>>>>
>>>>>>>>>>> analyzerWrapper
>>>>>>>>>>> .addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>>>>> newKeywordAnalyzer());
>>>>>>>>>>>
>>>>>>>>>>> queryParser =
>>>>>>>>>>> newMultiFieldQueryParser
>>>>>>>>>>> (FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>>>> analyzerWrapper);
>>>>>>>>>>>
>>>>>>>>>>> try {
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>>>>
>>>>>>>>>>> documentSearcherManagers = new
>>>>>>>>>>> DocumentSearcherManager[searchers.size()];
>>>>>>>>>>>
>>>>>>>>>>> for (int i = 0; i < searchers.size() ;i++) {
>>>>>>>>>>>
>>>>>>>>>>> IndexSearcher indexSearcher = searchers.get(i);
>>>>>>>>>>>
>>>>>>>>>>> Directory directory =
>>>>>>>>>>> indexSearcher.getIndexReader().directory();
>>>>>>>>>>>
>>>>>>>>>>> DocumentSearcherManager documentSearcherManager =
>>>>>>>>>>> newDocumentSearcherManager(directory);
>>>>>>>>>>>
>>>>>>>>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>>>>
>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>
>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This initialises search managers. I then have methods:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> private void maybeReopen() throws Exception {
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>>>>>>>
>>>>>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>>>>>> documentSearcherManagers) {
>>>>>>>>>>>
>>>>>>>>>>> documentSearcherManager.maybeReopen();
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> private void release() throws Exception {
>>>>>>>>>>>
>>>>>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>>>>>> documentSearcherManagers) {
>>>>>>>>>>>
>>>>>>>>>>> documentSearcherManager
>>>>>>>>>>> .release(documentSearcherManager.get());
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> private MultiSearcher get() {
>>>>>>>>>>>
>>>>>>>>>>> List<IndexSearcher> listOfIndexSeachers = new
>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>
>>>>>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>>>>>> documentSearcherManagers) {
>>>>>>>>>>>
>>>>>>>>>>> listOfIndexSeachers.add(documentSearcherManager.get());
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> try {
>>>>>>>>>>>
>>>>>>>>>>> multiSearcher = new
>>>>>>>>>>> MultiSearcher
>>>>>>>>>>> (listOfIndexSeachers.toArray(newIndexSearcher[] {}));
>>>>>>>>>>>
>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>
>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> These methods are used in the following manner in the
>>>>>>>>>>> search code:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>>>>> throwsSearchExecutionException {
>>>>>>>>>>>
>>>>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>>>>
>>>>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>>>>
>>>>>>>>>>> throw new SearchExecutionException("Search string cannot
>>>>>>>>>>> be empty.
>>>>>>>>>>> There
>>>>>>>>>>> will be too many results to process.");
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>>>
>>>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>>>
>>>>>>>>>>> stopWatch.start();
>>>>>>>>>>>
>>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>
>>>>>>>>>>> try {
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("Ensuring all index readers are up to
>>>>>>>>>>> date...");
>>>>>>>>>>>
>>>>>>>>>>> maybeReopen();
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of
>>>>>>>>>>> index
>>>>>>>>>>> searchers
>>>>>>>>>>> '"
>>>>>>>>>>> +
>>>>>>>>>>> indexSearchers.size() +"'");
>>>>>>>>>>>
>>>>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>>>>>>>>>>> Query
>>>>>>>>>>> '"
>>>>>>>>>>> +
>>>>>>>>>>> query.toString() +"'");
>>>>>>>>>>>
>>>>>>>>>>> Sort sort = null;
>>>>>>>>>>>
>>>>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>>>>
>>>>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>>>>
>>>>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>>>>
>>>>>>>>>>> if (filters != null) {
>>>>>>>>>>>
>>>>>>>>>>> chainedFilter = new ChainedFilter(filters,
>>>>>>>>>>> ChainedFilter.OR);
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,
>>>>>>>>>>> 100,sort);
>>>>>>>>>>>
>>>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("total number of hits for [" +
>>>>>>>>>>> query.toString() + " ]
>>>>>>>>>>> =
>>>>>>>>>>> "+topDocs.
>>>>>>>>>>> totalHits);
>>>>>>>>>>>
>>>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>>>
>>>>>>>>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>>>>>>>>
>>>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>>>
>>>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc,
>>>>>>>>>>> score);
>>>>>>>>>>>
>>>>>>>>>>> Summary documentSummary = new
>>>>>>>>>>> DocumentSummaryImpl(baseDocument);
>>>>>>>>>>>
>>>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> release();
>>>>>>>>>>>
>>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>>
>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> stopWatch.stop();
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>>>
>>>>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Does this look better? Again..I really really appreciate
>>>>>>>>>>> your
>>>>>>>>>>> help!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Mar 1, 2009 at 4:18 PM, Michael McCandless <
>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This is not quite right -- you should only create
>>>>>>>>>>> SearcherManager
>>>>>>>>>>> once
>>>>>>>>>>>
>>>>>>>>>>> (per Direcotry) at startup/app load, not with every search
>>>>>>>>>>> request.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> And I don't see release -- it must call
>>>>>>>>>>>> SearcherManager.release
>>>>>>>>>>>> of
>>>>>>>>>>>> each of the IndexSearchers previously returned from get().
>>>>>>>>>>>>
>>>>>>>>>>>> Mike
>>>>>>>>>>>>
>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks again for helping on a Sunday!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I have now modified my maybeOpen() to do the following:
>>>>>>>>>>>>>
>>>>>>>>>>>>> private void maybeReopen() throws Exception {
>>>>>>>>>>>>>
>>>>>>>>>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>>>>>>>>>
>>>>>>>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>>>>>>>>>>> multiSearcher
>>>>>>>>>>>>> .getSearchables();
>>>>>>>>>>>>>
>>>>>>>>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>>>>>>>>
>>>>>>>>>>>>> SearcherManager documentSearcherManager = new
>>>>>>>>>>>>> SearcherManager(indexReader.directory());
>>>>>>>>>>>>>
>>>>>>>>>>>>> documentSearcherManager.maybeReopen();
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> And get() to:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>>>
>>>>>>>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>>>>>>>>>>> multiSearcher
>>>>>>>>>>>>> .getSearchables();
>>>>>>>>>>>>>
>>>>>>>>>>>>> List<IndexSearcher> indexSearchersList = new
>>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>>
>>>>>>>>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>>>>>>>>
>>>>>>>>>>>>> SearcherManager documentSearcherManager = null;
>>>>>>>>>>>>>
>>>>>>>>>>>>> try {
>>>>>>>>>>>>>
>>>>>>>>>>>>> documentSearcherManager = new
>>>>>>>>>>>>> SearcherManager(indexReader.directory());
>>>>>>>>>>>>>
>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> indexSearchersList.add(documentSearcherManager.get());
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> try {
>>>>>>>>>>>>>
>>>>>>>>>>>>> multiSearcher = new
>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>> (indexSearchersList.toArray(newIndexSearcher[]
>>>>>>>>>>>>> {}));
>>>>>>>>>>>>>
>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This makes all my test pass. I am using the
>>>>>>>>>>>>> SearchManager that
>>>>>>>>>>>>> you
>>>>>>>>>>>>> recommended. Does this look ok?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Your maybeReopen has an excess incRef().
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm not sure how you open the searchers in the first
>>>>>>>>>>>>> place? The
>>>>>>>>>>>>>
>>>>>>>>>>>>> list
>>>>>>>>>>>>>> starts as empty, and nothing populates it?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When you do the initial population, you need an incRef.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think you're hitting IllegalStateException because
>>>>>>>>>>>>>> maybeReopen
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> closing a reader before get() can get it (since they
>>>>>>>>>>>>>> synchronize
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>> different objects).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'd recommend switching to the SearcherManager class.
>>>>>>>>>>>>>> Instantiate
>>>>>>>>>>>>>> one
>>>>>>>>>>>>>> for each of your searchers. On each search request, go
>>>>>>>>>>>>>> through
>>>>>>>>>>>>>> them
>>>>>>>>>>>>>> and call maybeReopen(), and then call get() and gather
>>>>>>>>>>>>>> each
>>>>>>>>>>>>>> IndexSearcher instance into a new array. Then, make a
>>>>>>>>>>>>>> new
>>>>>>>>>>>>>> MultiSearcher (opposite of what I said before): while
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>> creates
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>> small amount of garbage, it'll keep your code simpler
>>>>>>>>>>>>>> (good
>>>>>>>>>>>>>> tradeoff).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> sorrry I added
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> release(multiSearcher);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> instead of multiSearcher.close();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I've now done the following:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> public Summary[] search(final SearchRequest
>>>>>>>>>>>>>>> searchRequest)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> throwsSearchExecutionException {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> final String searchTerm =
>>>>>>>>>>>>>>>> searchRequest.getSearchTerm();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> throw new SearchExecutionException("Search string
>>>>>>>>>>>>>>>> cannot be
>>>>>>>>>>>>>>>> empty.
>>>>>>>>>>>>>>>> There
>>>>>>>>>>>>>>>> will be too many results to process.");
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> stopWatch.start();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> LOGGER.debug("Ensuring all index readers are up to
>>>>>>>>>>>>>>>> date...");
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> maybeReopen();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No
>>>>>>>>>>>>>>>> of index
>>>>>>>>>>>>>>>> searchers
>>>>>>>>>>>>>>>> '"+ indexSearchers.size() +
>>>>>>>>>>>>>>>> "'");
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ---->
>>>>>>>>>>>>>>>> Lucene
>>>>>>>>>>>>>>>> Query
>>>>>>>>>>>>>>>> '"
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>> query.toString() +"'");
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sort sort = null;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Filter[] filters
>>>>>>>>>>>>>>>> =applyFiltersIfApplicable(searchRequest);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> if (filters != null) {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> chainedFilter = new ChainedFilter(filters,
>>>>>>>>>>>>>>>> ChainedFilter.OR);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter
>>>>>>>>>>>>>>>> ,100,sort);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> LOGGER.debug("total number of hits for [" +
>>>>>>>>>>>>>>>> query.toString()
>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>> "
>>>>>>>>>>>>>>>> ]
>>>>>>>>>>>>>>>> =
>>>>>>>>>>>>>>>> "+topDocs.
>>>>>>>>>>>>>>>> totalHits);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc,
>>>>>>>>>>>>>>>> score);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Summary documentSummary = new
>>>>>>>>>>>>>>>> DocumentSummaryImpl(baseDocument);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> multiSearcher.close();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> stopWatch.stop();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> And have the following methods:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> @PostConstruct
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> public void initialiseQueryParser() {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>>>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>>>>>>>>>> analyzer);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> analyzerWrapper
>>>>>>>>>>>>>>>> .addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>>>>>>>>>> newKeywordAnalyzer());
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> queryParser =
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> newMultiFieldQueryParser
>>>>>>>>>>>>>>>> (FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> analyzerWrapper);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> this.multiSearcher = new
>>>>>>>>>>>>>>>> MultiSearcher(searchers.toArray(newIndexSearcher[]
>>>>>>>>>>>>>>>> {}));
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Initialises mutltisearcher when this class is creared
>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>> spring.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> private synchronized void
>>>>>>>>>>>>>>>> swapMultiSearcher(MultiSearcher
>>>>>>>>>>>>>>>> newMultiSearcher) {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> release(multiSearcher);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> multiSearcher = newMultiSearcher;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> public void maybeReopen() throws IOException {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> MultiSearcher newMultiSeacher = null;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> boolean refreshMultiSeacher = false;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> synchronized (searchers) {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> IndexReader reader = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> reader.incRef();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Directory directory = reader.directory();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> long currentVersion = reader.getVersion();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> if (IndexReader.getCurrentVersion(directory) !=
>>>>>>>>>>>>>>>> currentVersion)
>>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> IndexReader newReader =
>>>>>>>>>>>>>>>> indexSearcher.getIndexReader().reopen();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> if (newReader != reader) {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> reader.decRef();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> refreshMultiSeacher = true;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> reader = newReader;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> IndexSearcher newSearcher = new
>>>>>>>>>>>>>>>> IndexSearcher(newReader);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> indexSearchers.add(newSearcher);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> if (refreshMultiSeacher) {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> newMultiSeacher = new
>>>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>>>> (indexSearchers.toArray(newIndexSearcher[] {}));
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> warm(newMultiSeacher);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> swapMultiSearcher(newMultiSeacher);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> private void warm(MultiSearcher newMultiSeacher) {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> private synchronized void release(MultiSearcher
>>>>>>>>>>>>>>>> multiSearcher)
>>>>>>>>>>>>>>>> throwsIOException {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> indexSearcher.getIndexReader().decRef();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> However I am now getting
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> java.lang.IllegalStateException:
>>>>>>>>>>>>>>>> org.apache.lucene.store.AlreadyClosedException: this
>>>>>>>>>>>>>>>> IndexReader
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> closed
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> on the call:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm doing something wrong ..obviously..not sure where
>>>>>>>>>>>>>>>> though..
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I was wondering the same thing ;)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It's best to call this method from a single BG
>>>>>>>>>>>>>>>> "warming"
>>>>>>>>>>>>>>>> thread,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>> case it would not need its own synchronization.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> But, to be safe, I'll add internal synchronization
>>>>>>>>>>>>>>>>> to it.
>>>>>>>>>>>>>>>>> You
>>>>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>> simply put synchronized in front of the method,
>>>>>>>>>>>>>>>>> since you
>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>>>> this to
>>>>>>>>>>>>>>>>> block searching.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> just a quick point:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> public void maybeReopen() throws IOException {
>>>>>>>>>>>>>>>>> //D
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> long currentVersion =
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> currentSearcher.getIndexReader().getVersion();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> if (IndexReader.getCurrentVersion(dir) !=
>>>>>>>>>>>>>>>>>> currentVersion) {
>>>>>>>>>>>>>>>>>> IndexReader newReader =
>>>>>>>>>>>>>>>>>> currentSearcher.getIndexReader().reopen();
>>>>>>>>>>>>>>>>>> assert newReader != currentSearcher.getIndexReader();
>>>>>>>>>>>>>>>>>> IndexSearcher newSearcher = new
>>>>>>>>>>>>>>>>>> IndexSearcher(newReader);
>>>>>>>>>>>>>>>>>> warm(newSearcher);
>>>>>>>>>>>>>>>>>> swapSearcher(newSearcher);
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> should the above be synchronised?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-
>>>>>>>>>>>>>>>>>> Coleman <
>>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> thanks. i will rewrite..in between giving my baby
>>>>>>>>>>>>>>>>>> her feed
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> playing
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> with the other child and my wife who wants me to do
>>>>>>>>>>>>>>>>>> several
>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> things!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>>>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for your input. I would like to have a go at
>>>>>>>>>>>>>>>>>>>> doing
>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> first, Solr may be an option.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser
>>>>>>>>>>>>>>>>>>>>> every
>>>>>>>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>> creating unnecessary garbage; instead, they
>>>>>>>>>>>>>>>>>>>>> should be
>>>>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -- I can moved the code out so that it is only
>>>>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> reused.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>>>>>>> when nothing has changed. This just generates
>>>>>>>>>>>>>>>>>>>>> unnecessary
>>>>>>>>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -- This was something I thought about. I could
>>>>>>>>>>>>>>>>>>>>> move it
>>>>>>>>>>>>>>>>>>>>> out
>>>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>> created once. However I presume inside my code
>>>>>>>>>>>>>>>>>>>>> i need
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>>>>>> whether
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> indexreaders are update to date. This needs to be
>>>>>>>>>>>>>>>>>>>>> synchronized
>>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>> well I
>>>>>>>>>>>>>>>>>>>>> guess(?)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Yes you should synchronize the check for whether
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> IndexReader
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> current.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * I don't see any synchronization -- it looks
>>>>>>>>>>>>>>>>>>>>> like two
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> search
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> requests are allowed into this method at the same
>>>>>>>>>>>>>>>>>>>> time?
>>>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully
>>>>>>>>>>>>>>>>>>>> reopen
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -- So i need to extract the logic for reopening
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>> synchronisation mechanism.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Yes.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Ok. So I have some work to do. I'll refactor
>>>>>>>>>>>>>>>>>>>>> the code
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> inline to your recommendations.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael
>>>>>>>>>>>>>>>>>>>> McCandless <
>>>>>>>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On a quick look, I think there are a few
>>>>>>>>>>>>>>>>>>>>> problems with
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> code:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * I don't see any synchronization -- it looks
>>>>>>>>>>>>>>>>>>>>> like two
>>>>>>>>>>>>>>>>>>>>> search
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> requests are allowed into this method at the
>>>>>>>>>>>>>>>>>>>>> same time?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully
>>>>>>>>>>>>>>>>>>>>>> reopen
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * You are over-incRef'ing (the reader.incRef
>>>>>>>>>>>>>>>>>>>>>> inside the
>>>>>>>>>>>>>>>>>>>>>> loop)
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>> don't see a corresponding decRef.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * You reopen and warm your searchers "live" (vs
>>>>>>>>>>>>>>>>>>>>>> with BG
>>>>>>>>>>>>>>>>>>>>>> thread);
>>>>>>>>>>>>>>>>>>>>>> meaning the unlucky search request that hits a
>>>>>>>>>>>>>>>>>>>>>> reopen
>>>>>>>>>>>>>>>>>>>>>> pays
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> cost. This might be OK if the index is small
>>>>>>>>>>>>>>>>>>>>>> enough
>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>> reopening & warming takes very little time.
>>>>>>>>>>>>>>>>>>>>>> But if
>>>>>>>>>>>>>>>>>>>>>> index
>>>>>>>>>>>>>>>>>>>>>> gets
>>>>>>>>>>>>>>>>>>>>>> large, making a random search pay that warming
>>>>>>>>>>>>>>>>>>>>>> cost is
>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>> nice
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> the end user. It erodes their trust in you.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>>>>>>>> when nothing has changed. This just generates
>>>>>>>>>>>>>>>>>>>>>> unnecessary
>>>>>>>>>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser
>>>>>>>>>>>>>>>>>>>>>> every
>>>>>>>>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>> creating unnecessary garbage; instead, they
>>>>>>>>>>>>>>>>>>>>>> should be
>>>>>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> You should consider simply using Solr -- it
>>>>>>>>>>>>>>>>>>>>>> handles all
>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>> logic
>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>> you and has been well debugged with time...
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> The reason for the indexreader.reopen is
>>>>>>>>>>>>>>>>>>>>>> because I have
>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>> webapp
>>>>>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> enables users to upload files and then search
>>>>>>>>>>>>>>>>>>>>>> for the
>>>>>>>>>>>>>>>>>>>>>> documents.
>>>>>>>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> reopen i'm concerned that the facet hit counter
>>>>>>>>>>>>>>>>>>>>>> won't
>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> updated.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-
>>>>>>>>>>>>>>>>>>>>>>> Coleman
>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I have been able to get the code working for my
>>>>>>>>>>>>>>>>>>>>>>>> scenario,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> however
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> question and I was wondering if I could get
>>>>>>>>>>>>>>>>>>>>>>> some help.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>> IndexSearchers which are used in a
>>>>>>>>>>>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>>>>>>>>>>>> class.
>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> indexsearchers to get each indexreader and
>>>>>>>>>>>>>>>>>>>>>>>> put them
>>>>>>>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>> MultiIndexReader.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> IndexReader[] readers = new
>>>>>>>>>>>>>>>>>>>>>>>> IndexReader[searchables.length];
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> IndexSearcher indexSearcher =
>>>>>>>>>>>>>>>>>>>>>>>> (IndexSearcher)searchables[i];
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> readers[i].close();
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> readers[i] = newReader;
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>>>>>>>>>>>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> IndexSearcher indexSearcher = new
>>>>>>>>>>>>>>>>>>>>>>>> IndexSearcher(multiReader);
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I then use the indexseacher to do the facet
>>>>>>>>>>>>>>>>>>>>>>>> stuff. I
>>>>>>>>>>>>>>>>>>>>>>>> end
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>> closing the multireader. This is causing
>>>>>>>>>>>>>>>>>>>>>>>> problems in
>>>>>>>>>>>>>>>>>>>>>>>> another
>>>>>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>>> where I
>>>>>>>>>>>>>>>>>>>>>>>> do some other search as the indexreaders are
>>>>>>>>>>>>>>>>>>>>>>>> closed.
>>>>>>>>>>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>> close
>>>>>>>>>>>>>>>>>>>>>>>> the multiindexreader or should I do some
>>>>>>>>>>>>>>>>>>>>>>>> additional
>>>>>>>>>>>>>>>>>>>>>>>> checks
>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin
>>>>>>>>>>>>>>>>>>>>>>>> Mohammed-Coleman
>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks just what I needed!
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>>>>>>>>>>>>>>>>>>>>>>>>> marcelo.ochoa@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi Amin:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin
>>>>>>>>>>>>>>>>>>>>>>>>>> Mohammed-Coleman
>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry to re send this email but I was
>>>>>>>>>>>>>>>>>>>>>>>>>> wondering if
>>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> advice
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-
>>>>>>>>>>>>>>>>>>>>>>>>>>> Coleman <
>>>>>>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I am looking at building a faceted search
>>>>>>>>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>>>>>>>> Lucene.
>>>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Solr
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> comes with this built in, however I would
>>>>>>>>>>>>>>>>>>>>>>>>>>> like to
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> try
>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>>>>>>>>>>>> (something to add to my CV!). I have been
>>>>>>>>>>>>>>>>>>>>>>>>>>>> looking
>>>>>>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> you can use the IndexReader and use
>>>>>>>>>>>>>>>>>>>>>>>>>>>> TermVectors.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>> sure how to filter the results so that a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> particular
>>>>>>>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> subset of results. The next option I was
>>>>>>>>>>>>>>>>>>>>>>>>>>>> looking
>>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Term[] termsArray = new Term[] { term1,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> term2
>>>>>>>>>>>>>>>>>>>>>>>>>>>> };un
>>>>>>>>>>>>>>>>>>>>>>>>>>>> int[] docFreqs =
>>>>>>>>>>>>>>>>>>>>>>>>>>>> indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The only problem here is that I have to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> brand
>>>>>>>>>>>>>>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>>>>>>>>>>>>> time a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> new brand is created. Again I'm not sure
>>>>>>>>>>>>>>>>>>>>>>>>>>>> how I
>>>>>>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>>>> filter
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> results
>>>>>>>>>>>>>>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> It may be that I'm using the wrong api
>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would be grateful if I could get some
>>>>>>>>>>>>>>>>>>>>>>>>>>>> advice on
>>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> P.S. I am basically trying to do
>>>>>>>>>>>>>>>>>>>>>>>>>>>> something that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> displays
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Personal Contact (23) Business Contact
>>>>>>>>>>>>>>>>>>>>>>>>>>>> (45) and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>>>>>>>>>> on..
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> ______________
>>>>>>>>>>>>>>>>>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>>>>>>>>>>>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>
>>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>
>>>>>>>>> To unsubscribe, e-mail: java-user-
>>>>>>>>> unsubscribe@lucene.apache.org
>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
In my test case I have a set up method that should populate the indexes
before I start using the document searcher. I will start adding some more
debug statements. So basically I should be able to do: get() followed by
maybeReopen.
I will let you know what the outcome is.
Cheers
Amin
On Mon, Mar 2, 2009 at 1:39 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
>
> Is it possible that when you first create the SearcherManager, there is no
> index in each Directory?
>
> If not... you better start adding diagnostics. EG inside your get(), print
> out the numDocs() of each IndexReader you get from the SearcherManager?
>
> Something is wrong and it's best to explain it...
>
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> Nope. If i remove the maybeReopen the search doesn't work. It only works
>> when i cal maybeReopen followed by get().
>>
>> Cheers
>> Amin
>>
>> On Mon, Mar 2, 2009 at 12:56 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>
>>> That's not right; something must be wrong.
>>>
>>> get() before maybeReopen() should simply let you search based on the
>>> searcher before reopening.
>>>
>>> If you just do get() and don't call maybeReopen() does it work?
>>>
>>>
>>> Mike
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> I noticed that if i do the get() before the maybeReopen then I get no
>>>
>>>> results. But otherwise I can change it further.
>>>>
>>>> On Mon, Mar 2, 2009 at 11:46 AM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>>
>>>> There is no such thing as final code -- code is alive and is always
>>>>> changing ;)
>>>>>
>>>>> It looks good to me.
>>>>>
>>>>> Though one trivial thing is: I would move the code in the try clause up
>>>>> to
>>>>> and including the multiSearcher=get() out above the try. I always
>>>>> attempt
>>>>> to "shrink wrap" what's inside a try clause to the minimum that needs
>>>>> to
>>>>> be
>>>>> there. Ie, your code that creates a query, finds the right sort &
>>>>> filter
>>>>> to
>>>>> use, etc, can all happen outside the try, because you have not yet
>>>>> acquired
>>>>> the multiSearcher.
>>>>>
>>>>> If you do that, you also don't need the null check in the finally
>>>>> clause,
>>>>> because multiSearcher must be non-null on entering the try.
>>>>>
>>>>> Mike
>>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>> Hi there
>>>>>
>>>>> Good morning! Here is the final search code:
>>>>>>
>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>> throwsSearchExecutionException {
>>>>>>
>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>
>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>
>>>>>> throw new SearchExecutionException("Search string cannot be empty.
>>>>>> There
>>>>>> will be too many results to process.");
>>>>>>
>>>>>> }
>>>>>>
>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>
>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>
>>>>>> stopWatch.start();
>>>>>>
>>>>>> MultiSearcher multiSearcher = null;
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>
>>>>>> maybeReopen();
>>>>>>
>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>
>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
>>>>>> query.toString() +"'");
>>>>>>
>>>>>> Sort sort = null;
>>>>>>
>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>
>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>
>>>>>> ChainedFilter chainedFilter = null;
>>>>>>
>>>>>> if (filters != null) {
>>>>>>
>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> multiSearcher = get();
>>>>>>
>>>>>> TopDocs topDocs = multiSearcher.search(query,chainedFilter ,100,sort);
>>>>>>
>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>
>>>>>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>>>>>> "+topDocs.
>>>>>> totalHits);
>>>>>>
>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>
>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>
>>>>>> float score = scoreDoc.score;
>>>>>>
>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>
>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>
>>>>>> summaryList.add(documentSummary);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> } catch (Exception e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> } finally {
>>>>>>
>>>>>> if (multiSearcher != null) {
>>>>>>
>>>>>> release(multiSearcher);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> stopWatch.stop();
>>>>>>
>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>
>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> I hope this makes sense...thanks again!
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> Amin
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Mar 1, 2009 at 8:09 PM, Michael McCandless <
>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>
>>>>>>
>>>>>> You're calling get() too many times. For every call to get() you must
>>>>>>
>>>>>>> match with a call to release().
>>>>>>>
>>>>>>> So, once at the front of your search method you should:
>>>>>>>
>>>>>>> MultiSearcher searcher = get();
>>>>>>>
>>>>>>> then use that searcher to do searching, retrieve docs, etc.
>>>>>>>
>>>>>>> Then in the finally clause, pass that searcher to release.
>>>>>>>
>>>>>>> So, only one call to get() and one matching call to release().
>>>>>>>
>>>>>>> Mike
>>>>>>>
>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> The searchers are injected into the class via Spring. So when a
>>>>>>>
>>>>>>>> client
>>>>>>>> calls the class it is fully configured with a list of index
>>>>>>>> searchers.
>>>>>>>> However I have removed this list and instead injecting a list of
>>>>>>>> directories which are passed to the DocumentSearchManager.
>>>>>>>> DocumentSearchManager is SearchManager (should've mentioned that
>>>>>>>> earlier).
>>>>>>>> So finally I have modified by release code to do the following:
>>>>>>>>
>>>>>>>> private void release(MultiSearcher multiSeacher) throws Exception {
>>>>>>>>
>>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>>>>>> multiSeacher.getSearchables();
>>>>>>>>
>>>>>>>> for(int i =0 ; i < indexSearchers.length;i++) {
>>>>>>>>
>>>>>>>> documentSearcherManagers[i].release(indexSearchers[i]);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> and it's use looks like this:
>>>>>>>>
>>>>>>>>
>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>> throwsSearchExecutionException {
>>>>>>>>
>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>
>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>
>>>>>>>> throw new SearchExecutionException("Search string cannot be empty.
>>>>>>>> There
>>>>>>>> will be too many results to process.");
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>
>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>
>>>>>>>> stopWatch.start();
>>>>>>>>
>>>>>>>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>>>>>>>
>>>>>>>> try {
>>>>>>>>
>>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>>
>>>>>>>> maybeReopen();
>>>>>>>>
>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>>>> searchers
>>>>>>>> '"
>>>>>>>> +
>>>>>>>> indexSearchers.size() +"'");
>>>>>>>>
>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>
>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '"
>>>>>>>> +
>>>>>>>> query.toString() +"'");
>>>>>>>>
>>>>>>>> Sort sort = null;
>>>>>>>>
>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>
>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>
>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>
>>>>>>>> if (filters != null) {
>>>>>>>>
>>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>>>
>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>
>>>>>>>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>>>>>>>> "+topDocs.
>>>>>>>> totalHits);
>>>>>>>>
>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>
>>>>>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>>>>>
>>>>>>>> float score = scoreDoc.score;
>>>>>>>>
>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>>>
>>>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>>>
>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> } catch (Exception e) {
>>>>>>>>
>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>
>>>>>>>> } finally {
>>>>>>>>
>>>>>>>> release(get());
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> stopWatch.stop();
>>>>>>>>
>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>
>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> So the final post construct constructs the DocumentSearchMangers
>>>>>>>> with
>>>>>>>> the
>>>>>>>> list of directories..looking like this
>>>>>>>>
>>>>>>>>
>>>>>>>> @PostConstruct
>>>>>>>>
>>>>>>>> public void initialiseDocumentSearcher() {
>>>>>>>>
>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>> analyzer);
>>>>>>>>
>>>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>> newKeywordAnalyzer());
>>>>>>>>
>>>>>>>> queryParser =
>>>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>>> analyzerWrapper);
>>>>>>>>
>>>>>>>> try {
>>>>>>>>
>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>
>>>>>>>> documentSearcherManagers = new
>>>>>>>> DocumentSearcherManager[directories.size()];
>>>>>>>>
>>>>>>>> for (int i = 0; i < directories.size() ;i++) {
>>>>>>>>
>>>>>>>> Directory directory = directories.get(i);
>>>>>>>>
>>>>>>>> DocumentSearcherManager documentSearcherManager =
>>>>>>>> newDocumentSearcherManager(directory);
>>>>>>>>
>>>>>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>
>>>>>>>> } catch (IOException e) {
>>>>>>>>
>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>>
>>>>>>>> Amin
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Mar 1, 2009 at 6:15 PM, Michael McCandless <
>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> I don't understand where searchers comes from, prior to
>>>>>>>>
>>>>>>>> initializeDocumentSearcher? You should, instead, simply create the
>>>>>>>>> SearcherManager (from your Directory instances). You don't need
>>>>>>>>> any
>>>>>>>>> searchers during initialize.
>>>>>>>>>
>>>>>>>>> Is DocumentSearcherManager the same as SearcherManager (just
>>>>>>>>> renamed)?
>>>>>>>>>
>>>>>>>>> The release method is wrong -- you're calling .get() and then
>>>>>>>>> immediately release. Instead, you should step through the
>>>>>>>>> searchers
>>>>>>>>> from your MultiSearcher and release them to each SearcherManager.
>>>>>>>>>
>>>>>>>>> You should call your release() in a finally clause.
>>>>>>>>>
>>>>>>>>> Mike
>>>>>>>>>
>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>
>>>>>>>>> Sorry...i'm getting slightly confused.
>>>>>>>>>
>>>>>>>>> I have a PostConstruct which is where I should create an array of
>>>>>>>>>
>>>>>>>>> SearchManagers (per indexSeacher). From there I initialise the
>>>>>>>>>> multisearcher using the get(). After which I need to call
>>>>>>>>>> maybeReopen
>>>>>>>>>> for
>>>>>>>>>> each IndexSearcher. So I'll do the following:
>>>>>>>>>>
>>>>>>>>>> @PostConstruct
>>>>>>>>>>
>>>>>>>>>> public void initialiseDocumentSearcher() {
>>>>>>>>>>
>>>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>>>> analyzer);
>>>>>>>>>>
>>>>>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>>>> newKeywordAnalyzer());
>>>>>>>>>>
>>>>>>>>>> queryParser =
>>>>>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>>> analyzerWrapper);
>>>>>>>>>>
>>>>>>>>>> try {
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>>>
>>>>>>>>>> documentSearcherManagers = new
>>>>>>>>>> DocumentSearcherManager[searchers.size()];
>>>>>>>>>>
>>>>>>>>>> for (int i = 0; i < searchers.size() ;i++) {
>>>>>>>>>>
>>>>>>>>>> IndexSearcher indexSearcher = searchers.get(i);
>>>>>>>>>>
>>>>>>>>>> Directory directory = indexSearcher.getIndexReader().directory();
>>>>>>>>>>
>>>>>>>>>> DocumentSearcherManager documentSearcherManager =
>>>>>>>>>> newDocumentSearcherManager(directory);
>>>>>>>>>>
>>>>>>>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>>>
>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>
>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This initialises search managers. I then have methods:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> private void maybeReopen() throws Exception {
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>>>>>>
>>>>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>>>>> documentSearcherManagers) {
>>>>>>>>>>
>>>>>>>>>> documentSearcherManager.maybeReopen();
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> private void release() throws Exception {
>>>>>>>>>>
>>>>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>>>>> documentSearcherManagers) {
>>>>>>>>>>
>>>>>>>>>> documentSearcherManager.release(documentSearcherManager.get());
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> private MultiSearcher get() {
>>>>>>>>>>
>>>>>>>>>> List<IndexSearcher> listOfIndexSeachers = new
>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>
>>>>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>>>>> documentSearcherManagers) {
>>>>>>>>>>
>>>>>>>>>> listOfIndexSeachers.add(documentSearcherManager.get());
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> try {
>>>>>>>>>>
>>>>>>>>>> multiSearcher = new
>>>>>>>>>> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[] {}));
>>>>>>>>>>
>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>
>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> return multiSearcher;
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> These methods are used in the following manner in the search code:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>>>> throwsSearchExecutionException {
>>>>>>>>>>
>>>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>>>
>>>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>>>
>>>>>>>>>> throw new SearchExecutionException("Search string cannot be empty.
>>>>>>>>>> There
>>>>>>>>>> will be too many results to process.");
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>>
>>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>>
>>>>>>>>>> stopWatch.start();
>>>>>>>>>>
>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>
>>>>>>>>>> try {
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>>>>
>>>>>>>>>> maybeReopen();
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>>>>>> searchers
>>>>>>>>>> '"
>>>>>>>>>> +
>>>>>>>>>> indexSearchers.size() +"'");
>>>>>>>>>>
>>>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query
>>>>>>>>>> '"
>>>>>>>>>> +
>>>>>>>>>> query.toString() +"'");
>>>>>>>>>>
>>>>>>>>>> Sort sort = null;
>>>>>>>>>>
>>>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>>>
>>>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>>>
>>>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>>>
>>>>>>>>>> if (filters != null) {
>>>>>>>>>>
>>>>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>>>>>
>>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("total number of hits for [" + query.toString() + " ]
>>>>>>>>>> =
>>>>>>>>>> "+topDocs.
>>>>>>>>>> totalHits);
>>>>>>>>>>
>>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>>
>>>>>>>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>>>>>>>
>>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>>
>>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>>>>>
>>>>>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>>>>>
>>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> release();
>>>>>>>>>>
>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>
>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> stopWatch.stop();
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>>
>>>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Does this look better? Again..I really really appreciate your
>>>>>>>>>> help!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Mar 1, 2009 at 4:18 PM, Michael McCandless <
>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This is not quite right -- you should only create SearcherManager
>>>>>>>>>> once
>>>>>>>>>>
>>>>>>>>>> (per Direcotry) at startup/app load, not with every search
>>>>>>>>>> request.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> And I don't see release -- it must call SearcherManager.release
>>>>>>>>>>> of
>>>>>>>>>>> each of the IndexSearchers previously returned from get().
>>>>>>>>>>>
>>>>>>>>>>> Mike
>>>>>>>>>>>
>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi
>>>>>>>>>>>
>>>>>>>>>>> Thanks again for helping on a Sunday!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I have now modified my maybeOpen() to do the following:
>>>>>>>>>>>>
>>>>>>>>>>>> private void maybeReopen() throws Exception {
>>>>>>>>>>>>
>>>>>>>>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>>>>>>>>
>>>>>>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>>>>>>>>>>> .getSearchables();
>>>>>>>>>>>>
>>>>>>>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>>>>>>>
>>>>>>>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>>>>>>>
>>>>>>>>>>>> SearcherManager documentSearcherManager = new
>>>>>>>>>>>> SearcherManager(indexReader.directory());
>>>>>>>>>>>>
>>>>>>>>>>>> documentSearcherManager.maybeReopen();
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> And get() to:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>>
>>>>>>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>>>>>>>>>>> .getSearchables();
>>>>>>>>>>>>
>>>>>>>>>>>> List<IndexSearcher> indexSearchersList = new
>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>
>>>>>>>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>>>>>>>
>>>>>>>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>>>>>>>
>>>>>>>>>>>> SearcherManager documentSearcherManager = null;
>>>>>>>>>>>>
>>>>>>>>>>>> try {
>>>>>>>>>>>>
>>>>>>>>>>>> documentSearcherManager = new
>>>>>>>>>>>> SearcherManager(indexReader.directory());
>>>>>>>>>>>>
>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>
>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> indexSearchersList.add(documentSearcherManager.get());
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> try {
>>>>>>>>>>>>
>>>>>>>>>>>> multiSearcher = new
>>>>>>>>>>>> MultiSearcher(indexSearchersList.toArray(newIndexSearcher[]
>>>>>>>>>>>> {}));
>>>>>>>>>>>>
>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>
>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This makes all my test pass. I am using the SearchManager that
>>>>>>>>>>>> you
>>>>>>>>>>>> recommended. Does this look ok?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Your maybeReopen has an excess incRef().
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I'm not sure how you open the searchers in the first place? The
>>>>>>>>>>>>
>>>>>>>>>>>> list
>>>>>>>>>>>>> starts as empty, and nothing populates it?
>>>>>>>>>>>>>
>>>>>>>>>>>>> When you do the initial population, you need an incRef.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think you're hitting IllegalStateException because
>>>>>>>>>>>>> maybeReopen
>>>>>>>>>>>>> is
>>>>>>>>>>>>> closing a reader before get() can get it (since they
>>>>>>>>>>>>> synchronize
>>>>>>>>>>>>> on
>>>>>>>>>>>>> different objects).
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'd recommend switching to the SearcherManager class.
>>>>>>>>>>>>> Instantiate
>>>>>>>>>>>>> one
>>>>>>>>>>>>> for each of your searchers. On each search request, go through
>>>>>>>>>>>>> them
>>>>>>>>>>>>> and call maybeReopen(), and then call get() and gather each
>>>>>>>>>>>>> IndexSearcher instance into a new array. Then, make a new
>>>>>>>>>>>>> MultiSearcher (opposite of what I said before): while that
>>>>>>>>>>>>> creates
>>>>>>>>>>>>> a
>>>>>>>>>>>>> small amount of garbage, it'll keep your code simpler (good
>>>>>>>>>>>>> tradeoff).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>
>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> sorrry I added
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> release(multiSearcher);
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> instead of multiSearcher.close();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I've now done the following:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> throwsSearchExecutionException {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> throw new SearchExecutionException("Search string cannot be
>>>>>>>>>>>>>>> empty.
>>>>>>>>>>>>>>> There
>>>>>>>>>>>>>>> will be too many results to process.");
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> stopWatch.start();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> maybeReopen();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>>>>>>>>>>> searchers
>>>>>>>>>>>>>>> '"+ indexSearchers.size() +
>>>>>>>>>>>>>>> "'");
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>>>>>>>>>>>>>>> Query
>>>>>>>>>>>>>>> '"
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>> query.toString() +"'");
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sort sort = null;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> if (filters != null) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter
>>>>>>>>>>>>>>> ,100,sort);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> LOGGER.debug("total number of hits for [" + query.toString()
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>> "
>>>>>>>>>>>>>>> ]
>>>>>>>>>>>>>>> =
>>>>>>>>>>>>>>> "+topDocs.
>>>>>>>>>>>>>>> totalHits);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc,
>>>>>>>>>>>>>>> score);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Summary documentSummary = new
>>>>>>>>>>>>>>> DocumentSummaryImpl(baseDocument);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> multiSearcher.close();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> stopWatch.stop();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> And have the following methods:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> @PostConstruct
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> public void initialiseQueryParser() {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>>>>>>>>> analyzer);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>>>>>>>>> newKeywordAnalyzer());
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> queryParser =
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> analyzerWrapper);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> this.multiSearcher = new
>>>>>>>>>>>>>>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Initialises mutltisearcher when this class is creared by
>>>>>>>>>>>>>>> spring.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> private synchronized void swapMultiSearcher(MultiSearcher
>>>>>>>>>>>>>>> newMultiSearcher) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> release(multiSearcher);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> multiSearcher = newMultiSearcher;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> public void maybeReopen() throws IOException {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> MultiSearcher newMultiSeacher = null;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> boolean refreshMultiSeacher = false;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> synchronized (searchers) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> IndexReader reader = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> reader.incRef();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Directory directory = reader.directory();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> long currentVersion = reader.getVersion();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> if (IndexReader.getCurrentVersion(directory) !=
>>>>>>>>>>>>>>> currentVersion)
>>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> IndexReader newReader =
>>>>>>>>>>>>>>> indexSearcher.getIndexReader().reopen();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> if (newReader != reader) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> reader.decRef();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> refreshMultiSeacher = true;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> reader = newReader;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> indexSearchers.add(newSearcher);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> if (refreshMultiSeacher) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> newMultiSeacher = new
>>>>>>>>>>>>>>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> warm(newMultiSeacher);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> swapMultiSearcher(newMultiSeacher);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> private void warm(MultiSearcher newMultiSeacher) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> private synchronized void release(MultiSearcher
>>>>>>>>>>>>>>> multiSearcher)
>>>>>>>>>>>>>>> throwsIOException {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> indexSearcher.getIndexReader().decRef();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> However I am now getting
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java.lang.IllegalStateException:
>>>>>>>>>>>>>>> org.apache.lucene.store.AlreadyClosedException: this
>>>>>>>>>>>>>>> IndexReader
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> closed
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> on the call:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm doing something wrong ..obviously..not sure where
>>>>>>>>>>>>>>> though..
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I was wondering the same thing ;)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It's best to call this method from a single BG "warming"
>>>>>>>>>>>>>>> thread,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>> case it would not need its own synchronization.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> But, to be safe, I'll add internal synchronization to it.
>>>>>>>>>>>>>>>> You
>>>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>> simply put synchronized in front of the method, since you
>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>>> this to
>>>>>>>>>>>>>>>> block searching.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> just a quick point:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> public void maybeReopen() throws IOException {
>>>>>>>>>>>>>>>> //D
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> long currentVersion =
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> currentSearcher.getIndexReader().getVersion();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>>>>>>>>>>>>>>>>> IndexReader newReader =
>>>>>>>>>>>>>>>>> currentSearcher.getIndexReader().reopen();
>>>>>>>>>>>>>>>>> assert newReader != currentSearcher.getIndexReader();
>>>>>>>>>>>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>>>>>>>>>>> warm(newSearcher);
>>>>>>>>>>>>>>>>> swapSearcher(newSearcher);
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> should the above be synchronised?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> thanks. i will rewrite..in between giving my baby her feed
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> playing
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> with the other child and my wife who wants me to do several
>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> things!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for your input. I would like to have a go at
>>>>>>>>>>>>>>>>>>> doing
>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> first, Solr may be an option.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser every
>>>>>>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>> creating unnecessary garbage; instead, they should be
>>>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -- I can moved the code out so that it is only created
>>>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> reused.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>>>>>> when nothing has changed. This just generates
>>>>>>>>>>>>>>>>>>>> unnecessary
>>>>>>>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -- This was something I thought about. I could move it
>>>>>>>>>>>>>>>>>>>> out
>>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>> created once. However I presume inside my code i need
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>>>>> whether
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> indexreaders are update to date. This needs to be
>>>>>>>>>>>>>>>>>>>> synchronized
>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>> well I
>>>>>>>>>>>>>>>>>>>> guess(?)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Yes you should synchronize the check for whether the
>>>>>>>>>>>>>>>>>>>> IndexReader
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> current.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * I don't see any synchronization -- it looks like two
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> search
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> requests are allowed into this method at the same time?
>>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -- So i need to extract the logic for reopening and
>>>>>>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> synchronisation mechanism.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Yes.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Ok. So I have some work to do. I'll refactor the code
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> inline to your recommendations.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>>>>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On a quick look, I think there are a few problems with
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> code:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * I don't see any synchronization -- it looks like two
>>>>>>>>>>>>>>>>>>>> search
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> requests are allowed into this method at the same time?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * You are over-incRef'ing (the reader.incRef inside the
>>>>>>>>>>>>>>>>>>>>> loop)
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>> don't see a corresponding decRef.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * You reopen and warm your searchers "live" (vs with BG
>>>>>>>>>>>>>>>>>>>>> thread);
>>>>>>>>>>>>>>>>>>>>> meaning the unlucky search request that hits a reopen
>>>>>>>>>>>>>>>>>>>>> pays
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> cost. This might be OK if the index is small enough
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> reopening & warming takes very little time. But if
>>>>>>>>>>>>>>>>>>>>> index
>>>>>>>>>>>>>>>>>>>>> gets
>>>>>>>>>>>>>>>>>>>>> large, making a random search pay that warming cost is
>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>> nice
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> the end user. It erodes their trust in you.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>>>>>>> when nothing has changed. This just generates
>>>>>>>>>>>>>>>>>>>>> unnecessary
>>>>>>>>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser every
>>>>>>>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>> creating unnecessary garbage; instead, they should be
>>>>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> You should consider simply using Solr -- it handles all
>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>> logic
>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> you and has been well debugged with time...
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The reason for the indexreader.reopen is because I have
>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>> webapp
>>>>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> enables users to upload files and then search for the
>>>>>>>>>>>>>>>>>>>>> documents.
>>>>>>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> reopen i'm concerned that the facet hit counter won't
>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> updated.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman
>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I have been able to get the code working for my
>>>>>>>>>>>>>>>>>>>>>>> scenario,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> however
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> question and I was wondering if I could get some help.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>> IndexSearchers which are used in a MultiSearcher
>>>>>>>>>>>>>>>>>>>>>>> class.
>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> indexsearchers to get each indexreader and put them
>>>>>>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>> MultiIndexReader.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> IndexReader[] readers = new
>>>>>>>>>>>>>>>>>>>>>>> IndexReader[searchables.length];
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> IndexSearcher indexSearcher =
>>>>>>>>>>>>>>>>>>>>>>> (IndexSearcher)searchables[i];
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> readers[i].close();
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> readers[i] = newReader;
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>>>>>>>>>>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> IndexSearcher indexSearcher = new
>>>>>>>>>>>>>>>>>>>>>>> IndexSearcher(multiReader);
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I then use the indexseacher to do the facet stuff. I
>>>>>>>>>>>>>>>>>>>>>>> end
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>> closing the multireader. This is causing problems in
>>>>>>>>>>>>>>>>>>>>>>> another
>>>>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>> where I
>>>>>>>>>>>>>>>>>>>>>>> do some other search as the indexreaders are closed.
>>>>>>>>>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>> close
>>>>>>>>>>>>>>>>>>>>>>> the multiindexreader or should I do some additional
>>>>>>>>>>>>>>>>>>>>>>> checks
>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin
>>>>>>>>>>>>>>>>>>>>>>> Mohammed-Coleman
>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks just what I needed!
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>>>>>>>>>>>>>>>>>>>>>>>> marcelo.ochoa@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi Amin:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>>>>>>>>>>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin
>>>>>>>>>>>>>>>>>>>>>>>>> Mohammed-Coleman
>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Sorry to re send this email but I was wondering if
>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> advice
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I am looking at building a faceted search using
>>>>>>>>>>>>>>>>>>>>>>>>>> Lucene.
>>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Solr
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> comes with this built in, however I would like to
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> try
>>>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>>>>>>>>>>> (something to add to my CV!). I have been
>>>>>>>>>>>>>>>>>>>>>>>>>>> looking
>>>>>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>> you can use the IndexReader and use TermVectors.
>>>>>>>>>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>> sure how to filter the results so that a
>>>>>>>>>>>>>>>>>>>>>>>>>>> particular
>>>>>>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>> subset of results. The next option I was looking
>>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>>>>>>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>>>>>>>>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2
>>>>>>>>>>>>>>>>>>>>>>>>>>> };un
>>>>>>>>>>>>>>>>>>>>>>>>>>> int[] docFreqs =
>>>>>>>>>>>>>>>>>>>>>>>>>>> indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> The only problem here is that I have to provide
>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> brand
>>>>>>>>>>>>>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>>>>>>>>>>>> time a
>>>>>>>>>>>>>>>>>>>>>>>>>>> new brand is created. Again I'm not sure how I
>>>>>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>>> filter
>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> results
>>>>>>>>>>>>>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>>>>>>>>>>>> It may be that I'm using the wrong api methods to
>>>>>>>>>>>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I would be grateful if I could get some advice on
>>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> P.S. I am basically trying to do something that
>>>>>>>>>>>>>>>>>>>>>>>>>>> displays
>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Personal Contact (23) Business Contact (45) and
>>>>>>>>>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>>>>>>>>> on..
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> ______________
>>>>>>>>>>>>>>>>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>>>>>>>>>>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>
>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>
>>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>
>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>
>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Faceted Search using Lucene
Posted by Michael McCandless <lu...@mikemccandless.com>.
Is it possible that when you first create the SearcherManager, there
is no index in each Directory?
If not... you better start adding diagnostics. EG inside your get(),
print out the numDocs() of each IndexReader you get from the
SearcherManager?
Something is wrong and it's best to explain it...
Mike
Amin Mohammed-Coleman wrote:
> Nope. If i remove the maybeReopen the search doesn't work. It only
> works
> when i cal maybeReopen followed by get().
>
> Cheers
> Amin
>
> On Mon, Mar 2, 2009 at 12:56 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>>
>> That's not right; something must be wrong.
>>
>> get() before maybeReopen() should simply let you search based on the
>> searcher before reopening.
>>
>> If you just do get() and don't call maybeReopen() does it work?
>>
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> I noticed that if i do the get() before the maybeReopen then I get no
>>> results. But otherwise I can change it further.
>>>
>>> On Mon, Mar 2, 2009 at 11:46 AM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>>> There is no such thing as final code -- code is alive and is always
>>>> changing ;)
>>>>
>>>> It looks good to me.
>>>>
>>>> Though one trivial thing is: I would move the code in the try
>>>> clause up
>>>> to
>>>> and including the multiSearcher=get() out above the try. I always
>>>> attempt
>>>> to "shrink wrap" what's inside a try clause to the minimum that
>>>> needs to
>>>> be
>>>> there. Ie, your code that creates a query, finds the right sort
>>>> & filter
>>>> to
>>>> use, etc, can all happen outside the try, because you have not yet
>>>> acquired
>>>> the multiSearcher.
>>>>
>>>> If you do that, you also don't need the null check in the finally
>>>> clause,
>>>> because multiSearcher must be non-null on entering the try.
>>>>
>>>> Mike
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> Hi there
>>>>
>>>>> Good morning! Here is the final search code:
>>>>>
>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>> throwsSearchExecutionException {
>>>>>
>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>
>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>
>>>>> throw new SearchExecutionException("Search string cannot be
>>>>> empty. There
>>>>> will be too many results to process.");
>>>>>
>>>>> }
>>>>>
>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>
>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>
>>>>> stopWatch.start();
>>>>>
>>>>> MultiSearcher multiSearcher = null;
>>>>>
>>>>> try {
>>>>>
>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>
>>>>> maybeReopen();
>>>>>
>>>>> Query query = queryParser.parse(searchTerm);
>>>>>
>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query
>>>>> '" +
>>>>> query.toString() +"'");
>>>>>
>>>>> Sort sort = null;
>>>>>
>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>
>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>
>>>>> ChainedFilter chainedFilter = null;
>>>>>
>>>>> if (filters != null) {
>>>>>
>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>
>>>>> }
>>>>>
>>>>> multiSearcher = get();
>>>>>
>>>>> TopDocs topDocs = multiSearcher.search(query,chainedFilter ,
>>>>> 100,sort);
>>>>>
>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>
>>>>> LOGGER.debug("total number of hits for [" + query.toString() +
>>>>> " ] =
>>>>> "+topDocs.
>>>>> totalHits);
>>>>>
>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>
>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>
>>>>> float score = scoreDoc.score;
>>>>>
>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>
>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>
>>>>> summaryList.add(documentSummary);
>>>>>
>>>>> }
>>>>>
>>>>> } catch (Exception e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> } finally {
>>>>>
>>>>> if (multiSearcher != null) {
>>>>>
>>>>> release(multiSearcher);
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> stopWatch.stop();
>>>>>
>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>
>>>>> return summaryList.toArray(new Summary[] {});
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> I hope this makes sense...thanks again!
>>>>>
>>>>>
>>>>> Cheers
>>>>>
>>>>> Amin
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Mar 1, 2009 at 8:09 PM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>>
>>>>> You're calling get() too many times. For every call to get()
>>>>> you must
>>>>>> match with a call to release().
>>>>>>
>>>>>> So, once at the front of your search method you should:
>>>>>>
>>>>>> MultiSearcher searcher = get();
>>>>>>
>>>>>> then use that searcher to do searching, retrieve docs, etc.
>>>>>>
>>>>>> Then in the finally clause, pass that searcher to release.
>>>>>>
>>>>>> So, only one call to get() and one matching call to release().
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> The searchers are injected into the class via Spring. So when a
>>>>>>> client
>>>>>>> calls the class it is fully configured with a list of index
>>>>>>> searchers.
>>>>>>> However I have removed this list and instead injecting a list of
>>>>>>> directories which are passed to the DocumentSearchManager.
>>>>>>> DocumentSearchManager is SearchManager (should've mentioned that
>>>>>>> earlier).
>>>>>>> So finally I have modified by release code to do the following:
>>>>>>>
>>>>>>> private void release(MultiSearcher multiSeacher) throws
>>>>>>> Exception {
>>>>>>>
>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>>>>> multiSeacher.getSearchables();
>>>>>>>
>>>>>>> for(int i =0 ; i < indexSearchers.length;i++) {
>>>>>>>
>>>>>>> documentSearcherManagers[i].release(indexSearchers[i]);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> and it's use looks like this:
>>>>>>>
>>>>>>>
>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>> throwsSearchExecutionException {
>>>>>>>
>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>
>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>
>>>>>>> throw new SearchExecutionException("Search string cannot be
>>>>>>> empty.
>>>>>>> There
>>>>>>> will be too many results to process.");
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>
>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>
>>>>>>> stopWatch.start();
>>>>>>>
>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>
>>>>>>> try {
>>>>>>>
>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>
>>>>>>> maybeReopen();
>>>>>>>
>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>>> searchers
>>>>>>> '"
>>>>>>> +
>>>>>>> indexSearchers.size() +"'");
>>>>>>>
>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>
>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>>>>>>> Query '" +
>>>>>>> query.toString() +"'");
>>>>>>>
>>>>>>> Sort sort = null;
>>>>>>>
>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>
>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>
>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>
>>>>>>> if (filters != null) {
>>>>>>>
>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>>
>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>
>>>>>>> LOGGER.debug("total number of hits for [" + query.toString() +
>>>>>>> " ] =
>>>>>>> "+topDocs.
>>>>>>> totalHits);
>>>>>>>
>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>
>>>>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>>>>
>>>>>>> float score = scoreDoc.score;
>>>>>>>
>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>>
>>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>>
>>>>>>> summaryList.add(documentSummary);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> } catch (Exception e) {
>>>>>>>
>>>>>>> throw new IllegalStateException(e);
>>>>>>>
>>>>>>> } finally {
>>>>>>>
>>>>>>> release(get());
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> stopWatch.stop();
>>>>>>>
>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>
>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> So the final post construct constructs the
>>>>>>> DocumentSearchMangers with
>>>>>>> the
>>>>>>> list of directories..looking like this
>>>>>>>
>>>>>>>
>>>>>>> @PostConstruct
>>>>>>>
>>>>>>> public void initialiseDocumentSearcher() {
>>>>>>>
>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>> analyzer);
>>>>>>>
>>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>> newKeywordAnalyzer());
>>>>>>>
>>>>>>> queryParser =
>>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>> analyzerWrapper);
>>>>>>>
>>>>>>> try {
>>>>>>>
>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>
>>>>>>> documentSearcherManagers = new
>>>>>>> DocumentSearcherManager[directories.size()];
>>>>>>>
>>>>>>> for (int i = 0; i < directories.size() ;i++) {
>>>>>>>
>>>>>>> Directory directory = directories.get(i);
>>>>>>>
>>>>>>> DocumentSearcherManager documentSearcherManager =
>>>>>>> newDocumentSearcherManager(directory);
>>>>>>>
>>>>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>
>>>>>>> } catch (IOException e) {
>>>>>>>
>>>>>>> throw new IllegalStateException(e);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>> Amin
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Mar 1, 2009 at 6:15 PM, Michael McCandless <
>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> I don't understand where searchers comes from, prior to
>>>>>>>
>>>>>>>> initializeDocumentSearcher? You should, instead, simply
>>>>>>>> create the
>>>>>>>> SearcherManager (from your Directory instances). You don't
>>>>>>>> need any
>>>>>>>> searchers during initialize.
>>>>>>>>
>>>>>>>> Is DocumentSearcherManager the same as SearcherManager (just
>>>>>>>> renamed)?
>>>>>>>>
>>>>>>>> The release method is wrong -- you're calling .get() and then
>>>>>>>> immediately release. Instead, you should step through the
>>>>>>>> searchers
>>>>>>>> from your MultiSearcher and release them to each
>>>>>>>> SearcherManager.
>>>>>>>>
>>>>>>>> You should call your release() in a finally clause.
>>>>>>>>
>>>>>>>> Mike
>>>>>>>>
>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>
>>>>>>>> Sorry...i'm getting slightly confused.
>>>>>>>>
>>>>>>>> I have a PostConstruct which is where I should create an
>>>>>>>> array of
>>>>>>>>
>>>>>>>>> SearchManagers (per indexSeacher). From there I initialise
>>>>>>>>> the
>>>>>>>>> multisearcher using the get(). After which I need to call
>>>>>>>>> maybeReopen
>>>>>>>>> for
>>>>>>>>> each IndexSearcher. So I'll do the following:
>>>>>>>>>
>>>>>>>>> @PostConstruct
>>>>>>>>>
>>>>>>>>> public void initialiseDocumentSearcher() {
>>>>>>>>>
>>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>>> analyzer);
>>>>>>>>>
>>>>>>>>> analyzerWrapper
>>>>>>>>> .addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>>> newKeywordAnalyzer());
>>>>>>>>>
>>>>>>>>> queryParser =
>>>>>>>>> newMultiFieldQueryParser
>>>>>>>>> (FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>> analyzerWrapper);
>>>>>>>>>
>>>>>>>>> try {
>>>>>>>>>
>>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>>
>>>>>>>>> documentSearcherManagers = new
>>>>>>>>> DocumentSearcherManager[searchers.size()];
>>>>>>>>>
>>>>>>>>> for (int i = 0; i < searchers.size() ;i++) {
>>>>>>>>>
>>>>>>>>> IndexSearcher indexSearcher = searchers.get(i);
>>>>>>>>>
>>>>>>>>> Directory directory =
>>>>>>>>> indexSearcher.getIndexReader().directory();
>>>>>>>>>
>>>>>>>>> DocumentSearcherManager documentSearcherManager =
>>>>>>>>> newDocumentSearcherManager(directory);
>>>>>>>>>
>>>>>>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>>
>>>>>>>>> } catch (IOException e) {
>>>>>>>>>
>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This initialises search managers. I then have methods:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> private void maybeReopen() throws Exception {
>>>>>>>>>
>>>>>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>>>>>
>>>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>>>> documentSearcherManagers) {
>>>>>>>>>
>>>>>>>>> documentSearcherManager.maybeReopen();
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> private void release() throws Exception {
>>>>>>>>>
>>>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>>>> documentSearcherManagers) {
>>>>>>>>>
>>>>>>>>> documentSearcherManager
>>>>>>>>> .release(documentSearcherManager.get());
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> private MultiSearcher get() {
>>>>>>>>>
>>>>>>>>> List<IndexSearcher> listOfIndexSeachers = new
>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>
>>>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>>>> documentSearcherManagers) {
>>>>>>>>>
>>>>>>>>> listOfIndexSeachers.add(documentSearcherManager.get());
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> try {
>>>>>>>>>
>>>>>>>>> multiSearcher = new
>>>>>>>>> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[]
>>>>>>>>> {}));
>>>>>>>>>
>>>>>>>>> } catch (IOException e) {
>>>>>>>>>
>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> return multiSearcher;
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> These methods are used in the following manner in the search
>>>>>>>>> code:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>>> throwsSearchExecutionException {
>>>>>>>>>
>>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>>
>>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>>
>>>>>>>>> throw new SearchExecutionException("Search string cannot be
>>>>>>>>> empty.
>>>>>>>>> There
>>>>>>>>> will be too many results to process.");
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>
>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>
>>>>>>>>> stopWatch.start();
>>>>>>>>>
>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>
>>>>>>>>> try {
>>>>>>>>>
>>>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>>>
>>>>>>>>> maybeReopen();
>>>>>>>>>
>>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>>>>> searchers
>>>>>>>>> '"
>>>>>>>>> +
>>>>>>>>> indexSearchers.size() +"'");
>>>>>>>>>
>>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>>
>>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>>>>>>>>> Query '"
>>>>>>>>> +
>>>>>>>>> query.toString() +"'");
>>>>>>>>>
>>>>>>>>> Sort sort = null;
>>>>>>>>>
>>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>>
>>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>>
>>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>>
>>>>>>>>> if (filters != null) {
>>>>>>>>>
>>>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>>>>
>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>
>>>>>>>>> LOGGER.debug("total number of hits for [" + query.toString()
>>>>>>>>> + " ] =
>>>>>>>>> "+topDocs.
>>>>>>>>> totalHits);
>>>>>>>>>
>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>
>>>>>>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>>>>>>
>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>
>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc,
>>>>>>>>> score);
>>>>>>>>>
>>>>>>>>> Summary documentSummary = new
>>>>>>>>> DocumentSummaryImpl(baseDocument);
>>>>>>>>>
>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> release();
>>>>>>>>>
>>>>>>>>> } catch (Exception e) {
>>>>>>>>>
>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> stopWatch.stop();
>>>>>>>>>
>>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>
>>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Does this look better? Again..I really really appreciate
>>>>>>>>> your help!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Mar 1, 2009 at 4:18 PM, Michael McCandless <
>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is not quite right -- you should only create
>>>>>>>>> SearcherManager
>>>>>>>>> once
>>>>>>>>>
>>>>>>>>> (per Direcotry) at startup/app load, not with every search
>>>>>>>>> request.
>>>>>>>>>>
>>>>>>>>>> And I don't see release -- it must call
>>>>>>>>>> SearcherManager.release of
>>>>>>>>>> each of the IndexSearchers previously returned from get().
>>>>>>>>>>
>>>>>>>>>> Mike
>>>>>>>>>>
>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> Thanks again for helping on a Sunday!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I have now modified my maybeOpen() to do the following:
>>>>>>>>>>>
>>>>>>>>>>> private void maybeReopen() throws Exception {
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>>>>>>>
>>>>>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>>>>>>>>> multiSearcher
>>>>>>>>>>> .getSearchables();
>>>>>>>>>>>
>>>>>>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>>>>>>
>>>>>>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>>>>>>
>>>>>>>>>>> SearcherManager documentSearcherManager = new
>>>>>>>>>>> SearcherManager(indexReader.directory());
>>>>>>>>>>>
>>>>>>>>>>> documentSearcherManager.maybeReopen();
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> And get() to:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>
>>>>>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>>>>>>>>> multiSearcher
>>>>>>>>>>> .getSearchables();
>>>>>>>>>>>
>>>>>>>>>>> List<IndexSearcher> indexSearchersList = new
>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>
>>>>>>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>>>>>>
>>>>>>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>>>>>>
>>>>>>>>>>> SearcherManager documentSearcherManager = null;
>>>>>>>>>>>
>>>>>>>>>>> try {
>>>>>>>>>>>
>>>>>>>>>>> documentSearcherManager = new
>>>>>>>>>>> SearcherManager(indexReader.directory());
>>>>>>>>>>>
>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>
>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> indexSearchersList.add(documentSearcherManager.get());
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> try {
>>>>>>>>>>>
>>>>>>>>>>> multiSearcher = new
>>>>>>>>>>> MultiSearcher
>>>>>>>>>>> (indexSearchersList.toArray(newIndexSearcher[] {}));
>>>>>>>>>>>
>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>
>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This makes all my test pass. I am using the SearchManager
>>>>>>>>>>> that
>>>>>>>>>>> you
>>>>>>>>>>> recommended. Does this look ok?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Your maybeReopen has an excess incRef().
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I'm not sure how you open the searchers in the first
>>>>>>>>>>> place? The
>>>>>>>>>>>
>>>>>>>>>>>> list
>>>>>>>>>>>> starts as empty, and nothing populates it?
>>>>>>>>>>>>
>>>>>>>>>>>> When you do the initial population, you need an incRef.
>>>>>>>>>>>>
>>>>>>>>>>>> I think you're hitting IllegalStateException because
>>>>>>>>>>>> maybeReopen
>>>>>>>>>>>> is
>>>>>>>>>>>> closing a reader before get() can get it (since they
>>>>>>>>>>>> synchronize
>>>>>>>>>>>> on
>>>>>>>>>>>> different objects).
>>>>>>>>>>>>
>>>>>>>>>>>> I'd recommend switching to the SearcherManager class.
>>>>>>>>>>>> Instantiate
>>>>>>>>>>>> one
>>>>>>>>>>>> for each of your searchers. On each search request, go
>>>>>>>>>>>> through
>>>>>>>>>>>> them
>>>>>>>>>>>> and call maybeReopen(), and then call get() and gather each
>>>>>>>>>>>> IndexSearcher instance into a new array. Then, make a new
>>>>>>>>>>>> MultiSearcher (opposite of what I said before): while that
>>>>>>>>>>>> creates
>>>>>>>>>>>> a
>>>>>>>>>>>> small amount of garbage, it'll keep your code simpler (good
>>>>>>>>>>>> tradeoff).
>>>>>>>>>>>>
>>>>>>>>>>>> Mike
>>>>>>>>>>>>
>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> sorrry I added
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> release(multiSearcher);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> instead of multiSearcher.close();
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> I've now done the following:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>>>>>>>> throwsSearchExecutionException {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> throw new SearchExecutionException("Search string
>>>>>>>>>>>>>> cannot be
>>>>>>>>>>>>>> empty.
>>>>>>>>>>>>>> There
>>>>>>>>>>>>>> will be too many results to process.");
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> stopWatch.start();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> LOGGER.debug("Ensuring all index readers are up to
>>>>>>>>>>>>>> date...");
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> maybeReopen();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of
>>>>>>>>>>>>>> index
>>>>>>>>>>>>>> searchers
>>>>>>>>>>>>>> '"+ indexSearchers.size() +
>>>>>>>>>>>>>> "'");
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ---->
>>>>>>>>>>>>>> Lucene
>>>>>>>>>>>>>> Query
>>>>>>>>>>>>>> '"
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>> query.toString() +"'");
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sort sort = null;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Filter[] filters
>>>>>>>>>>>>>> =applyFiltersIfApplicable(searchRequest);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> if (filters != null) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> chainedFilter = new ChainedFilter(filters,
>>>>>>>>>>>>>> ChainedFilter.OR);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,
>>>>>>>>>>>>>> 100,sort);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> LOGGER.debug("total number of hits for [" +
>>>>>>>>>>>>>> query.toString() +
>>>>>>>>>>>>>> "
>>>>>>>>>>>>>> ]
>>>>>>>>>>>>>> =
>>>>>>>>>>>>>> "+topDocs.
>>>>>>>>>>>>>> totalHits);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc,
>>>>>>>>>>>>>> score);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Summary documentSummary = new
>>>>>>>>>>>>>> DocumentSummaryImpl(baseDocument);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> multiSearcher.close();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> stopWatch.stop();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And have the following methods:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> @PostConstruct
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> public void initialiseQueryParser() {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>>>>>>>> analyzer);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> analyzerWrapper
>>>>>>>>>>>>>> .addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>>>>>>>> newKeywordAnalyzer());
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> queryParser =
>>>>>>>>>>>>>> newMultiFieldQueryParser
>>>>>>>>>>>>>> (FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> analyzerWrapper);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> this.multiSearcher = new
>>>>>>>>>>>>>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Initialises mutltisearcher when this class is creared by
>>>>>>>>>>>>>> spring.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> private synchronized void swapMultiSearcher(MultiSearcher
>>>>>>>>>>>>>> newMultiSearcher) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> try {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> release(multiSearcher);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> multiSearcher = newMultiSearcher;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> public void maybeReopen() throws IOException {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> MultiSearcher newMultiSeacher = null;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> boolean refreshMultiSeacher = false;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> synchronized (searchers) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> IndexReader reader = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> reader.incRef();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Directory directory = reader.directory();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> long currentVersion = reader.getVersion();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> if (IndexReader.getCurrentVersion(directory) !=
>>>>>>>>>>>>>> currentVersion)
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> IndexReader newReader =
>>>>>>>>>>>>>> indexSearcher.getIndexReader().reopen();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> if (newReader != reader) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> reader.decRef();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> refreshMultiSeacher = true;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> reader = newReader;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> indexSearchers.add(newSearcher);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> if (refreshMultiSeacher) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> newMultiSeacher = new
>>>>>>>>>>>>>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[]
>>>>>>>>>>>>>> {}));
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> warm(newMultiSeacher);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> swapMultiSearcher(newMultiSeacher);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> private void warm(MultiSearcher newMultiSeacher) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> private synchronized void release(MultiSearcher
>>>>>>>>>>>>>> multiSearcher)
>>>>>>>>>>>>>> throwsIOException {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> indexSearcher.getIndexReader().decRef();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> However I am now getting
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java.lang.IllegalStateException:
>>>>>>>>>>>>>> org.apache.lucene.store.AlreadyClosedException: this
>>>>>>>>>>>>>> IndexReader
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> closed
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> on the call:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm doing something wrong ..obviously..not sure where
>>>>>>>>>>>>>> though..
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I was wondering the same thing ;)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It's best to call this method from a single BG "warming"
>>>>>>>>>>>>>> thread,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>> case it would not need its own synchronization.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> But, to be safe, I'll add internal synchronization to
>>>>>>>>>>>>>>> it. You
>>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>> simply put synchronized in front of the method, since
>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>> this to
>>>>>>>>>>>>>>> block searching.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> just a quick point:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> public void maybeReopen() throws IOException {
>>>>>>>>>>>>>>> //D
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> long currentVersion =
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> currentSearcher.getIndexReader().getVersion();
>>>>>>>>>>>>>>>> if (IndexReader.getCurrentVersion(dir) !=
>>>>>>>>>>>>>>>> currentVersion) {
>>>>>>>>>>>>>>>> IndexReader newReader =
>>>>>>>>>>>>>>>> currentSearcher.getIndexReader().reopen();
>>>>>>>>>>>>>>>> assert newReader != currentSearcher.getIndexReader();
>>>>>>>>>>>>>>>> IndexSearcher newSearcher = new
>>>>>>>>>>>>>>>> IndexSearcher(newReader);
>>>>>>>>>>>>>>>> warm(newSearcher);
>>>>>>>>>>>>>>>> swapSearcher(newSearcher);
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> should the above be synchronised?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> thanks. i will rewrite..in between giving my baby
>>>>>>>>>>>>>>>> her feed
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> playing
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> with the other child and my wife who wants me to do
>>>>>>>>>>>>>>>> several
>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> things!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for your input. I would like to have a go
>>>>>>>>>>>>>>>>>> at doing
>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> first, Solr may be an option.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser
>>>>>>>>>>>>>>>>>>> every
>>>>>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>> creating unnecessary garbage; instead, they should
>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -- I can moved the code out so that it is only
>>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> reused.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>>>>> when nothing has changed. This just generates
>>>>>>>>>>>>>>>>>>> unnecessary
>>>>>>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -- This was something I thought about. I could
>>>>>>>>>>>>>>>>>>> move it
>>>>>>>>>>>>>>>>>>> out
>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>> created once. However I presume inside my code i
>>>>>>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>>>> whether
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> indexreaders are update to date. This needs to be
>>>>>>>>>>>>>>>>>>> synchronized
>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>> well I
>>>>>>>>>>>>>>>>>>> guess(?)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Yes you should synchronize the check for whether the
>>>>>>>>>>>>>>>>>>> IndexReader
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> current.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * I don't see any synchronization -- it looks like
>>>>>>>>>>>>>>>>>>> two
>>>>>>>>>>>>>>>>>> search
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> requests are allowed into this method at the same
>>>>>>>>>>>>>>>>>> time?
>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully
>>>>>>>>>>>>>>>>>> reopen the
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -- So i need to extract the logic for reopening and
>>>>>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> synchronisation mechanism.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Yes.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Ok. So I have some work to do. I'll refactor the
>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> inline to your recommendations.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael
>>>>>>>>>>>>>>>>>>> McCandless <
>>>>>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On a quick look, I think there are a few problems
>>>>>>>>>>>>>>>>>>> with the
>>>>>>>>>>>>>>>>>>> code:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * I don't see any synchronization -- it looks like
>>>>>>>>>>>>>>>>>>> two
>>>>>>>>>>>>>>>>>>> search
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> requests are allowed into this method at the same
>>>>>>>>>>>>>>>>>>> time?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully
>>>>>>>>>>>>>>>>>>>> reopen
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * You are over-incRef'ing (the reader.incRef
>>>>>>>>>>>>>>>>>>>> inside the
>>>>>>>>>>>>>>>>>>>> loop)
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>> don't see a corresponding decRef.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * You reopen and warm your searchers "live" (vs
>>>>>>>>>>>>>>>>>>>> with BG
>>>>>>>>>>>>>>>>>>>> thread);
>>>>>>>>>>>>>>>>>>>> meaning the unlucky search request that hits a
>>>>>>>>>>>>>>>>>>>> reopen
>>>>>>>>>>>>>>>>>>>> pays
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> cost. This might be OK if the index is small
>>>>>>>>>>>>>>>>>>>> enough that
>>>>>>>>>>>>>>>>>>>> reopening & warming takes very little time. But
>>>>>>>>>>>>>>>>>>>> if index
>>>>>>>>>>>>>>>>>>>> gets
>>>>>>>>>>>>>>>>>>>> large, making a random search pay that warming
>>>>>>>>>>>>>>>>>>>> cost is
>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>> nice
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> the end user. It erodes their trust in you.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>>>>>> when nothing has changed. This just generates
>>>>>>>>>>>>>>>>>>>> unnecessary
>>>>>>>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser
>>>>>>>>>>>>>>>>>>>> every
>>>>>>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>> creating unnecessary garbage; instead, they
>>>>>>>>>>>>>>>>>>>> should be
>>>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> You should consider simply using Solr -- it
>>>>>>>>>>>>>>>>>>>> handles all
>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> logic
>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>> you and has been well debugged with time...
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The reason for the indexreader.reopen is because
>>>>>>>>>>>>>>>>>>>> I have a
>>>>>>>>>>>>>>>>>>>> webapp
>>>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> enables users to upload files and then search for
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> documents.
>>>>>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> reopen i'm concerned that the facet hit counter
>>>>>>>>>>>>>>>>>>>> won't be
>>>>>>>>>>>>>>>>>>>>> updated.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-
>>>>>>>>>>>>>>>>>>>>> Coleman <
>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I have been able to get the code working for my
>>>>>>>>>>>>>>>>>>>>>> scenario,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> however
>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> question and I was wondering if I could get some
>>>>>>>>>>>>>>>>>>>>> help.
>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>> IndexSearchers which are used in a
>>>>>>>>>>>>>>>>>>>>>> MultiSearcher class.
>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> indexsearchers to get each indexreader and put
>>>>>>>>>>>>>>>>>>>>>> them
>>>>>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>> MultiIndexReader.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> IndexReader[] readers = new
>>>>>>>>>>>>>>>>>>>>>> IndexReader[searchables.length];
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> IndexSearcher indexSearcher =
>>>>>>>>>>>>>>>>>>>>>> (IndexSearcher)searchables[i];
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> readers[i].close();
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> readers[i] = newReader;
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>>>>>>>>>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> IndexSearcher indexSearcher = new
>>>>>>>>>>>>>>>>>>>>>> IndexSearcher(multiReader);
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I then use the indexseacher to do the facet
>>>>>>>>>>>>>>>>>>>>>> stuff. I
>>>>>>>>>>>>>>>>>>>>>> end
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>> closing the multireader. This is causing
>>>>>>>>>>>>>>>>>>>>>> problems in
>>>>>>>>>>>>>>>>>>>>>> another
>>>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>> where I
>>>>>>>>>>>>>>>>>>>>>> do some other search as the indexreaders are
>>>>>>>>>>>>>>>>>>>>>> closed.
>>>>>>>>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>> close
>>>>>>>>>>>>>>>>>>>>>> the multiindexreader or should I do some
>>>>>>>>>>>>>>>>>>>>>> additional
>>>>>>>>>>>>>>>>>>>>>> checks
>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-
>>>>>>>>>>>>>>>>>>>>>> Coleman
>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks just what I needed!
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>>>>>>>>>>>>>>>>>>>>>>> marcelo.ochoa@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi Amin:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>>>>>>>>>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin
>>>>>>>>>>>>>>>>>>>>>>>> Mohammed-Coleman
>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Sorry to re send this email but I was
>>>>>>>>>>>>>>>>>>>>>>>> wondering if I
>>>>>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> advice
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-
>>>>>>>>>>>>>>>>>>>>>>>>> Coleman <
>>>>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I am looking at building a faceted search
>>>>>>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>>>>>> Lucene.
>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Solr
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> comes with this built in, however I would
>>>>>>>>>>>>>>>>>>>>>>>>> like to
>>>>>>>>>>>>>>>>>>>>>>>>>> try
>>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>>>>>>>>>> (something to add to my CV!). I have been
>>>>>>>>>>>>>>>>>>>>>>>>>> looking
>>>>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>> you can use the IndexReader and use
>>>>>>>>>>>>>>>>>>>>>>>>>> TermVectors.
>>>>>>>>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>> sure how to filter the results so that a
>>>>>>>>>>>>>>>>>>>>>>>>>> particular
>>>>>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>> subset of results. The next option I was
>>>>>>>>>>>>>>>>>>>>>>>>>> looking
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>>>>>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>>>>>>>>>>>>>>>> Term[] termsArray = new Term[] { term1,
>>>>>>>>>>>>>>>>>>>>>>>>>> term2 };un
>>>>>>>>>>>>>>>>>>>>>>>>>> int[] docFreqs =
>>>>>>>>>>>>>>>>>>>>>>>>>> indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> The only problem here is that I have to
>>>>>>>>>>>>>>>>>>>>>>>>>> provide the
>>>>>>>>>>>>>>>>>>>>>>>>>> brand
>>>>>>>>>>>>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>>>>>>>>>>> time a
>>>>>>>>>>>>>>>>>>>>>>>>>> new brand is created. Again I'm not sure
>>>>>>>>>>>>>>>>>>>>>>>>>> how I can
>>>>>>>>>>>>>>>>>>>>>>>>>> filter
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> results
>>>>>>>>>>>>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>>>>>>>>>>> It may be that I'm using the wrong api
>>>>>>>>>>>>>>>>>>>>>>>>>> methods to
>>>>>>>>>>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I would be grateful if I could get some
>>>>>>>>>>>>>>>>>>>>>>>>>> advice on
>>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> P.S. I am basically trying to do something
>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>> displays
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Personal Contact (23) Business Contact (45)
>>>>>>>>>>>>>>>>>>>>>>>>>> and so
>>>>>>>>>>>>>>>>>>>>>>>>>> on..
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>>>>>>>>>>>>>>>> ______________
>>>>>>>>>>>>>>>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>>>>>>>>>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>
>>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>
>>>>>>>>> To unsubscribe, e-mail: java-user-
>>>>>>>>> unsubscribe@lucene.apache.org
>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
Nope. If i remove the maybeReopen the search doesn't work. It only works
when i cal maybeReopen followed by get().
Cheers
Amin
On Mon, Mar 2, 2009 at 12:56 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
>
> That's not right; something must be wrong.
>
> get() before maybeReopen() should simply let you search based on the
> searcher before reopening.
>
> If you just do get() and don't call maybeReopen() does it work?
>
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> I noticed that if i do the get() before the maybeReopen then I get no
>> results. But otherwise I can change it further.
>>
>> On Mon, Mar 2, 2009 at 11:46 AM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>
>>> There is no such thing as final code -- code is alive and is always
>>> changing ;)
>>>
>>> It looks good to me.
>>>
>>> Though one trivial thing is: I would move the code in the try clause up
>>> to
>>> and including the multiSearcher=get() out above the try. I always
>>> attempt
>>> to "shrink wrap" what's inside a try clause to the minimum that needs to
>>> be
>>> there. Ie, your code that creates a query, finds the right sort & filter
>>> to
>>> use, etc, can all happen outside the try, because you have not yet
>>> acquired
>>> the multiSearcher.
>>>
>>> If you do that, you also don't need the null check in the finally clause,
>>> because multiSearcher must be non-null on entering the try.
>>>
>>> Mike
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> Hi there
>>>
>>>> Good morning! Here is the final search code:
>>>>
>>>> public Summary[] search(final SearchRequest searchRequest)
>>>> throwsSearchExecutionException {
>>>>
>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>
>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>
>>>> throw new SearchExecutionException("Search string cannot be empty. There
>>>> will be too many results to process.");
>>>>
>>>> }
>>>>
>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>
>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>
>>>> stopWatch.start();
>>>>
>>>> MultiSearcher multiSearcher = null;
>>>>
>>>> try {
>>>>
>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>
>>>> maybeReopen();
>>>>
>>>> Query query = queryParser.parse(searchTerm);
>>>>
>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
>>>> query.toString() +"'");
>>>>
>>>> Sort sort = null;
>>>>
>>>> sort = applySortIfApplicable(searchRequest);
>>>>
>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>
>>>> ChainedFilter chainedFilter = null;
>>>>
>>>> if (filters != null) {
>>>>
>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>
>>>> }
>>>>
>>>> multiSearcher = get();
>>>>
>>>> TopDocs topDocs = multiSearcher.search(query,chainedFilter ,100,sort);
>>>>
>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>
>>>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>>>> "+topDocs.
>>>> totalHits);
>>>>
>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>
>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>
>>>> float score = scoreDoc.score;
>>>>
>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>
>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>
>>>> summaryList.add(documentSummary);
>>>>
>>>> }
>>>>
>>>> } catch (Exception e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> } finally {
>>>>
>>>> if (multiSearcher != null) {
>>>>
>>>> release(multiSearcher);
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> stopWatch.stop();
>>>>
>>>> LOGGER.debug("total time taken for document seach: " +
>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>
>>>> return summaryList.toArray(new Summary[] {});
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> I hope this makes sense...thanks again!
>>>>
>>>>
>>>> Cheers
>>>>
>>>> Amin
>>>>
>>>>
>>>>
>>>> On Sun, Mar 1, 2009 at 8:09 PM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>>
>>>> You're calling get() too many times. For every call to get() you must
>>>>> match with a call to release().
>>>>>
>>>>> So, once at the front of your search method you should:
>>>>>
>>>>> MultiSearcher searcher = get();
>>>>>
>>>>> then use that searcher to do searching, retrieve docs, etc.
>>>>>
>>>>> Then in the finally clause, pass that searcher to release.
>>>>>
>>>>> So, only one call to get() and one matching call to release().
>>>>>
>>>>> Mike
>>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> The searchers are injected into the class via Spring. So when a
>>>>>> client
>>>>>> calls the class it is fully configured with a list of index searchers.
>>>>>> However I have removed this list and instead injecting a list of
>>>>>> directories which are passed to the DocumentSearchManager.
>>>>>> DocumentSearchManager is SearchManager (should've mentioned that
>>>>>> earlier).
>>>>>> So finally I have modified by release code to do the following:
>>>>>>
>>>>>> private void release(MultiSearcher multiSeacher) throws Exception {
>>>>>>
>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>>>> multiSeacher.getSearchables();
>>>>>>
>>>>>> for(int i =0 ; i < indexSearchers.length;i++) {
>>>>>>
>>>>>> documentSearcherManagers[i].release(indexSearchers[i]);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> and it's use looks like this:
>>>>>>
>>>>>>
>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>> throwsSearchExecutionException {
>>>>>>
>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>
>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>
>>>>>> throw new SearchExecutionException("Search string cannot be empty.
>>>>>> There
>>>>>> will be too many results to process.");
>>>>>>
>>>>>> }
>>>>>>
>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>
>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>
>>>>>> stopWatch.start();
>>>>>>
>>>>>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>
>>>>>> maybeReopen();
>>>>>>
>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>> searchers
>>>>>> '"
>>>>>> +
>>>>>> indexSearchers.size() +"'");
>>>>>>
>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>
>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
>>>>>> query.toString() +"'");
>>>>>>
>>>>>> Sort sort = null;
>>>>>>
>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>
>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>
>>>>>> ChainedFilter chainedFilter = null;
>>>>>>
>>>>>> if (filters != null) {
>>>>>>
>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>
>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>
>>>>>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>>>>>> "+topDocs.
>>>>>> totalHits);
>>>>>>
>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>
>>>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>>>
>>>>>> float score = scoreDoc.score;
>>>>>>
>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>
>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>
>>>>>> summaryList.add(documentSummary);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> } catch (Exception e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> } finally {
>>>>>>
>>>>>> release(get());
>>>>>>
>>>>>> }
>>>>>>
>>>>>> stopWatch.stop();
>>>>>>
>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>
>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> So the final post construct constructs the DocumentSearchMangers with
>>>>>> the
>>>>>> list of directories..looking like this
>>>>>>
>>>>>>
>>>>>> @PostConstruct
>>>>>>
>>>>>> public void initialiseDocumentSearcher() {
>>>>>>
>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new PerFieldAnalyzerWrapper(
>>>>>> analyzer);
>>>>>>
>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>> newKeywordAnalyzer());
>>>>>>
>>>>>> queryParser =
>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>> analyzerWrapper);
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>
>>>>>> documentSearcherManagers = new
>>>>>> DocumentSearcherManager[directories.size()];
>>>>>>
>>>>>> for (int i = 0; i < directories.size() ;i++) {
>>>>>>
>>>>>> Directory directory = directories.get(i);
>>>>>>
>>>>>> DocumentSearcherManager documentSearcherManager =
>>>>>> newDocumentSearcherManager(directory);
>>>>>>
>>>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>>>
>>>>>> }
>>>>>>
>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>
>>>>>> } catch (IOException e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> Amin
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Mar 1, 2009 at 6:15 PM, Michael McCandless <
>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>
>>>>>>
>>>>>> I don't understand where searchers comes from, prior to
>>>>>>
>>>>>>> initializeDocumentSearcher? You should, instead, simply create the
>>>>>>> SearcherManager (from your Directory instances). You don't need any
>>>>>>> searchers during initialize.
>>>>>>>
>>>>>>> Is DocumentSearcherManager the same as SearcherManager (just
>>>>>>> renamed)?
>>>>>>>
>>>>>>> The release method is wrong -- you're calling .get() and then
>>>>>>> immediately release. Instead, you should step through the searchers
>>>>>>> from your MultiSearcher and release them to each SearcherManager.
>>>>>>>
>>>>>>> You should call your release() in a finally clause.
>>>>>>>
>>>>>>> Mike
>>>>>>>
>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>
>>>>>>> Sorry...i'm getting slightly confused.
>>>>>>>
>>>>>>> I have a PostConstruct which is where I should create an array of
>>>>>>>
>>>>>>>> SearchManagers (per indexSeacher). From there I initialise the
>>>>>>>> multisearcher using the get(). After which I need to call
>>>>>>>> maybeReopen
>>>>>>>> for
>>>>>>>> each IndexSearcher. So I'll do the following:
>>>>>>>>
>>>>>>>> @PostConstruct
>>>>>>>>
>>>>>>>> public void initialiseDocumentSearcher() {
>>>>>>>>
>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>> analyzer);
>>>>>>>>
>>>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>> newKeywordAnalyzer());
>>>>>>>>
>>>>>>>> queryParser =
>>>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>>> analyzerWrapper);
>>>>>>>>
>>>>>>>> try {
>>>>>>>>
>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>
>>>>>>>> documentSearcherManagers = new
>>>>>>>> DocumentSearcherManager[searchers.size()];
>>>>>>>>
>>>>>>>> for (int i = 0; i < searchers.size() ;i++) {
>>>>>>>>
>>>>>>>> IndexSearcher indexSearcher = searchers.get(i);
>>>>>>>>
>>>>>>>> Directory directory = indexSearcher.getIndexReader().directory();
>>>>>>>>
>>>>>>>> DocumentSearcherManager documentSearcherManager =
>>>>>>>> newDocumentSearcherManager(directory);
>>>>>>>>
>>>>>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>
>>>>>>>> } catch (IOException e) {
>>>>>>>>
>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> This initialises search managers. I then have methods:
>>>>>>>>
>>>>>>>>
>>>>>>>> private void maybeReopen() throws Exception {
>>>>>>>>
>>>>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>>>>
>>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>>> documentSearcherManagers) {
>>>>>>>>
>>>>>>>> documentSearcherManager.maybeReopen();
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> private void release() throws Exception {
>>>>>>>>
>>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>>> documentSearcherManagers) {
>>>>>>>>
>>>>>>>> documentSearcherManager.release(documentSearcherManager.get());
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> private MultiSearcher get() {
>>>>>>>>
>>>>>>>> List<IndexSearcher> listOfIndexSeachers = new
>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>
>>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>>> documentSearcherManagers) {
>>>>>>>>
>>>>>>>> listOfIndexSeachers.add(documentSearcherManager.get());
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> try {
>>>>>>>>
>>>>>>>> multiSearcher = new
>>>>>>>> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[] {}));
>>>>>>>>
>>>>>>>> } catch (IOException e) {
>>>>>>>>
>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> return multiSearcher;
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> These methods are used in the following manner in the search code:
>>>>>>>>
>>>>>>>>
>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>> throwsSearchExecutionException {
>>>>>>>>
>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>
>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>
>>>>>>>> throw new SearchExecutionException("Search string cannot be empty.
>>>>>>>> There
>>>>>>>> will be too many results to process.");
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>
>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>
>>>>>>>> stopWatch.start();
>>>>>>>>
>>>>>>>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>>>>>>>
>>>>>>>> try {
>>>>>>>>
>>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>>
>>>>>>>> maybeReopen();
>>>>>>>>
>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>>>> searchers
>>>>>>>> '"
>>>>>>>> +
>>>>>>>> indexSearchers.size() +"'");
>>>>>>>>
>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>
>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '"
>>>>>>>> +
>>>>>>>> query.toString() +"'");
>>>>>>>>
>>>>>>>> Sort sort = null;
>>>>>>>>
>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>
>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>
>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>
>>>>>>>> if (filters != null) {
>>>>>>>>
>>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>>>
>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>
>>>>>>>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>>>>>>>> "+topDocs.
>>>>>>>> totalHits);
>>>>>>>>
>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>
>>>>>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>>>>>
>>>>>>>> float score = scoreDoc.score;
>>>>>>>>
>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>>>
>>>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>>>
>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> release();
>>>>>>>>
>>>>>>>> } catch (Exception e) {
>>>>>>>>
>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> stopWatch.stop();
>>>>>>>>
>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>
>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> Does this look better? Again..I really really appreciate your help!
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Mar 1, 2009 at 4:18 PM, Michael McCandless <
>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> This is not quite right -- you should only create SearcherManager
>>>>>>>> once
>>>>>>>>
>>>>>>>> (per Direcotry) at startup/app load, not with every search request.
>>>>>>>>>
>>>>>>>>> And I don't see release -- it must call SearcherManager.release of
>>>>>>>>> each of the IndexSearchers previously returned from get().
>>>>>>>>>
>>>>>>>>> Mike
>>>>>>>>>
>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> Thanks again for helping on a Sunday!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I have now modified my maybeOpen() to do the following:
>>>>>>>>>>
>>>>>>>>>> private void maybeReopen() throws Exception {
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>>>>>>
>>>>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>>>>>>>>> .getSearchables();
>>>>>>>>>>
>>>>>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>>>>>
>>>>>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>>>>>
>>>>>>>>>> SearcherManager documentSearcherManager = new
>>>>>>>>>> SearcherManager(indexReader.directory());
>>>>>>>>>>
>>>>>>>>>> documentSearcherManager.maybeReopen();
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> And get() to:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>
>>>>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>>>>>>>>> .getSearchables();
>>>>>>>>>>
>>>>>>>>>> List<IndexSearcher> indexSearchersList = new
>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>
>>>>>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>>>>>
>>>>>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>>>>>
>>>>>>>>>> SearcherManager documentSearcherManager = null;
>>>>>>>>>>
>>>>>>>>>> try {
>>>>>>>>>>
>>>>>>>>>> documentSearcherManager = new
>>>>>>>>>> SearcherManager(indexReader.directory());
>>>>>>>>>>
>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>
>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> indexSearchersList.add(documentSearcherManager.get());
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> try {
>>>>>>>>>>
>>>>>>>>>> multiSearcher = new
>>>>>>>>>> MultiSearcher(indexSearchersList.toArray(newIndexSearcher[] {}));
>>>>>>>>>>
>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>
>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> return multiSearcher;
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This makes all my test pass. I am using the SearchManager that
>>>>>>>>>> you
>>>>>>>>>> recommended. Does this look ok?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Your maybeReopen has an excess incRef().
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I'm not sure how you open the searchers in the first place? The
>>>>>>>>>>
>>>>>>>>>>> list
>>>>>>>>>>> starts as empty, and nothing populates it?
>>>>>>>>>>>
>>>>>>>>>>> When you do the initial population, you need an incRef.
>>>>>>>>>>>
>>>>>>>>>>> I think you're hitting IllegalStateException because maybeReopen
>>>>>>>>>>> is
>>>>>>>>>>> closing a reader before get() can get it (since they synchronize
>>>>>>>>>>> on
>>>>>>>>>>> different objects).
>>>>>>>>>>>
>>>>>>>>>>> I'd recommend switching to the SearcherManager class.
>>>>>>>>>>> Instantiate
>>>>>>>>>>> one
>>>>>>>>>>> for each of your searchers. On each search request, go through
>>>>>>>>>>> them
>>>>>>>>>>> and call maybeReopen(), and then call get() and gather each
>>>>>>>>>>> IndexSearcher instance into a new array. Then, make a new
>>>>>>>>>>> MultiSearcher (opposite of what I said before): while that
>>>>>>>>>>> creates
>>>>>>>>>>> a
>>>>>>>>>>> small amount of garbage, it'll keep your code simpler (good
>>>>>>>>>>> tradeoff).
>>>>>>>>>>>
>>>>>>>>>>> Mike
>>>>>>>>>>>
>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>
>>>>>>>>>>> sorrry I added
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> release(multiSearcher);
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> instead of multiSearcher.close();
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> I've now done the following:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>>>>>>> throwsSearchExecutionException {
>>>>>>>>>>>>>
>>>>>>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>>>>>>
>>>>>>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> throw new SearchExecutionException("Search string cannot be
>>>>>>>>>>>>> empty.
>>>>>>>>>>>>> There
>>>>>>>>>>>>> will be too many results to process.");
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>>>>>
>>>>>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>>>>>
>>>>>>>>>>>>> stopWatch.start();
>>>>>>>>>>>>>
>>>>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>>
>>>>>>>>>>>>> try {
>>>>>>>>>>>>>
>>>>>>>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>>>>>>>
>>>>>>>>>>>>> maybeReopen();
>>>>>>>>>>>>>
>>>>>>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>>>>>>>>> searchers
>>>>>>>>>>>>> '"+ indexSearchers.size() +
>>>>>>>>>>>>> "'");
>>>>>>>>>>>>>
>>>>>>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>>>>>>
>>>>>>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>>>>>>>>>>>>> Query
>>>>>>>>>>>>> '"
>>>>>>>>>>>>> +
>>>>>>>>>>>>> query.toString() +"'");
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sort sort = null;
>>>>>>>>>>>>>
>>>>>>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>>>>>>
>>>>>>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>>>>>>
>>>>>>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>>>>>>
>>>>>>>>>>>>> if (filters != null) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>>>>>>>>
>>>>>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>>>>>
>>>>>>>>>>>>> LOGGER.debug("total number of hits for [" + query.toString() +
>>>>>>>>>>>>> "
>>>>>>>>>>>>> ]
>>>>>>>>>>>>> =
>>>>>>>>>>>>> "+topDocs.
>>>>>>>>>>>>> totalHits);
>>>>>>>>>>>>>
>>>>>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>>>>>>>>
>>>>>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>>>>>
>>>>>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>>>>>>>>
>>>>>>>>>>>>> Summary documentSummary = new
>>>>>>>>>>>>> DocumentSummaryImpl(baseDocument);
>>>>>>>>>>>>>
>>>>>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> multiSearcher.close();
>>>>>>>>>>>>>
>>>>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> stopWatch.stop();
>>>>>>>>>>>>>
>>>>>>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>>>>>
>>>>>>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> And have the following methods:
>>>>>>>>>>>>>
>>>>>>>>>>>>> @PostConstruct
>>>>>>>>>>>>>
>>>>>>>>>>>>> public void initialiseQueryParser() {
>>>>>>>>>>>>>
>>>>>>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>>>>>>> analyzer);
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>>>>>>> newKeywordAnalyzer());
>>>>>>>>>>>>>
>>>>>>>>>>>>> queryParser =
>>>>>>>>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>>>>>>
>>>>>>>>>>>>> analyzerWrapper);
>>>>>>>>>>>>>
>>>>>>>>>>>>> try {
>>>>>>>>>>>>>
>>>>>>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>>>>>>
>>>>>>>>>>>>> this.multiSearcher = new
>>>>>>>>>>>>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>>>>>>>>>>>>
>>>>>>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>>>>>>
>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Initialises mutltisearcher when this class is creared by
>>>>>>>>>>>>> spring.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> private synchronized void swapMultiSearcher(MultiSearcher
>>>>>>>>>>>>> newMultiSearcher) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> try {
>>>>>>>>>>>>>
>>>>>>>>>>>>> release(multiSearcher);
>>>>>>>>>>>>>
>>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> multiSearcher = newMultiSearcher;
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> public void maybeReopen() throws IOException {
>>>>>>>>>>>>>
>>>>>>>>>>>>> MultiSearcher newMultiSeacher = null;
>>>>>>>>>>>>>
>>>>>>>>>>>>> boolean refreshMultiSeacher = false;
>>>>>>>>>>>>>
>>>>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>>
>>>>>>>>>>>>> synchronized (searchers) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> IndexReader reader = indexSearcher.getIndexReader();
>>>>>>>>>>>>>
>>>>>>>>>>>>> reader.incRef();
>>>>>>>>>>>>>
>>>>>>>>>>>>> Directory directory = reader.directory();
>>>>>>>>>>>>>
>>>>>>>>>>>>> long currentVersion = reader.getVersion();
>>>>>>>>>>>>>
>>>>>>>>>>>>> if (IndexReader.getCurrentVersion(directory) != currentVersion)
>>>>>>>>>>>>> {
>>>>>>>>>>>>>
>>>>>>>>>>>>> IndexReader newReader =
>>>>>>>>>>>>> indexSearcher.getIndexReader().reopen();
>>>>>>>>>>>>>
>>>>>>>>>>>>> if (newReader != reader) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> reader.decRef();
>>>>>>>>>>>>>
>>>>>>>>>>>>> refreshMultiSeacher = true;
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> reader = newReader;
>>>>>>>>>>>>>
>>>>>>>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>>>>>>>
>>>>>>>>>>>>> indexSearchers.add(newSearcher);
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> if (refreshMultiSeacher) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> newMultiSeacher = new
>>>>>>>>>>>>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
>>>>>>>>>>>>>
>>>>>>>>>>>>> warm(newMultiSeacher);
>>>>>>>>>>>>>
>>>>>>>>>>>>> swapMultiSearcher(newMultiSeacher);
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> private void warm(MultiSearcher newMultiSeacher) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>>>
>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> private synchronized void release(MultiSearcher multiSearcher)
>>>>>>>>>>>>> throwsIOException {
>>>>>>>>>>>>>
>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> indexSearcher.getIndexReader().decRef();
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> However I am now getting
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.lang.IllegalStateException:
>>>>>>>>>>>>> org.apache.lucene.store.AlreadyClosedException: this
>>>>>>>>>>>>> IndexReader
>>>>>>>>>>>>> is
>>>>>>>>>>>>> closed
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> on the call:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>>>
>>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm doing something wrong ..obviously..not sure where though..
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I was wondering the same thing ;)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> It's best to call this method from a single BG "warming"
>>>>>>>>>>>>> thread,
>>>>>>>>>>>>>
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> which
>>>>>>>>>>>>>> case it would not need its own synchronization.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> But, to be safe, I'll add internal synchronization to it. You
>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>> simply put synchronized in front of the method, since you
>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>> want
>>>>>>>>>>>>>> this to
>>>>>>>>>>>>>> block searching.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> just a quick point:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> public void maybeReopen() throws IOException {
>>>>>>>>>>>>>> //D
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> long currentVersion =
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> currentSearcher.getIndexReader().getVersion();
>>>>>>>>>>>>>>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>>>>>>>>>>>>>>> IndexReader newReader =
>>>>>>>>>>>>>>> currentSearcher.getIndexReader().reopen();
>>>>>>>>>>>>>>> assert newReader != currentSearcher.getIndexReader();
>>>>>>>>>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>>>>>>>>> warm(newSearcher);
>>>>>>>>>>>>>>> swapSearcher(newSearcher);
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> should the above be synchronised?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> thanks. i will rewrite..in between giving my baby her feed
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> playing
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> with the other child and my wife who wants me to do several
>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> things!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for your input. I would like to have a go at doing
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> first, Solr may be an option.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser every
>>>>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>> creating unnecessary garbage; instead, they should be
>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -- I can moved the code out so that it is only created
>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> reused.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>>>> when nothing has changed. This just generates unnecessary
>>>>>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -- This was something I thought about. I could move it
>>>>>>>>>>>>>>>>>> out
>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>> created once. However I presume inside my code i need to
>>>>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>>> whether
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> indexreaders are update to date. This needs to be
>>>>>>>>>>>>>>>>>> synchronized
>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>> well I
>>>>>>>>>>>>>>>>>> guess(?)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Yes you should synchronize the check for whether the
>>>>>>>>>>>>>>>>>> IndexReader
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> current.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * I don't see any synchronization -- it looks like two
>>>>>>>>>>>>>>>>> search
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> requests are allowed into this method at the same time?
>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -- So i need to extract the logic for reopening and
>>>>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> synchronisation mechanism.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Yes.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Ok. So I have some work to do. I'll refactor the code
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> inline to your recommendations.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On a quick look, I think there are a few problems with the
>>>>>>>>>>>>>>>>>> code:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * I don't see any synchronization -- it looks like two
>>>>>>>>>>>>>>>>>> search
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> requests are allowed into this method at the same time?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * You are over-incRef'ing (the reader.incRef inside the
>>>>>>>>>>>>>>>>>>> loop)
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>> don't see a corresponding decRef.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * You reopen and warm your searchers "live" (vs with BG
>>>>>>>>>>>>>>>>>>> thread);
>>>>>>>>>>>>>>>>>>> meaning the unlucky search request that hits a reopen
>>>>>>>>>>>>>>>>>>> pays
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> cost. This might be OK if the index is small enough that
>>>>>>>>>>>>>>>>>>> reopening & warming takes very little time. But if index
>>>>>>>>>>>>>>>>>>> gets
>>>>>>>>>>>>>>>>>>> large, making a random search pay that warming cost is
>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>> nice
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> the end user. It erodes their trust in you.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>>>>> when nothing has changed. This just generates
>>>>>>>>>>>>>>>>>>> unnecessary
>>>>>>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser every
>>>>>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>> creating unnecessary garbage; instead, they should be
>>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> You should consider simply using Solr -- it handles all
>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>> logic
>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> you and has been well debugged with time...
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The reason for the indexreader.reopen is because I have a
>>>>>>>>>>>>>>>>>>> webapp
>>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> enables users to upload files and then search for the
>>>>>>>>>>>>>>>>>>> documents.
>>>>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> reopen i'm concerned that the facet hit counter won't be
>>>>>>>>>>>>>>>>>>>> updated.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I have been able to get the code working for my
>>>>>>>>>>>>>>>>>>>>> scenario,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> however
>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> question and I was wondering if I could get some help.
>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>> IndexSearchers which are used in a MultiSearcher class.
>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> indexsearchers to get each indexreader and put them
>>>>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>> MultiIndexReader.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> IndexReader[] readers = new
>>>>>>>>>>>>>>>>>>>>> IndexReader[searchables.length];
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> IndexSearcher indexSearcher =
>>>>>>>>>>>>>>>>>>>>> (IndexSearcher)searchables[i];
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> readers[i].close();
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> readers[i] = newReader;
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>>>>>>>>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> IndexSearcher indexSearcher = new
>>>>>>>>>>>>>>>>>>>>> IndexSearcher(multiReader);
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I then use the indexseacher to do the facet stuff. I
>>>>>>>>>>>>>>>>>>>>> end
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>> closing the multireader. This is causing problems in
>>>>>>>>>>>>>>>>>>>>> another
>>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>> where I
>>>>>>>>>>>>>>>>>>>>> do some other search as the indexreaders are closed.
>>>>>>>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>> close
>>>>>>>>>>>>>>>>>>>>> the multiindexreader or should I do some additional
>>>>>>>>>>>>>>>>>>>>> checks
>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman
>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks just what I needed!
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>>>>>>>>>>>>>>>>>>>>>> marcelo.ochoa@gmail.com>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi Amin:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>>>>>>>>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin
>>>>>>>>>>>>>>>>>>>>>>> Mohammed-Coleman
>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Sorry to re send this email but I was wondering if I
>>>>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> advice
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I am looking at building a faceted search using
>>>>>>>>>>>>>>>>>>>>>>>> Lucene.
>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Solr
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> comes with this built in, however I would like to
>>>>>>>>>>>>>>>>>>>>>>>>> try
>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>>>>>>>>> (something to add to my CV!). I have been looking
>>>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> you can use the IndexReader and use TermVectors.
>>>>>>>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>> sure how to filter the results so that a particular
>>>>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>> subset of results. The next option I was looking
>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>>>>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>>>>>>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>>>>>>>>>>>>>>>>>> int[] docFreqs =
>>>>>>>>>>>>>>>>>>>>>>>>> indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> The only problem here is that I have to provide the
>>>>>>>>>>>>>>>>>>>>>>>>> brand
>>>>>>>>>>>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>>>>>>>>>> time a
>>>>>>>>>>>>>>>>>>>>>>>>> new brand is created. Again I'm not sure how I can
>>>>>>>>>>>>>>>>>>>>>>>>> filter
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> results
>>>>>>>>>>>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>>>>>>>>>> It may be that I'm using the wrong api methods to
>>>>>>>>>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I would be grateful if I could get some advice on
>>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> P.S. I am basically trying to do something that
>>>>>>>>>>>>>>>>>>>>>>>>> displays
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Personal Contact (23) Business Contact (45) and so
>>>>>>>>>>>>>>>>>>>>>>>>> on..
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>>>>>>>>>>>>>>> ______________
>>>>>>>>>>>>>>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>>>>>>>>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>
>>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>
>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>
>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Faceted Search using Lucene
Posted by Michael McCandless <lu...@mikemccandless.com>.
That's not right; something must be wrong.
get() before maybeReopen() should simply let you search based on the
searcher before reopening.
If you just do get() and don't call maybeReopen() does it work?
Mike
Amin Mohammed-Coleman wrote:
> I noticed that if i do the get() before the maybeReopen then I get no
> results. But otherwise I can change it further.
>
> On Mon, Mar 2, 2009 at 11:46 AM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>>
>> There is no such thing as final code -- code is alive and is always
>> changing ;)
>>
>> It looks good to me.
>>
>> Though one trivial thing is: I would move the code in the try
>> clause up to
>> and including the multiSearcher=get() out above the try. I always
>> attempt
>> to "shrink wrap" what's inside a try clause to the minimum that
>> needs to be
>> there. Ie, your code that creates a query, finds the right sort &
>> filter to
>> use, etc, can all happen outside the try, because you have not yet
>> acquired
>> the multiSearcher.
>>
>> If you do that, you also don't need the null check in the finally
>> clause,
>> because multiSearcher must be non-null on entering the try.
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> Hi there
>>> Good morning! Here is the final search code:
>>>
>>> public Summary[] search(final SearchRequest searchRequest)
>>> throwsSearchExecutionException {
>>>
>>> final String searchTerm = searchRequest.getSearchTerm();
>>>
>>> if (StringUtils.isBlank(searchTerm)) {
>>>
>>> throw new SearchExecutionException("Search string cannot be empty.
>>> There
>>> will be too many results to process.");
>>>
>>> }
>>>
>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>
>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>
>>> stopWatch.start();
>>>
>>> MultiSearcher multiSearcher = null;
>>>
>>> try {
>>>
>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>
>>> maybeReopen();
>>>
>>> Query query = queryParser.parse(searchTerm);
>>>
>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query
>>> '" +
>>> query.toString() +"'");
>>>
>>> Sort sort = null;
>>>
>>> sort = applySortIfApplicable(searchRequest);
>>>
>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>
>>> ChainedFilter chainedFilter = null;
>>>
>>> if (filters != null) {
>>>
>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>
>>> }
>>>
>>> multiSearcher = get();
>>>
>>> TopDocs topDocs = multiSearcher.search(query,chainedFilter ,
>>> 100,sort);
>>>
>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>
>>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>>> "+topDocs.
>>> totalHits);
>>>
>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>
>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>
>>> float score = scoreDoc.score;
>>>
>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>
>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>
>>> summaryList.add(documentSummary);
>>>
>>> }
>>>
>>> } catch (Exception e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> } finally {
>>>
>>> if (multiSearcher != null) {
>>>
>>> release(multiSearcher);
>>>
>>> }
>>>
>>> }
>>>
>>> stopWatch.stop();
>>>
>>> LOGGER.debug("total time taken for document seach: " +
>>> stopWatch.getTotalTimeMillis() + " ms");
>>>
>>> return summaryList.toArray(new Summary[] {});
>>>
>>> }
>>>
>>>
>>>
>>> I hope this makes sense...thanks again!
>>>
>>>
>>> Cheers
>>>
>>> Amin
>>>
>>>
>>>
>>> On Sun, Mar 1, 2009 at 8:09 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>>> You're calling get() too many times. For every call to get() you
>>>> must
>>>> match with a call to release().
>>>>
>>>> So, once at the front of your search method you should:
>>>>
>>>> MultiSearcher searcher = get();
>>>>
>>>> then use that searcher to do searching, retrieve docs, etc.
>>>>
>>>> Then in the finally clause, pass that searcher to release.
>>>>
>>>> So, only one call to get() and one matching call to release().
>>>>
>>>> Mike
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> Hi
>>>>
>>>>> The searchers are injected into the class via Spring. So when a
>>>>> client
>>>>> calls the class it is fully configured with a list of index
>>>>> searchers.
>>>>> However I have removed this list and instead injecting a list of
>>>>> directories which are passed to the DocumentSearchManager.
>>>>> DocumentSearchManager is SearchManager (should've mentioned that
>>>>> earlier).
>>>>> So finally I have modified by release code to do the following:
>>>>>
>>>>> private void release(MultiSearcher multiSeacher) throws
>>>>> Exception {
>>>>>
>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>>> multiSeacher.getSearchables();
>>>>>
>>>>> for(int i =0 ; i < indexSearchers.length;i++) {
>>>>>
>>>>> documentSearcherManagers[i].release(indexSearchers[i]);
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> and it's use looks like this:
>>>>>
>>>>>
>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>> throwsSearchExecutionException {
>>>>>
>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>
>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>
>>>>> throw new SearchExecutionException("Search string cannot be
>>>>> empty. There
>>>>> will be too many results to process.");
>>>>>
>>>>> }
>>>>>
>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>
>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>
>>>>> stopWatch.start();
>>>>>
>>>>> List<IndexSearcher> indexSearchers = new
>>>>> ArrayList<IndexSearcher>();
>>>>>
>>>>> try {
>>>>>
>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>
>>>>> maybeReopen();
>>>>>
>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>> searchers
>>>>> '"
>>>>> +
>>>>> indexSearchers.size() +"'");
>>>>>
>>>>> Query query = queryParser.parse(searchTerm);
>>>>>
>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query
>>>>> '" +
>>>>> query.toString() +"'");
>>>>>
>>>>> Sort sort = null;
>>>>>
>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>
>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>
>>>>> ChainedFilter chainedFilter = null;
>>>>>
>>>>> if (filters != null) {
>>>>>
>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>
>>>>> }
>>>>>
>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>
>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>
>>>>> LOGGER.debug("total number of hits for [" + query.toString() +
>>>>> " ] =
>>>>> "+topDocs.
>>>>> totalHits);
>>>>>
>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>
>>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>>
>>>>> float score = scoreDoc.score;
>>>>>
>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>
>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>
>>>>> summaryList.add(documentSummary);
>>>>>
>>>>> }
>>>>>
>>>>> } catch (Exception e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> } finally {
>>>>>
>>>>> release(get());
>>>>>
>>>>> }
>>>>>
>>>>> stopWatch.stop();
>>>>>
>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>
>>>>> return summaryList.toArray(new Summary[] {});
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> So the final post construct constructs the DocumentSearchMangers
>>>>> with
>>>>> the
>>>>> list of directories..looking like this
>>>>>
>>>>>
>>>>> @PostConstruct
>>>>>
>>>>> public void initialiseDocumentSearcher() {
>>>>>
>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>> PerFieldAnalyzerWrapper(
>>>>> analyzer);
>>>>>
>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>> newKeywordAnalyzer());
>>>>>
>>>>> queryParser =
>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>> analyzerWrapper);
>>>>>
>>>>> try {
>>>>>
>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>
>>>>> documentSearcherManagers = new
>>>>> DocumentSearcherManager[directories.size()];
>>>>>
>>>>> for (int i = 0; i < directories.size() ;i++) {
>>>>>
>>>>> Directory directory = directories.get(i);
>>>>>
>>>>> DocumentSearcherManager documentSearcherManager =
>>>>> newDocumentSearcherManager(directory);
>>>>>
>>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>>
>>>>> }
>>>>>
>>>>> LOGGER.debug("multi searcher initialised");
>>>>>
>>>>> } catch (IOException e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> Cheers
>>>>>
>>>>> Amin
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Mar 1, 2009 at 6:15 PM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>>
>>>>> I don't understand where searchers comes from, prior to
>>>>>> initializeDocumentSearcher? You should, instead, simply create
>>>>>> the
>>>>>> SearcherManager (from your Directory instances). You don't
>>>>>> need any
>>>>>> searchers during initialize.
>>>>>>
>>>>>> Is DocumentSearcherManager the same as SearcherManager (just
>>>>>> renamed)?
>>>>>>
>>>>>> The release method is wrong -- you're calling .get() and then
>>>>>> immediately release. Instead, you should step through the
>>>>>> searchers
>>>>>> from your MultiSearcher and release them to each SearcherManager.
>>>>>>
>>>>>> You should call your release() in a finally clause.
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>> Sorry...i'm getting slightly confused.
>>>>>>
>>>>>> I have a PostConstruct which is where I should create an array of
>>>>>>> SearchManagers (per indexSeacher). From there I initialise the
>>>>>>> multisearcher using the get(). After which I need to call
>>>>>>> maybeReopen
>>>>>>> for
>>>>>>> each IndexSearcher. So I'll do the following:
>>>>>>>
>>>>>>> @PostConstruct
>>>>>>>
>>>>>>> public void initialiseDocumentSearcher() {
>>>>>>>
>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>> analyzer);
>>>>>>>
>>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>> newKeywordAnalyzer());
>>>>>>>
>>>>>>> queryParser =
>>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>> analyzerWrapper);
>>>>>>>
>>>>>>> try {
>>>>>>>
>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>
>>>>>>> documentSearcherManagers = new
>>>>>>> DocumentSearcherManager[searchers.size()];
>>>>>>>
>>>>>>> for (int i = 0; i < searchers.size() ;i++) {
>>>>>>>
>>>>>>> IndexSearcher indexSearcher = searchers.get(i);
>>>>>>>
>>>>>>> Directory directory =
>>>>>>> indexSearcher.getIndexReader().directory();
>>>>>>>
>>>>>>> DocumentSearcherManager documentSearcherManager =
>>>>>>> newDocumentSearcherManager(directory);
>>>>>>>
>>>>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>
>>>>>>> } catch (IOException e) {
>>>>>>>
>>>>>>> throw new IllegalStateException(e);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> This initialises search managers. I then have methods:
>>>>>>>
>>>>>>>
>>>>>>> private void maybeReopen() throws Exception {
>>>>>>>
>>>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>>>
>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>> documentSearcherManagers) {
>>>>>>>
>>>>>>> documentSearcherManager.maybeReopen();
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> private void release() throws Exception {
>>>>>>>
>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>> documentSearcherManagers) {
>>>>>>>
>>>>>>> documentSearcherManager.release(documentSearcherManager.get());
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> private MultiSearcher get() {
>>>>>>>
>>>>>>> List<IndexSearcher> listOfIndexSeachers = new
>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>
>>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>>> documentSearcherManagers) {
>>>>>>>
>>>>>>> listOfIndexSeachers.add(documentSearcherManager.get());
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> try {
>>>>>>>
>>>>>>> multiSearcher = new
>>>>>>> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[]
>>>>>>> {}));
>>>>>>>
>>>>>>> } catch (IOException e) {
>>>>>>>
>>>>>>> throw new IllegalStateException(e);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> return multiSearcher;
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> These methods are used in the following manner in the search
>>>>>>> code:
>>>>>>>
>>>>>>>
>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>> throwsSearchExecutionException {
>>>>>>>
>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>
>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>
>>>>>>> throw new SearchExecutionException("Search string cannot be
>>>>>>> empty.
>>>>>>> There
>>>>>>> will be too many results to process.");
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>
>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>
>>>>>>> stopWatch.start();
>>>>>>>
>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>
>>>>>>> try {
>>>>>>>
>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>
>>>>>>> maybeReopen();
>>>>>>>
>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>>> searchers
>>>>>>> '"
>>>>>>> +
>>>>>>> indexSearchers.size() +"'");
>>>>>>>
>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>
>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>>>>>>> Query '" +
>>>>>>> query.toString() +"'");
>>>>>>>
>>>>>>> Sort sort = null;
>>>>>>>
>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>
>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>
>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>
>>>>>>> if (filters != null) {
>>>>>>>
>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>>
>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>
>>>>>>> LOGGER.debug("total number of hits for [" + query.toString() +
>>>>>>> " ] =
>>>>>>> "+topDocs.
>>>>>>> totalHits);
>>>>>>>
>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>
>>>>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>>>>
>>>>>>> float score = scoreDoc.score;
>>>>>>>
>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>>
>>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>>
>>>>>>> summaryList.add(documentSummary);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> release();
>>>>>>>
>>>>>>> } catch (Exception e) {
>>>>>>>
>>>>>>> throw new IllegalStateException(e);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> stopWatch.stop();
>>>>>>>
>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>
>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> Does this look better? Again..I really really appreciate your
>>>>>>> help!
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Mar 1, 2009 at 4:18 PM, Michael McCandless <
>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> This is not quite right -- you should only create
>>>>>>> SearcherManager once
>>>>>>>
>>>>>>>> (per Direcotry) at startup/app load, not with every search
>>>>>>>> request.
>>>>>>>>
>>>>>>>> And I don't see release -- it must call
>>>>>>>> SearcherManager.release of
>>>>>>>> each of the IndexSearchers previously returned from get().
>>>>>>>>
>>>>>>>> Mike
>>>>>>>>
>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> Thanks again for helping on a Sunday!
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have now modified my maybeOpen() to do the following:
>>>>>>>>>
>>>>>>>>> private void maybeReopen() throws Exception {
>>>>>>>>>
>>>>>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>>>>>
>>>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>>>>>>> multiSearcher
>>>>>>>>> .getSearchables();
>>>>>>>>>
>>>>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>>>>
>>>>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>>>>
>>>>>>>>> SearcherManager documentSearcherManager = new
>>>>>>>>> SearcherManager(indexReader.directory());
>>>>>>>>>
>>>>>>>>> documentSearcherManager.maybeReopen();
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> And get() to:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>
>>>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>>>>>>> multiSearcher
>>>>>>>>> .getSearchables();
>>>>>>>>>
>>>>>>>>> List<IndexSearcher> indexSearchersList = new
>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>
>>>>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>>>>
>>>>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>>>>
>>>>>>>>> SearcherManager documentSearcherManager = null;
>>>>>>>>>
>>>>>>>>> try {
>>>>>>>>>
>>>>>>>>> documentSearcherManager = new
>>>>>>>>> SearcherManager(indexReader.directory());
>>>>>>>>>
>>>>>>>>> } catch (IOException e) {
>>>>>>>>>
>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> indexSearchersList.add(documentSearcherManager.get());
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> try {
>>>>>>>>>
>>>>>>>>> multiSearcher = new
>>>>>>>>> MultiSearcher(indexSearchersList.toArray(newIndexSearcher[]
>>>>>>>>> {}));
>>>>>>>>>
>>>>>>>>> } catch (IOException e) {
>>>>>>>>>
>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> return multiSearcher;
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This makes all my test pass. I am using the SearchManager
>>>>>>>>> that you
>>>>>>>>> recommended. Does this look ok?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>
>>>>>>>>> Your maybeReopen has an excess incRef().
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm not sure how you open the searchers in the first place?
>>>>>>>>> The
>>>>>>>>>> list
>>>>>>>>>> starts as empty, and nothing populates it?
>>>>>>>>>>
>>>>>>>>>> When you do the initial population, you need an incRef.
>>>>>>>>>>
>>>>>>>>>> I think you're hitting IllegalStateException because
>>>>>>>>>> maybeReopen is
>>>>>>>>>> closing a reader before get() can get it (since they
>>>>>>>>>> synchronize on
>>>>>>>>>> different objects).
>>>>>>>>>>
>>>>>>>>>> I'd recommend switching to the SearcherManager class.
>>>>>>>>>> Instantiate
>>>>>>>>>> one
>>>>>>>>>> for each of your searchers. On each search request, go
>>>>>>>>>> through
>>>>>>>>>> them
>>>>>>>>>> and call maybeReopen(), and then call get() and gather each
>>>>>>>>>> IndexSearcher instance into a new array. Then, make a new
>>>>>>>>>> MultiSearcher (opposite of what I said before): while that
>>>>>>>>>> creates
>>>>>>>>>> a
>>>>>>>>>> small amount of garbage, it'll keep your code simpler (good
>>>>>>>>>> tradeoff).
>>>>>>>>>>
>>>>>>>>>> Mike
>>>>>>>>>>
>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>
>>>>>>>>>> sorrry I added
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> release(multiSearcher);
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> instead of multiSearcher.close();
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <
>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I've now done the following:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>>>>>> throwsSearchExecutionException {
>>>>>>>>>>>>
>>>>>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>>>>>
>>>>>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>>>>>
>>>>>>>>>>>> throw new SearchExecutionException("Search string cannot be
>>>>>>>>>>>> empty.
>>>>>>>>>>>> There
>>>>>>>>>>>> will be too many results to process.");
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>>>>
>>>>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>>>>
>>>>>>>>>>>> stopWatch.start();
>>>>>>>>>>>>
>>>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>
>>>>>>>>>>>> try {
>>>>>>>>>>>>
>>>>>>>>>>>> LOGGER.debug("Ensuring all index readers are up to
>>>>>>>>>>>> date...");
>>>>>>>>>>>>
>>>>>>>>>>>> maybeReopen();
>>>>>>>>>>>>
>>>>>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of
>>>>>>>>>>>> index
>>>>>>>>>>>> searchers
>>>>>>>>>>>> '"+ indexSearchers.size() +
>>>>>>>>>>>> "'");
>>>>>>>>>>>>
>>>>>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>>>>>
>>>>>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ---->
>>>>>>>>>>>> Lucene Query
>>>>>>>>>>>> '"
>>>>>>>>>>>> +
>>>>>>>>>>>> query.toString() +"'");
>>>>>>>>>>>>
>>>>>>>>>>>> Sort sort = null;
>>>>>>>>>>>>
>>>>>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>>>>>
>>>>>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>>>>>
>>>>>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>>>>>
>>>>>>>>>>>> if (filters != null) {
>>>>>>>>>>>>
>>>>>>>>>>>> chainedFilter = new ChainedFilter(filters,
>>>>>>>>>>>> ChainedFilter.OR);
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,
>>>>>>>>>>>> 100,sort);
>>>>>>>>>>>>
>>>>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>>>>
>>>>>>>>>>>> LOGGER.debug("total number of hits for [" +
>>>>>>>>>>>> query.toString() + "
>>>>>>>>>>>> ]
>>>>>>>>>>>> =
>>>>>>>>>>>> "+topDocs.
>>>>>>>>>>>> totalHits);
>>>>>>>>>>>>
>>>>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>>>>
>>>>>>>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>>>>>>>
>>>>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>>>>
>>>>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc,
>>>>>>>>>>>> score);
>>>>>>>>>>>>
>>>>>>>>>>>> Summary documentSummary = new
>>>>>>>>>>>> DocumentSummaryImpl(baseDocument);
>>>>>>>>>>>>
>>>>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> multiSearcher.close();
>>>>>>>>>>>>
>>>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>>>
>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> stopWatch.stop();
>>>>>>>>>>>>
>>>>>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>>>>
>>>>>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> And have the following methods:
>>>>>>>>>>>>
>>>>>>>>>>>> @PostConstruct
>>>>>>>>>>>>
>>>>>>>>>>>> public void initialiseQueryParser() {
>>>>>>>>>>>>
>>>>>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>>>>>> analyzer);
>>>>>>>>>>>>
>>>>>>>>>>>> analyzerWrapper
>>>>>>>>>>>> .addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>>>>>> newKeywordAnalyzer());
>>>>>>>>>>>>
>>>>>>>>>>>> queryParser =
>>>>>>>>>>>> newMultiFieldQueryParser
>>>>>>>>>>>> (FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>>>>>
>>>>>>>>>>>> analyzerWrapper);
>>>>>>>>>>>>
>>>>>>>>>>>> try {
>>>>>>>>>>>>
>>>>>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>>>>>
>>>>>>>>>>>> this.multiSearcher = new
>>>>>>>>>>>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>>>>>>>>>>>
>>>>>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>>>>>
>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>
>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Initialises mutltisearcher when this class is creared by
>>>>>>>>>>>> spring.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> private synchronized void swapMultiSearcher(MultiSearcher
>>>>>>>>>>>> newMultiSearcher) {
>>>>>>>>>>>>
>>>>>>>>>>>> try {
>>>>>>>>>>>>
>>>>>>>>>>>> release(multiSearcher);
>>>>>>>>>>>>
>>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>>
>>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> multiSearcher = newMultiSearcher;
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> public void maybeReopen() throws IOException {
>>>>>>>>>>>>
>>>>>>>>>>>> MultiSearcher newMultiSeacher = null;
>>>>>>>>>>>>
>>>>>>>>>>>> boolean refreshMultiSeacher = false;
>>>>>>>>>>>>
>>>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>>
>>>>>>>>>>>> synchronized (searchers) {
>>>>>>>>>>>>
>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>
>>>>>>>>>>>> IndexReader reader = indexSearcher.getIndexReader();
>>>>>>>>>>>>
>>>>>>>>>>>> reader.incRef();
>>>>>>>>>>>>
>>>>>>>>>>>> Directory directory = reader.directory();
>>>>>>>>>>>>
>>>>>>>>>>>> long currentVersion = reader.getVersion();
>>>>>>>>>>>>
>>>>>>>>>>>> if (IndexReader.getCurrentVersion(directory) !=
>>>>>>>>>>>> currentVersion) {
>>>>>>>>>>>>
>>>>>>>>>>>> IndexReader newReader =
>>>>>>>>>>>> indexSearcher.getIndexReader().reopen();
>>>>>>>>>>>>
>>>>>>>>>>>> if (newReader != reader) {
>>>>>>>>>>>>
>>>>>>>>>>>> reader.decRef();
>>>>>>>>>>>>
>>>>>>>>>>>> refreshMultiSeacher = true;
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> reader = newReader;
>>>>>>>>>>>>
>>>>>>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>>>>>>
>>>>>>>>>>>> indexSearchers.add(newSearcher);
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> if (refreshMultiSeacher) {
>>>>>>>>>>>>
>>>>>>>>>>>> newMultiSeacher = new
>>>>>>>>>>>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[]
>>>>>>>>>>>> {}));
>>>>>>>>>>>>
>>>>>>>>>>>> warm(newMultiSeacher);
>>>>>>>>>>>>
>>>>>>>>>>>> swapMultiSearcher(newMultiSeacher);
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> private void warm(MultiSearcher newMultiSeacher) {
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>>
>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>
>>>>>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> private synchronized void release(MultiSearcher
>>>>>>>>>>>> multiSearcher)
>>>>>>>>>>>> throwsIOException {
>>>>>>>>>>>>
>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>
>>>>>>>>>>>> indexSearcher.getIndexReader().decRef();
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> However I am now getting
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> java.lang.IllegalStateException:
>>>>>>>>>>>> org.apache.lucene.store.AlreadyClosedException: this
>>>>>>>>>>>> IndexReader
>>>>>>>>>>>> is
>>>>>>>>>>>> closed
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> on the call:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>>
>>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>>
>>>>>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I'm doing something wrong ..obviously..not sure where
>>>>>>>>>>>> though..
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I was wondering the same thing ;)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> It's best to call this method from a single BG "warming"
>>>>>>>>>>>> thread,
>>>>>>>>>>>>> in
>>>>>>>>>>>>> which
>>>>>>>>>>>>> case it would not need its own synchronization.
>>>>>>>>>>>>>
>>>>>>>>>>>>> But, to be safe, I'll add internal synchronization to
>>>>>>>>>>>>> it. You
>>>>>>>>>>>>> can't
>>>>>>>>>>>>> simply put synchronized in front of the method, since
>>>>>>>>>>>>> you don't
>>>>>>>>>>>>> want
>>>>>>>>>>>>> this to
>>>>>>>>>>>>> block searching.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>
>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> just a quick point:
>>>>>>>>>>>>>
>>>>>>>>>>>>> public void maybeReopen() throws IOException {
>>>>>>>>>>>>> //D
>>>>>>>>>>>>>
>>>>>>>>>>>>> long currentVersion =
>>>>>>>>>>>>>
>>>>>>>>>>>>>> currentSearcher.getIndexReader().getVersion();
>>>>>>>>>>>>>> if (IndexReader.getCurrentVersion(dir) !=
>>>>>>>>>>>>>> currentVersion) {
>>>>>>>>>>>>>> IndexReader newReader =
>>>>>>>>>>>>>> currentSearcher.getIndexReader().reopen();
>>>>>>>>>>>>>> assert newReader != currentSearcher.getIndexReader();
>>>>>>>>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>>>>>>>> warm(newSearcher);
>>>>>>>>>>>>>> swapSearcher(newSearcher);
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> should the above be synchronised?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> thanks. i will rewrite..in between giving my baby her
>>>>>>>>>>>>>>> feed
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> playing
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> with the other child and my wife who wants me to do
>>>>>>>>>>>>>> several
>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> things!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for your input. I would like to have a go at
>>>>>>>>>>>>>>>> doing
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> first, Solr may be an option.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser
>>>>>>>>>>>>>>>>> every time,
>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>> creating unnecessary garbage; instead, they should be
>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -- I can moved the code out so that it is only
>>>>>>>>>>>>>>>>> created once
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> reused.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>>> when nothing has changed. This just generates
>>>>>>>>>>>>>>>>> unnecessary
>>>>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -- This was something I thought about. I could move
>>>>>>>>>>>>>>>>> it out
>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>> created once. However I presume inside my code i
>>>>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>> whether
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> indexreaders are update to date. This needs to be
>>>>>>>>>>>>>>>>> synchronized
>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> well I
>>>>>>>>>>>>>>>>> guess(?)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yes you should synchronize the check for whether the
>>>>>>>>>>>>>>>>> IndexReader
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> current.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * I don't see any synchronization -- it looks like
>>>>>>>>>>>>>>>> two search
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> requests are allowed into this method at the same time?
>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully
>>>>>>>>>>>>>>>> reopen the
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>>>>> -- So i need to extract the logic for reopening and
>>>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>> synchronisation mechanism.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yes.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Ok. So I have some work to do. I'll refactor the
>>>>>>>>>>>>>>>>> code and
>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> inline to your recommendations.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On a quick look, I think there are a few problems
>>>>>>>>>>>>>>>>> with the
>>>>>>>>>>>>>>>>> code:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * I don't see any synchronization -- it looks like two
>>>>>>>>>>>>>>>>> search
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> requests are allowed into this method at the same
>>>>>>>>>>>>>>>>> time?
>>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully
>>>>>>>>>>>>>>>>>> reopen the
>>>>>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * You are over-incRef'ing (the reader.incRef inside
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> loop)
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>> don't see a corresponding decRef.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * You reopen and warm your searchers "live" (vs
>>>>>>>>>>>>>>>>>> with BG
>>>>>>>>>>>>>>>>>> thread);
>>>>>>>>>>>>>>>>>> meaning the unlucky search request that hits a
>>>>>>>>>>>>>>>>>> reopen pays
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> cost. This might be OK if the index is small
>>>>>>>>>>>>>>>>>> enough that
>>>>>>>>>>>>>>>>>> reopening & warming takes very little time. But if
>>>>>>>>>>>>>>>>>> index
>>>>>>>>>>>>>>>>>> gets
>>>>>>>>>>>>>>>>>> large, making a random search pay that warming cost
>>>>>>>>>>>>>>>>>> is not
>>>>>>>>>>>>>>>>>> nice
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> the end user. It erodes their trust in you.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>>>> when nothing has changed. This just generates
>>>>>>>>>>>>>>>>>> unnecessary
>>>>>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser
>>>>>>>>>>>>>>>>>> every time,
>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>> creating unnecessary garbage; instead, they should be
>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> You should consider simply using Solr -- it handles
>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> logic
>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> you and has been well debugged with time...
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The reason for the indexreader.reopen is because I
>>>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>>>> webapp
>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> enables users to upload files and then search for the
>>>>>>>>>>>>>>>>>> documents.
>>>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> reopen i'm concerned that the facet hit counter
>>>>>>>>>>>>>>>>>>> won't be
>>>>>>>>>>>>>>>>>>> updated.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-
>>>>>>>>>>>>>>>>>>> Coleman <
>>>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I have been able to get the code working for my
>>>>>>>>>>>>>>>>>>>> scenario,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> however
>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> question and I was wondering if I could get some
>>>>>>>>>>>>>>>>>>>> help. I
>>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>> IndexSearchers which are used in a MultiSearcher
>>>>>>>>>>>>>>>>>>>> class.
>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> indexsearchers to get each indexreader and put
>>>>>>>>>>>>>>>>>>>> them into
>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> MultiIndexReader.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> IndexReader[] readers = new
>>>>>>>>>>>>>>>>>>>> IndexReader[searchables.length];
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> IndexSearcher indexSearcher =
>>>>>>>>>>>>>>>>>>>> (IndexSearcher)searchables[i];
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> readers[i].close();
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> readers[i] = newReader;
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>>>>>>>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> IndexSearcher indexSearcher = new
>>>>>>>>>>>>>>>>>>>> IndexSearcher(multiReader);
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I then use the indexseacher to do the facet
>>>>>>>>>>>>>>>>>>>> stuff. I end
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>> closing the multireader. This is causing
>>>>>>>>>>>>>>>>>>>> problems in
>>>>>>>>>>>>>>>>>>>> another
>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>> where I
>>>>>>>>>>>>>>>>>>>> do some other search as the indexreaders are
>>>>>>>>>>>>>>>>>>>> closed. Is
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>> close
>>>>>>>>>>>>>>>>>>>> the multiindexreader or should I do some additional
>>>>>>>>>>>>>>>>>>>> checks
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-
>>>>>>>>>>>>>>>>>>>> Coleman <
>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks just what I needed!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>>>>>>>>>>>>>>>>>>>>> marcelo.ochoa@gmail.com>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Amin:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>>>>>>>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-
>>>>>>>>>>>>>>>>>>>>>> Coleman
>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Sorry to re send this email but I was wondering
>>>>>>>>>>>>>>>>>>>>>> if I
>>>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> advice
>>>>>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-
>>>>>>>>>>>>>>>>>>>>>>> Coleman <
>>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I am looking at building a faceted search using
>>>>>>>>>>>>>>>>>>>>>>> Lucene.
>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Solr
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> comes with this built in, however I would
>>>>>>>>>>>>>>>>>>>>>>>> like to try
>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>>>>>>>> (something to add to my CV!). I have been
>>>>>>>>>>>>>>>>>>>>>>>> looking
>>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>> you can use the IndexReader and use
>>>>>>>>>>>>>>>>>>>>>>>> TermVectors.
>>>>>>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>> sure how to filter the results so that a
>>>>>>>>>>>>>>>>>>>>>>>> particular
>>>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>> subset of results. The next option I was
>>>>>>>>>>>>>>>>>>>>>>>> looking at
>>>>>>>>>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>>>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>>>>>>>>>>>>>> Term[] termsArray = new Term[] { term1,
>>>>>>>>>>>>>>>>>>>>>>>> term2 };un
>>>>>>>>>>>>>>>>>>>>>>>> int[] docFreqs =
>>>>>>>>>>>>>>>>>>>>>>>> indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> The only problem here is that I have to
>>>>>>>>>>>>>>>>>>>>>>>> provide the
>>>>>>>>>>>>>>>>>>>>>>>> brand
>>>>>>>>>>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>>>>>>>>> time a
>>>>>>>>>>>>>>>>>>>>>>>> new brand is created. Again I'm not sure how
>>>>>>>>>>>>>>>>>>>>>>>> I can
>>>>>>>>>>>>>>>>>>>>>>>> filter
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> results
>>>>>>>>>>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>>>>>>>>> It may be that I'm using the wrong api
>>>>>>>>>>>>>>>>>>>>>>>> methods to do
>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I would be grateful if I could get some
>>>>>>>>>>>>>>>>>>>>>>>> advice on
>>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> P.S. I am basically trying to do something
>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>> displays
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Personal Contact (23) Business Contact (45)
>>>>>>>>>>>>>>>>>>>>>>>> and so
>>>>>>>>>>>>>>>>>>>>>>>> on..
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>>>>>>>>>>>>>> ______________
>>>>>>>>>>>>>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>>>>>>>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>
>>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>
>>>>>>>>> To unsubscribe, e-mail: java-user-
>>>>>>>>> unsubscribe@lucene.apache.org
>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
I noticed that if i do the get() before the maybeReopen then I get no
results. But otherwise I can change it further.
On Mon, Mar 2, 2009 at 11:46 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:
>
> There is no such thing as final code -- code is alive and is always
> changing ;)
>
> It looks good to me.
>
> Though one trivial thing is: I would move the code in the try clause up to
> and including the multiSearcher=get() out above the try. I always attempt
> to "shrink wrap" what's inside a try clause to the minimum that needs to be
> there. Ie, your code that creates a query, finds the right sort & filter to
> use, etc, can all happen outside the try, because you have not yet acquired
> the multiSearcher.
>
> If you do that, you also don't need the null check in the finally clause,
> because multiSearcher must be non-null on entering the try.
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> Hi there
>> Good morning! Here is the final search code:
>>
>> public Summary[] search(final SearchRequest searchRequest)
>> throwsSearchExecutionException {
>>
>> final String searchTerm = searchRequest.getSearchTerm();
>>
>> if (StringUtils.isBlank(searchTerm)) {
>>
>> throw new SearchExecutionException("Search string cannot be empty. There
>> will be too many results to process.");
>>
>> }
>>
>> List<Summary> summaryList = new ArrayList<Summary>();
>>
>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>
>> stopWatch.start();
>>
>> MultiSearcher multiSearcher = null;
>>
>> try {
>>
>> LOGGER.debug("Ensuring all index readers are up to date...");
>>
>> maybeReopen();
>>
>> Query query = queryParser.parse(searchTerm);
>>
>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
>> query.toString() +"'");
>>
>> Sort sort = null;
>>
>> sort = applySortIfApplicable(searchRequest);
>>
>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>
>> ChainedFilter chainedFilter = null;
>>
>> if (filters != null) {
>>
>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>
>> }
>>
>> multiSearcher = get();
>>
>> TopDocs topDocs = multiSearcher.search(query,chainedFilter ,100,sort);
>>
>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>
>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>> "+topDocs.
>> totalHits);
>>
>> for (ScoreDoc scoreDoc : scoreDocs) {
>>
>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>
>> float score = scoreDoc.score;
>>
>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>
>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>
>> summaryList.add(documentSummary);
>>
>> }
>>
>> } catch (Exception e) {
>>
>> throw new IllegalStateException(e);
>>
>> } finally {
>>
>> if (multiSearcher != null) {
>>
>> release(multiSearcher);
>>
>> }
>>
>> }
>>
>> stopWatch.stop();
>>
>> LOGGER.debug("total time taken for document seach: " +
>> stopWatch.getTotalTimeMillis() + " ms");
>>
>> return summaryList.toArray(new Summary[] {});
>>
>> }
>>
>>
>>
>> I hope this makes sense...thanks again!
>>
>>
>> Cheers
>>
>> Amin
>>
>>
>>
>> On Sun, Mar 1, 2009 at 8:09 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>
>>> You're calling get() too many times. For every call to get() you must
>>> match with a call to release().
>>>
>>> So, once at the front of your search method you should:
>>>
>>> MultiSearcher searcher = get();
>>>
>>> then use that searcher to do searching, retrieve docs, etc.
>>>
>>> Then in the finally clause, pass that searcher to release.
>>>
>>> So, only one call to get() and one matching call to release().
>>>
>>> Mike
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> Hi
>>>
>>>> The searchers are injected into the class via Spring. So when a client
>>>> calls the class it is fully configured with a list of index searchers.
>>>> However I have removed this list and instead injecting a list of
>>>> directories which are passed to the DocumentSearchManager.
>>>> DocumentSearchManager is SearchManager (should've mentioned that
>>>> earlier).
>>>> So finally I have modified by release code to do the following:
>>>>
>>>> private void release(MultiSearcher multiSeacher) throws Exception {
>>>>
>>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>>> multiSeacher.getSearchables();
>>>>
>>>> for(int i =0 ; i < indexSearchers.length;i++) {
>>>>
>>>> documentSearcherManagers[i].release(indexSearchers[i]);
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> and it's use looks like this:
>>>>
>>>>
>>>> public Summary[] search(final SearchRequest searchRequest)
>>>> throwsSearchExecutionException {
>>>>
>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>
>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>
>>>> throw new SearchExecutionException("Search string cannot be empty. There
>>>> will be too many results to process.");
>>>>
>>>> }
>>>>
>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>
>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>
>>>> stopWatch.start();
>>>>
>>>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>>>
>>>> try {
>>>>
>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>
>>>> maybeReopen();
>>>>
>>>> LOGGER.debug("All Index Searchers are up to date. No of index searchers
>>>> '"
>>>> +
>>>> indexSearchers.size() +"'");
>>>>
>>>> Query query = queryParser.parse(searchTerm);
>>>>
>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
>>>> query.toString() +"'");
>>>>
>>>> Sort sort = null;
>>>>
>>>> sort = applySortIfApplicable(searchRequest);
>>>>
>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>
>>>> ChainedFilter chainedFilter = null;
>>>>
>>>> if (filters != null) {
>>>>
>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>
>>>> }
>>>>
>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>
>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>
>>>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>>>> "+topDocs.
>>>> totalHits);
>>>>
>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>
>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>
>>>> float score = scoreDoc.score;
>>>>
>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>
>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>
>>>> summaryList.add(documentSummary);
>>>>
>>>> }
>>>>
>>>> } catch (Exception e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> } finally {
>>>>
>>>> release(get());
>>>>
>>>> }
>>>>
>>>> stopWatch.stop();
>>>>
>>>> LOGGER.debug("total time taken for document seach: " +
>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>
>>>> return summaryList.toArray(new Summary[] {});
>>>>
>>>> }
>>>>
>>>>
>>>> So the final post construct constructs the DocumentSearchMangers with
>>>> the
>>>> list of directories..looking like this
>>>>
>>>>
>>>> @PostConstruct
>>>>
>>>> public void initialiseDocumentSearcher() {
>>>>
>>>> PerFieldAnalyzerWrapper analyzerWrapper = new PerFieldAnalyzerWrapper(
>>>> analyzer);
>>>>
>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>> newKeywordAnalyzer());
>>>>
>>>> queryParser =
>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>> analyzerWrapper);
>>>>
>>>> try {
>>>>
>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>
>>>> documentSearcherManagers = new
>>>> DocumentSearcherManager[directories.size()];
>>>>
>>>> for (int i = 0; i < directories.size() ;i++) {
>>>>
>>>> Directory directory = directories.get(i);
>>>>
>>>> DocumentSearcherManager documentSearcherManager =
>>>> newDocumentSearcherManager(directory);
>>>>
>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>
>>>> }
>>>>
>>>> LOGGER.debug("multi searcher initialised");
>>>>
>>>> } catch (IOException e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> Cheers
>>>>
>>>> Amin
>>>>
>>>>
>>>>
>>>> On Sun, Mar 1, 2009 at 6:15 PM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>>
>>>> I don't understand where searchers comes from, prior to
>>>>> initializeDocumentSearcher? You should, instead, simply create the
>>>>> SearcherManager (from your Directory instances). You don't need any
>>>>> searchers during initialize.
>>>>>
>>>>> Is DocumentSearcherManager the same as SearcherManager (just renamed)?
>>>>>
>>>>> The release method is wrong -- you're calling .get() and then
>>>>> immediately release. Instead, you should step through the searchers
>>>>> from your MultiSearcher and release them to each SearcherManager.
>>>>>
>>>>> You should call your release() in a finally clause.
>>>>>
>>>>> Mike
>>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>> Sorry...i'm getting slightly confused.
>>>>>
>>>>> I have a PostConstruct which is where I should create an array of
>>>>>> SearchManagers (per indexSeacher). From there I initialise the
>>>>>> multisearcher using the get(). After which I need to call maybeReopen
>>>>>> for
>>>>>> each IndexSearcher. So I'll do the following:
>>>>>>
>>>>>> @PostConstruct
>>>>>>
>>>>>> public void initialiseDocumentSearcher() {
>>>>>>
>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new PerFieldAnalyzerWrapper(
>>>>>> analyzer);
>>>>>>
>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>> newKeywordAnalyzer());
>>>>>>
>>>>>> queryParser =
>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>> analyzerWrapper);
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>
>>>>>> documentSearcherManagers = new
>>>>>> DocumentSearcherManager[searchers.size()];
>>>>>>
>>>>>> for (int i = 0; i < searchers.size() ;i++) {
>>>>>>
>>>>>> IndexSearcher indexSearcher = searchers.get(i);
>>>>>>
>>>>>> Directory directory = indexSearcher.getIndexReader().directory();
>>>>>>
>>>>>> DocumentSearcherManager documentSearcherManager =
>>>>>> newDocumentSearcherManager(directory);
>>>>>>
>>>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>>>
>>>>>> }
>>>>>>
>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>
>>>>>> } catch (IOException e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> This initialises search managers. I then have methods:
>>>>>>
>>>>>>
>>>>>> private void maybeReopen() throws Exception {
>>>>>>
>>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>>
>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>> documentSearcherManagers) {
>>>>>>
>>>>>> documentSearcherManager.maybeReopen();
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> private void release() throws Exception {
>>>>>>
>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>> documentSearcherManagers) {
>>>>>>
>>>>>> documentSearcherManager.release(documentSearcherManager.get());
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> private MultiSearcher get() {
>>>>>>
>>>>>> List<IndexSearcher> listOfIndexSeachers = new
>>>>>> ArrayList<IndexSearcher>();
>>>>>>
>>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>>> documentSearcherManagers) {
>>>>>>
>>>>>> listOfIndexSeachers.add(documentSearcherManager.get());
>>>>>>
>>>>>> }
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> multiSearcher = new
>>>>>> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[] {}));
>>>>>>
>>>>>> } catch (IOException e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> return multiSearcher;
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> These methods are used in the following manner in the search code:
>>>>>>
>>>>>>
>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>> throwsSearchExecutionException {
>>>>>>
>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>
>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>
>>>>>> throw new SearchExecutionException("Search string cannot be empty.
>>>>>> There
>>>>>> will be too many results to process.");
>>>>>>
>>>>>> }
>>>>>>
>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>
>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>
>>>>>> stopWatch.start();
>>>>>>
>>>>>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>
>>>>>> maybeReopen();
>>>>>>
>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>> searchers
>>>>>> '"
>>>>>> +
>>>>>> indexSearchers.size() +"'");
>>>>>>
>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>
>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
>>>>>> query.toString() +"'");
>>>>>>
>>>>>> Sort sort = null;
>>>>>>
>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>
>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>
>>>>>> ChainedFilter chainedFilter = null;
>>>>>>
>>>>>> if (filters != null) {
>>>>>>
>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>
>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>
>>>>>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>>>>>> "+topDocs.
>>>>>> totalHits);
>>>>>>
>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>
>>>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>>>
>>>>>> float score = scoreDoc.score;
>>>>>>
>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>
>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>
>>>>>> summaryList.add(documentSummary);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> release();
>>>>>>
>>>>>> } catch (Exception e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> stopWatch.stop();
>>>>>>
>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>
>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> Does this look better? Again..I really really appreciate your help!
>>>>>>
>>>>>>
>>>>>> On Sun, Mar 1, 2009 at 4:18 PM, Michael McCandless <
>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>
>>>>>>
>>>>>> This is not quite right -- you should only create SearcherManager once
>>>>>>
>>>>>>> (per Direcotry) at startup/app load, not with every search request.
>>>>>>>
>>>>>>> And I don't see release -- it must call SearcherManager.release of
>>>>>>> each of the IndexSearchers previously returned from get().
>>>>>>>
>>>>>>> Mike
>>>>>>>
>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> Thanks again for helping on a Sunday!
>>>>>>>
>>>>>>>>
>>>>>>>> I have now modified my maybeOpen() to do the following:
>>>>>>>>
>>>>>>>> private void maybeReopen() throws Exception {
>>>>>>>>
>>>>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>>>>
>>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>>>>>>> .getSearchables();
>>>>>>>>
>>>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>>>
>>>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>>>
>>>>>>>> SearcherManager documentSearcherManager = new
>>>>>>>> SearcherManager(indexReader.directory());
>>>>>>>>
>>>>>>>> documentSearcherManager.maybeReopen();
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> And get() to:
>>>>>>>>
>>>>>>>>
>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>
>>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>>>>>>> .getSearchables();
>>>>>>>>
>>>>>>>> List<IndexSearcher> indexSearchersList = new
>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>
>>>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>>>
>>>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>>>
>>>>>>>> SearcherManager documentSearcherManager = null;
>>>>>>>>
>>>>>>>> try {
>>>>>>>>
>>>>>>>> documentSearcherManager = new
>>>>>>>> SearcherManager(indexReader.directory());
>>>>>>>>
>>>>>>>> } catch (IOException e) {
>>>>>>>>
>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> indexSearchersList.add(documentSearcherManager.get());
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> try {
>>>>>>>>
>>>>>>>> multiSearcher = new
>>>>>>>> MultiSearcher(indexSearchersList.toArray(newIndexSearcher[] {}));
>>>>>>>>
>>>>>>>> } catch (IOException e) {
>>>>>>>>
>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> return multiSearcher;
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> This makes all my test pass. I am using the SearchManager that you
>>>>>>>> recommended. Does this look ok?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>
>>>>>>>> Your maybeReopen has an excess incRef().
>>>>>>>>
>>>>>>>>
>>>>>>>> I'm not sure how you open the searchers in the first place? The
>>>>>>>>> list
>>>>>>>>> starts as empty, and nothing populates it?
>>>>>>>>>
>>>>>>>>> When you do the initial population, you need an incRef.
>>>>>>>>>
>>>>>>>>> I think you're hitting IllegalStateException because maybeReopen is
>>>>>>>>> closing a reader before get() can get it (since they synchronize on
>>>>>>>>> different objects).
>>>>>>>>>
>>>>>>>>> I'd recommend switching to the SearcherManager class. Instantiate
>>>>>>>>> one
>>>>>>>>> for each of your searchers. On each search request, go through
>>>>>>>>> them
>>>>>>>>> and call maybeReopen(), and then call get() and gather each
>>>>>>>>> IndexSearcher instance into a new array. Then, make a new
>>>>>>>>> MultiSearcher (opposite of what I said before): while that creates
>>>>>>>>> a
>>>>>>>>> small amount of garbage, it'll keep your code simpler (good
>>>>>>>>> tradeoff).
>>>>>>>>>
>>>>>>>>> Mike
>>>>>>>>>
>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>
>>>>>>>>> sorrry I added
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> release(multiSearcher);
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> instead of multiSearcher.close();
>>>>>>>>>>
>>>>>>>>>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <
>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I've now done the following:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>>>>> throwsSearchExecutionException {
>>>>>>>>>>>
>>>>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>>>>
>>>>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>>>>
>>>>>>>>>>> throw new SearchExecutionException("Search string cannot be
>>>>>>>>>>> empty.
>>>>>>>>>>> There
>>>>>>>>>>> will be too many results to process.");
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>>>
>>>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>>>
>>>>>>>>>>> stopWatch.start();
>>>>>>>>>>>
>>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>
>>>>>>>>>>> try {
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>>>>>
>>>>>>>>>>> maybeReopen();
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>>>>>>> searchers
>>>>>>>>>>> '"+ indexSearchers.size() +
>>>>>>>>>>> "'");
>>>>>>>>>>>
>>>>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query
>>>>>>>>>>> '"
>>>>>>>>>>> +
>>>>>>>>>>> query.toString() +"'");
>>>>>>>>>>>
>>>>>>>>>>> Sort sort = null;
>>>>>>>>>>>
>>>>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>>>>
>>>>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>>>>
>>>>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>>>>
>>>>>>>>>>> if (filters != null) {
>>>>>>>>>>>
>>>>>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>>>>>>
>>>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("total number of hits for [" + query.toString() + "
>>>>>>>>>>> ]
>>>>>>>>>>> =
>>>>>>>>>>> "+topDocs.
>>>>>>>>>>> totalHits);
>>>>>>>>>>>
>>>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>>>
>>>>>>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>>>>>>
>>>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>>>
>>>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>>>>>>
>>>>>>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>>>>>>
>>>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> multiSearcher.close();
>>>>>>>>>>>
>>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>>
>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> stopWatch.stop();
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>>>
>>>>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> And have the following methods:
>>>>>>>>>>>
>>>>>>>>>>> @PostConstruct
>>>>>>>>>>>
>>>>>>>>>>> public void initialiseQueryParser() {
>>>>>>>>>>>
>>>>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>>>>> analyzer);
>>>>>>>>>>>
>>>>>>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>>>>> newKeywordAnalyzer());
>>>>>>>>>>>
>>>>>>>>>>> queryParser =
>>>>>>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>>>>
>>>>>>>>>>> analyzerWrapper);
>>>>>>>>>>>
>>>>>>>>>>> try {
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>>>>
>>>>>>>>>>> this.multiSearcher = new
>>>>>>>>>>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>>>>>>>>>>
>>>>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>>>>
>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>
>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Initialises mutltisearcher when this class is creared by spring.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> private synchronized void swapMultiSearcher(MultiSearcher
>>>>>>>>>>> newMultiSearcher) {
>>>>>>>>>>>
>>>>>>>>>>> try {
>>>>>>>>>>>
>>>>>>>>>>> release(multiSearcher);
>>>>>>>>>>>
>>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>>
>>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> multiSearcher = newMultiSearcher;
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> public void maybeReopen() throws IOException {
>>>>>>>>>>>
>>>>>>>>>>> MultiSearcher newMultiSeacher = null;
>>>>>>>>>>>
>>>>>>>>>>> boolean refreshMultiSeacher = false;
>>>>>>>>>>>
>>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>>
>>>>>>>>>>> synchronized (searchers) {
>>>>>>>>>>>
>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>
>>>>>>>>>>> IndexReader reader = indexSearcher.getIndexReader();
>>>>>>>>>>>
>>>>>>>>>>> reader.incRef();
>>>>>>>>>>>
>>>>>>>>>>> Directory directory = reader.directory();
>>>>>>>>>>>
>>>>>>>>>>> long currentVersion = reader.getVersion();
>>>>>>>>>>>
>>>>>>>>>>> if (IndexReader.getCurrentVersion(directory) != currentVersion) {
>>>>>>>>>>>
>>>>>>>>>>> IndexReader newReader = indexSearcher.getIndexReader().reopen();
>>>>>>>>>>>
>>>>>>>>>>> if (newReader != reader) {
>>>>>>>>>>>
>>>>>>>>>>> reader.decRef();
>>>>>>>>>>>
>>>>>>>>>>> refreshMultiSeacher = true;
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> reader = newReader;
>>>>>>>>>>>
>>>>>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>>>>>
>>>>>>>>>>> indexSearchers.add(newSearcher);
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> if (refreshMultiSeacher) {
>>>>>>>>>>>
>>>>>>>>>>> newMultiSeacher = new
>>>>>>>>>>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
>>>>>>>>>>>
>>>>>>>>>>> warm(newMultiSeacher);
>>>>>>>>>>>
>>>>>>>>>>> swapMultiSearcher(newMultiSeacher);
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> private void warm(MultiSearcher newMultiSeacher) {
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>
>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>
>>>>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> private synchronized void release(MultiSearcher multiSearcher)
>>>>>>>>>>> throwsIOException {
>>>>>>>>>>>
>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>
>>>>>>>>>>> indexSearcher.getIndexReader().decRef();
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> However I am now getting
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> java.lang.IllegalStateException:
>>>>>>>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexReader
>>>>>>>>>>> is
>>>>>>>>>>> closed
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> on the call:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>>
>>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>>
>>>>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> return multiSearcher;
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I'm doing something wrong ..obviously..not sure where though..
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Cheers
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I was wondering the same thing ;)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> It's best to call this method from a single BG "warming" thread,
>>>>>>>>>>>> in
>>>>>>>>>>>> which
>>>>>>>>>>>> case it would not need its own synchronization.
>>>>>>>>>>>>
>>>>>>>>>>>> But, to be safe, I'll add internal synchronization to it. You
>>>>>>>>>>>> can't
>>>>>>>>>>>> simply put synchronized in front of the method, since you don't
>>>>>>>>>>>> want
>>>>>>>>>>>> this to
>>>>>>>>>>>> block searching.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Mike
>>>>>>>>>>>>
>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> just a quick point:
>>>>>>>>>>>>
>>>>>>>>>>>> public void maybeReopen() throws IOException {
>>>>>>>>>>>> //D
>>>>>>>>>>>>
>>>>>>>>>>>> long currentVersion =
>>>>>>>>>>>>
>>>>>>>>>>>>> currentSearcher.getIndexReader().getVersion();
>>>>>>>>>>>>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>>>>>>>>>>>>> IndexReader newReader =
>>>>>>>>>>>>> currentSearcher.getIndexReader().reopen();
>>>>>>>>>>>>> assert newReader != currentSearcher.getIndexReader();
>>>>>>>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>>>>>>> warm(newSearcher);
>>>>>>>>>>>>> swapSearcher(newSearcher);
>>>>>>>>>>>>> }
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> should the above be synchronised?
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> thanks. i will rewrite..in between giving my baby her feed
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> playing
>>>>>>>>>>>>>
>>>>>>>>>>>>> with the other child and my wife who wants me to do several
>>>>>>>>>>>>> other
>>>>>>>>>>>>>
>>>>>>>>>>>>> things!
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for your input. I would like to have a go at doing
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> first, Solr may be an option.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser every time,
>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>> creating unnecessary garbage; instead, they should be
>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -- I can moved the code out so that it is only created once
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> reused.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>> when nothing has changed. This just generates unnecessary
>>>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -- This was something I thought about. I could move it out
>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>> created once. However I presume inside my code i need to
>>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>> whether
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> indexreaders are update to date. This needs to be
>>>>>>>>>>>>>>>> synchronized
>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>> well I
>>>>>>>>>>>>>>>> guess(?)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes you should synchronize the check for whether the
>>>>>>>>>>>>>>>> IndexReader
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> current.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * I don't see any synchronization -- it looks like two search
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> requests are allowed into this method at the same time?
>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>>>> -- So i need to extract the logic for reopening and provide
>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> synchronisation mechanism.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ok. So I have some work to do. I'll refactor the code and
>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> inline to your recommendations.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On a quick look, I think there are a few problems with the
>>>>>>>>>>>>>>>> code:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * I don't see any synchronization -- it looks like two
>>>>>>>>>>>>>>>> search
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> requests are allowed into this method at the same time?
>>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * You are over-incRef'ing (the reader.incRef inside the
>>>>>>>>>>>>>>>>> loop)
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>> don't see a corresponding decRef.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * You reopen and warm your searchers "live" (vs with BG
>>>>>>>>>>>>>>>>> thread);
>>>>>>>>>>>>>>>>> meaning the unlucky search request that hits a reopen pays
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> cost. This might be OK if the index is small enough that
>>>>>>>>>>>>>>>>> reopening & warming takes very little time. But if index
>>>>>>>>>>>>>>>>> gets
>>>>>>>>>>>>>>>>> large, making a random search pay that warming cost is not
>>>>>>>>>>>>>>>>> nice
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> the end user. It erodes their trust in you.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>>> when nothing has changed. This just generates unnecessary
>>>>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser every time,
>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>> creating unnecessary garbage; instead, they should be
>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> You should consider simply using Solr -- it handles all
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> logic
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> you and has been well debugged with time...
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The reason for the indexreader.reopen is because I have a
>>>>>>>>>>>>>>>>> webapp
>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> enables users to upload files and then search for the
>>>>>>>>>>>>>>>>> documents.
>>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> reopen i'm concerned that the facet hit counter won't be
>>>>>>>>>>>>>>>>>> updated.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I have been able to get the code working for my scenario,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> however
>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> question and I was wondering if I could get some help. I
>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>> IndexSearchers which are used in a MultiSearcher class.
>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> indexsearchers to get each indexreader and put them into
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> MultiIndexReader.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> IndexReader[] readers = new
>>>>>>>>>>>>>>>>>>> IndexReader[searchables.length];
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> IndexSearcher indexSearcher =
>>>>>>>>>>>>>>>>>>> (IndexSearcher)searchables[i];
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> readers[i].close();
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> readers[i] = newReader;
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>>>>>>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> IndexSearcher indexSearcher = new
>>>>>>>>>>>>>>>>>>> IndexSearcher(multiReader);
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I then use the indexseacher to do the facet stuff. I end
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>> closing the multireader. This is causing problems in
>>>>>>>>>>>>>>>>>>> another
>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>> where I
>>>>>>>>>>>>>>>>>>> do some other search as the indexreaders are closed. Is
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>> close
>>>>>>>>>>>>>>>>>>> the multiindexreader or should I do some additional
>>>>>>>>>>>>>>>>>>> checks
>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks just what I needed!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>>>>>>>>>>>>>>>>>>>> marcelo.ochoa@gmail.com>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Amin:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>>>>>>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman
>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Sorry to re send this email but I was wondering if I
>>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> advice
>>>>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I am looking at building a faceted search using
>>>>>>>>>>>>>>>>>>>>>> Lucene.
>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Solr
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> comes with this built in, however I would like to try
>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>>>>>>> (something to add to my CV!). I have been looking
>>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>> you can use the IndexReader and use TermVectors.
>>>>>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>> sure how to filter the results so that a particular
>>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>> subset of results. The next option I was looking at
>>>>>>>>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>>>>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>>>>>>>>>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> The only problem here is that I have to provide the
>>>>>>>>>>>>>>>>>>>>>>> brand
>>>>>>>>>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>>>>>>>> time a
>>>>>>>>>>>>>>>>>>>>>>> new brand is created. Again I'm not sure how I can
>>>>>>>>>>>>>>>>>>>>>>> filter
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> results
>>>>>>>>>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>>>>>>>> It may be that I'm using the wrong api methods to do
>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I would be grateful if I could get some advice on
>>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> P.S. I am basically trying to do something that
>>>>>>>>>>>>>>>>>>>>>>> displays
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Personal Contact (23) Business Contact (45) and so
>>>>>>>>>>>>>>>>>>>>>>> on..
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>>>>>>>>>>>>> ______________
>>>>>>>>>>>>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>>>>>>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>
>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>
>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>
>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Faceted Search using Lucene
Posted by Michael McCandless <lu...@mikemccandless.com>.
There is no such thing as final code -- code is alive and is always
changing ;)
It looks good to me.
Though one trivial thing is: I would move the code in the try clause
up to and including the multiSearcher=get() out above the try. I
always attempt to "shrink wrap" what's inside a try clause to the
minimum that needs to be there. Ie, your code that creates a query,
finds the right sort & filter to use, etc, can all happen outside the
try, because you have not yet acquired the multiSearcher.
If you do that, you also don't need the null check in the finally
clause, because multiSearcher must be non-null on entering the try.
Mike
Amin Mohammed-Coleman wrote:
> Hi there
> Good morning! Here is the final search code:
>
> public Summary[] search(final SearchRequest searchRequest)
> throwsSearchExecutionException {
>
> final String searchTerm = searchRequest.getSearchTerm();
>
> if (StringUtils.isBlank(searchTerm)) {
>
> throw new SearchExecutionException("Search string cannot be empty.
> There
> will be too many results to process.");
>
> }
>
> List<Summary> summaryList = new ArrayList<Summary>();
>
> StopWatch stopWatch = new StopWatch("searchStopWatch");
>
> stopWatch.start();
>
> MultiSearcher multiSearcher = null;
>
> try {
>
> LOGGER.debug("Ensuring all index readers are up to date...");
>
> maybeReopen();
>
> Query query = queryParser.parse(searchTerm);
>
> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
> query.toString() +"'");
>
> Sort sort = null;
>
> sort = applySortIfApplicable(searchRequest);
>
> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>
> ChainedFilter chainedFilter = null;
>
> if (filters != null) {
>
> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>
> }
>
> multiSearcher = get();
>
> TopDocs topDocs = multiSearcher.search(query,chainedFilter ,100,sort);
>
> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>
> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
> "+topDocs.
> totalHits);
>
> for (ScoreDoc scoreDoc : scoreDocs) {
>
> final Document doc = multiSearcher.doc(scoreDoc.doc);
>
> float score = scoreDoc.score;
>
> final BaseDocument baseDocument = new BaseDocument(doc, score);
>
> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>
> summaryList.add(documentSummary);
>
> }
>
> } catch (Exception e) {
>
> throw new IllegalStateException(e);
>
> } finally {
>
> if (multiSearcher != null) {
>
> release(multiSearcher);
>
> }
>
> }
>
> stopWatch.stop();
>
> LOGGER.debug("total time taken for document seach: " +
> stopWatch.getTotalTimeMillis() + " ms");
>
> return summaryList.toArray(new Summary[] {});
>
> }
>
>
>
> I hope this makes sense...thanks again!
>
>
> Cheers
>
> Amin
>
>
>
> On Sun, Mar 1, 2009 at 8:09 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>>
>> You're calling get() too many times. For every call to get() you
>> must
>> match with a call to release().
>>
>> So, once at the front of your search method you should:
>>
>> MultiSearcher searcher = get();
>>
>> then use that searcher to do searching, retrieve docs, etc.
>>
>> Then in the finally clause, pass that searcher to release.
>>
>> So, only one call to get() and one matching call to release().
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> Hi
>>> The searchers are injected into the class via Spring. So when a
>>> client
>>> calls the class it is fully configured with a list of index
>>> searchers.
>>> However I have removed this list and instead injecting a list of
>>> directories which are passed to the DocumentSearchManager.
>>> DocumentSearchManager is SearchManager (should've mentioned that
>>> earlier).
>>> So finally I have modified by release code to do the following:
>>>
>>> private void release(MultiSearcher multiSeacher) throws Exception {
>>>
>>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>>> multiSeacher.getSearchables();
>>>
>>> for(int i =0 ; i < indexSearchers.length;i++) {
>>>
>>> documentSearcherManagers[i].release(indexSearchers[i]);
>>>
>>> }
>>>
>>> }
>>>
>>>
>>> and it's use looks like this:
>>>
>>>
>>> public Summary[] search(final SearchRequest searchRequest)
>>> throwsSearchExecutionException {
>>>
>>> final String searchTerm = searchRequest.getSearchTerm();
>>>
>>> if (StringUtils.isBlank(searchTerm)) {
>>>
>>> throw new SearchExecutionException("Search string cannot be empty.
>>> There
>>> will be too many results to process.");
>>>
>>> }
>>>
>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>
>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>
>>> stopWatch.start();
>>>
>>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>>
>>> try {
>>>
>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>
>>> maybeReopen();
>>>
>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>> searchers '"
>>> +
>>> indexSearchers.size() +"'");
>>>
>>> Query query = queryParser.parse(searchTerm);
>>>
>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query
>>> '" +
>>> query.toString() +"'");
>>>
>>> Sort sort = null;
>>>
>>> sort = applySortIfApplicable(searchRequest);
>>>
>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>
>>> ChainedFilter chainedFilter = null;
>>>
>>> if (filters != null) {
>>>
>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>
>>> }
>>>
>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>
>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>
>>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>>> "+topDocs.
>>> totalHits);
>>>
>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>
>>> final Document doc = get().doc(scoreDoc.doc);
>>>
>>> float score = scoreDoc.score;
>>>
>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>
>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>
>>> summaryList.add(documentSummary);
>>>
>>> }
>>>
>>> } catch (Exception e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> } finally {
>>>
>>> release(get());
>>>
>>> }
>>>
>>> stopWatch.stop();
>>>
>>> LOGGER.debug("total time taken for document seach: " +
>>> stopWatch.getTotalTimeMillis() + " ms");
>>>
>>> return summaryList.toArray(new Summary[] {});
>>>
>>> }
>>>
>>>
>>> So the final post construct constructs the DocumentSearchMangers
>>> with the
>>> list of directories..looking like this
>>>
>>>
>>> @PostConstruct
>>>
>>> public void initialiseDocumentSearcher() {
>>>
>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>> PerFieldAnalyzerWrapper(
>>> analyzer);
>>>
>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>> newKeywordAnalyzer());
>>>
>>> queryParser =
>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>> analyzerWrapper);
>>>
>>> try {
>>>
>>> LOGGER.debug("Initialising multi searcher ....");
>>>
>>> documentSearcherManagers = new
>>> DocumentSearcherManager[directories.size()];
>>>
>>> for (int i = 0; i < directories.size() ;i++) {
>>>
>>> Directory directory = directories.get(i);
>>>
>>> DocumentSearcherManager documentSearcherManager =
>>> newDocumentSearcherManager(directory);
>>>
>>> documentSearcherManagers[i]=documentSearcherManager;
>>>
>>> }
>>>
>>> LOGGER.debug("multi searcher initialised");
>>>
>>> } catch (IOException e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> }
>>>
>>> }
>>>
>>>
>>>
>>> Cheers
>>>
>>> Amin
>>>
>>>
>>>
>>> On Sun, Mar 1, 2009 at 6:15 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>>> I don't understand where searchers comes from, prior to
>>>> initializeDocumentSearcher? You should, instead, simply create the
>>>> SearcherManager (from your Directory instances). You don't need
>>>> any
>>>> searchers during initialize.
>>>>
>>>> Is DocumentSearcherManager the same as SearcherManager (just
>>>> renamed)?
>>>>
>>>> The release method is wrong -- you're calling .get() and then
>>>> immediately release. Instead, you should step through the
>>>> searchers
>>>> from your MultiSearcher and release them to each SearcherManager.
>>>>
>>>> You should call your release() in a finally clause.
>>>>
>>>> Mike
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> Sorry...i'm getting slightly confused.
>>>>
>>>>> I have a PostConstruct which is where I should create an array of
>>>>> SearchManagers (per indexSeacher). From there I initialise the
>>>>> multisearcher using the get(). After which I need to call
>>>>> maybeReopen
>>>>> for
>>>>> each IndexSearcher. So I'll do the following:
>>>>>
>>>>> @PostConstruct
>>>>>
>>>>> public void initialiseDocumentSearcher() {
>>>>>
>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>> PerFieldAnalyzerWrapper(
>>>>> analyzer);
>>>>>
>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>> newKeywordAnalyzer());
>>>>>
>>>>> queryParser =
>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>> analyzerWrapper);
>>>>>
>>>>> try {
>>>>>
>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>
>>>>> documentSearcherManagers = new
>>>>> DocumentSearcherManager[searchers.size()];
>>>>>
>>>>> for (int i = 0; i < searchers.size() ;i++) {
>>>>>
>>>>> IndexSearcher indexSearcher = searchers.get(i);
>>>>>
>>>>> Directory directory = indexSearcher.getIndexReader().directory();
>>>>>
>>>>> DocumentSearcherManager documentSearcherManager =
>>>>> newDocumentSearcherManager(directory);
>>>>>
>>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>>
>>>>> }
>>>>>
>>>>> LOGGER.debug("multi searcher initialised");
>>>>>
>>>>> } catch (IOException e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> This initialises search managers. I then have methods:
>>>>>
>>>>>
>>>>> private void maybeReopen() throws Exception {
>>>>>
>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>
>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>> documentSearcherManagers) {
>>>>>
>>>>> documentSearcherManager.maybeReopen();
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> private void release() throws Exception {
>>>>>
>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>> documentSearcherManagers) {
>>>>>
>>>>> documentSearcherManager.release(documentSearcherManager.get());
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> private MultiSearcher get() {
>>>>>
>>>>> List<IndexSearcher> listOfIndexSeachers = new
>>>>> ArrayList<IndexSearcher>();
>>>>>
>>>>> for (DocumentSearcherManager documentSearcherManager :
>>>>> documentSearcherManagers) {
>>>>>
>>>>> listOfIndexSeachers.add(documentSearcherManager.get());
>>>>>
>>>>> }
>>>>>
>>>>> try {
>>>>>
>>>>> multiSearcher = new
>>>>> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[] {}));
>>>>>
>>>>> } catch (IOException e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> }
>>>>>
>>>>> return multiSearcher;
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> These methods are used in the following manner in the search code:
>>>>>
>>>>>
>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>> throwsSearchExecutionException {
>>>>>
>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>
>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>
>>>>> throw new SearchExecutionException("Search string cannot be
>>>>> empty. There
>>>>> will be too many results to process.");
>>>>>
>>>>> }
>>>>>
>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>
>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>
>>>>> stopWatch.start();
>>>>>
>>>>> List<IndexSearcher> indexSearchers = new
>>>>> ArrayList<IndexSearcher>();
>>>>>
>>>>> try {
>>>>>
>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>
>>>>> maybeReopen();
>>>>>
>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>> searchers
>>>>> '"
>>>>> +
>>>>> indexSearchers.size() +"'");
>>>>>
>>>>> Query query = queryParser.parse(searchTerm);
>>>>>
>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query
>>>>> '" +
>>>>> query.toString() +"'");
>>>>>
>>>>> Sort sort = null;
>>>>>
>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>
>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>
>>>>> ChainedFilter chainedFilter = null;
>>>>>
>>>>> if (filters != null) {
>>>>>
>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>
>>>>> }
>>>>>
>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>
>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>
>>>>> LOGGER.debug("total number of hits for [" + query.toString() +
>>>>> " ] =
>>>>> "+topDocs.
>>>>> totalHits);
>>>>>
>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>
>>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>>
>>>>> float score = scoreDoc.score;
>>>>>
>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>
>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>
>>>>> summaryList.add(documentSummary);
>>>>>
>>>>> }
>>>>>
>>>>> release();
>>>>>
>>>>> } catch (Exception e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> }
>>>>>
>>>>> stopWatch.stop();
>>>>>
>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>
>>>>> return summaryList.toArray(new Summary[] {});
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> Does this look better? Again..I really really appreciate your
>>>>> help!
>>>>>
>>>>>
>>>>> On Sun, Mar 1, 2009 at 4:18 PM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>>
>>>>> This is not quite right -- you should only create
>>>>> SearcherManager once
>>>>>> (per Direcotry) at startup/app load, not with every search
>>>>>> request.
>>>>>>
>>>>>> And I don't see release -- it must call SearcherManager.release
>>>>>> of
>>>>>> each of the IndexSearchers previously returned from get().
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> Thanks again for helping on a Sunday!
>>>>>>>
>>>>>>> I have now modified my maybeOpen() to do the following:
>>>>>>>
>>>>>>> private void maybeReopen() throws Exception {
>>>>>>>
>>>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>>>
>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>>>>>> .getSearchables();
>>>>>>>
>>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>>
>>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>>
>>>>>>> SearcherManager documentSearcherManager = new
>>>>>>> SearcherManager(indexReader.directory());
>>>>>>>
>>>>>>> documentSearcherManager.maybeReopen();
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> And get() to:
>>>>>>>
>>>>>>>
>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>
>>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>>>>>> .getSearchables();
>>>>>>>
>>>>>>> List<IndexSearcher> indexSearchersList = new
>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>
>>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>>
>>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>>
>>>>>>> SearcherManager documentSearcherManager = null;
>>>>>>>
>>>>>>> try {
>>>>>>>
>>>>>>> documentSearcherManager = new
>>>>>>> SearcherManager(indexReader.directory());
>>>>>>>
>>>>>>> } catch (IOException e) {
>>>>>>>
>>>>>>> throw new IllegalStateException(e);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> indexSearchersList.add(documentSearcherManager.get());
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> try {
>>>>>>>
>>>>>>> multiSearcher = new
>>>>>>> MultiSearcher(indexSearchersList.toArray(newIndexSearcher[]
>>>>>>> {}));
>>>>>>>
>>>>>>> } catch (IOException e) {
>>>>>>>
>>>>>>> throw new IllegalStateException(e);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> return multiSearcher;
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This makes all my test pass. I am using the SearchManager
>>>>>>> that you
>>>>>>> recommended. Does this look ok?
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>
>>>>>>> Your maybeReopen has an excess incRef().
>>>>>>>
>>>>>>>
>>>>>>>> I'm not sure how you open the searchers in the first place?
>>>>>>>> The list
>>>>>>>> starts as empty, and nothing populates it?
>>>>>>>>
>>>>>>>> When you do the initial population, you need an incRef.
>>>>>>>>
>>>>>>>> I think you're hitting IllegalStateException because
>>>>>>>> maybeReopen is
>>>>>>>> closing a reader before get() can get it (since they
>>>>>>>> synchronize on
>>>>>>>> different objects).
>>>>>>>>
>>>>>>>> I'd recommend switching to the SearcherManager class.
>>>>>>>> Instantiate
>>>>>>>> one
>>>>>>>> for each of your searchers. On each search request, go
>>>>>>>> through them
>>>>>>>> and call maybeReopen(), and then call get() and gather each
>>>>>>>> IndexSearcher instance into a new array. Then, make a new
>>>>>>>> MultiSearcher (opposite of what I said before): while that
>>>>>>>> creates a
>>>>>>>> small amount of garbage, it'll keep your code simpler (good
>>>>>>>> tradeoff).
>>>>>>>>
>>>>>>>> Mike
>>>>>>>>
>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>
>>>>>>>> sorrry I added
>>>>>>>>
>>>>>>>>
>>>>>>>> release(multiSearcher);
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> instead of multiSearcher.close();
>>>>>>>>>
>>>>>>>>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <
>>>>>>>>> aminmc@gmail.com
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> I've now done the following:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>>>> throwsSearchExecutionException {
>>>>>>>>>>
>>>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>>>
>>>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>>>
>>>>>>>>>> throw new SearchExecutionException("Search string cannot be
>>>>>>>>>> empty.
>>>>>>>>>> There
>>>>>>>>>> will be too many results to process.");
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>>
>>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>>
>>>>>>>>>> stopWatch.start();
>>>>>>>>>>
>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>
>>>>>>>>>> try {
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>>>>
>>>>>>>>>> maybeReopen();
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>>>>>> searchers
>>>>>>>>>> '"+ indexSearchers.size() +
>>>>>>>>>> "'");
>>>>>>>>>>
>>>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>>>>>>>>>> Query '"
>>>>>>>>>> +
>>>>>>>>>> query.toString() +"'");
>>>>>>>>>>
>>>>>>>>>> Sort sort = null;
>>>>>>>>>>
>>>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>>>
>>>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>>>
>>>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>>>
>>>>>>>>>> if (filters != null) {
>>>>>>>>>>
>>>>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,
>>>>>>>>>> 100,sort);
>>>>>>>>>>
>>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("total number of hits for [" +
>>>>>>>>>> query.toString() + " ]
>>>>>>>>>> =
>>>>>>>>>> "+topDocs.
>>>>>>>>>> totalHits);
>>>>>>>>>>
>>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>>
>>>>>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>>>>>
>>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>>
>>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc,
>>>>>>>>>> score);
>>>>>>>>>>
>>>>>>>>>> Summary documentSummary = new
>>>>>>>>>> DocumentSummaryImpl(baseDocument);
>>>>>>>>>>
>>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> multiSearcher.close();
>>>>>>>>>>
>>>>>>>>>> } catch (Exception e) {
>>>>>>>>>>
>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> stopWatch.stop();
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>>
>>>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> And have the following methods:
>>>>>>>>>>
>>>>>>>>>> @PostConstruct
>>>>>>>>>>
>>>>>>>>>> public void initialiseQueryParser() {
>>>>>>>>>>
>>>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>>>> analyzer);
>>>>>>>>>>
>>>>>>>>>> analyzerWrapper
>>>>>>>>>> .addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>>>> newKeywordAnalyzer());
>>>>>>>>>>
>>>>>>>>>> queryParser =
>>>>>>>>>> newMultiFieldQueryParser
>>>>>>>>>> (FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>>>
>>>>>>>>>> analyzerWrapper);
>>>>>>>>>>
>>>>>>>>>> try {
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>>>
>>>>>>>>>> this.multiSearcher = new
>>>>>>>>>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>>>>>>>>>
>>>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>>>
>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>
>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Initialises mutltisearcher when this class is creared by
>>>>>>>>>> spring.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> private synchronized void swapMultiSearcher(MultiSearcher
>>>>>>>>>> newMultiSearcher) {
>>>>>>>>>>
>>>>>>>>>> try {
>>>>>>>>>>
>>>>>>>>>> release(multiSearcher);
>>>>>>>>>>
>>>>>>>>>> } catch (IOException e) {
>>>>>>>>>>
>>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> multiSearcher = newMultiSearcher;
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> public void maybeReopen() throws IOException {
>>>>>>>>>>
>>>>>>>>>> MultiSearcher newMultiSeacher = null;
>>>>>>>>>>
>>>>>>>>>> boolean refreshMultiSeacher = false;
>>>>>>>>>>
>>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>>
>>>>>>>>>> synchronized (searchers) {
>>>>>>>>>>
>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>
>>>>>>>>>> IndexReader reader = indexSearcher.getIndexReader();
>>>>>>>>>>
>>>>>>>>>> reader.incRef();
>>>>>>>>>>
>>>>>>>>>> Directory directory = reader.directory();
>>>>>>>>>>
>>>>>>>>>> long currentVersion = reader.getVersion();
>>>>>>>>>>
>>>>>>>>>> if (IndexReader.getCurrentVersion(directory) !=
>>>>>>>>>> currentVersion) {
>>>>>>>>>>
>>>>>>>>>> IndexReader newReader =
>>>>>>>>>> indexSearcher.getIndexReader().reopen();
>>>>>>>>>>
>>>>>>>>>> if (newReader != reader) {
>>>>>>>>>>
>>>>>>>>>> reader.decRef();
>>>>>>>>>>
>>>>>>>>>> refreshMultiSeacher = true;
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> reader = newReader;
>>>>>>>>>>
>>>>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>>>>
>>>>>>>>>> indexSearchers.add(newSearcher);
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> if (refreshMultiSeacher) {
>>>>>>>>>>
>>>>>>>>>> newMultiSeacher = new
>>>>>>>>>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
>>>>>>>>>>
>>>>>>>>>> warm(newMultiSeacher);
>>>>>>>>>>
>>>>>>>>>> swapMultiSearcher(newMultiSeacher);
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> private void warm(MultiSearcher newMultiSeacher) {
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>
>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>
>>>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> return multiSearcher;
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> private synchronized void release(MultiSearcher
>>>>>>>>>> multiSearcher)
>>>>>>>>>> throwsIOException {
>>>>>>>>>>
>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>
>>>>>>>>>> indexSearcher.getIndexReader().decRef();
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> However I am now getting
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> java.lang.IllegalStateException:
>>>>>>>>>> org.apache.lucene.store.AlreadyClosedException: this
>>>>>>>>>> IndexReader is
>>>>>>>>>> closed
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> on the call:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>>
>>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>>
>>>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> return multiSearcher;
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I'm doing something wrong ..obviously..not sure where
>>>>>>>>>> though..
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I was wondering the same thing ;)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> It's best to call this method from a single BG "warming"
>>>>>>>>>>> thread,
>>>>>>>>>>> in
>>>>>>>>>>> which
>>>>>>>>>>> case it would not need its own synchronization.
>>>>>>>>>>>
>>>>>>>>>>> But, to be safe, I'll add internal synchronization to it.
>>>>>>>>>>> You
>>>>>>>>>>> can't
>>>>>>>>>>> simply put synchronized in front of the method, since you
>>>>>>>>>>> don't
>>>>>>>>>>> want
>>>>>>>>>>> this to
>>>>>>>>>>> block searching.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Mike
>>>>>>>>>>>
>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>
>>>>>>>>>>> just a quick point:
>>>>>>>>>>>
>>>>>>>>>>> public void maybeReopen() throws IOException
>>>>>>>>>>> { //D
>>>>>>>>>>>
>>>>>>>>>>> long currentVersion =
>>>>>>>>>>>> currentSearcher.getIndexReader().getVersion();
>>>>>>>>>>>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>>>>>>>>>>>> IndexReader newReader =
>>>>>>>>>>>> currentSearcher.getIndexReader().reopen();
>>>>>>>>>>>> assert newReader != currentSearcher.getIndexReader();
>>>>>>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>>>>>> warm(newSearcher);
>>>>>>>>>>>> swapSearcher(newSearcher);
>>>>>>>>>>>> }
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> should the above be synchronised?
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> thanks. i will rewrite..in between giving my baby her
>>>>>>>>>>>>> feed and
>>>>>>>>>>>>>
>>>>>>>>>>>> playing
>>>>>>>>>>>>
>>>>>>>>>>>> with the other child and my wife who wants me to do
>>>>>>>>>>>> several other
>>>>>>>>>>>>
>>>>>>>>>>>> things!
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for your input. I would like to have a go at
>>>>>>>>>>>>>> doing this
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> first, Solr may be an option.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser every
>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>> creating unnecessary garbage; instead, they should be
>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -- I can moved the code out so that it is only created
>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> reused.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>> when nothing has changed. This just generates
>>>>>>>>>>>>>>> unnecessary
>>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -- This was something I thought about. I could move
>>>>>>>>>>>>>>> it out so
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>> created once. However I presume inside my code i need
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>> whether
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> indexreaders are update to date. This needs to be
>>>>>>>>>>>>>>> synchronized
>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>> well I
>>>>>>>>>>>>>>> guess(?)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes you should synchronize the check for whether the
>>>>>>>>>>>>>>> IndexReader
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> current.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * I don't see any synchronization -- it looks like two
>>>>>>>>>>>>>> search
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> requests are allowed into this method at the same
>>>>>>>>>>>>>> time? Which
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>>> -- So i need to extract the logic for reopening and
>>>>>>>>>>>>>>> provide a
>>>>>>>>>>>>>>> synchronisation mechanism.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ok. So I have some work to do. I'll refactor the code
>>>>>>>>>>>>>> and see
>>>>>>>>>>>>>> if
>>>>>>>>>>>>>> I
>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> inline to your recommendations.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On a quick look, I think there are a few problems with
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> code:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * I don't see any synchronization -- it looks like two
>>>>>>>>>>>>>>> search
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> requests are allowed into this method at the same time?
>>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully
>>>>>>>>>>>>>>>> reopen the
>>>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * You are over-incRef'ing (the reader.incRef inside
>>>>>>>>>>>>>>>> the loop)
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>> don't see a corresponding decRef.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * You reopen and warm your searchers "live" (vs with BG
>>>>>>>>>>>>>>>> thread);
>>>>>>>>>>>>>>>> meaning the unlucky search request that hits a reopen
>>>>>>>>>>>>>>>> pays
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> cost. This might be OK if the index is small enough
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> reopening & warming takes very little time. But if
>>>>>>>>>>>>>>>> index
>>>>>>>>>>>>>>>> gets
>>>>>>>>>>>>>>>> large, making a random search pay that warming cost
>>>>>>>>>>>>>>>> is not
>>>>>>>>>>>>>>>> nice
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> the end user. It erodes their trust in you.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>> when nothing has changed. This just generates
>>>>>>>>>>>>>>>> unnecessary
>>>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser every
>>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>> creating unnecessary garbage; instead, they should be
>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> You should consider simply using Solr -- it handles
>>>>>>>>>>>>>>>> all this
>>>>>>>>>>>>>>>> logic
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> you and has been well debugged with time...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The reason for the indexreader.reopen is because I
>>>>>>>>>>>>>>>> have a
>>>>>>>>>>>>>>>> webapp
>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> enables users to upload files and then search for the
>>>>>>>>>>>>>>>> documents.
>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>> reopen i'm concerned that the facet hit counter
>>>>>>>>>>>>>>>>> won't be
>>>>>>>>>>>>>>>>> updated.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-
>>>>>>>>>>>>>>>>> Coleman <
>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I have been able to get the code working for my
>>>>>>>>>>>>>>>>>> scenario,
>>>>>>>>>>>>>>>>> however
>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> question and I was wondering if I could get some
>>>>>>>>>>>>>>>>>> help. I
>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> IndexSearchers which are used in a MultiSearcher
>>>>>>>>>>>>>>>>>> class. I
>>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> indexsearchers to get each indexreader and put them
>>>>>>>>>>>>>>>>>> into a
>>>>>>>>>>>>>>>>>> MultiIndexReader.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> IndexReader[] readers = new
>>>>>>>>>>>>>>>>>> IndexReader[searchables.length];
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> IndexSearcher indexSearcher =
>>>>>>>>>>>>>>>>>> (IndexSearcher)searchables[i];
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> readers[i].close();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> readers[i] = newReader;
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>>>>>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> IndexSearcher indexSearcher = new
>>>>>>>>>>>>>>>>>> IndexSearcher(multiReader);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I then use the indexseacher to do the facet stuff.
>>>>>>>>>>>>>>>>>> I end
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>> closing the multireader. This is causing problems in
>>>>>>>>>>>>>>>>>> another
>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>> where I
>>>>>>>>>>>>>>>>>> do some other search as the indexreaders are
>>>>>>>>>>>>>>>>>> closed. Is it
>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>> close
>>>>>>>>>>>>>>>>>> the multiindexreader or should I do some additional
>>>>>>>>>>>>>>>>>> checks
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-
>>>>>>>>>>>>>>>>>> Coleman <
>>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks just what I needed!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>>>>>>>>>>>>>>>>>>> marcelo.ochoa@gmail.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Amin:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>>>>>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-
>>>>>>>>>>>>>>>>>>>> Coleman <
>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Sorry to re send this email but I was wondering
>>>>>>>>>>>>>>>>>>>> if I
>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> advice
>>>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I am looking at building a faceted search using
>>>>>>>>>>>>>>>>>>>>> Lucene.
>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Solr
>>>>>>>>>>>>>>>>>>>>>> comes with this built in, however I would like
>>>>>>>>>>>>>>>>>>>>>> to try
>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>>>>>> (something to add to my CV!). I have been
>>>>>>>>>>>>>>>>>>>>>> looking
>>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>> you can use the IndexReader and use
>>>>>>>>>>>>>>>>>>>>>> TermVectors. This
>>>>>>>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>> sure how to filter the results so that a
>>>>>>>>>>>>>>>>>>>>>> particular
>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>> subset of results. The next option I was
>>>>>>>>>>>>>>>>>>>>>> looking at
>>>>>>>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>>>>>>>>>>>> Term[] termsArray = new Term[] { term1,
>>>>>>>>>>>>>>>>>>>>>> term2 };un
>>>>>>>>>>>>>>>>>>>>>> int[] docFreqs =
>>>>>>>>>>>>>>>>>>>>>> indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> The only problem here is that I have to provide
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> brand
>>>>>>>>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>>>>>>> time a
>>>>>>>>>>>>>>>>>>>>>> new brand is created. Again I'm not sure how I
>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>> filter
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> results
>>>>>>>>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>>>>>>> It may be that I'm using the wrong api methods
>>>>>>>>>>>>>>>>>>>>>> to do
>>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I would be grateful if I could get some advice
>>>>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> P.S. I am basically trying to do something that
>>>>>>>>>>>>>>>>>>>>>> displays
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Personal Contact (23) Business Contact (45) and
>>>>>>>>>>>>>>>>>>>>>> so on..
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>>>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>>>>>>>>>>>> ______________
>>>>>>>>>>>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>>>>>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>
>>>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>
>>>>>>>>> To unsubscribe, e-mail: java-user-
>>>>>>>>> unsubscribe@lucene.apache.org
>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
Hi there
Good morning! Here is the final search code:
public Summary[] search(final SearchRequest searchRequest)
throwsSearchExecutionException {
final String searchTerm = searchRequest.getSearchTerm();
if (StringUtils.isBlank(searchTerm)) {
throw new SearchExecutionException("Search string cannot be empty. There
will be too many results to process.");
}
List<Summary> summaryList = new ArrayList<Summary>();
StopWatch stopWatch = new StopWatch("searchStopWatch");
stopWatch.start();
MultiSearcher multiSearcher = null;
try {
LOGGER.debug("Ensuring all index readers are up to date...");
maybeReopen();
Query query = queryParser.parse(searchTerm);
LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
query.toString() +"'");
Sort sort = null;
sort = applySortIfApplicable(searchRequest);
Filter[] filters =applyFiltersIfApplicable(searchRequest);
ChainedFilter chainedFilter = null;
if (filters != null) {
chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
}
multiSearcher = get();
TopDocs topDocs = multiSearcher.search(query,chainedFilter ,100,sort);
ScoreDoc[] scoreDocs = topDocs.scoreDocs;
LOGGER.debug("total number of hits for [" + query.toString() + " ] = "+topDocs.
totalHits);
for (ScoreDoc scoreDoc : scoreDocs) {
final Document doc = multiSearcher.doc(scoreDoc.doc);
float score = scoreDoc.score;
final BaseDocument baseDocument = new BaseDocument(doc, score);
Summary documentSummary = new DocumentSummaryImpl(baseDocument);
summaryList.add(documentSummary);
}
} catch (Exception e) {
throw new IllegalStateException(e);
} finally {
if (multiSearcher != null) {
release(multiSearcher);
}
}
stopWatch.stop();
LOGGER.debug("total time taken for document seach: " +
stopWatch.getTotalTimeMillis() + " ms");
return summaryList.toArray(new Summary[] {});
}
I hope this makes sense...thanks again!
Cheers
Amin
On Sun, Mar 1, 2009 at 8:09 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
>
> You're calling get() too many times. For every call to get() you must
> match with a call to release().
>
> So, once at the front of your search method you should:
>
> MultiSearcher searcher = get();
>
> then use that searcher to do searching, retrieve docs, etc.
>
> Then in the finally clause, pass that searcher to release.
>
> So, only one call to get() and one matching call to release().
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> Hi
>> The searchers are injected into the class via Spring. So when a client
>> calls the class it is fully configured with a list of index searchers.
>> However I have removed this list and instead injecting a list of
>> directories which are passed to the DocumentSearchManager.
>> DocumentSearchManager is SearchManager (should've mentioned that earlier).
>> So finally I have modified by release code to do the following:
>>
>> private void release(MultiSearcher multiSeacher) throws Exception {
>>
>> IndexSearcher[] indexSearchers = (IndexSearcher[])
>> multiSeacher.getSearchables();
>>
>> for(int i =0 ; i < indexSearchers.length;i++) {
>>
>> documentSearcherManagers[i].release(indexSearchers[i]);
>>
>> }
>>
>> }
>>
>>
>> and it's use looks like this:
>>
>>
>> public Summary[] search(final SearchRequest searchRequest)
>> throwsSearchExecutionException {
>>
>> final String searchTerm = searchRequest.getSearchTerm();
>>
>> if (StringUtils.isBlank(searchTerm)) {
>>
>> throw new SearchExecutionException("Search string cannot be empty. There
>> will be too many results to process.");
>>
>> }
>>
>> List<Summary> summaryList = new ArrayList<Summary>();
>>
>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>
>> stopWatch.start();
>>
>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>
>> try {
>>
>> LOGGER.debug("Ensuring all index readers are up to date...");
>>
>> maybeReopen();
>>
>> LOGGER.debug("All Index Searchers are up to date. No of index searchers '"
>> +
>> indexSearchers.size() +"'");
>>
>> Query query = queryParser.parse(searchTerm);
>>
>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
>> query.toString() +"'");
>>
>> Sort sort = null;
>>
>> sort = applySortIfApplicable(searchRequest);
>>
>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>
>> ChainedFilter chainedFilter = null;
>>
>> if (filters != null) {
>>
>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>
>> }
>>
>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>
>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>
>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>> "+topDocs.
>> totalHits);
>>
>> for (ScoreDoc scoreDoc : scoreDocs) {
>>
>> final Document doc = get().doc(scoreDoc.doc);
>>
>> float score = scoreDoc.score;
>>
>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>
>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>
>> summaryList.add(documentSummary);
>>
>> }
>>
>> } catch (Exception e) {
>>
>> throw new IllegalStateException(e);
>>
>> } finally {
>>
>> release(get());
>>
>> }
>>
>> stopWatch.stop();
>>
>> LOGGER.debug("total time taken for document seach: " +
>> stopWatch.getTotalTimeMillis() + " ms");
>>
>> return summaryList.toArray(new Summary[] {});
>>
>> }
>>
>>
>> So the final post construct constructs the DocumentSearchMangers with the
>> list of directories..looking like this
>>
>>
>> @PostConstruct
>>
>> public void initialiseDocumentSearcher() {
>>
>> PerFieldAnalyzerWrapper analyzerWrapper = new PerFieldAnalyzerWrapper(
>> analyzer);
>>
>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>> newKeywordAnalyzer());
>>
>> queryParser =
>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>> analyzerWrapper);
>>
>> try {
>>
>> LOGGER.debug("Initialising multi searcher ....");
>>
>> documentSearcherManagers = new
>> DocumentSearcherManager[directories.size()];
>>
>> for (int i = 0; i < directories.size() ;i++) {
>>
>> Directory directory = directories.get(i);
>>
>> DocumentSearcherManager documentSearcherManager =
>> newDocumentSearcherManager(directory);
>>
>> documentSearcherManagers[i]=documentSearcherManager;
>>
>> }
>>
>> LOGGER.debug("multi searcher initialised");
>>
>> } catch (IOException e) {
>>
>> throw new IllegalStateException(e);
>>
>> }
>>
>> }
>>
>>
>>
>> Cheers
>>
>> Amin
>>
>>
>>
>> On Sun, Mar 1, 2009 at 6:15 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>
>>> I don't understand where searchers comes from, prior to
>>> initializeDocumentSearcher? You should, instead, simply create the
>>> SearcherManager (from your Directory instances). You don't need any
>>> searchers during initialize.
>>>
>>> Is DocumentSearcherManager the same as SearcherManager (just renamed)?
>>>
>>> The release method is wrong -- you're calling .get() and then
>>> immediately release. Instead, you should step through the searchers
>>> from your MultiSearcher and release them to each SearcherManager.
>>>
>>> You should call your release() in a finally clause.
>>>
>>> Mike
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> Sorry...i'm getting slightly confused.
>>>
>>>> I have a PostConstruct which is where I should create an array of
>>>> SearchManagers (per indexSeacher). From there I initialise the
>>>> multisearcher using the get(). After which I need to call maybeReopen
>>>> for
>>>> each IndexSearcher. So I'll do the following:
>>>>
>>>> @PostConstruct
>>>>
>>>> public void initialiseDocumentSearcher() {
>>>>
>>>> PerFieldAnalyzerWrapper analyzerWrapper = new PerFieldAnalyzerWrapper(
>>>> analyzer);
>>>>
>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>> newKeywordAnalyzer());
>>>>
>>>> queryParser =
>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>> analyzerWrapper);
>>>>
>>>> try {
>>>>
>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>
>>>> documentSearcherManagers = new
>>>> DocumentSearcherManager[searchers.size()];
>>>>
>>>> for (int i = 0; i < searchers.size() ;i++) {
>>>>
>>>> IndexSearcher indexSearcher = searchers.get(i);
>>>>
>>>> Directory directory = indexSearcher.getIndexReader().directory();
>>>>
>>>> DocumentSearcherManager documentSearcherManager =
>>>> newDocumentSearcherManager(directory);
>>>>
>>>> documentSearcherManagers[i]=documentSearcherManager;
>>>>
>>>> }
>>>>
>>>> LOGGER.debug("multi searcher initialised");
>>>>
>>>> } catch (IOException e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> This initialises search managers. I then have methods:
>>>>
>>>>
>>>> private void maybeReopen() throws Exception {
>>>>
>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>
>>>> for (DocumentSearcherManager documentSearcherManager :
>>>> documentSearcherManagers) {
>>>>
>>>> documentSearcherManager.maybeReopen();
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> private void release() throws Exception {
>>>>
>>>> for (DocumentSearcherManager documentSearcherManager :
>>>> documentSearcherManagers) {
>>>>
>>>> documentSearcherManager.release(documentSearcherManager.get());
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> private MultiSearcher get() {
>>>>
>>>> List<IndexSearcher> listOfIndexSeachers = new
>>>> ArrayList<IndexSearcher>();
>>>>
>>>> for (DocumentSearcherManager documentSearcherManager :
>>>> documentSearcherManagers) {
>>>>
>>>> listOfIndexSeachers.add(documentSearcherManager.get());
>>>>
>>>> }
>>>>
>>>> try {
>>>>
>>>> multiSearcher = new
>>>> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[] {}));
>>>>
>>>> } catch (IOException e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> }
>>>>
>>>> return multiSearcher;
>>>>
>>>> }
>>>>
>>>>
>>>> These methods are used in the following manner in the search code:
>>>>
>>>>
>>>> public Summary[] search(final SearchRequest searchRequest)
>>>> throwsSearchExecutionException {
>>>>
>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>
>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>
>>>> throw new SearchExecutionException("Search string cannot be empty. There
>>>> will be too many results to process.");
>>>>
>>>> }
>>>>
>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>
>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>
>>>> stopWatch.start();
>>>>
>>>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>>>
>>>> try {
>>>>
>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>
>>>> maybeReopen();
>>>>
>>>> LOGGER.debug("All Index Searchers are up to date. No of index searchers
>>>> '"
>>>> +
>>>> indexSearchers.size() +"'");
>>>>
>>>> Query query = queryParser.parse(searchTerm);
>>>>
>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
>>>> query.toString() +"'");
>>>>
>>>> Sort sort = null;
>>>>
>>>> sort = applySortIfApplicable(searchRequest);
>>>>
>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>
>>>> ChainedFilter chainedFilter = null;
>>>>
>>>> if (filters != null) {
>>>>
>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>
>>>> }
>>>>
>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>
>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>
>>>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>>>> "+topDocs.
>>>> totalHits);
>>>>
>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>
>>>> final Document doc = get().doc(scoreDoc.doc);
>>>>
>>>> float score = scoreDoc.score;
>>>>
>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>
>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>
>>>> summaryList.add(documentSummary);
>>>>
>>>> }
>>>>
>>>> release();
>>>>
>>>> } catch (Exception e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> }
>>>>
>>>> stopWatch.stop();
>>>>
>>>> LOGGER.debug("total time taken for document seach: " +
>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>
>>>> return summaryList.toArray(new Summary[] {});
>>>>
>>>> }
>>>>
>>>>
>>>> Does this look better? Again..I really really appreciate your help!
>>>>
>>>>
>>>> On Sun, Mar 1, 2009 at 4:18 PM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>>
>>>> This is not quite right -- you should only create SearcherManager once
>>>>> (per Direcotry) at startup/app load, not with every search request.
>>>>>
>>>>> And I don't see release -- it must call SearcherManager.release of
>>>>> each of the IndexSearchers previously returned from get().
>>>>>
>>>>> Mike
>>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> Thanks again for helping on a Sunday!
>>>>>>
>>>>>> I have now modified my maybeOpen() to do the following:
>>>>>>
>>>>>> private void maybeReopen() throws Exception {
>>>>>>
>>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>>
>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>>>>> .getSearchables();
>>>>>>
>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>
>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>
>>>>>> SearcherManager documentSearcherManager = new
>>>>>> SearcherManager(indexReader.directory());
>>>>>>
>>>>>> documentSearcherManager.maybeReopen();
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> And get() to:
>>>>>>
>>>>>>
>>>>>> private synchronized MultiSearcher get() {
>>>>>>
>>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>>>>> .getSearchables();
>>>>>>
>>>>>> List<IndexSearcher> indexSearchersList = new
>>>>>> ArrayList<IndexSearcher>();
>>>>>>
>>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>>
>>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>>
>>>>>> SearcherManager documentSearcherManager = null;
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> documentSearcherManager = new
>>>>>> SearcherManager(indexReader.directory());
>>>>>>
>>>>>> } catch (IOException e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> indexSearchersList.add(documentSearcherManager.get());
>>>>>>
>>>>>> }
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> multiSearcher = new
>>>>>> MultiSearcher(indexSearchersList.toArray(newIndexSearcher[] {}));
>>>>>>
>>>>>> } catch (IOException e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> return multiSearcher;
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> This makes all my test pass. I am using the SearchManager that you
>>>>>> recommended. Does this look ok?
>>>>>>
>>>>>>
>>>>>> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>
>>>>>> Your maybeReopen has an excess incRef().
>>>>>>
>>>>>>
>>>>>>> I'm not sure how you open the searchers in the first place? The list
>>>>>>> starts as empty, and nothing populates it?
>>>>>>>
>>>>>>> When you do the initial population, you need an incRef.
>>>>>>>
>>>>>>> I think you're hitting IllegalStateException because maybeReopen is
>>>>>>> closing a reader before get() can get it (since they synchronize on
>>>>>>> different objects).
>>>>>>>
>>>>>>> I'd recommend switching to the SearcherManager class. Instantiate
>>>>>>> one
>>>>>>> for each of your searchers. On each search request, go through them
>>>>>>> and call maybeReopen(), and then call get() and gather each
>>>>>>> IndexSearcher instance into a new array. Then, make a new
>>>>>>> MultiSearcher (opposite of what I said before): while that creates a
>>>>>>> small amount of garbage, it'll keep your code simpler (good
>>>>>>> tradeoff).
>>>>>>>
>>>>>>> Mike
>>>>>>>
>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>
>>>>>>> sorrry I added
>>>>>>>
>>>>>>>
>>>>>>> release(multiSearcher);
>>>>>>>>
>>>>>>>>
>>>>>>>> instead of multiSearcher.close();
>>>>>>>>
>>>>>>>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <
>>>>>>>> aminmc@gmail.com
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>
>>>>>>>> I've now done the following:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>>> throwsSearchExecutionException {
>>>>>>>>>
>>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>>
>>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>>
>>>>>>>>> throw new SearchExecutionException("Search string cannot be empty.
>>>>>>>>> There
>>>>>>>>> will be too many results to process.");
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>>
>>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>>
>>>>>>>>> stopWatch.start();
>>>>>>>>>
>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>
>>>>>>>>> try {
>>>>>>>>>
>>>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>>>
>>>>>>>>> maybeReopen();
>>>>>>>>>
>>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>>>>> searchers
>>>>>>>>> '"+ indexSearchers.size() +
>>>>>>>>> "'");
>>>>>>>>>
>>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>>
>>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '"
>>>>>>>>> +
>>>>>>>>> query.toString() +"'");
>>>>>>>>>
>>>>>>>>> Sort sort = null;
>>>>>>>>>
>>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>>
>>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>>
>>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>>
>>>>>>>>> if (filters != null) {
>>>>>>>>>
>>>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>>>>
>>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>>
>>>>>>>>> LOGGER.debug("total number of hits for [" + query.toString() + " ]
>>>>>>>>> =
>>>>>>>>> "+topDocs.
>>>>>>>>> totalHits);
>>>>>>>>>
>>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>>
>>>>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>>>>
>>>>>>>>> float score = scoreDoc.score;
>>>>>>>>>
>>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>>>>
>>>>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>>>>
>>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> multiSearcher.close();
>>>>>>>>>
>>>>>>>>> } catch (Exception e) {
>>>>>>>>>
>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> stopWatch.stop();
>>>>>>>>>
>>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>>
>>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> And have the following methods:
>>>>>>>>>
>>>>>>>>> @PostConstruct
>>>>>>>>>
>>>>>>>>> public void initialiseQueryParser() {
>>>>>>>>>
>>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>>> analyzer);
>>>>>>>>>
>>>>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>>> newKeywordAnalyzer());
>>>>>>>>>
>>>>>>>>> queryParser =
>>>>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>>
>>>>>>>>> analyzerWrapper);
>>>>>>>>>
>>>>>>>>> try {
>>>>>>>>>
>>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>>
>>>>>>>>> this.multiSearcher = new
>>>>>>>>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>>>>>>>>
>>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>>
>>>>>>>>> } catch (IOException e) {
>>>>>>>>>
>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Initialises mutltisearcher when this class is creared by spring.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> private synchronized void swapMultiSearcher(MultiSearcher
>>>>>>>>> newMultiSearcher) {
>>>>>>>>>
>>>>>>>>> try {
>>>>>>>>>
>>>>>>>>> release(multiSearcher);
>>>>>>>>>
>>>>>>>>> } catch (IOException e) {
>>>>>>>>>
>>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> multiSearcher = newMultiSearcher;
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> public void maybeReopen() throws IOException {
>>>>>>>>>
>>>>>>>>> MultiSearcher newMultiSeacher = null;
>>>>>>>>>
>>>>>>>>> boolean refreshMultiSeacher = false;
>>>>>>>>>
>>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>>
>>>>>>>>> synchronized (searchers) {
>>>>>>>>>
>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>
>>>>>>>>> IndexReader reader = indexSearcher.getIndexReader();
>>>>>>>>>
>>>>>>>>> reader.incRef();
>>>>>>>>>
>>>>>>>>> Directory directory = reader.directory();
>>>>>>>>>
>>>>>>>>> long currentVersion = reader.getVersion();
>>>>>>>>>
>>>>>>>>> if (IndexReader.getCurrentVersion(directory) != currentVersion) {
>>>>>>>>>
>>>>>>>>> IndexReader newReader = indexSearcher.getIndexReader().reopen();
>>>>>>>>>
>>>>>>>>> if (newReader != reader) {
>>>>>>>>>
>>>>>>>>> reader.decRef();
>>>>>>>>>
>>>>>>>>> refreshMultiSeacher = true;
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> reader = newReader;
>>>>>>>>>
>>>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>>>
>>>>>>>>> indexSearchers.add(newSearcher);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> if (refreshMultiSeacher) {
>>>>>>>>>
>>>>>>>>> newMultiSeacher = new
>>>>>>>>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
>>>>>>>>>
>>>>>>>>> warm(newMultiSeacher);
>>>>>>>>>
>>>>>>>>> swapMultiSearcher(newMultiSeacher);
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> private void warm(MultiSearcher newMultiSeacher) {
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>
>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>
>>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> return multiSearcher;
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> private synchronized void release(MultiSearcher multiSearcher)
>>>>>>>>> throwsIOException {
>>>>>>>>>
>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>
>>>>>>>>> indexSearcher.getIndexReader().decRef();
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> However I am now getting
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> java.lang.IllegalStateException:
>>>>>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexReader is
>>>>>>>>> closed
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> on the call:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>>
>>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>>
>>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> return multiSearcher;
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm doing something wrong ..obviously..not sure where though..
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I was wondering the same thing ;)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> It's best to call this method from a single BG "warming" thread,
>>>>>>>>>> in
>>>>>>>>>> which
>>>>>>>>>> case it would not need its own synchronization.
>>>>>>>>>>
>>>>>>>>>> But, to be safe, I'll add internal synchronization to it. You
>>>>>>>>>> can't
>>>>>>>>>> simply put synchronized in front of the method, since you don't
>>>>>>>>>> want
>>>>>>>>>> this to
>>>>>>>>>> block searching.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Mike
>>>>>>>>>>
>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>
>>>>>>>>>> just a quick point:
>>>>>>>>>>
>>>>>>>>>> public void maybeReopen() throws IOException { //D
>>>>>>>>>>
>>>>>>>>>> long currentVersion =
>>>>>>>>>>> currentSearcher.getIndexReader().getVersion();
>>>>>>>>>>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>>>>>>>>>>> IndexReader newReader =
>>>>>>>>>>> currentSearcher.getIndexReader().reopen();
>>>>>>>>>>> assert newReader != currentSearcher.getIndexReader();
>>>>>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>>>>> warm(newSearcher);
>>>>>>>>>>> swapSearcher(newSearcher);
>>>>>>>>>>> }
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> should the above be synchronised?
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> thanks. i will rewrite..in between giving my baby her feed and
>>>>>>>>>>>>
>>>>>>>>>>> playing
>>>>>>>>>>>
>>>>>>>>>>> with the other child and my wife who wants me to do several other
>>>>>>>>>>>
>>>>>>>>>>> things!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for your input. I would like to have a go at doing this
>>>>>>>>>>>>>
>>>>>>>>>>>>> myself
>>>>>>>>>>>>>
>>>>>>>>>>>>>> first, Solr may be an option.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser every time,
>>>>>>>>>>>>>> also
>>>>>>>>>>>>>> creating unnecessary garbage; instead, they should be created
>>>>>>>>>>>>>> once
>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -- I can moved the code out so that it is only created once
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> reused.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new MultiSearcher
>>>>>>>>>>>>>> even
>>>>>>>>>>>>>> when nothing has changed. This just generates unnecessary
>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -- This was something I thought about. I could move it out so
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>> created once. However I presume inside my code i need to
>>>>>>>>>>>>>> check
>>>>>>>>>>>>>> whether
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> indexreaders are update to date. This needs to be
>>>>>>>>>>>>>> synchronized
>>>>>>>>>>>>>> as
>>>>>>>>>>>>>> well I
>>>>>>>>>>>>>> guess(?)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes you should synchronize the check for whether the
>>>>>>>>>>>>>> IndexReader
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> current.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> * I don't see any synchronization -- it looks like two search
>>>>>>>>>>>>>
>>>>>>>>>>>>> requests are allowed into this method at the same time? Which
>>>>>>>>>>>>> is
>>>>>>>>>>>>>
>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>>>>>>>>>
>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>> -- So i need to extract the logic for reopening and provide a
>>>>>>>>>>>>>> synchronisation mechanism.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Ok. So I have some work to do. I'll refactor the code and see
>>>>>>>>>>>>> if
>>>>>>>>>>>>> I
>>>>>>>>>>>>> can
>>>>>>>>>>>>>
>>>>>>>>>>>>> get
>>>>>>>>>>>>>
>>>>>>>>>>>>> inline to your recommendations.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On a quick look, I think there are a few problems with the
>>>>>>>>>>>>>> code:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * I don't see any synchronization -- it looks like two search
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> requests are allowed into this method at the same time?
>>>>>>>>>>>>>>> Which
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * You are over-incRef'ing (the reader.incRef inside the loop)
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>> don't see a corresponding decRef.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * You reopen and warm your searchers "live" (vs with BG
>>>>>>>>>>>>>>> thread);
>>>>>>>>>>>>>>> meaning the unlucky search request that hits a reopen pays
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> cost. This might be OK if the index is small enough that
>>>>>>>>>>>>>>> reopening & warming takes very little time. But if index
>>>>>>>>>>>>>>> gets
>>>>>>>>>>>>>>> large, making a random search pay that warming cost is not
>>>>>>>>>>>>>>> nice
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> the end user. It erodes their trust in you.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new MultiSearcher
>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>> when nothing has changed. This just generates unnecessary
>>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser every time,
>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>> creating unnecessary garbage; instead, they should be created
>>>>>>>>>>>>>>> once
>>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You should consider simply using Solr -- it handles all this
>>>>>>>>>>>>>>> logic
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> you and has been well debugged with time...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The reason for the indexreader.reopen is because I have a
>>>>>>>>>>>>>>> webapp
>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> enables users to upload files and then search for the
>>>>>>>>>>>>>>> documents.
>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>> reopen i'm concerned that the facet hit counter won't be
>>>>>>>>>>>>>>>> updated.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I have been able to get the code working for my scenario,
>>>>>>>>>>>>>>>> however
>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>> question and I was wondering if I could get some help. I
>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> IndexSearchers which are used in a MultiSearcher class. I
>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> indexsearchers to get each indexreader and put them into a
>>>>>>>>>>>>>>>>> MultiIndexReader.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> IndexReader[] readers = new
>>>>>>>>>>>>>>>>> IndexReader[searchables.length];
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> IndexSearcher indexSearcher =
>>>>>>>>>>>>>>>>> (IndexSearcher)searchables[i];
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> readers[i].close();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> readers[i] = newReader;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>>>>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> IndexSearcher indexSearcher = new
>>>>>>>>>>>>>>>>> IndexSearcher(multiReader);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I then use the indexseacher to do the facet stuff. I end
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> closing the multireader. This is causing problems in
>>>>>>>>>>>>>>>>> another
>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>> where I
>>>>>>>>>>>>>>>>> do some other search as the indexreaders are closed. Is it
>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>> close
>>>>>>>>>>>>>>>>> the multiindexreader or should I do some additional checks
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks just what I needed!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>>>>>>>>>>>>>>>>>> marcelo.ochoa@gmail.com>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Amin:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>>>>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Sorry to re send this email but I was wondering if I
>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> advice
>>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I am looking at building a faceted search using Lucene.
>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Solr
>>>>>>>>>>>>>>>>>>>>> comes with this built in, however I would like to try
>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>>>>> (something to add to my CV!). I have been looking
>>>>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> you can use the IndexReader and use TermVectors. This
>>>>>>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>> sure how to filter the results so that a particular
>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>> subset of results. The next option I was looking at
>>>>>>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>>>>>>>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The only problem here is that I have to provide the
>>>>>>>>>>>>>>>>>>>>> brand
>>>>>>>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>>>>>> time a
>>>>>>>>>>>>>>>>>>>>> new brand is created. Again I'm not sure how I can
>>>>>>>>>>>>>>>>>>>>> filter
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> results
>>>>>>>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>>>>>> It may be that I'm using the wrong api methods to do
>>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> P.S. I am basically trying to do something that
>>>>>>>>>>>>>>>>>>>>> displays
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>>>>>>>>>>> ______________
>>>>>>>>>>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>>>>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>
>>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>
>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Faceted Search using Lucene
Posted by Michael McCandless <lu...@mikemccandless.com>.
You're calling get() too many times. For every call to get() you must
match with a call to release().
So, once at the front of your search method you should:
MultiSearcher searcher = get();
then use that searcher to do searching, retrieve docs, etc.
Then in the finally clause, pass that searcher to release.
So, only one call to get() and one matching call to release().
Mike
Amin Mohammed-Coleman wrote:
> Hi
> The searchers are injected into the class via Spring. So when a
> client
> calls the class it is fully configured with a list of index searchers.
> However I have removed this list and instead injecting a list of
> directories which are passed to the DocumentSearchManager.
> DocumentSearchManager is SearchManager (should've mentioned that
> earlier).
> So finally I have modified by release code to do the following:
>
> private void release(MultiSearcher multiSeacher) throws Exception {
>
> IndexSearcher[] indexSearchers = (IndexSearcher[])
> multiSeacher.getSearchables();
>
> for(int i =0 ; i < indexSearchers.length;i++) {
>
> documentSearcherManagers[i].release(indexSearchers[i]);
>
> }
>
> }
>
>
> and it's use looks like this:
>
>
> public Summary[] search(final SearchRequest searchRequest)
> throwsSearchExecutionException {
>
> final String searchTerm = searchRequest.getSearchTerm();
>
> if (StringUtils.isBlank(searchTerm)) {
>
> throw new SearchExecutionException("Search string cannot be empty.
> There
> will be too many results to process.");
>
> }
>
> List<Summary> summaryList = new ArrayList<Summary>();
>
> StopWatch stopWatch = new StopWatch("searchStopWatch");
>
> stopWatch.start();
>
> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>
> try {
>
> LOGGER.debug("Ensuring all index readers are up to date...");
>
> maybeReopen();
>
> LOGGER.debug("All Index Searchers are up to date. No of index
> searchers '" +
> indexSearchers.size() +"'");
>
> Query query = queryParser.parse(searchTerm);
>
> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
> query.toString() +"'");
>
> Sort sort = null;
>
> sort = applySortIfApplicable(searchRequest);
>
> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>
> ChainedFilter chainedFilter = null;
>
> if (filters != null) {
>
> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>
> }
>
> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>
> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>
> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
> "+topDocs.
> totalHits);
>
> for (ScoreDoc scoreDoc : scoreDocs) {
>
> final Document doc = get().doc(scoreDoc.doc);
>
> float score = scoreDoc.score;
>
> final BaseDocument baseDocument = new BaseDocument(doc, score);
>
> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>
> summaryList.add(documentSummary);
>
> }
>
> } catch (Exception e) {
>
> throw new IllegalStateException(e);
>
> } finally {
>
> release(get());
>
> }
>
> stopWatch.stop();
>
> LOGGER.debug("total time taken for document seach: " +
> stopWatch.getTotalTimeMillis() + " ms");
>
> return summaryList.toArray(new Summary[] {});
>
> }
>
>
> So the final post construct constructs the DocumentSearchMangers
> with the
> list of directories..looking like this
>
>
> @PostConstruct
>
> public void initialiseDocumentSearcher() {
>
> PerFieldAnalyzerWrapper analyzerWrapper = new PerFieldAnalyzerWrapper(
> analyzer);
>
> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
> newKeywordAnalyzer());
>
> queryParser =
> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
> analyzerWrapper);
>
> try {
>
> LOGGER.debug("Initialising multi searcher ....");
>
> documentSearcherManagers = new
> DocumentSearcherManager[directories.size()];
>
> for (int i = 0; i < directories.size() ;i++) {
>
> Directory directory = directories.get(i);
>
> DocumentSearcherManager documentSearcherManager =
> newDocumentSearcherManager(directory);
>
> documentSearcherManagers[i]=documentSearcherManager;
>
> }
>
> LOGGER.debug("multi searcher initialised");
>
> } catch (IOException e) {
>
> throw new IllegalStateException(e);
>
> }
>
> }
>
>
>
> Cheers
>
> Amin
>
>
>
> On Sun, Mar 1, 2009 at 6:15 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>>
>> I don't understand where searchers comes from, prior to
>> initializeDocumentSearcher? You should, instead, simply create the
>> SearcherManager (from your Directory instances). You don't need any
>> searchers during initialize.
>>
>> Is DocumentSearcherManager the same as SearcherManager (just
>> renamed)?
>>
>> The release method is wrong -- you're calling .get() and then
>> immediately release. Instead, you should step through the searchers
>> from your MultiSearcher and release them to each SearcherManager.
>>
>> You should call your release() in a finally clause.
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> Sorry...i'm getting slightly confused.
>>> I have a PostConstruct which is where I should create an array of
>>> SearchManagers (per indexSeacher). From there I initialise the
>>> multisearcher using the get(). After which I need to call
>>> maybeReopen for
>>> each IndexSearcher. So I'll do the following:
>>>
>>> @PostConstruct
>>>
>>> public void initialiseDocumentSearcher() {
>>>
>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>> PerFieldAnalyzerWrapper(
>>> analyzer);
>>>
>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>> newKeywordAnalyzer());
>>>
>>> queryParser =
>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>> analyzerWrapper);
>>>
>>> try {
>>>
>>> LOGGER.debug("Initialising multi searcher ....");
>>>
>>> documentSearcherManagers = new
>>> DocumentSearcherManager[searchers.size()];
>>>
>>> for (int i = 0; i < searchers.size() ;i++) {
>>>
>>> IndexSearcher indexSearcher = searchers.get(i);
>>>
>>> Directory directory = indexSearcher.getIndexReader().directory();
>>>
>>> DocumentSearcherManager documentSearcherManager =
>>> newDocumentSearcherManager(directory);
>>>
>>> documentSearcherManagers[i]=documentSearcherManager;
>>>
>>> }
>>>
>>> LOGGER.debug("multi searcher initialised");
>>>
>>> } catch (IOException e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> }
>>>
>>> }
>>>
>>>
>>> This initialises search managers. I then have methods:
>>>
>>>
>>> private void maybeReopen() throws Exception {
>>>
>>> LOGGER.debug("Initiating reopening of index readers...");
>>>
>>> for (DocumentSearcherManager documentSearcherManager :
>>> documentSearcherManagers) {
>>>
>>> documentSearcherManager.maybeReopen();
>>>
>>> }
>>>
>>> }
>>>
>>>
>>>
>>> private void release() throws Exception {
>>>
>>> for (DocumentSearcherManager documentSearcherManager :
>>> documentSearcherManagers) {
>>>
>>> documentSearcherManager.release(documentSearcherManager.get());
>>>
>>> }
>>>
>>> }
>>>
>>>
>>> private MultiSearcher get() {
>>>
>>> List<IndexSearcher> listOfIndexSeachers = new
>>> ArrayList<IndexSearcher>();
>>>
>>> for (DocumentSearcherManager documentSearcherManager :
>>> documentSearcherManagers) {
>>>
>>> listOfIndexSeachers.add(documentSearcherManager.get());
>>>
>>> }
>>>
>>> try {
>>>
>>> multiSearcher = new
>>> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[] {}));
>>>
>>> } catch (IOException e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> }
>>>
>>> return multiSearcher;
>>>
>>> }
>>>
>>>
>>> These methods are used in the following manner in the search code:
>>>
>>>
>>> public Summary[] search(final SearchRequest searchRequest)
>>> throwsSearchExecutionException {
>>>
>>> final String searchTerm = searchRequest.getSearchTerm();
>>>
>>> if (StringUtils.isBlank(searchTerm)) {
>>>
>>> throw new SearchExecutionException("Search string cannot be empty.
>>> There
>>> will be too many results to process.");
>>>
>>> }
>>>
>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>
>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>
>>> stopWatch.start();
>>>
>>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>>
>>> try {
>>>
>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>
>>> maybeReopen();
>>>
>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>> searchers '"
>>> +
>>> indexSearchers.size() +"'");
>>>
>>> Query query = queryParser.parse(searchTerm);
>>>
>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query
>>> '" +
>>> query.toString() +"'");
>>>
>>> Sort sort = null;
>>>
>>> sort = applySortIfApplicable(searchRequest);
>>>
>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>
>>> ChainedFilter chainedFilter = null;
>>>
>>> if (filters != null) {
>>>
>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>
>>> }
>>>
>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>
>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>
>>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>>> "+topDocs.
>>> totalHits);
>>>
>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>
>>> final Document doc = get().doc(scoreDoc.doc);
>>>
>>> float score = scoreDoc.score;
>>>
>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>
>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>
>>> summaryList.add(documentSummary);
>>>
>>> }
>>>
>>> release();
>>>
>>> } catch (Exception e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> }
>>>
>>> stopWatch.stop();
>>>
>>> LOGGER.debug("total time taken for document seach: " +
>>> stopWatch.getTotalTimeMillis() + " ms");
>>>
>>> return summaryList.toArray(new Summary[] {});
>>>
>>> }
>>>
>>>
>>> Does this look better? Again..I really really appreciate your help!
>>>
>>>
>>> On Sun, Mar 1, 2009 at 4:18 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>>> This is not quite right -- you should only create SearcherManager
>>>> once
>>>> (per Direcotry) at startup/app load, not with every search request.
>>>>
>>>> And I don't see release -- it must call SearcherManager.release of
>>>> each of the IndexSearchers previously returned from get().
>>>>
>>>> Mike
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> Hi
>>>>
>>>>> Thanks again for helping on a Sunday!
>>>>>
>>>>> I have now modified my maybeOpen() to do the following:
>>>>>
>>>>> private void maybeReopen() throws Exception {
>>>>>
>>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>>
>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>>>> .getSearchables();
>>>>>
>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>
>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>
>>>>> SearcherManager documentSearcherManager = new
>>>>> SearcherManager(indexReader.directory());
>>>>>
>>>>> documentSearcherManager.maybeReopen();
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> And get() to:
>>>>>
>>>>>
>>>>> private synchronized MultiSearcher get() {
>>>>>
>>>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>>>> .getSearchables();
>>>>>
>>>>> List<IndexSearcher> indexSearchersList = new
>>>>> ArrayList<IndexSearcher>();
>>>>>
>>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>>
>>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>>
>>>>> SearcherManager documentSearcherManager = null;
>>>>>
>>>>> try {
>>>>>
>>>>> documentSearcherManager = new
>>>>> SearcherManager(indexReader.directory());
>>>>>
>>>>> } catch (IOException e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> }
>>>>>
>>>>> indexSearchersList.add(documentSearcherManager.get());
>>>>>
>>>>> }
>>>>>
>>>>> try {
>>>>>
>>>>> multiSearcher = new
>>>>> MultiSearcher(indexSearchersList.toArray(newIndexSearcher[] {}));
>>>>>
>>>>> } catch (IOException e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> }
>>>>>
>>>>> return multiSearcher;
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> This makes all my test pass. I am using the SearchManager that
>>>>> you
>>>>> recommended. Does this look ok?
>>>>>
>>>>>
>>>>> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>> Your maybeReopen has an excess incRef().
>>>>>
>>>>>>
>>>>>> I'm not sure how you open the searchers in the first place?
>>>>>> The list
>>>>>> starts as empty, and nothing populates it?
>>>>>>
>>>>>> When you do the initial population, you need an incRef.
>>>>>>
>>>>>> I think you're hitting IllegalStateException because
>>>>>> maybeReopen is
>>>>>> closing a reader before get() can get it (since they
>>>>>> synchronize on
>>>>>> different objects).
>>>>>>
>>>>>> I'd recommend switching to the SearcherManager class.
>>>>>> Instantiate one
>>>>>> for each of your searchers. On each search request, go through
>>>>>> them
>>>>>> and call maybeReopen(), and then call get() and gather each
>>>>>> IndexSearcher instance into a new array. Then, make a new
>>>>>> MultiSearcher (opposite of what I said before): while that
>>>>>> creates a
>>>>>> small amount of garbage, it'll keep your code simpler (good
>>>>>> tradeoff).
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>> sorrry I added
>>>>>>
>>>>>>
>>>>>>> release(multiSearcher);
>>>>>>>
>>>>>>>
>>>>>>> instead of multiSearcher.close();
>>>>>>>
>>>>>>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <
>>>>>>> aminmc@gmail.com
>>>>>>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> I've now done the following:
>>>>>>>>
>>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>>> throwsSearchExecutionException {
>>>>>>>>
>>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>>
>>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>>
>>>>>>>> throw new SearchExecutionException("Search string cannot be
>>>>>>>> empty.
>>>>>>>> There
>>>>>>>> will be too many results to process.");
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>>
>>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>>
>>>>>>>> stopWatch.start();
>>>>>>>>
>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>
>>>>>>>> try {
>>>>>>>>
>>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>>
>>>>>>>> maybeReopen();
>>>>>>>>
>>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>>>> searchers
>>>>>>>> '"+ indexSearchers.size() +
>>>>>>>> "'");
>>>>>>>>
>>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>>
>>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>>>>>>>> Query '" +
>>>>>>>> query.toString() +"'");
>>>>>>>>
>>>>>>>> Sort sort = null;
>>>>>>>>
>>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>>
>>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>>
>>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>>
>>>>>>>> if (filters != null) {
>>>>>>>>
>>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>>>
>>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>>
>>>>>>>> LOGGER.debug("total number of hits for [" + query.toString()
>>>>>>>> + " ] =
>>>>>>>> "+topDocs.
>>>>>>>> totalHits);
>>>>>>>>
>>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>>
>>>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>>>
>>>>>>>> float score = scoreDoc.score;
>>>>>>>>
>>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>>>
>>>>>>>> Summary documentSummary = new
>>>>>>>> DocumentSummaryImpl(baseDocument);
>>>>>>>>
>>>>>>>> summaryList.add(documentSummary);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> multiSearcher.close();
>>>>>>>>
>>>>>>>> } catch (Exception e) {
>>>>>>>>
>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> stopWatch.stop();
>>>>>>>>
>>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>>
>>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> And have the following methods:
>>>>>>>>
>>>>>>>> @PostConstruct
>>>>>>>>
>>>>>>>> public void initialiseQueryParser() {
>>>>>>>>
>>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>>> analyzer);
>>>>>>>>
>>>>>>>> analyzerWrapper
>>>>>>>> .addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>>> newKeywordAnalyzer());
>>>>>>>>
>>>>>>>> queryParser =
>>>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>>>
>>>>>>>> analyzerWrapper);
>>>>>>>>
>>>>>>>> try {
>>>>>>>>
>>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>>
>>>>>>>> this.multiSearcher = new
>>>>>>>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>>>>>>>
>>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>>
>>>>>>>> } catch (IOException e) {
>>>>>>>>
>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> Initialises mutltisearcher when this class is creared by
>>>>>>>> spring.
>>>>>>>>
>>>>>>>>
>>>>>>>> private synchronized void swapMultiSearcher(MultiSearcher
>>>>>>>> newMultiSearcher) {
>>>>>>>>
>>>>>>>> try {
>>>>>>>>
>>>>>>>> release(multiSearcher);
>>>>>>>>
>>>>>>>> } catch (IOException e) {
>>>>>>>>
>>>>>>>> throw new IllegalStateException(e);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> multiSearcher = newMultiSearcher;
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> public void maybeReopen() throws IOException {
>>>>>>>>
>>>>>>>> MultiSearcher newMultiSeacher = null;
>>>>>>>>
>>>>>>>> boolean refreshMultiSeacher = false;
>>>>>>>>
>>>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>>>> ArrayList<IndexSearcher>();
>>>>>>>>
>>>>>>>> synchronized (searchers) {
>>>>>>>>
>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>
>>>>>>>> IndexReader reader = indexSearcher.getIndexReader();
>>>>>>>>
>>>>>>>> reader.incRef();
>>>>>>>>
>>>>>>>> Directory directory = reader.directory();
>>>>>>>>
>>>>>>>> long currentVersion = reader.getVersion();
>>>>>>>>
>>>>>>>> if (IndexReader.getCurrentVersion(directory) !=
>>>>>>>> currentVersion) {
>>>>>>>>
>>>>>>>> IndexReader newReader =
>>>>>>>> indexSearcher.getIndexReader().reopen();
>>>>>>>>
>>>>>>>> if (newReader != reader) {
>>>>>>>>
>>>>>>>> reader.decRef();
>>>>>>>>
>>>>>>>> refreshMultiSeacher = true;
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> reader = newReader;
>>>>>>>>
>>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>>
>>>>>>>> indexSearchers.add(newSearcher);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> if (refreshMultiSeacher) {
>>>>>>>>
>>>>>>>> newMultiSeacher = new
>>>>>>>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
>>>>>>>>
>>>>>>>> warm(newMultiSeacher);
>>>>>>>>
>>>>>>>> swapMultiSearcher(newMultiSeacher);
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> private void warm(MultiSearcher newMultiSeacher) {
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>
>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>
>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> return multiSearcher;
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> private synchronized void release(MultiSearcher multiSearcher)
>>>>>>>> throwsIOException {
>>>>>>>>
>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>
>>>>>>>> indexSearcher.getIndexReader().decRef();
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> However I am now getting
>>>>>>>>
>>>>>>>>
>>>>>>>> java.lang.IllegalStateException:
>>>>>>>> org.apache.lucene.store.AlreadyClosedException: this
>>>>>>>> IndexReader is
>>>>>>>> closed
>>>>>>>>
>>>>>>>>
>>>>>>>> on the call:
>>>>>>>>
>>>>>>>>
>>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>>
>>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>>
>>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> return multiSearcher;
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> I'm doing something wrong ..obviously..not sure where though..
>>>>>>>>
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> I was wondering the same thing ;)
>>>>>>>>
>>>>>>>>>
>>>>>>>>> It's best to call this method from a single BG "warming"
>>>>>>>>> thread, in
>>>>>>>>> which
>>>>>>>>> case it would not need its own synchronization.
>>>>>>>>>
>>>>>>>>> But, to be safe, I'll add internal synchronization to it.
>>>>>>>>> You can't
>>>>>>>>> simply put synchronized in front of the method, since you
>>>>>>>>> don't want
>>>>>>>>> this to
>>>>>>>>> block searching.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Mike
>>>>>>>>>
>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>
>>>>>>>>> just a quick point:
>>>>>>>>>
>>>>>>>>> public void maybeReopen() throws IOException
>>>>>>>>> { //D
>>>>>>>>>
>>>>>>>>>> long currentVersion =
>>>>>>>>>> currentSearcher.getIndexReader().getVersion();
>>>>>>>>>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>>>>>>>>>> IndexReader newReader =
>>>>>>>>>> currentSearcher.getIndexReader().reopen();
>>>>>>>>>> assert newReader != currentSearcher.getIndexReader();
>>>>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>>>> warm(newSearcher);
>>>>>>>>>> swapSearcher(newSearcher);
>>>>>>>>>> }
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> should the above be synchronised?
>>>>>>>>>>
>>>>>>>>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> thanks. i will rewrite..in between giving my baby her
>>>>>>>>>>> feed and
>>>>>>>>>> playing
>>>>>>>>>>
>>>>>>>>>> with the other child and my wife who wants me to do several
>>>>>>>>>> other
>>>>>>>>>>
>>>>>>>>>>> things!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Hi
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for your input. I would like to have a go at
>>>>>>>>>>>> doing this
>>>>>>>>>>>>
>>>>>>>>>>>> myself
>>>>>>>>>>>>> first, Solr may be an option.
>>>>>>>>>>>>>
>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser every
>>>>>>>>>>>>> time, also
>>>>>>>>>>>>> creating unnecessary garbage; instead, they should be
>>>>>>>>>>>>> created
>>>>>>>>>>>>> once
>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -- I can moved the code out so that it is only created
>>>>>>>>>>>>> once and
>>>>>>>>>>>>> reused.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>> even
>>>>>>>>>>>>> when nothing has changed. This just generates unnecessary
>>>>>>>>>>>>> garbage
>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -- This was something I thought about. I could move it
>>>>>>>>>>>>> out so
>>>>>>>>>>>>> that
>>>>>>>>>>>>> it's
>>>>>>>>>>>>> created once. However I presume inside my code i need
>>>>>>>>>>>>> to check
>>>>>>>>>>>>> whether
>>>>>>>>>>>>> the
>>>>>>>>>>>>> indexreaders are update to date. This needs to be
>>>>>>>>>>>>> synchronized
>>>>>>>>>>>>> as
>>>>>>>>>>>>> well I
>>>>>>>>>>>>> guess(?)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes you should synchronize the check for whether the
>>>>>>>>>>>>> IndexReader
>>>>>>>>>>>>> is
>>>>>>>>>>>>>
>>>>>>>>>>>>> current.
>>>>>>>>>>>>
>>>>>>>>>>>> * I don't see any synchronization -- it looks like two
>>>>>>>>>>>> search
>>>>>>>>>>>>
>>>>>>>>>>>> requests are allowed into this method at the same time?
>>>>>>>>>>>> Which is
>>>>>>>>>>>>
>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>>>>>>>>> readers.
>>>>>>>>>>>>> -- So i need to extract the logic for reopening and
>>>>>>>>>>>>> provide a
>>>>>>>>>>>>> synchronisation mechanism.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Ok. So I have some work to do. I'll refactor the code
>>>>>>>>>>>> and see
>>>>>>>>>>>> if
>>>>>>>>>>>> I
>>>>>>>>>>>> can
>>>>>>>>>>>>
>>>>>>>>>>>> get
>>>>>>>>>>>>
>>>>>>>>>>>> inline to your recommendations.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On a quick look, I think there are a few problems with
>>>>>>>>>>>>> the code:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> * I don't see any synchronization -- it looks like two
>>>>>>>>>>>>> search
>>>>>>>>>>>>>> requests are allowed into this method at the same
>>>>>>>>>>>>>> time? Which
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * You are over-incRef'ing (the reader.incRef inside the
>>>>>>>>>>>>>> loop)
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> I
>>>>>>>>>>>>>> don't see a corresponding decRef.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * You reopen and warm your searchers "live" (vs with BG
>>>>>>>>>>>>>> thread);
>>>>>>>>>>>>>> meaning the unlucky search request that hits a reopen
>>>>>>>>>>>>>> pays the
>>>>>>>>>>>>>> cost. This might be OK if the index is small enough that
>>>>>>>>>>>>>> reopening & warming takes very little time. But if
>>>>>>>>>>>>>> index gets
>>>>>>>>>>>>>> large, making a random search pay that warming cost is
>>>>>>>>>>>>>> not nice
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> the end user. It erodes their trust in you.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>>>> even
>>>>>>>>>>>>>> when nothing has changed. This just generates
>>>>>>>>>>>>>> unnecessary
>>>>>>>>>>>>>> garbage
>>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser every
>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>> also
>>>>>>>>>>>>>> creating unnecessary garbage; instead, they should be
>>>>>>>>>>>>>> created
>>>>>>>>>>>>>> once
>>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You should consider simply using Solr -- it handles all
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> logic
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>> you and has been well debugged with time...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The reason for the indexreader.reopen is because I have a
>>>>>>>>>>>>>> webapp
>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> enables users to upload files and then search for the
>>>>>>>>>>>>>> documents.
>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>> reopen i'm concerned that the facet hit counter won't be
>>>>>>>>>>>>>>> updated.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have been able to get the code working for my
>>>>>>>>>>>>>>> scenario,
>>>>>>>>>>>>>>> however
>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> question and I was wondering if I could get some
>>>>>>>>>>>>>>>> help. I
>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> IndexSearchers which are used in a MultiSearcher
>>>>>>>>>>>>>>>> class. I
>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> indexsearchers to get each indexreader and put them
>>>>>>>>>>>>>>>> into a
>>>>>>>>>>>>>>>> MultiIndexReader.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> IndexReader[] readers = new
>>>>>>>>>>>>>>>> IndexReader[searchables.length];
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> IndexSearcher indexSearcher =
>>>>>>>>>>>>>>>> (IndexSearcher)searchables[i];
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> readers[i].close();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> readers[i] = newReader;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>>>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> IndexSearcher indexSearcher = new
>>>>>>>>>>>>>>>> IndexSearcher(multiReader);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I then use the indexseacher to do the facet stuff. I
>>>>>>>>>>>>>>>> end the
>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>> closing the multireader. This is causing problems in
>>>>>>>>>>>>>>>> another
>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>> where I
>>>>>>>>>>>>>>>> do some other search as the indexreaders are closed.
>>>>>>>>>>>>>>>> Is it
>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>> close
>>>>>>>>>>>>>>>> the multiindexreader or should I do some additional
>>>>>>>>>>>>>>>> checks in
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-
>>>>>>>>>>>>>>>> Coleman <
>>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks just what I needed!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>>>>>>>>>>>>>>>>> marcelo.ochoa@gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Amin:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>>>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-
>>>>>>>>>>>>>>>>>> Coleman <
>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Sorry to re send this email but I was wondering if
>>>>>>>>>>>>>>>>>> I could
>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>> advice
>>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I am looking at building a faceted search using
>>>>>>>>>>>>>>>>>>> Lucene. I
>>>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Solr
>>>>>>>>>>>>>>>>>>>> comes with this built in, however I would like to
>>>>>>>>>>>>>>>>>>>> try
>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>>>> (something to add to my CV!). I have been
>>>>>>>>>>>>>>>>>>>> looking around
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> you can use the IndexReader and use TermVectors.
>>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>> sure how to filter the results so that a
>>>>>>>>>>>>>>>>>>>> particular user
>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> subset of results. The next option I was looking
>>>>>>>>>>>>>>>>>>>> at was
>>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>>>>>>>>>>>>> int[] docFreqs =
>>>>>>>>>>>>>>>>>>>> indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The only problem here is that I have to provide
>>>>>>>>>>>>>>>>>>>> the brand
>>>>>>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>>>>> time a
>>>>>>>>>>>>>>>>>>>> new brand is created. Again I'm not sure how I can
>>>>>>>>>>>>>>>>>>>> filter
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> results
>>>>>>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>>>>> It may be that I'm using the wrong api methods to
>>>>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I would be grateful if I could get some advice on
>>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> P.S. I am basically trying to do something that
>>>>>>>>>>>>>>>>>>>> displays
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Personal Contact (23) Business Contact (45) and
>>>>>>>>>>>>>>>>>>>> so on..
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>>>>>>>>>> ______________
>>>>>>>>>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>>>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>
>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
Hi
The searchers are injected into the class via Spring. So when a client
calls the class it is fully configured with a list of index searchers.
However I have removed this list and instead injecting a list of
directories which are passed to the DocumentSearchManager.
DocumentSearchManager is SearchManager (should've mentioned that earlier).
So finally I have modified by release code to do the following:
private void release(MultiSearcher multiSeacher) throws Exception {
IndexSearcher[] indexSearchers = (IndexSearcher[])
multiSeacher.getSearchables();
for(int i =0 ; i < indexSearchers.length;i++) {
documentSearcherManagers[i].release(indexSearchers[i]);
}
}
and it's use looks like this:
public Summary[] search(final SearchRequest searchRequest)
throwsSearchExecutionException {
final String searchTerm = searchRequest.getSearchTerm();
if (StringUtils.isBlank(searchTerm)) {
throw new SearchExecutionException("Search string cannot be empty. There
will be too many results to process.");
}
List<Summary> summaryList = new ArrayList<Summary>();
StopWatch stopWatch = new StopWatch("searchStopWatch");
stopWatch.start();
List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
try {
LOGGER.debug("Ensuring all index readers are up to date...");
maybeReopen();
LOGGER.debug("All Index Searchers are up to date. No of index searchers '" +
indexSearchers.size() +"'");
Query query = queryParser.parse(searchTerm);
LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
query.toString() +"'");
Sort sort = null;
sort = applySortIfApplicable(searchRequest);
Filter[] filters =applyFiltersIfApplicable(searchRequest);
ChainedFilter chainedFilter = null;
if (filters != null) {
chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
}
TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
ScoreDoc[] scoreDocs = topDocs.scoreDocs;
LOGGER.debug("total number of hits for [" + query.toString() + " ] = "+topDocs.
totalHits);
for (ScoreDoc scoreDoc : scoreDocs) {
final Document doc = get().doc(scoreDoc.doc);
float score = scoreDoc.score;
final BaseDocument baseDocument = new BaseDocument(doc, score);
Summary documentSummary = new DocumentSummaryImpl(baseDocument);
summaryList.add(documentSummary);
}
} catch (Exception e) {
throw new IllegalStateException(e);
} finally {
release(get());
}
stopWatch.stop();
LOGGER.debug("total time taken for document seach: " +
stopWatch.getTotalTimeMillis() + " ms");
return summaryList.toArray(new Summary[] {});
}
So the final post construct constructs the DocumentSearchMangers with the
list of directories..looking like this
@PostConstruct
public void initialiseDocumentSearcher() {
PerFieldAnalyzerWrapper analyzerWrapper = new PerFieldAnalyzerWrapper(
analyzer);
analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
newKeywordAnalyzer());
queryParser = newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
analyzerWrapper);
try {
LOGGER.debug("Initialising multi searcher ....");
documentSearcherManagers = new DocumentSearcherManager[directories.size()];
for (int i = 0; i < directories.size() ;i++) {
Directory directory = directories.get(i);
DocumentSearcherManager documentSearcherManager =
newDocumentSearcherManager(directory);
documentSearcherManagers[i]=documentSearcherManager;
}
LOGGER.debug("multi searcher initialised");
} catch (IOException e) {
throw new IllegalStateException(e);
}
}
Cheers
Amin
On Sun, Mar 1, 2009 at 6:15 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
>
> I don't understand where searchers comes from, prior to
> initializeDocumentSearcher? You should, instead, simply create the
> SearcherManager (from your Directory instances). You don't need any
> searchers during initialize.
>
> Is DocumentSearcherManager the same as SearcherManager (just renamed)?
>
> The release method is wrong -- you're calling .get() and then
> immediately release. Instead, you should step through the searchers
> from your MultiSearcher and release them to each SearcherManager.
>
> You should call your release() in a finally clause.
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> Sorry...i'm getting slightly confused.
>> I have a PostConstruct which is where I should create an array of
>> SearchManagers (per indexSeacher). From there I initialise the
>> multisearcher using the get(). After which I need to call maybeReopen for
>> each IndexSearcher. So I'll do the following:
>>
>> @PostConstruct
>>
>> public void initialiseDocumentSearcher() {
>>
>> PerFieldAnalyzerWrapper analyzerWrapper = new PerFieldAnalyzerWrapper(
>> analyzer);
>>
>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>> newKeywordAnalyzer());
>>
>> queryParser =
>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>> analyzerWrapper);
>>
>> try {
>>
>> LOGGER.debug("Initialising multi searcher ....");
>>
>> documentSearcherManagers = new DocumentSearcherManager[searchers.size()];
>>
>> for (int i = 0; i < searchers.size() ;i++) {
>>
>> IndexSearcher indexSearcher = searchers.get(i);
>>
>> Directory directory = indexSearcher.getIndexReader().directory();
>>
>> DocumentSearcherManager documentSearcherManager =
>> newDocumentSearcherManager(directory);
>>
>> documentSearcherManagers[i]=documentSearcherManager;
>>
>> }
>>
>> LOGGER.debug("multi searcher initialised");
>>
>> } catch (IOException e) {
>>
>> throw new IllegalStateException(e);
>>
>> }
>>
>> }
>>
>>
>> This initialises search managers. I then have methods:
>>
>>
>> private void maybeReopen() throws Exception {
>>
>> LOGGER.debug("Initiating reopening of index readers...");
>>
>> for (DocumentSearcherManager documentSearcherManager :
>> documentSearcherManagers) {
>>
>> documentSearcherManager.maybeReopen();
>>
>> }
>>
>> }
>>
>>
>>
>> private void release() throws Exception {
>>
>> for (DocumentSearcherManager documentSearcherManager :
>> documentSearcherManagers) {
>>
>> documentSearcherManager.release(documentSearcherManager.get());
>>
>> }
>>
>> }
>>
>>
>> private MultiSearcher get() {
>>
>> List<IndexSearcher> listOfIndexSeachers = new ArrayList<IndexSearcher>();
>>
>> for (DocumentSearcherManager documentSearcherManager :
>> documentSearcherManagers) {
>>
>> listOfIndexSeachers.add(documentSearcherManager.get());
>>
>> }
>>
>> try {
>>
>> multiSearcher = new
>> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[] {}));
>>
>> } catch (IOException e) {
>>
>> throw new IllegalStateException(e);
>>
>> }
>>
>> return multiSearcher;
>>
>> }
>>
>>
>> These methods are used in the following manner in the search code:
>>
>>
>> public Summary[] search(final SearchRequest searchRequest)
>> throwsSearchExecutionException {
>>
>> final String searchTerm = searchRequest.getSearchTerm();
>>
>> if (StringUtils.isBlank(searchTerm)) {
>>
>> throw new SearchExecutionException("Search string cannot be empty. There
>> will be too many results to process.");
>>
>> }
>>
>> List<Summary> summaryList = new ArrayList<Summary>();
>>
>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>
>> stopWatch.start();
>>
>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>
>> try {
>>
>> LOGGER.debug("Ensuring all index readers are up to date...");
>>
>> maybeReopen();
>>
>> LOGGER.debug("All Index Searchers are up to date. No of index searchers '"
>> +
>> indexSearchers.size() +"'");
>>
>> Query query = queryParser.parse(searchTerm);
>>
>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
>> query.toString() +"'");
>>
>> Sort sort = null;
>>
>> sort = applySortIfApplicable(searchRequest);
>>
>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>
>> ChainedFilter chainedFilter = null;
>>
>> if (filters != null) {
>>
>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>
>> }
>>
>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>
>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>
>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>> "+topDocs.
>> totalHits);
>>
>> for (ScoreDoc scoreDoc : scoreDocs) {
>>
>> final Document doc = get().doc(scoreDoc.doc);
>>
>> float score = scoreDoc.score;
>>
>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>
>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>
>> summaryList.add(documentSummary);
>>
>> }
>>
>> release();
>>
>> } catch (Exception e) {
>>
>> throw new IllegalStateException(e);
>>
>> }
>>
>> stopWatch.stop();
>>
>> LOGGER.debug("total time taken for document seach: " +
>> stopWatch.getTotalTimeMillis() + " ms");
>>
>> return summaryList.toArray(new Summary[] {});
>>
>> }
>>
>>
>> Does this look better? Again..I really really appreciate your help!
>>
>>
>> On Sun, Mar 1, 2009 at 4:18 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>
>>> This is not quite right -- you should only create SearcherManager once
>>> (per Direcotry) at startup/app load, not with every search request.
>>>
>>> And I don't see release -- it must call SearcherManager.release of
>>> each of the IndexSearchers previously returned from get().
>>>
>>> Mike
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> Hi
>>>
>>>> Thanks again for helping on a Sunday!
>>>>
>>>> I have now modified my maybeOpen() to do the following:
>>>>
>>>> private void maybeReopen() throws Exception {
>>>>
>>>> LOGGER.debug("Initiating reopening of index readers...");
>>>>
>>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>>> .getSearchables();
>>>>
>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>
>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>
>>>> SearcherManager documentSearcherManager = new
>>>> SearcherManager(indexReader.directory());
>>>>
>>>> documentSearcherManager.maybeReopen();
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> And get() to:
>>>>
>>>>
>>>> private synchronized MultiSearcher get() {
>>>>
>>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>>> .getSearchables();
>>>>
>>>> List<IndexSearcher> indexSearchersList = new
>>>> ArrayList<IndexSearcher>();
>>>>
>>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>>
>>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>>
>>>> SearcherManager documentSearcherManager = null;
>>>>
>>>> try {
>>>>
>>>> documentSearcherManager = new SearcherManager(indexReader.directory());
>>>>
>>>> } catch (IOException e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> }
>>>>
>>>> indexSearchersList.add(documentSearcherManager.get());
>>>>
>>>> }
>>>>
>>>> try {
>>>>
>>>> multiSearcher = new
>>>> MultiSearcher(indexSearchersList.toArray(newIndexSearcher[] {}));
>>>>
>>>> } catch (IOException e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> }
>>>>
>>>> return multiSearcher;
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> This makes all my test pass. I am using the SearchManager that you
>>>> recommended. Does this look ok?
>>>>
>>>>
>>>> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>> Your maybeReopen has an excess incRef().
>>>>
>>>>>
>>>>> I'm not sure how you open the searchers in the first place? The list
>>>>> starts as empty, and nothing populates it?
>>>>>
>>>>> When you do the initial population, you need an incRef.
>>>>>
>>>>> I think you're hitting IllegalStateException because maybeReopen is
>>>>> closing a reader before get() can get it (since they synchronize on
>>>>> different objects).
>>>>>
>>>>> I'd recommend switching to the SearcherManager class. Instantiate one
>>>>> for each of your searchers. On each search request, go through them
>>>>> and call maybeReopen(), and then call get() and gather each
>>>>> IndexSearcher instance into a new array. Then, make a new
>>>>> MultiSearcher (opposite of what I said before): while that creates a
>>>>> small amount of garbage, it'll keep your code simpler (good
>>>>> tradeoff).
>>>>>
>>>>> Mike
>>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>> sorrry I added
>>>>>
>>>>>
>>>>>> release(multiSearcher);
>>>>>>
>>>>>>
>>>>>> instead of multiSearcher.close();
>>>>>>
>>>>>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <
>>>>>> aminmc@gmail.com
>>>>>>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> I've now done the following:
>>>>>>>
>>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>>> throwsSearchExecutionException {
>>>>>>>
>>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>>
>>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>>
>>>>>>> throw new SearchExecutionException("Search string cannot be empty.
>>>>>>> There
>>>>>>> will be too many results to process.");
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>>
>>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>>
>>>>>>> stopWatch.start();
>>>>>>>
>>>>>>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>>>>>>
>>>>>>> try {
>>>>>>>
>>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>>
>>>>>>> maybeReopen();
>>>>>>>
>>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>>> searchers
>>>>>>> '"+ indexSearchers.size() +
>>>>>>> "'");
>>>>>>>
>>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>>
>>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
>>>>>>> query.toString() +"'");
>>>>>>>
>>>>>>> Sort sort = null;
>>>>>>>
>>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>>
>>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>>
>>>>>>> ChainedFilter chainedFilter = null;
>>>>>>>
>>>>>>> if (filters != null) {
>>>>>>>
>>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>>
>>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>>
>>>>>>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>>>>>>> "+topDocs.
>>>>>>> totalHits);
>>>>>>>
>>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>>
>>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>>
>>>>>>> float score = scoreDoc.score;
>>>>>>>
>>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>>
>>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>>
>>>>>>> summaryList.add(documentSummary);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> multiSearcher.close();
>>>>>>>
>>>>>>> } catch (Exception e) {
>>>>>>>
>>>>>>> throw new IllegalStateException(e);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> stopWatch.stop();
>>>>>>>
>>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>>
>>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> And have the following methods:
>>>>>>>
>>>>>>> @PostConstruct
>>>>>>>
>>>>>>> public void initialiseQueryParser() {
>>>>>>>
>>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>>> PerFieldAnalyzerWrapper(
>>>>>>> analyzer);
>>>>>>>
>>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>>> newKeywordAnalyzer());
>>>>>>>
>>>>>>> queryParser =
>>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>>
>>>>>>> analyzerWrapper);
>>>>>>>
>>>>>>> try {
>>>>>>>
>>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>>
>>>>>>> this.multiSearcher = new
>>>>>>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>>>>>>
>>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>>
>>>>>>> } catch (IOException e) {
>>>>>>>
>>>>>>> throw new IllegalStateException(e);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> Initialises mutltisearcher when this class is creared by spring.
>>>>>>>
>>>>>>>
>>>>>>> private synchronized void swapMultiSearcher(MultiSearcher
>>>>>>> newMultiSearcher) {
>>>>>>>
>>>>>>> try {
>>>>>>>
>>>>>>> release(multiSearcher);
>>>>>>>
>>>>>>> } catch (IOException e) {
>>>>>>>
>>>>>>> throw new IllegalStateException(e);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> multiSearcher = newMultiSearcher;
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> public void maybeReopen() throws IOException {
>>>>>>>
>>>>>>> MultiSearcher newMultiSeacher = null;
>>>>>>>
>>>>>>> boolean refreshMultiSeacher = false;
>>>>>>>
>>>>>>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>>>>>>
>>>>>>> synchronized (searchers) {
>>>>>>>
>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>
>>>>>>> IndexReader reader = indexSearcher.getIndexReader();
>>>>>>>
>>>>>>> reader.incRef();
>>>>>>>
>>>>>>> Directory directory = reader.directory();
>>>>>>>
>>>>>>> long currentVersion = reader.getVersion();
>>>>>>>
>>>>>>> if (IndexReader.getCurrentVersion(directory) != currentVersion) {
>>>>>>>
>>>>>>> IndexReader newReader = indexSearcher.getIndexReader().reopen();
>>>>>>>
>>>>>>> if (newReader != reader) {
>>>>>>>
>>>>>>> reader.decRef();
>>>>>>>
>>>>>>> refreshMultiSeacher = true;
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> reader = newReader;
>>>>>>>
>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>
>>>>>>> indexSearchers.add(newSearcher);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> if (refreshMultiSeacher) {
>>>>>>>
>>>>>>> newMultiSeacher = new
>>>>>>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
>>>>>>>
>>>>>>> warm(newMultiSeacher);
>>>>>>>
>>>>>>> swapMultiSearcher(newMultiSeacher);
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> private void warm(MultiSearcher newMultiSeacher) {
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>
>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>
>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> return multiSearcher;
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> private synchronized void release(MultiSearcher multiSearcher)
>>>>>>> throwsIOException {
>>>>>>>
>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>
>>>>>>> indexSearcher.getIndexReader().decRef();
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> However I am now getting
>>>>>>>
>>>>>>>
>>>>>>> java.lang.IllegalStateException:
>>>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexReader is
>>>>>>> closed
>>>>>>>
>>>>>>>
>>>>>>> on the call:
>>>>>>>
>>>>>>>
>>>>>>> private synchronized MultiSearcher get() {
>>>>>>>
>>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>>
>>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> return multiSearcher;
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> I'm doing something wrong ..obviously..not sure where though..
>>>>>>>
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> I was wondering the same thing ;)
>>>>>>>
>>>>>>>>
>>>>>>>> It's best to call this method from a single BG "warming" thread, in
>>>>>>>> which
>>>>>>>> case it would not need its own synchronization.
>>>>>>>>
>>>>>>>> But, to be safe, I'll add internal synchronization to it. You can't
>>>>>>>> simply put synchronized in front of the method, since you don't want
>>>>>>>> this to
>>>>>>>> block searching.
>>>>>>>>
>>>>>>>>
>>>>>>>> Mike
>>>>>>>>
>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>
>>>>>>>> just a quick point:
>>>>>>>>
>>>>>>>> public void maybeReopen() throws IOException { //D
>>>>>>>>
>>>>>>>>> long currentVersion =
>>>>>>>>> currentSearcher.getIndexReader().getVersion();
>>>>>>>>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>>>>>>>>> IndexReader newReader = currentSearcher.getIndexReader().reopen();
>>>>>>>>> assert newReader != currentSearcher.getIndexReader();
>>>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>>> warm(newSearcher);
>>>>>>>>> swapSearcher(newSearcher);
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> should the above be synchronised?
>>>>>>>>>
>>>>>>>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
>>>>>>>>> aminmc@gmail.com
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> thanks. i will rewrite..in between giving my baby her feed and
>>>>>>>>> playing
>>>>>>>>>
>>>>>>>>> with the other child and my wife who wants me to do several other
>>>>>>>>>
>>>>>>>>>> things!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Hi
>>>>>>>>>>>
>>>>>>>>>>> Thanks for your input. I would like to have a go at doing this
>>>>>>>>>>>
>>>>>>>>>>> myself
>>>>>>>>>>>> first, Solr may be an option.
>>>>>>>>>>>>
>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser every time, also
>>>>>>>>>>>> creating unnecessary garbage; instead, they should be created
>>>>>>>>>>>> once
>>>>>>>>>>>> & reused.
>>>>>>>>>>>>
>>>>>>>>>>>> -- I can moved the code out so that it is only created once and
>>>>>>>>>>>> reused.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> * You always make a new IndexSearcher and a new MultiSearcher
>>>>>>>>>>>> even
>>>>>>>>>>>> when nothing has changed. This just generates unnecessary
>>>>>>>>>>>> garbage
>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>
>>>>>>>>>>>> -- This was something I thought about. I could move it out so
>>>>>>>>>>>> that
>>>>>>>>>>>> it's
>>>>>>>>>>>> created once. However I presume inside my code i need to check
>>>>>>>>>>>> whether
>>>>>>>>>>>> the
>>>>>>>>>>>> indexreaders are update to date. This needs to be synchronized
>>>>>>>>>>>> as
>>>>>>>>>>>> well I
>>>>>>>>>>>> guess(?)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Yes you should synchronize the check for whether the IndexReader
>>>>>>>>>>>> is
>>>>>>>>>>>>
>>>>>>>>>>>> current.
>>>>>>>>>>>
>>>>>>>>>>> * I don't see any synchronization -- it looks like two search
>>>>>>>>>>>
>>>>>>>>>>> requests are allowed into this method at the same time? Which is
>>>>>>>>>>>
>>>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>>>>>>>> readers.
>>>>>>>>>>>> -- So i need to extract the logic for reopening and provide a
>>>>>>>>>>>> synchronisation mechanism.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Yes.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Ok. So I have some work to do. I'll refactor the code and see
>>>>>>>>>>> if
>>>>>>>>>>> I
>>>>>>>>>>> can
>>>>>>>>>>>
>>>>>>>>>>> get
>>>>>>>>>>>
>>>>>>>>>>> inline to your recommendations.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On a quick look, I think there are a few problems with the code:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> * I don't see any synchronization -- it looks like two search
>>>>>>>>>>>>> requests are allowed into this method at the same time? Which
>>>>>>>>>>>>> is
>>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>>>>>>>>> readers.
>>>>>>>>>>>>>
>>>>>>>>>>>>> * You are over-incRef'ing (the reader.incRef inside the loop)
>>>>>>>>>>>>> --
>>>>>>>>>>>>> I
>>>>>>>>>>>>> don't see a corresponding decRef.
>>>>>>>>>>>>>
>>>>>>>>>>>>> * You reopen and warm your searchers "live" (vs with BG
>>>>>>>>>>>>> thread);
>>>>>>>>>>>>> meaning the unlucky search request that hits a reopen pays the
>>>>>>>>>>>>> cost. This might be OK if the index is small enough that
>>>>>>>>>>>>> reopening & warming takes very little time. But if index gets
>>>>>>>>>>>>> large, making a random search pay that warming cost is not nice
>>>>>>>>>>>>> to
>>>>>>>>>>>>> the end user. It erodes their trust in you.
>>>>>>>>>>>>>
>>>>>>>>>>>>> * You always make a new IndexSearcher and a new MultiSearcher
>>>>>>>>>>>>> even
>>>>>>>>>>>>> when nothing has changed. This just generates unnecessary
>>>>>>>>>>>>> garbage
>>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>>
>>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser every time,
>>>>>>>>>>>>> also
>>>>>>>>>>>>> creating unnecessary garbage; instead, they should be created
>>>>>>>>>>>>> once
>>>>>>>>>>>>> & reused.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You should consider simply using Solr -- it handles all this
>>>>>>>>>>>>> logic
>>>>>>>>>>>>> for
>>>>>>>>>>>>> you and has been well debugged with time...
>>>>>>>>>>>>>
>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>
>>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> The reason for the indexreader.reopen is because I have a
>>>>>>>>>>>>> webapp
>>>>>>>>>>>>> which
>>>>>>>>>>>>>
>>>>>>>>>>>>> enables users to upload files and then search for the
>>>>>>>>>>>>> documents.
>>>>>>>>>>>>> If
>>>>>>>>>>>>>
>>>>>>>>>>>>> I
>>>>>>>>>>>>>
>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>> reopen i'm concerned that the facet hit counter won't be
>>>>>>>>>>>>>> updated.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have been able to get the code working for my scenario,
>>>>>>>>>>>>>> however
>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>> question and I was wondering if I could get some help. I
>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> IndexSearchers which are used in a MultiSearcher class. I
>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> indexsearchers to get each indexreader and put them into a
>>>>>>>>>>>>>>> MultiIndexReader.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> IndexReader[] readers = new IndexReader[searchables.length];
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> readers[i].close();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> readers[i] = newReader;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I then use the indexseacher to do the facet stuff. I end the
>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>> closing the multireader. This is causing problems in another
>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>> where I
>>>>>>>>>>>>>>> do some other search as the indexreaders are closed. Is it
>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>> close
>>>>>>>>>>>>>>> the multiindexreader or should I do some additional checks in
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks just what I needed!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>>>>>>>>>>>>>>>> marcelo.ochoa@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Amin:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Sorry to re send this email but I was wondering if I could
>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>> advice
>>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I am looking at building a faceted search using Lucene. I
>>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Solr
>>>>>>>>>>>>>>>>>>> comes with this built in, however I would like to try
>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>>> (something to add to my CV!). I have been looking around
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> you can use the IndexReader and use TermVectors. This
>>>>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>> sure how to filter the results so that a particular user
>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> subset of results. The next option I was looking at was
>>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>>>>>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The only problem here is that I have to provide the brand
>>>>>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>>>> time a
>>>>>>>>>>>>>>>>>>> new brand is created. Again I'm not sure how I can
>>>>>>>>>>>>>>>>>>> filter
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> results
>>>>>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>>>> It may be that I'm using the wrong api methods to do
>>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> P.S. I am basically trying to do something that displays
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>>>>>>>>> ______________
>>>>>>>>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>
>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>
>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Faceted Search using Lucene
Posted by Michael McCandless <lu...@mikemccandless.com>.
I don't understand where searchers comes from, prior to
initializeDocumentSearcher? You should, instead, simply create the
SearcherManager (from your Directory instances). You don't need any
searchers during initialize.
Is DocumentSearcherManager the same as SearcherManager (just renamed)?
The release method is wrong -- you're calling .get() and then
immediately release. Instead, you should step through the searchers
from your MultiSearcher and release them to each SearcherManager.
You should call your release() in a finally clause.
Mike
Amin Mohammed-Coleman wrote:
> Sorry...i'm getting slightly confused.
> I have a PostConstruct which is where I should create an array of
> SearchManagers (per indexSeacher). From there I initialise the
> multisearcher using the get(). After which I need to call
> maybeReopen for
> each IndexSearcher. So I'll do the following:
>
> @PostConstruct
>
> public void initialiseDocumentSearcher() {
>
> PerFieldAnalyzerWrapper analyzerWrapper = new PerFieldAnalyzerWrapper(
> analyzer);
>
> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
> newKeywordAnalyzer());
>
> queryParser =
> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
> analyzerWrapper);
>
> try {
>
> LOGGER.debug("Initialising multi searcher ....");
>
> documentSearcherManagers = new
> DocumentSearcherManager[searchers.size()];
>
> for (int i = 0; i < searchers.size() ;i++) {
>
> IndexSearcher indexSearcher = searchers.get(i);
>
> Directory directory = indexSearcher.getIndexReader().directory();
>
> DocumentSearcherManager documentSearcherManager =
> newDocumentSearcherManager(directory);
>
> documentSearcherManagers[i]=documentSearcherManager;
>
> }
>
> LOGGER.debug("multi searcher initialised");
>
> } catch (IOException e) {
>
> throw new IllegalStateException(e);
>
> }
>
> }
>
>
> This initialises search managers. I then have methods:
>
>
> private void maybeReopen() throws Exception {
>
> LOGGER.debug("Initiating reopening of index readers...");
>
> for (DocumentSearcherManager documentSearcherManager :
> documentSearcherManagers) {
>
> documentSearcherManager.maybeReopen();
>
> }
>
> }
>
>
>
> private void release() throws Exception {
>
> for (DocumentSearcherManager documentSearcherManager :
> documentSearcherManagers) {
>
> documentSearcherManager.release(documentSearcherManager.get());
>
> }
>
> }
>
>
> private MultiSearcher get() {
>
> List<IndexSearcher> listOfIndexSeachers = new
> ArrayList<IndexSearcher>();
>
> for (DocumentSearcherManager documentSearcherManager :
> documentSearcherManagers) {
>
> listOfIndexSeachers.add(documentSearcherManager.get());
>
> }
>
> try {
>
> multiSearcher = new
> MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[] {}));
>
> } catch (IOException e) {
>
> throw new IllegalStateException(e);
>
> }
>
> return multiSearcher;
>
> }
>
>
> These methods are used in the following manner in the search code:
>
>
> public Summary[] search(final SearchRequest searchRequest)
> throwsSearchExecutionException {
>
> final String searchTerm = searchRequest.getSearchTerm();
>
> if (StringUtils.isBlank(searchTerm)) {
>
> throw new SearchExecutionException("Search string cannot be empty.
> There
> will be too many results to process.");
>
> }
>
> List<Summary> summaryList = new ArrayList<Summary>();
>
> StopWatch stopWatch = new StopWatch("searchStopWatch");
>
> stopWatch.start();
>
> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>
> try {
>
> LOGGER.debug("Ensuring all index readers are up to date...");
>
> maybeReopen();
>
> LOGGER.debug("All Index Searchers are up to date. No of index
> searchers '" +
> indexSearchers.size() +"'");
>
> Query query = queryParser.parse(searchTerm);
>
> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
> query.toString() +"'");
>
> Sort sort = null;
>
> sort = applySortIfApplicable(searchRequest);
>
> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>
> ChainedFilter chainedFilter = null;
>
> if (filters != null) {
>
> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>
> }
>
> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>
> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>
> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
> "+topDocs.
> totalHits);
>
> for (ScoreDoc scoreDoc : scoreDocs) {
>
> final Document doc = get().doc(scoreDoc.doc);
>
> float score = scoreDoc.score;
>
> final BaseDocument baseDocument = new BaseDocument(doc, score);
>
> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>
> summaryList.add(documentSummary);
>
> }
>
> release();
>
> } catch (Exception e) {
>
> throw new IllegalStateException(e);
>
> }
>
> stopWatch.stop();
>
> LOGGER.debug("total time taken for document seach: " +
> stopWatch.getTotalTimeMillis() + " ms");
>
> return summaryList.toArray(new Summary[] {});
>
> }
>
>
> Does this look better? Again..I really really appreciate your help!
>
>
> On Sun, Mar 1, 2009 at 4:18 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>>
>> This is not quite right -- you should only create SearcherManager
>> once
>> (per Direcotry) at startup/app load, not with every search request.
>>
>> And I don't see release -- it must call SearcherManager.release of
>> each of the IndexSearchers previously returned from get().
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> Hi
>>> Thanks again for helping on a Sunday!
>>>
>>> I have now modified my maybeOpen() to do the following:
>>>
>>> private void maybeReopen() throws Exception {
>>>
>>> LOGGER.debug("Initiating reopening of index readers...");
>>>
>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>> .getSearchables();
>>>
>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>
>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>
>>> SearcherManager documentSearcherManager = new
>>> SearcherManager(indexReader.directory());
>>>
>>> documentSearcherManager.maybeReopen();
>>>
>>> }
>>>
>>> }
>>>
>>>
>>> And get() to:
>>>
>>>
>>> private synchronized MultiSearcher get() {
>>>
>>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>>> .getSearchables();
>>>
>>> List<IndexSearcher> indexSearchersList = new
>>> ArrayList<IndexSearcher>();
>>>
>>> for (IndexSearcher indexSearcher : indexSearchers) {
>>>
>>> IndexReader indexReader = indexSearcher.getIndexReader();
>>>
>>> SearcherManager documentSearcherManager = null;
>>>
>>> try {
>>>
>>> documentSearcherManager = new
>>> SearcherManager(indexReader.directory());
>>>
>>> } catch (IOException e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> }
>>>
>>> indexSearchersList.add(documentSearcherManager.get());
>>>
>>> }
>>>
>>> try {
>>>
>>> multiSearcher = new
>>> MultiSearcher(indexSearchersList.toArray(newIndexSearcher[] {}));
>>>
>>> } catch (IOException e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> }
>>>
>>> return multiSearcher;
>>>
>>> }
>>>
>>>
>>>
>>> This makes all my test pass. I am using the SearchManager that you
>>> recommended. Does this look ok?
>>>
>>>
>>> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>> Your maybeReopen has an excess incRef().
>>>>
>>>> I'm not sure how you open the searchers in the first place? The
>>>> list
>>>> starts as empty, and nothing populates it?
>>>>
>>>> When you do the initial population, you need an incRef.
>>>>
>>>> I think you're hitting IllegalStateException because maybeReopen is
>>>> closing a reader before get() can get it (since they synchronize on
>>>> different objects).
>>>>
>>>> I'd recommend switching to the SearcherManager class.
>>>> Instantiate one
>>>> for each of your searchers. On each search request, go through
>>>> them
>>>> and call maybeReopen(), and then call get() and gather each
>>>> IndexSearcher instance into a new array. Then, make a new
>>>> MultiSearcher (opposite of what I said before): while that
>>>> creates a
>>>> small amount of garbage, it'll keep your code simpler (good
>>>> tradeoff).
>>>>
>>>> Mike
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> sorrry I added
>>>>
>>>>>
>>>>> release(multiSearcher);
>>>>>
>>>>>
>>>>> instead of multiSearcher.close();
>>>>>
>>>>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <aminmc@gmail.com
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>
>>>>> Hi
>>>>>
>>>>>> I've now done the following:
>>>>>>
>>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>>> throwsSearchExecutionException {
>>>>>>
>>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>>
>>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>>
>>>>>> throw new SearchExecutionException("Search string cannot be
>>>>>> empty.
>>>>>> There
>>>>>> will be too many results to process.");
>>>>>>
>>>>>> }
>>>>>>
>>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>>
>>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>>
>>>>>> stopWatch.start();
>>>>>>
>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>> ArrayList<IndexSearcher>();
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>>
>>>>>> maybeReopen();
>>>>>>
>>>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>>>> searchers
>>>>>> '"+ indexSearchers.size() +
>>>>>> "'");
>>>>>>
>>>>>> Query query = queryParser.parse(searchTerm);
>>>>>>
>>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene
>>>>>> Query '" +
>>>>>> query.toString() +"'");
>>>>>>
>>>>>> Sort sort = null;
>>>>>>
>>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>>
>>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>>
>>>>>> ChainedFilter chainedFilter = null;
>>>>>>
>>>>>> if (filters != null) {
>>>>>>
>>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>>
>>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>>
>>>>>> LOGGER.debug("total number of hits for [" + query.toString() +
>>>>>> " ] =
>>>>>> "+topDocs.
>>>>>> totalHits);
>>>>>>
>>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>>
>>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>>
>>>>>> float score = scoreDoc.score;
>>>>>>
>>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>>
>>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>>
>>>>>> summaryList.add(documentSummary);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> multiSearcher.close();
>>>>>>
>>>>>> } catch (Exception e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> stopWatch.stop();
>>>>>>
>>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>>
>>>>>> return summaryList.toArray(new Summary[] {});
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> And have the following methods:
>>>>>>
>>>>>> @PostConstruct
>>>>>>
>>>>>> public void initialiseQueryParser() {
>>>>>>
>>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>>>> PerFieldAnalyzerWrapper(
>>>>>> analyzer);
>>>>>>
>>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>>> newKeywordAnalyzer());
>>>>>>
>>>>>> queryParser =
>>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>>
>>>>>> analyzerWrapper);
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>>
>>>>>> this.multiSearcher = new
>>>>>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>>>>>
>>>>>> LOGGER.debug("multi searcher initialised");
>>>>>>
>>>>>> } catch (IOException e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> Initialises mutltisearcher when this class is creared by spring.
>>>>>>
>>>>>>
>>>>>> private synchronized void swapMultiSearcher(MultiSearcher
>>>>>> newMultiSearcher) {
>>>>>>
>>>>>> try {
>>>>>>
>>>>>> release(multiSearcher);
>>>>>>
>>>>>> } catch (IOException e) {
>>>>>>
>>>>>> throw new IllegalStateException(e);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> multiSearcher = newMultiSearcher;
>>>>>>
>>>>>> }
>>>>>>
>>>>>> public void maybeReopen() throws IOException {
>>>>>>
>>>>>> MultiSearcher newMultiSeacher = null;
>>>>>>
>>>>>> boolean refreshMultiSeacher = false;
>>>>>>
>>>>>> List<IndexSearcher> indexSearchers = new
>>>>>> ArrayList<IndexSearcher>();
>>>>>>
>>>>>> synchronized (searchers) {
>>>>>>
>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>
>>>>>> IndexReader reader = indexSearcher.getIndexReader();
>>>>>>
>>>>>> reader.incRef();
>>>>>>
>>>>>> Directory directory = reader.directory();
>>>>>>
>>>>>> long currentVersion = reader.getVersion();
>>>>>>
>>>>>> if (IndexReader.getCurrentVersion(directory) != currentVersion) {
>>>>>>
>>>>>> IndexReader newReader = indexSearcher.getIndexReader().reopen();
>>>>>>
>>>>>> if (newReader != reader) {
>>>>>>
>>>>>> reader.decRef();
>>>>>>
>>>>>> refreshMultiSeacher = true;
>>>>>>
>>>>>> }
>>>>>>
>>>>>> reader = newReader;
>>>>>>
>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>
>>>>>> indexSearchers.add(newSearcher);
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> if (refreshMultiSeacher) {
>>>>>>
>>>>>> newMultiSeacher = new
>>>>>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
>>>>>>
>>>>>> warm(newMultiSeacher);
>>>>>>
>>>>>> swapMultiSearcher(newMultiSeacher);
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> private void warm(MultiSearcher newMultiSeacher) {
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> private synchronized MultiSearcher get() {
>>>>>>
>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>
>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>
>>>>>> }
>>>>>>
>>>>>> return multiSearcher;
>>>>>>
>>>>>> }
>>>>>>
>>>>>> private synchronized void release(MultiSearcher multiSearcher)
>>>>>> throwsIOException {
>>>>>>
>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>
>>>>>> indexSearcher.getIndexReader().decRef();
>>>>>>
>>>>>> }
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> However I am now getting
>>>>>>
>>>>>>
>>>>>> java.lang.IllegalStateException:
>>>>>> org.apache.lucene.store.AlreadyClosedException: this
>>>>>> IndexReader is
>>>>>> closed
>>>>>>
>>>>>>
>>>>>> on the call:
>>>>>>
>>>>>>
>>>>>> private synchronized MultiSearcher get() {
>>>>>>
>>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>>
>>>>>> indexSearcher.getIndexReader().incRef();
>>>>>>
>>>>>> }
>>>>>>
>>>>>> return multiSearcher;
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> I'm doing something wrong ..obviously..not sure where though..
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>>>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>
>>>>>>
>>>>>> I was wondering the same thing ;)
>>>>>>>
>>>>>>> It's best to call this method from a single BG "warming"
>>>>>>> thread, in
>>>>>>> which
>>>>>>> case it would not need its own synchronization.
>>>>>>>
>>>>>>> But, to be safe, I'll add internal synchronization to it. You
>>>>>>> can't
>>>>>>> simply put synchronized in front of the method, since you
>>>>>>> don't want
>>>>>>> this to
>>>>>>> block searching.
>>>>>>>
>>>>>>>
>>>>>>> Mike
>>>>>>>
>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>
>>>>>>> just a quick point:
>>>>>>>
>>>>>>> public void maybeReopen() throws IOException
>>>>>>> { //D
>>>>>>>> long currentVersion =
>>>>>>>> currentSearcher.getIndexReader().getVersion();
>>>>>>>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>>>>>>>> IndexReader newReader =
>>>>>>>> currentSearcher.getIndexReader().reopen();
>>>>>>>> assert newReader != currentSearcher.getIndexReader();
>>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>>> warm(newSearcher);
>>>>>>>> swapSearcher(newSearcher);
>>>>>>>> }
>>>>>>>> }
>>>>>>>>
>>>>>>>> should the above be synchronised?
>>>>>>>>
>>>>>>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
>>>>>>>> aminmc@gmail.com
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>> thanks. i will rewrite..in between giving my baby her feed and
>>>>>>>> playing
>>>>>>>>
>>>>>>>> with the other child and my wife who wants me to do several
>>>>>>>> other
>>>>>>>>> things!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> Thanks for your input. I would like to have a go at doing
>>>>>>>>>> this
>>>>>>>>>>
>>>>>>>>>>> myself
>>>>>>>>>>> first, Solr may be an option.
>>>>>>>>>>>
>>>>>>>>>>> * You are creating a new Analyzer & QueryParser every
>>>>>>>>>>> time, also
>>>>>>>>>>> creating unnecessary garbage; instead, they should be
>>>>>>>>>>> created once
>>>>>>>>>>> & reused.
>>>>>>>>>>>
>>>>>>>>>>> -- I can moved the code out so that it is only created
>>>>>>>>>>> once and
>>>>>>>>>>> reused.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>> MultiSearcher even
>>>>>>>>>>> when nothing has changed. This just generates unnecessary
>>>>>>>>>>> garbage
>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>
>>>>>>>>>>> -- This was something I thought about. I could move it
>>>>>>>>>>> out so
>>>>>>>>>>> that
>>>>>>>>>>> it's
>>>>>>>>>>> created once. However I presume inside my code i need to
>>>>>>>>>>> check
>>>>>>>>>>> whether
>>>>>>>>>>> the
>>>>>>>>>>> indexreaders are update to date. This needs to be
>>>>>>>>>>> synchronized as
>>>>>>>>>>> well I
>>>>>>>>>>> guess(?)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Yes you should synchronize the check for whether the
>>>>>>>>>>> IndexReader
>>>>>>>>>>> is
>>>>>>>>>>>
>>>>>>>>>> current.
>>>>>>>>>>
>>>>>>>>>> * I don't see any synchronization -- it looks like two search
>>>>>>>>>>
>>>>>>>>>> requests are allowed into this method at the same time?
>>>>>>>>>> Which is
>>>>>>>>>>
>>>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>>>>>>> readers.
>>>>>>>>>>> -- So i need to extract the logic for reopening and
>>>>>>>>>>> provide a
>>>>>>>>>>> synchronisation mechanism.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Yes.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Ok. So I have some work to do. I'll refactor the code and
>>>>>>>>>> see if
>>>>>>>>>> I
>>>>>>>>>> can
>>>>>>>>>>
>>>>>>>>>> get
>>>>>>>>>>
>>>>>>>>>>> inline to your recommendations.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On a quick look, I think there are a few problems with the
>>>>>>>>>>> code:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> * I don't see any synchronization -- it looks like two
>>>>>>>>>>>> search
>>>>>>>>>>>> requests are allowed into this method at the same time?
>>>>>>>>>>>> Which is
>>>>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>>>>>>>> readers.
>>>>>>>>>>>>
>>>>>>>>>>>> * You are over-incRef'ing (the reader.incRef inside the
>>>>>>>>>>>> loop) --
>>>>>>>>>>>> I
>>>>>>>>>>>> don't see a corresponding decRef.
>>>>>>>>>>>>
>>>>>>>>>>>> * You reopen and warm your searchers "live" (vs with BG
>>>>>>>>>>>> thread);
>>>>>>>>>>>> meaning the unlucky search request that hits a reopen
>>>>>>>>>>>> pays the
>>>>>>>>>>>> cost. This might be OK if the index is small enough that
>>>>>>>>>>>> reopening & warming takes very little time. But if index
>>>>>>>>>>>> gets
>>>>>>>>>>>> large, making a random search pay that warming cost is
>>>>>>>>>>>> not nice
>>>>>>>>>>>> to
>>>>>>>>>>>> the end user. It erodes their trust in you.
>>>>>>>>>>>>
>>>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>>>> MultiSearcher
>>>>>>>>>>>> even
>>>>>>>>>>>> when nothing has changed. This just generates unnecessary
>>>>>>>>>>>> garbage
>>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>>
>>>>>>>>>>>> * You are creating a new Analyzer & QueryParser every
>>>>>>>>>>>> time, also
>>>>>>>>>>>> creating unnecessary garbage; instead, they should be
>>>>>>>>>>>> created
>>>>>>>>>>>> once
>>>>>>>>>>>> & reused.
>>>>>>>>>>>>
>>>>>>>>>>>> You should consider simply using Solr -- it handles all
>>>>>>>>>>>> this
>>>>>>>>>>>> logic
>>>>>>>>>>>> for
>>>>>>>>>>>> you and has been well debugged with time...
>>>>>>>>>>>>
>>>>>>>>>>>> Mike
>>>>>>>>>>>>
>>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> The reason for the indexreader.reopen is because I have a
>>>>>>>>>>>> webapp
>>>>>>>>>>>> which
>>>>>>>>>>>>
>>>>>>>>>>>> enables users to upload files and then search for the
>>>>>>>>>>>> documents.
>>>>>>>>>>>> If
>>>>>>>>>>>>
>>>>>>>>>>>> I
>>>>>>>>>>>>> don't
>>>>>>>>>>>>> reopen i'm concerned that the facet hit counter won't be
>>>>>>>>>>>>> updated.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have been able to get the code working for my scenario,
>>>>>>>>>>>>> however
>>>>>>>>>>>>> I
>>>>>>>>>>>>>
>>>>>>>>>>>>> have
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>> question and I was wondering if I could get some help.
>>>>>>>>>>>>>> I have
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>> list
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> IndexSearchers which are used in a MultiSearcher
>>>>>>>>>>>>>> class. I use
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> indexsearchers to get each indexreader and put them
>>>>>>>>>>>>>> into a
>>>>>>>>>>>>>> MultiIndexReader.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> IndexReader[] readers = new
>>>>>>>>>>>>>> IndexReader[searchables.length];
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> IndexSearcher indexSearcher =
>>>>>>>>>>>>>> (IndexSearcher)searchables[i];
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> readers[i].close();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> readers[i] = newReader;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> IndexSearcher indexSearcher = new
>>>>>>>>>>>>>> IndexSearcher(multiReader);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I then use the indexseacher to do the facet stuff. I
>>>>>>>>>>>>>> end the
>>>>>>>>>>>>>> code
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>> closing the multireader. This is causing problems in
>>>>>>>>>>>>>> another
>>>>>>>>>>>>>> method
>>>>>>>>>>>>>> where I
>>>>>>>>>>>>>> do some other search as the indexreaders are closed.
>>>>>>>>>>>>>> Is it ok
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> not
>>>>>>>>>>>>>> close
>>>>>>>>>>>>>> the multiindexreader or should I do some additional
>>>>>>>>>>>>>> checks in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> other
>>>>>>>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks just what I needed!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>>>>>>>>>>>>>>> marcelo.ochoa@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Amin:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-
>>>>>>>>>>>>>>>> Coleman <
>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sorry to re send this email but I was wondering if I
>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>> advice
>>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I am looking at building a faceted search using
>>>>>>>>>>>>>>>>> Lucene. I
>>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> Solr
>>>>>>>>>>>>>>>>>> comes with this built in, however I would like to
>>>>>>>>>>>>>>>>>> try this
>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>> (something to add to my CV!). I have been looking
>>>>>>>>>>>>>>>>>> around
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> you can use the IndexReader and use TermVectors.
>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>> sure how to filter the results so that a particular
>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> subset of results. The next option I was looking
>>>>>>>>>>>>>>>>>> at was
>>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>>>>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The only problem here is that I have to provide the
>>>>>>>>>>>>>>>>>> brand
>>>>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>>> time a
>>>>>>>>>>>>>>>>>> new brand is created. Again I'm not sure how I can
>>>>>>>>>>>>>>>>>> filter
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> results
>>>>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>>> It may be that I'm using the wrong api methods to
>>>>>>>>>>>>>>>>>> do this.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I would be grateful if I could get some advice on
>>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> P.S. I am basically trying to do something that
>>>>>>>>>>>>>>>>>> displays
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Personal Contact (23) Business Contact (45) and so
>>>>>>>>>>>>>>>>>> on..
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>>>>>>>> ______________
>>>>>>>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-
>>>>>>> help@lucene.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
Sorry...i'm getting slightly confused.
I have a PostConstruct which is where I should create an array of
SearchManagers (per indexSeacher). From there I initialise the
multisearcher using the get(). After which I need to call maybeReopen for
each IndexSearcher. So I'll do the following:
@PostConstruct
public void initialiseDocumentSearcher() {
PerFieldAnalyzerWrapper analyzerWrapper = new PerFieldAnalyzerWrapper(
analyzer);
analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
newKeywordAnalyzer());
queryParser = newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
analyzerWrapper);
try {
LOGGER.debug("Initialising multi searcher ....");
documentSearcherManagers = new DocumentSearcherManager[searchers.size()];
for (int i = 0; i < searchers.size() ;i++) {
IndexSearcher indexSearcher = searchers.get(i);
Directory directory = indexSearcher.getIndexReader().directory();
DocumentSearcherManager documentSearcherManager =
newDocumentSearcherManager(directory);
documentSearcherManagers[i]=documentSearcherManager;
}
LOGGER.debug("multi searcher initialised");
} catch (IOException e) {
throw new IllegalStateException(e);
}
}
This initialises search managers. I then have methods:
private void maybeReopen() throws Exception {
LOGGER.debug("Initiating reopening of index readers...");
for (DocumentSearcherManager documentSearcherManager :
documentSearcherManagers) {
documentSearcherManager.maybeReopen();
}
}
private void release() throws Exception {
for (DocumentSearcherManager documentSearcherManager :
documentSearcherManagers) {
documentSearcherManager.release(documentSearcherManager.get());
}
}
private MultiSearcher get() {
List<IndexSearcher> listOfIndexSeachers = new ArrayList<IndexSearcher>();
for (DocumentSearcherManager documentSearcherManager :
documentSearcherManagers) {
listOfIndexSeachers.add(documentSearcherManager.get());
}
try {
multiSearcher = new
MultiSearcher(listOfIndexSeachers.toArray(newIndexSearcher[] {}));
} catch (IOException e) {
throw new IllegalStateException(e);
}
return multiSearcher;
}
These methods are used in the following manner in the search code:
public Summary[] search(final SearchRequest searchRequest)
throwsSearchExecutionException {
final String searchTerm = searchRequest.getSearchTerm();
if (StringUtils.isBlank(searchTerm)) {
throw new SearchExecutionException("Search string cannot be empty. There
will be too many results to process.");
}
List<Summary> summaryList = new ArrayList<Summary>();
StopWatch stopWatch = new StopWatch("searchStopWatch");
stopWatch.start();
List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
try {
LOGGER.debug("Ensuring all index readers are up to date...");
maybeReopen();
LOGGER.debug("All Index Searchers are up to date. No of index searchers '" +
indexSearchers.size() +"'");
Query query = queryParser.parse(searchTerm);
LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
query.toString() +"'");
Sort sort = null;
sort = applySortIfApplicable(searchRequest);
Filter[] filters =applyFiltersIfApplicable(searchRequest);
ChainedFilter chainedFilter = null;
if (filters != null) {
chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
}
TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
ScoreDoc[] scoreDocs = topDocs.scoreDocs;
LOGGER.debug("total number of hits for [" + query.toString() + " ] = "+topDocs.
totalHits);
for (ScoreDoc scoreDoc : scoreDocs) {
final Document doc = get().doc(scoreDoc.doc);
float score = scoreDoc.score;
final BaseDocument baseDocument = new BaseDocument(doc, score);
Summary documentSummary = new DocumentSummaryImpl(baseDocument);
summaryList.add(documentSummary);
}
release();
} catch (Exception e) {
throw new IllegalStateException(e);
}
stopWatch.stop();
LOGGER.debug("total time taken for document seach: " +
stopWatch.getTotalTimeMillis() + " ms");
return summaryList.toArray(new Summary[] {});
}
Does this look better? Again..I really really appreciate your help!
On Sun, Mar 1, 2009 at 4:18 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
>
> This is not quite right -- you should only create SearcherManager once
> (per Direcotry) at startup/app load, not with every search request.
>
> And I don't see release -- it must call SearcherManager.release of
> each of the IndexSearchers previously returned from get().
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> Hi
>> Thanks again for helping on a Sunday!
>>
>> I have now modified my maybeOpen() to do the following:
>>
>> private void maybeReopen() throws Exception {
>>
>> LOGGER.debug("Initiating reopening of index readers...");
>>
>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>> .getSearchables();
>>
>> for (IndexSearcher indexSearcher : indexSearchers) {
>>
>> IndexReader indexReader = indexSearcher.getIndexReader();
>>
>> SearcherManager documentSearcherManager = new
>> SearcherManager(indexReader.directory());
>>
>> documentSearcherManager.maybeReopen();
>>
>> }
>>
>> }
>>
>>
>> And get() to:
>>
>>
>> private synchronized MultiSearcher get() {
>>
>> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
>> .getSearchables();
>>
>> List<IndexSearcher> indexSearchersList = new ArrayList<IndexSearcher>();
>>
>> for (IndexSearcher indexSearcher : indexSearchers) {
>>
>> IndexReader indexReader = indexSearcher.getIndexReader();
>>
>> SearcherManager documentSearcherManager = null;
>>
>> try {
>>
>> documentSearcherManager = new SearcherManager(indexReader.directory());
>>
>> } catch (IOException e) {
>>
>> throw new IllegalStateException(e);
>>
>> }
>>
>> indexSearchersList.add(documentSearcherManager.get());
>>
>> }
>>
>> try {
>>
>> multiSearcher = new
>> MultiSearcher(indexSearchersList.toArray(newIndexSearcher[] {}));
>>
>> } catch (IOException e) {
>>
>> throw new IllegalStateException(e);
>>
>> }
>>
>> return multiSearcher;
>>
>> }
>>
>>
>>
>> This makes all my test pass. I am using the SearchManager that you
>> recommended. Does this look ok?
>>
>>
>> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>> Your maybeReopen has an excess incRef().
>>>
>>> I'm not sure how you open the searchers in the first place? The list
>>> starts as empty, and nothing populates it?
>>>
>>> When you do the initial population, you need an incRef.
>>>
>>> I think you're hitting IllegalStateException because maybeReopen is
>>> closing a reader before get() can get it (since they synchronize on
>>> different objects).
>>>
>>> I'd recommend switching to the SearcherManager class. Instantiate one
>>> for each of your searchers. On each search request, go through them
>>> and call maybeReopen(), and then call get() and gather each
>>> IndexSearcher instance into a new array. Then, make a new
>>> MultiSearcher (opposite of what I said before): while that creates a
>>> small amount of garbage, it'll keep your code simpler (good
>>> tradeoff).
>>>
>>> Mike
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> sorrry I added
>>>
>>>>
>>>> release(multiSearcher);
>>>>
>>>>
>>>> instead of multiSearcher.close();
>>>>
>>>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <aminmc@gmail.com
>>>>
>>>>> wrote:
>>>>>
>>>>
>>>> Hi
>>>>
>>>>> I've now done the following:
>>>>>
>>>>> public Summary[] search(final SearchRequest searchRequest)
>>>>> throwsSearchExecutionException {
>>>>>
>>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>>
>>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>>
>>>>> throw new SearchExecutionException("Search string cannot be empty.
>>>>> There
>>>>> will be too many results to process.");
>>>>>
>>>>> }
>>>>>
>>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>>
>>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>>
>>>>> stopWatch.start();
>>>>>
>>>>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>>>>
>>>>> try {
>>>>>
>>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>>
>>>>> maybeReopen();
>>>>>
>>>>> LOGGER.debug("All Index Searchers are up to date. No of index searchers
>>>>> '"+ indexSearchers.size() +
>>>>> "'");
>>>>>
>>>>> Query query = queryParser.parse(searchTerm);
>>>>>
>>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
>>>>> query.toString() +"'");
>>>>>
>>>>> Sort sort = null;
>>>>>
>>>>> sort = applySortIfApplicable(searchRequest);
>>>>>
>>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>>
>>>>> ChainedFilter chainedFilter = null;
>>>>>
>>>>> if (filters != null) {
>>>>>
>>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>>
>>>>> }
>>>>>
>>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>>
>>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>>
>>>>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>>>>> "+topDocs.
>>>>> totalHits);
>>>>>
>>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>>
>>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>>
>>>>> float score = scoreDoc.score;
>>>>>
>>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>>
>>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>>
>>>>> summaryList.add(documentSummary);
>>>>>
>>>>> }
>>>>>
>>>>> multiSearcher.close();
>>>>>
>>>>> } catch (Exception e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> }
>>>>>
>>>>> stopWatch.stop();
>>>>>
>>>>> LOGGER.debug("total time taken for document seach: " +
>>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>>
>>>>> return summaryList.toArray(new Summary[] {});
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> And have the following methods:
>>>>>
>>>>> @PostConstruct
>>>>>
>>>>> public void initialiseQueryParser() {
>>>>>
>>>>> PerFieldAnalyzerWrapper analyzerWrapper = new PerFieldAnalyzerWrapper(
>>>>> analyzer);
>>>>>
>>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>>> newKeywordAnalyzer());
>>>>>
>>>>> queryParser =
>>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>>
>>>>> analyzerWrapper);
>>>>>
>>>>> try {
>>>>>
>>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>>
>>>>> this.multiSearcher = new
>>>>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>>>>
>>>>> LOGGER.debug("multi searcher initialised");
>>>>>
>>>>> } catch (IOException e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> Initialises mutltisearcher when this class is creared by spring.
>>>>>
>>>>>
>>>>> private synchronized void swapMultiSearcher(MultiSearcher
>>>>> newMultiSearcher) {
>>>>>
>>>>> try {
>>>>>
>>>>> release(multiSearcher);
>>>>>
>>>>> } catch (IOException e) {
>>>>>
>>>>> throw new IllegalStateException(e);
>>>>>
>>>>> }
>>>>>
>>>>> multiSearcher = newMultiSearcher;
>>>>>
>>>>> }
>>>>>
>>>>> public void maybeReopen() throws IOException {
>>>>>
>>>>> MultiSearcher newMultiSeacher = null;
>>>>>
>>>>> boolean refreshMultiSeacher = false;
>>>>>
>>>>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>>>>
>>>>> synchronized (searchers) {
>>>>>
>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>
>>>>> IndexReader reader = indexSearcher.getIndexReader();
>>>>>
>>>>> reader.incRef();
>>>>>
>>>>> Directory directory = reader.directory();
>>>>>
>>>>> long currentVersion = reader.getVersion();
>>>>>
>>>>> if (IndexReader.getCurrentVersion(directory) != currentVersion) {
>>>>>
>>>>> IndexReader newReader = indexSearcher.getIndexReader().reopen();
>>>>>
>>>>> if (newReader != reader) {
>>>>>
>>>>> reader.decRef();
>>>>>
>>>>> refreshMultiSeacher = true;
>>>>>
>>>>> }
>>>>>
>>>>> reader = newReader;
>>>>>
>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>
>>>>> indexSearchers.add(newSearcher);
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> if (refreshMultiSeacher) {
>>>>>
>>>>> newMultiSeacher = new
>>>>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
>>>>>
>>>>> warm(newMultiSeacher);
>>>>>
>>>>> swapMultiSearcher(newMultiSeacher);
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> private void warm(MultiSearcher newMultiSeacher) {
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> private synchronized MultiSearcher get() {
>>>>>
>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>
>>>>> indexSearcher.getIndexReader().incRef();
>>>>>
>>>>> }
>>>>>
>>>>> return multiSearcher;
>>>>>
>>>>> }
>>>>>
>>>>> private synchronized void release(MultiSearcher multiSearcher)
>>>>> throwsIOException {
>>>>>
>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>
>>>>> indexSearcher.getIndexReader().decRef();
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> However I am now getting
>>>>>
>>>>>
>>>>> java.lang.IllegalStateException:
>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexReader is
>>>>> closed
>>>>>
>>>>>
>>>>> on the call:
>>>>>
>>>>>
>>>>> private synchronized MultiSearcher get() {
>>>>>
>>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>>
>>>>> indexSearcher.getIndexReader().incRef();
>>>>>
>>>>> }
>>>>>
>>>>> return multiSearcher;
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> I'm doing something wrong ..obviously..not sure where though..
>>>>>
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>>
>>>>> I was wondering the same thing ;)
>>>>>>
>>>>>> It's best to call this method from a single BG "warming" thread, in
>>>>>> which
>>>>>> case it would not need its own synchronization.
>>>>>>
>>>>>> But, to be safe, I'll add internal synchronization to it. You can't
>>>>>> simply put synchronized in front of the method, since you don't want
>>>>>> this to
>>>>>> block searching.
>>>>>>
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>> just a quick point:
>>>>>>
>>>>>> public void maybeReopen() throws IOException { //D
>>>>>>> long currentVersion = currentSearcher.getIndexReader().getVersion();
>>>>>>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>>>>>>> IndexReader newReader = currentSearcher.getIndexReader().reopen();
>>>>>>> assert newReader != currentSearcher.getIndexReader();
>>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>>> warm(newSearcher);
>>>>>>> swapSearcher(newSearcher);
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> should the above be synchronised?
>>>>>>>
>>>>>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
>>>>>>> aminmc@gmail.com
>>>>>>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>> thanks. i will rewrite..in between giving my baby her feed and
>>>>>>> playing
>>>>>>>
>>>>>>> with the other child and my wife who wants me to do several other
>>>>>>>> things!
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> Thanks for your input. I would like to have a go at doing this
>>>>>>>>>
>>>>>>>>>> myself
>>>>>>>>>> first, Solr may be an option.
>>>>>>>>>>
>>>>>>>>>> * You are creating a new Analyzer & QueryParser every time, also
>>>>>>>>>> creating unnecessary garbage; instead, they should be created once
>>>>>>>>>> & reused.
>>>>>>>>>>
>>>>>>>>>> -- I can moved the code out so that it is only created once and
>>>>>>>>>> reused.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> * You always make a new IndexSearcher and a new MultiSearcher even
>>>>>>>>>> when nothing has changed. This just generates unnecessary garbage
>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>
>>>>>>>>>> -- This was something I thought about. I could move it out so
>>>>>>>>>> that
>>>>>>>>>> it's
>>>>>>>>>> created once. However I presume inside my code i need to check
>>>>>>>>>> whether
>>>>>>>>>> the
>>>>>>>>>> indexreaders are update to date. This needs to be synchronized as
>>>>>>>>>> well I
>>>>>>>>>> guess(?)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Yes you should synchronize the check for whether the IndexReader
>>>>>>>>>> is
>>>>>>>>>>
>>>>>>>>> current.
>>>>>>>>>
>>>>>>>>> * I don't see any synchronization -- it looks like two search
>>>>>>>>>
>>>>>>>>> requests are allowed into this method at the same time? Which is
>>>>>>>>>
>>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>>>>>> readers.
>>>>>>>>>> -- So i need to extract the logic for reopening and provide a
>>>>>>>>>> synchronisation mechanism.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Yes.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ok. So I have some work to do. I'll refactor the code and see if
>>>>>>>>> I
>>>>>>>>> can
>>>>>>>>>
>>>>>>>>> get
>>>>>>>>>
>>>>>>>>>> inline to your recommendations.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On a quick look, I think there are a few problems with the code:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> * I don't see any synchronization -- it looks like two search
>>>>>>>>>>> requests are allowed into this method at the same time? Which is
>>>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>>>>>>> readers.
>>>>>>>>>>>
>>>>>>>>>>> * You are over-incRef'ing (the reader.incRef inside the loop) --
>>>>>>>>>>> I
>>>>>>>>>>> don't see a corresponding decRef.
>>>>>>>>>>>
>>>>>>>>>>> * You reopen and warm your searchers "live" (vs with BG thread);
>>>>>>>>>>> meaning the unlucky search request that hits a reopen pays the
>>>>>>>>>>> cost. This might be OK if the index is small enough that
>>>>>>>>>>> reopening & warming takes very little time. But if index gets
>>>>>>>>>>> large, making a random search pay that warming cost is not nice
>>>>>>>>>>> to
>>>>>>>>>>> the end user. It erodes their trust in you.
>>>>>>>>>>>
>>>>>>>>>>> * You always make a new IndexSearcher and a new MultiSearcher
>>>>>>>>>>> even
>>>>>>>>>>> when nothing has changed. This just generates unnecessary
>>>>>>>>>>> garbage
>>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>>
>>>>>>>>>>> * You are creating a new Analyzer & QueryParser every time, also
>>>>>>>>>>> creating unnecessary garbage; instead, they should be created
>>>>>>>>>>> once
>>>>>>>>>>> & reused.
>>>>>>>>>>>
>>>>>>>>>>> You should consider simply using Solr -- it handles all this
>>>>>>>>>>> logic
>>>>>>>>>>> for
>>>>>>>>>>> you and has been well debugged with time...
>>>>>>>>>>>
>>>>>>>>>>> Mike
>>>>>>>>>>>
>>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>>
>>>>>>>>>>> The reason for the indexreader.reopen is because I have a webapp
>>>>>>>>>>> which
>>>>>>>>>>>
>>>>>>>>>>> enables users to upload files and then search for the documents.
>>>>>>>>>>> If
>>>>>>>>>>>
>>>>>>>>>>> I
>>>>>>>>>>>> don't
>>>>>>>>>>>> reopen i'm concerned that the facet hit counter won't be
>>>>>>>>>>>> updated.
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I have been able to get the code working for my scenario,
>>>>>>>>>>>> however
>>>>>>>>>>>> I
>>>>>>>>>>>>
>>>>>>>>>>>> have
>>>>>>>>>>>>> a
>>>>>>>>>>>>> question and I was wondering if I could get some help. I have
>>>>>>>>>>>>> a
>>>>>>>>>>>>> list
>>>>>>>>>>>>> of
>>>>>>>>>>>>> IndexSearchers which are used in a MultiSearcher class. I use
>>>>>>>>>>>>> the
>>>>>>>>>>>>> indexsearchers to get each indexreader and put them into a
>>>>>>>>>>>>> MultiIndexReader.
>>>>>>>>>>>>>
>>>>>>>>>>>>> IndexReader[] readers = new IndexReader[searchables.length];
>>>>>>>>>>>>>
>>>>>>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>>>>>>>>>>>>>
>>>>>>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>>>>>>
>>>>>>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>>>>>>
>>>>>>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>>>>>>
>>>>>>>>>>>>> readers[i].close();
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> readers[i] = newReader;
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>>>>>>
>>>>>>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>>>>>>>
>>>>>>>>>>>>> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I then use the indexseacher to do the facet stuff. I end the
>>>>>>>>>>>>> code
>>>>>>>>>>>>> with
>>>>>>>>>>>>> closing the multireader. This is causing problems in another
>>>>>>>>>>>>> method
>>>>>>>>>>>>> where I
>>>>>>>>>>>>> do some other search as the indexreaders are closed. Is it ok
>>>>>>>>>>>>> to
>>>>>>>>>>>>> not
>>>>>>>>>>>>> close
>>>>>>>>>>>>> the multiindexreader or should I do some additional checks in
>>>>>>>>>>>>> the
>>>>>>>>>>>>> other
>>>>>>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
>>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks just what I needed!
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>>>>>>>>>>>>>> marcelo.ochoa@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Amin:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sorry to re send this email but I was wondering if I could
>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>>> advice
>>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I am looking at building a faceted search using Lucene. I
>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> Solr
>>>>>>>>>>>>>>>>> comes with this built in, however I would like to try this
>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>> (something to add to my CV!). I have been looking around
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> you can use the IndexReader and use TermVectors. This
>>>>>>>>>>>>>>>>> looks
>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>> sure how to filter the results so that a particular user
>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>> subset of results. The next option I was looking at was
>>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>>>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The only problem here is that I have to provide the brand
>>>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>>> time a
>>>>>>>>>>>>>>>>> new brand is created. Again I'm not sure how I can filter
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> results
>>>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>>> It may be that I'm using the wrong api methods to do this.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> P.S. I am basically trying to do something that displays
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>>>>>>> ______________
>>>>>>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>
>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>
>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Faceted Search using Lucene
Posted by Michael McCandless <lu...@mikemccandless.com>.
This is not quite right -- you should only create SearcherManager once
(per Direcotry) at startup/app load, not with every search request.
And I don't see release -- it must call SearcherManager.release of
each of the IndexSearchers previously returned from get().
Mike
Amin Mohammed-Coleman wrote:
> Hi
> Thanks again for helping on a Sunday!
>
> I have now modified my maybeOpen() to do the following:
>
> private void maybeReopen() throws Exception {
>
> LOGGER.debug("Initiating reopening of index readers...");
>
> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
> .getSearchables();
>
> for (IndexSearcher indexSearcher : indexSearchers) {
>
> IndexReader indexReader = indexSearcher.getIndexReader();
>
> SearcherManager documentSearcherManager = new
> SearcherManager(indexReader.directory());
>
> documentSearcherManager.maybeReopen();
>
> }
>
> }
>
>
> And get() to:
>
>
> private synchronized MultiSearcher get() {
>
> IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
> .getSearchables();
>
> List<IndexSearcher> indexSearchersList = new
> ArrayList<IndexSearcher>();
>
> for (IndexSearcher indexSearcher : indexSearchers) {
>
> IndexReader indexReader = indexSearcher.getIndexReader();
>
> SearcherManager documentSearcherManager = null;
>
> try {
>
> documentSearcherManager = new
> SearcherManager(indexReader.directory());
>
> } catch (IOException e) {
>
> throw new IllegalStateException(e);
>
> }
>
> indexSearchersList.add(documentSearcherManager.get());
>
> }
>
> try {
>
> multiSearcher = new
> MultiSearcher(indexSearchersList.toArray(newIndexSearcher[] {}));
>
> } catch (IOException e) {
>
> throw new IllegalStateException(e);
>
> }
>
> return multiSearcher;
>
> }
>
>
>
> This makes all my test pass. I am using the SearchManager that you
> recommended. Does this look ok?
>
>
> On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> Your maybeReopen has an excess incRef().
>>
>> I'm not sure how you open the searchers in the first place? The list
>> starts as empty, and nothing populates it?
>>
>> When you do the initial population, you need an incRef.
>>
>> I think you're hitting IllegalStateException because maybeReopen is
>> closing a reader before get() can get it (since they synchronize on
>> different objects).
>>
>> I'd recommend switching to the SearcherManager class. Instantiate
>> one
>> for each of your searchers. On each search request, go through them
>> and call maybeReopen(), and then call get() and gather each
>> IndexSearcher instance into a new array. Then, make a new
>> MultiSearcher (opposite of what I said before): while that creates a
>> small amount of garbage, it'll keep your code simpler (good
>> tradeoff).
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> sorrry I added
>>>
>>> release(multiSearcher);
>>>
>>>
>>> instead of multiSearcher.close();
>>>
>>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <aminmc@gmail.com
>>>> wrote:
>>>
>>> Hi
>>>> I've now done the following:
>>>>
>>>> public Summary[] search(final SearchRequest searchRequest)
>>>> throwsSearchExecutionException {
>>>>
>>>> final String searchTerm = searchRequest.getSearchTerm();
>>>>
>>>> if (StringUtils.isBlank(searchTerm)) {
>>>>
>>>> throw new SearchExecutionException("Search string cannot be
>>>> empty. There
>>>> will be too many results to process.");
>>>>
>>>> }
>>>>
>>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>>
>>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>>
>>>> stopWatch.start();
>>>>
>>>> List<IndexSearcher> indexSearchers = new
>>>> ArrayList<IndexSearcher>();
>>>>
>>>> try {
>>>>
>>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>>
>>>> maybeReopen();
>>>>
>>>> LOGGER.debug("All Index Searchers are up to date. No of index
>>>> searchers
>>>> '"+ indexSearchers.size() +
>>>> "'");
>>>>
>>>> Query query = queryParser.parse(searchTerm);
>>>>
>>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query
>>>> '" +
>>>> query.toString() +"'");
>>>>
>>>> Sort sort = null;
>>>>
>>>> sort = applySortIfApplicable(searchRequest);
>>>>
>>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>>
>>>> ChainedFilter chainedFilter = null;
>>>>
>>>> if (filters != null) {
>>>>
>>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>>
>>>> }
>>>>
>>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>>
>>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>>
>>>> LOGGER.debug("total number of hits for [" + query.toString() +
>>>> " ] =
>>>> "+topDocs.
>>>> totalHits);
>>>>
>>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>>
>>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>>
>>>> float score = scoreDoc.score;
>>>>
>>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>>
>>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>>
>>>> summaryList.add(documentSummary);
>>>>
>>>> }
>>>>
>>>> multiSearcher.close();
>>>>
>>>> } catch (Exception e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> }
>>>>
>>>> stopWatch.stop();
>>>>
>>>> LOGGER.debug("total time taken for document seach: " +
>>>> stopWatch.getTotalTimeMillis() + " ms");
>>>>
>>>> return summaryList.toArray(new Summary[] {});
>>>>
>>>> }
>>>>
>>>>
>>>> And have the following methods:
>>>>
>>>> @PostConstruct
>>>>
>>>> public void initialiseQueryParser() {
>>>>
>>>> PerFieldAnalyzerWrapper analyzerWrapper = new
>>>> PerFieldAnalyzerWrapper(
>>>> analyzer);
>>>>
>>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>>> newKeywordAnalyzer());
>>>>
>>>> queryParser =
>>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>>
>>>> analyzerWrapper);
>>>>
>>>> try {
>>>>
>>>> LOGGER.debug("Initialising multi searcher ....");
>>>>
>>>> this.multiSearcher = new
>>>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>>>
>>>> LOGGER.debug("multi searcher initialised");
>>>>
>>>> } catch (IOException e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> Initialises mutltisearcher when this class is creared by spring.
>>>>
>>>>
>>>> private synchronized void swapMultiSearcher(MultiSearcher
>>>> newMultiSearcher) {
>>>>
>>>> try {
>>>>
>>>> release(multiSearcher);
>>>>
>>>> } catch (IOException e) {
>>>>
>>>> throw new IllegalStateException(e);
>>>>
>>>> }
>>>>
>>>> multiSearcher = newMultiSearcher;
>>>>
>>>> }
>>>>
>>>> public void maybeReopen() throws IOException {
>>>>
>>>> MultiSearcher newMultiSeacher = null;
>>>>
>>>> boolean refreshMultiSeacher = false;
>>>>
>>>> List<IndexSearcher> indexSearchers = new
>>>> ArrayList<IndexSearcher>();
>>>>
>>>> synchronized (searchers) {
>>>>
>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>
>>>> IndexReader reader = indexSearcher.getIndexReader();
>>>>
>>>> reader.incRef();
>>>>
>>>> Directory directory = reader.directory();
>>>>
>>>> long currentVersion = reader.getVersion();
>>>>
>>>> if (IndexReader.getCurrentVersion(directory) != currentVersion) {
>>>>
>>>> IndexReader newReader = indexSearcher.getIndexReader().reopen();
>>>>
>>>> if (newReader != reader) {
>>>>
>>>> reader.decRef();
>>>>
>>>> refreshMultiSeacher = true;
>>>>
>>>> }
>>>>
>>>> reader = newReader;
>>>>
>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>
>>>> indexSearchers.add(newSearcher);
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> if (refreshMultiSeacher) {
>>>>
>>>> newMultiSeacher = new
>>>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
>>>>
>>>> warm(newMultiSeacher);
>>>>
>>>> swapMultiSearcher(newMultiSeacher);
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> }
>>>>
>>>>
>>>> private void warm(MultiSearcher newMultiSeacher) {
>>>>
>>>> }
>>>>
>>>>
>>>>
>>>> private synchronized MultiSearcher get() {
>>>>
>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>
>>>> indexSearcher.getIndexReader().incRef();
>>>>
>>>> }
>>>>
>>>> return multiSearcher;
>>>>
>>>> }
>>>>
>>>> private synchronized void release(MultiSearcher multiSearcher)
>>>> throwsIOException {
>>>>
>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>
>>>> indexSearcher.getIndexReader().decRef();
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> However I am now getting
>>>>
>>>>
>>>> java.lang.IllegalStateException:
>>>> org.apache.lucene.store.AlreadyClosedException: this IndexReader is
>>>> closed
>>>>
>>>>
>>>> on the call:
>>>>
>>>>
>>>> private synchronized MultiSearcher get() {
>>>>
>>>> for (IndexSearcher indexSearcher: searchers) {
>>>>
>>>> indexSearcher.getIndexReader().incRef();
>>>>
>>>> }
>>>>
>>>> return multiSearcher;
>>>>
>>>> }
>>>>
>>>>
>>>> I'm doing something wrong ..obviously..not sure where though..
>>>>
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>>
>>>>> I was wondering the same thing ;)
>>>>>
>>>>> It's best to call this method from a single BG "warming" thread,
>>>>> in
>>>>> which
>>>>> case it would not need its own synchronization.
>>>>>
>>>>> But, to be safe, I'll add internal synchronization to it. You
>>>>> can't
>>>>> simply put synchronized in front of the method, since you don't
>>>>> want
>>>>> this to
>>>>> block searching.
>>>>>
>>>>>
>>>>> Mike
>>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>> just a quick point:
>>>>>
>>>>>> public void maybeReopen() throws IOException
>>>>>> { //D
>>>>>> long currentVersion =
>>>>>> currentSearcher.getIndexReader().getVersion();
>>>>>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>>>>>> IndexReader newReader =
>>>>>> currentSearcher.getIndexReader().reopen();
>>>>>> assert newReader != currentSearcher.getIndexReader();
>>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>>> warm(newSearcher);
>>>>>> swapSearcher(newSearcher);
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> should the above be synchronised?
>>>>>>
>>>>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
>>>>>> aminmc@gmail.com
>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>
>>>>>> thanks. i will rewrite..in between giving my baby her feed and
>>>>>> playing
>>>>>>
>>>>>>> with the other child and my wife who wants me to do several
>>>>>>> other
>>>>>>> things!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> Thanks for your input. I would like to have a go at doing this
>>>>>>>>> myself
>>>>>>>>> first, Solr may be an option.
>>>>>>>>>
>>>>>>>>> * You are creating a new Analyzer & QueryParser every time,
>>>>>>>>> also
>>>>>>>>> creating unnecessary garbage; instead, they should be
>>>>>>>>> created once
>>>>>>>>> & reused.
>>>>>>>>>
>>>>>>>>> -- I can moved the code out so that it is only created once
>>>>>>>>> and
>>>>>>>>> reused.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>> MultiSearcher even
>>>>>>>>> when nothing has changed. This just generates unnecessary
>>>>>>>>> garbage
>>>>>>>>> which GC then must sweep up.
>>>>>>>>>
>>>>>>>>> -- This was something I thought about. I could move it out
>>>>>>>>> so that
>>>>>>>>> it's
>>>>>>>>> created once. However I presume inside my code i need to
>>>>>>>>> check
>>>>>>>>> whether
>>>>>>>>> the
>>>>>>>>> indexreaders are update to date. This needs to be
>>>>>>>>> synchronized as
>>>>>>>>> well I
>>>>>>>>> guess(?)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yes you should synchronize the check for whether the
>>>>>>>>> IndexReader is
>>>>>>>> current.
>>>>>>>>
>>>>>>>> * I don't see any synchronization -- it looks like two search
>>>>>>>>
>>>>>>>> requests are allowed into this method at the same time?
>>>>>>>> Which is
>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>>>>> readers.
>>>>>>>>> -- So i need to extract the logic for reopening and provide a
>>>>>>>>> synchronisation mechanism.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yes.
>>>>>>>>
>>>>>>>>
>>>>>>>> Ok. So I have some work to do. I'll refactor the code and
>>>>>>>> see if I
>>>>>>>> can
>>>>>>>>
>>>>>>>> get
>>>>>>>>> inline to your recommendations.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On a quick look, I think there are a few problems with the
>>>>>>>>> code:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> * I don't see any synchronization -- it looks like two search
>>>>>>>>>> requests are allowed into this method at the same time?
>>>>>>>>>> Which is
>>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>>>>>> readers.
>>>>>>>>>>
>>>>>>>>>> * You are over-incRef'ing (the reader.incRef inside the
>>>>>>>>>> loop) -- I
>>>>>>>>>> don't see a corresponding decRef.
>>>>>>>>>>
>>>>>>>>>> * You reopen and warm your searchers "live" (vs with BG
>>>>>>>>>> thread);
>>>>>>>>>> meaning the unlucky search request that hits a reopen pays
>>>>>>>>>> the
>>>>>>>>>> cost. This might be OK if the index is small enough that
>>>>>>>>>> reopening & warming takes very little time. But if index
>>>>>>>>>> gets
>>>>>>>>>> large, making a random search pay that warming cost is not
>>>>>>>>>> nice to
>>>>>>>>>> the end user. It erodes their trust in you.
>>>>>>>>>>
>>>>>>>>>> * You always make a new IndexSearcher and a new
>>>>>>>>>> MultiSearcher even
>>>>>>>>>> when nothing has changed. This just generates unnecessary
>>>>>>>>>> garbage
>>>>>>>>>> which GC then must sweep up.
>>>>>>>>>>
>>>>>>>>>> * You are creating a new Analyzer & QueryParser every time,
>>>>>>>>>> also
>>>>>>>>>> creating unnecessary garbage; instead, they should be
>>>>>>>>>> created once
>>>>>>>>>> & reused.
>>>>>>>>>>
>>>>>>>>>> You should consider simply using Solr -- it handles all
>>>>>>>>>> this logic
>>>>>>>>>> for
>>>>>>>>>> you and has been well debugged with time...
>>>>>>>>>>
>>>>>>>>>> Mike
>>>>>>>>>>
>>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>>
>>>>>>>>>> The reason for the indexreader.reopen is because I have a
>>>>>>>>>> webapp
>>>>>>>>>> which
>>>>>>>>>>
>>>>>>>>>> enables users to upload files and then search for the
>>>>>>>>>> documents.
>>>>>>>>>> If
>>>>>>>>>>
>>>>>>>>>>> I
>>>>>>>>>>> don't
>>>>>>>>>>> reopen i'm concerned that the facet hit counter won't be
>>>>>>>>>>> updated.
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <
>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi
>>>>>>>>>>>
>>>>>>>>>>> I have been able to get the code working for my scenario,
>>>>>>>>>>> however
>>>>>>>>>>> I
>>>>>>>>>>>
>>>>>>>>>>>> have
>>>>>>>>>>>> a
>>>>>>>>>>>> question and I was wondering if I could get some help. I
>>>>>>>>>>>> have a
>>>>>>>>>>>> list
>>>>>>>>>>>> of
>>>>>>>>>>>> IndexSearchers which are used in a MultiSearcher class.
>>>>>>>>>>>> I use
>>>>>>>>>>>> the
>>>>>>>>>>>> indexsearchers to get each indexreader and put them into a
>>>>>>>>>>>> MultiIndexReader.
>>>>>>>>>>>>
>>>>>>>>>>>> IndexReader[] readers = new
>>>>>>>>>>>> IndexReader[searchables.length];
>>>>>>>>>>>>
>>>>>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>>>>>
>>>>>>>>>>>> IndexSearcher indexSearcher =
>>>>>>>>>>>> (IndexSearcher)searchables[i];
>>>>>>>>>>>>
>>>>>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>>>>>
>>>>>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>>>>>
>>>>>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>>>>>
>>>>>>>>>>>> readers[i].close();
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> readers[i] = newReader;
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>>>>>
>>>>>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>>>>>>
>>>>>>>>>>>> IndexSearcher indexSearcher = new
>>>>>>>>>>>> IndexSearcher(multiReader);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I then use the indexseacher to do the facet stuff. I end
>>>>>>>>>>>> the
>>>>>>>>>>>> code
>>>>>>>>>>>> with
>>>>>>>>>>>> closing the multireader. This is causing problems in
>>>>>>>>>>>> another
>>>>>>>>>>>> method
>>>>>>>>>>>> where I
>>>>>>>>>>>> do some other search as the indexreaders are closed. Is
>>>>>>>>>>>> it ok to
>>>>>>>>>>>> not
>>>>>>>>>>>> close
>>>>>>>>>>>> the multiindexreader or should I do some additional
>>>>>>>>>>>> checks in the
>>>>>>>>>>>> other
>>>>>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
>>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks just what I needed!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>>>>>>>>>>>>> marcelo.ochoa@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Amin:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sorry to re send this email but I was wondering if I
>>>>>>>>>>>>>> could get
>>>>>>>>>>>>>>> some
>>>>>>>>>>>>>>> advice
>>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am looking at building a faceted search using
>>>>>>>>>>>>>>> Lucene. I
>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> Solr
>>>>>>>>>>>>>>>> comes with this built in, however I would like to try
>>>>>>>>>>>>>>>> this by
>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>> (something to add to my CV!). I have been looking
>>>>>>>>>>>>>>>> around and
>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> you can use the IndexReader and use TermVectors.
>>>>>>>>>>>>>>>> This looks
>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>> sure how to filter the results so that a particular
>>>>>>>>>>>>>>>> user can
>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> subset of results. The next option I was looking at
>>>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The only problem here is that I have to provide the
>>>>>>>>>>>>>>>> brand
>>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>>> time a
>>>>>>>>>>>>>>>> new brand is created. Again I'm not sure how I can
>>>>>>>>>>>>>>>> filter
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> results
>>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>>> It may be that I'm using the wrong api methods to do
>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> P.S. I am basically trying to do something that
>>>>>>>>>>>>>>>> displays the
>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>>>>>> ______________
>>>>>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
Hi
Thanks again for helping on a Sunday!
I have now modified my maybeOpen() to do the following:
private void maybeReopen() throws Exception {
LOGGER.debug("Initiating reopening of index readers...");
IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
.getSearchables();
for (IndexSearcher indexSearcher : indexSearchers) {
IndexReader indexReader = indexSearcher.getIndexReader();
SearcherManager documentSearcherManager = new
SearcherManager(indexReader.directory());
documentSearcherManager.maybeReopen();
}
}
And get() to:
private synchronized MultiSearcher get() {
IndexSearcher[] indexSearchers = (IndexSearcher[]) multiSearcher
.getSearchables();
List<IndexSearcher> indexSearchersList = new ArrayList<IndexSearcher>();
for (IndexSearcher indexSearcher : indexSearchers) {
IndexReader indexReader = indexSearcher.getIndexReader();
SearcherManager documentSearcherManager = null;
try {
documentSearcherManager = new SearcherManager(indexReader.directory());
} catch (IOException e) {
throw new IllegalStateException(e);
}
indexSearchersList.add(documentSearcherManager.get());
}
try {
multiSearcher = new
MultiSearcher(indexSearchersList.toArray(newIndexSearcher[] {}));
} catch (IOException e) {
throw new IllegalStateException(e);
}
return multiSearcher;
}
This makes all my test pass. I am using the SearchManager that you
recommended. Does this look ok?
On Sun, Mar 1, 2009 at 2:38 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> Your maybeReopen has an excess incRef().
>
> I'm not sure how you open the searchers in the first place? The list
> starts as empty, and nothing populates it?
>
> When you do the initial population, you need an incRef.
>
> I think you're hitting IllegalStateException because maybeReopen is
> closing a reader before get() can get it (since they synchronize on
> different objects).
>
> I'd recommend switching to the SearcherManager class. Instantiate one
> for each of your searchers. On each search request, go through them
> and call maybeReopen(), and then call get() and gather each
> IndexSearcher instance into a new array. Then, make a new
> MultiSearcher (opposite of what I said before): while that creates a
> small amount of garbage, it'll keep your code simpler (good
> tradeoff).
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> sorrry I added
>>
>> release(multiSearcher);
>>
>>
>> instead of multiSearcher.close();
>>
>> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <aminmc@gmail.com
>> >wrote:
>>
>> Hi
>>> I've now done the following:
>>>
>>> public Summary[] search(final SearchRequest searchRequest)
>>> throwsSearchExecutionException {
>>>
>>> final String searchTerm = searchRequest.getSearchTerm();
>>>
>>> if (StringUtils.isBlank(searchTerm)) {
>>>
>>> throw new SearchExecutionException("Search string cannot be empty. There
>>> will be too many results to process.");
>>>
>>> }
>>>
>>> List<Summary> summaryList = new ArrayList<Summary>();
>>>
>>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>>
>>> stopWatch.start();
>>>
>>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>>
>>> try {
>>>
>>> LOGGER.debug("Ensuring all index readers are up to date...");
>>>
>>> maybeReopen();
>>>
>>> LOGGER.debug("All Index Searchers are up to date. No of index searchers
>>> '"+ indexSearchers.size() +
>>> "'");
>>>
>>> Query query = queryParser.parse(searchTerm);
>>>
>>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
>>> query.toString() +"'");
>>>
>>> Sort sort = null;
>>>
>>> sort = applySortIfApplicable(searchRequest);
>>>
>>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>>
>>> ChainedFilter chainedFilter = null;
>>>
>>> if (filters != null) {
>>>
>>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>>
>>> }
>>>
>>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>>
>>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>>
>>> LOGGER.debug("total number of hits for [" + query.toString() + " ] =
>>> "+topDocs.
>>> totalHits);
>>>
>>> for (ScoreDoc scoreDoc : scoreDocs) {
>>>
>>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>>
>>> float score = scoreDoc.score;
>>>
>>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>>
>>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>>
>>> summaryList.add(documentSummary);
>>>
>>> }
>>>
>>> multiSearcher.close();
>>>
>>> } catch (Exception e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> }
>>>
>>> stopWatch.stop();
>>>
>>> LOGGER.debug("total time taken for document seach: " +
>>> stopWatch.getTotalTimeMillis() + " ms");
>>>
>>> return summaryList.toArray(new Summary[] {});
>>>
>>> }
>>>
>>>
>>> And have the following methods:
>>>
>>> @PostConstruct
>>>
>>> public void initialiseQueryParser() {
>>>
>>> PerFieldAnalyzerWrapper analyzerWrapper = new PerFieldAnalyzerWrapper(
>>> analyzer);
>>>
>>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>>> newKeywordAnalyzer());
>>>
>>> queryParser =
>>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>>>
>>> analyzerWrapper);
>>>
>>> try {
>>>
>>> LOGGER.debug("Initialising multi searcher ....");
>>>
>>> this.multiSearcher = new
>>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>>
>>> LOGGER.debug("multi searcher initialised");
>>>
>>> } catch (IOException e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> }
>>>
>>> }
>>>
>>>
>>> Initialises mutltisearcher when this class is creared by spring.
>>>
>>>
>>> private synchronized void swapMultiSearcher(MultiSearcher
>>> newMultiSearcher) {
>>>
>>> try {
>>>
>>> release(multiSearcher);
>>>
>>> } catch (IOException e) {
>>>
>>> throw new IllegalStateException(e);
>>>
>>> }
>>>
>>> multiSearcher = newMultiSearcher;
>>>
>>> }
>>>
>>> public void maybeReopen() throws IOException {
>>>
>>> MultiSearcher newMultiSeacher = null;
>>>
>>> boolean refreshMultiSeacher = false;
>>>
>>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>>
>>> synchronized (searchers) {
>>>
>>> for (IndexSearcher indexSearcher: searchers) {
>>>
>>> IndexReader reader = indexSearcher.getIndexReader();
>>>
>>> reader.incRef();
>>>
>>> Directory directory = reader.directory();
>>>
>>> long currentVersion = reader.getVersion();
>>>
>>> if (IndexReader.getCurrentVersion(directory) != currentVersion) {
>>>
>>> IndexReader newReader = indexSearcher.getIndexReader().reopen();
>>>
>>> if (newReader != reader) {
>>>
>>> reader.decRef();
>>>
>>> refreshMultiSeacher = true;
>>>
>>> }
>>>
>>> reader = newReader;
>>>
>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>
>>> indexSearchers.add(newSearcher);
>>>
>>> }
>>>
>>> }
>>>
>>> }
>>>
>>>
>>>
>>> if (refreshMultiSeacher) {
>>>
>>> newMultiSeacher = new
>>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
>>>
>>> warm(newMultiSeacher);
>>>
>>> swapMultiSearcher(newMultiSeacher);
>>>
>>> }
>>>
>>>
>>>
>>> }
>>>
>>>
>>> private void warm(MultiSearcher newMultiSeacher) {
>>>
>>> }
>>>
>>>
>>>
>>> private synchronized MultiSearcher get() {
>>>
>>> for (IndexSearcher indexSearcher: searchers) {
>>>
>>> indexSearcher.getIndexReader().incRef();
>>>
>>> }
>>>
>>> return multiSearcher;
>>>
>>> }
>>>
>>> private synchronized void release(MultiSearcher multiSearcher)
>>> throwsIOException {
>>>
>>> for (IndexSearcher indexSearcher: searchers) {
>>>
>>> indexSearcher.getIndexReader().decRef();
>>>
>>> }
>>>
>>> }
>>>
>>>
>>> However I am now getting
>>>
>>>
>>> java.lang.IllegalStateException:
>>> org.apache.lucene.store.AlreadyClosedException: this IndexReader is
>>> closed
>>>
>>>
>>> on the call:
>>>
>>>
>>> private synchronized MultiSearcher get() {
>>>
>>> for (IndexSearcher indexSearcher: searchers) {
>>>
>>> indexSearcher.getIndexReader().incRef();
>>>
>>> }
>>>
>>> return multiSearcher;
>>>
>>> }
>>>
>>>
>>> I'm doing something wrong ..obviously..not sure where though..
>>>
>>>
>>> Cheers
>>>
>>>
>>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>>> I was wondering the same thing ;)
>>>>
>>>> It's best to call this method from a single BG "warming" thread, in
>>>> which
>>>> case it would not need its own synchronization.
>>>>
>>>> But, to be safe, I'll add internal synchronization to it. You can't
>>>> simply put synchronized in front of the method, since you don't want
>>>> this to
>>>> block searching.
>>>>
>>>>
>>>> Mike
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> just a quick point:
>>>>
>>>>> public void maybeReopen() throws IOException { //D
>>>>> long currentVersion = currentSearcher.getIndexReader().getVersion();
>>>>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>>>>> IndexReader newReader = currentSearcher.getIndexReader().reopen();
>>>>> assert newReader != currentSearcher.getIndexReader();
>>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>>> warm(newSearcher);
>>>>> swapSearcher(newSearcher);
>>>>> }
>>>>> }
>>>>>
>>>>> should the above be synchronised?
>>>>>
>>>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <
>>>>> aminmc@gmail.com
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>
>>>>> thanks. i will rewrite..in between giving my baby her feed and playing
>>>>>
>>>>>> with the other child and my wife who wants me to do several other
>>>>>> things!
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> Thanks for your input. I would like to have a go at doing this
>>>>>>>> myself
>>>>>>>> first, Solr may be an option.
>>>>>>>>
>>>>>>>> * You are creating a new Analyzer & QueryParser every time, also
>>>>>>>> creating unnecessary garbage; instead, they should be created once
>>>>>>>> & reused.
>>>>>>>>
>>>>>>>> -- I can moved the code out so that it is only created once and
>>>>>>>> reused.
>>>>>>>>
>>>>>>>>
>>>>>>>> * You always make a new IndexSearcher and a new MultiSearcher even
>>>>>>>> when nothing has changed. This just generates unnecessary garbage
>>>>>>>> which GC then must sweep up.
>>>>>>>>
>>>>>>>> -- This was something I thought about. I could move it out so that
>>>>>>>> it's
>>>>>>>> created once. However I presume inside my code i need to check
>>>>>>>> whether
>>>>>>>> the
>>>>>>>> indexreaders are update to date. This needs to be synchronized as
>>>>>>>> well I
>>>>>>>> guess(?)
>>>>>>>>
>>>>>>>>
>>>>>>>> Yes you should synchronize the check for whether the IndexReader is
>>>>>>> current.
>>>>>>>
>>>>>>> * I don't see any synchronization -- it looks like two search
>>>>>>>
>>>>>>> requests are allowed into this method at the same time? Which is
>>>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>>>> readers.
>>>>>>>> -- So i need to extract the logic for reopening and provide a
>>>>>>>> synchronisation mechanism.
>>>>>>>>
>>>>>>>>
>>>>>>>> Yes.
>>>>>>>
>>>>>>>
>>>>>>> Ok. So I have some work to do. I'll refactor the code and see if I
>>>>>>> can
>>>>>>>
>>>>>>> get
>>>>>>>> inline to your recommendations.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On a quick look, I think there are a few problems with the code:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> * I don't see any synchronization -- it looks like two search
>>>>>>>>> requests are allowed into this method at the same time? Which is
>>>>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>>>>> readers.
>>>>>>>>>
>>>>>>>>> * You are over-incRef'ing (the reader.incRef inside the loop) -- I
>>>>>>>>> don't see a corresponding decRef.
>>>>>>>>>
>>>>>>>>> * You reopen and warm your searchers "live" (vs with BG thread);
>>>>>>>>> meaning the unlucky search request that hits a reopen pays the
>>>>>>>>> cost. This might be OK if the index is small enough that
>>>>>>>>> reopening & warming takes very little time. But if index gets
>>>>>>>>> large, making a random search pay that warming cost is not nice to
>>>>>>>>> the end user. It erodes their trust in you.
>>>>>>>>>
>>>>>>>>> * You always make a new IndexSearcher and a new MultiSearcher even
>>>>>>>>> when nothing has changed. This just generates unnecessary garbage
>>>>>>>>> which GC then must sweep up.
>>>>>>>>>
>>>>>>>>> * You are creating a new Analyzer & QueryParser every time, also
>>>>>>>>> creating unnecessary garbage; instead, they should be created once
>>>>>>>>> & reused.
>>>>>>>>>
>>>>>>>>> You should consider simply using Solr -- it handles all this logic
>>>>>>>>> for
>>>>>>>>> you and has been well debugged with time...
>>>>>>>>>
>>>>>>>>> Mike
>>>>>>>>>
>>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>>
>>>>>>>>> The reason for the indexreader.reopen is because I have a webapp
>>>>>>>>> which
>>>>>>>>>
>>>>>>>>> enables users to upload files and then search for the documents.
>>>>>>>>> If
>>>>>>>>>
>>>>>>>>>> I
>>>>>>>>>> don't
>>>>>>>>>> reopen i'm concerned that the facet hit counter won't be updated.
>>>>>>>>>>
>>>>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <
>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> I have been able to get the code working for my scenario, however
>>>>>>>>>> I
>>>>>>>>>>
>>>>>>>>>>> have
>>>>>>>>>>> a
>>>>>>>>>>> question and I was wondering if I could get some help. I have a
>>>>>>>>>>> list
>>>>>>>>>>> of
>>>>>>>>>>> IndexSearchers which are used in a MultiSearcher class. I use
>>>>>>>>>>> the
>>>>>>>>>>> indexsearchers to get each indexreader and put them into a
>>>>>>>>>>> MultiIndexReader.
>>>>>>>>>>>
>>>>>>>>>>> IndexReader[] readers = new IndexReader[searchables.length];
>>>>>>>>>>>
>>>>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>>>>
>>>>>>>>>>> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>>>>>>>>>>>
>>>>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>>>>
>>>>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>>>>
>>>>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>>>>
>>>>>>>>>>> readers[i].close();
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> readers[i] = newReader;
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>>>>
>>>>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>>>>>
>>>>>>>>>>> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I then use the indexseacher to do the facet stuff. I end the
>>>>>>>>>>> code
>>>>>>>>>>> with
>>>>>>>>>>> closing the multireader. This is causing problems in another
>>>>>>>>>>> method
>>>>>>>>>>> where I
>>>>>>>>>>> do some other search as the indexreaders are closed. Is it ok to
>>>>>>>>>>> not
>>>>>>>>>>> close
>>>>>>>>>>> the multiindexreader or should I do some additional checks in the
>>>>>>>>>>> other
>>>>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Cheers
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
>>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks just what I needed!
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers
>>>>>>>>>>>> Amin
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <
>>>>>>>>>>>> marcelo.ochoa@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Amin:
>>>>>>>>>>>>
>>>>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sorry to re send this email but I was wondering if I could get
>>>>>>>>>>>>>> some
>>>>>>>>>>>>>> advice
>>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am looking at building a faceted search using Lucene. I
>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> Solr
>>>>>>>>>>>>>>> comes with this built in, however I would like to try this by
>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>> (something to add to my CV!). I have been looking around and
>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> you can use the IndexReader and use TermVectors. This looks
>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>> sure how to filter the results so that a particular user can
>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>> subset of results. The next option I was looking at was
>>>>>>>>>>>>>>> something
>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The only problem here is that I have to provide the brand
>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>> each
>>>>>>>>>>>>>>> time a
>>>>>>>>>>>>>>> new brand is created. Again I'm not sure how I can filter
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> results
>>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>>> It may be that I'm using the wrong api methods to do this.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> P.S. I am basically trying to do something that displays the
>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>>>>> ______________
>>>>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>> java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>
>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Faceted Search using Lucene
Posted by Michael McCandless <lu...@mikemccandless.com>.
Your maybeReopen has an excess incRef().
I'm not sure how you open the searchers in the first place? The list
starts as empty, and nothing populates it?
When you do the initial population, you need an incRef.
I think you're hitting IllegalStateException because maybeReopen is
closing a reader before get() can get it (since they synchronize on
different objects).
I'd recommend switching to the SearcherManager class. Instantiate one
for each of your searchers. On each search request, go through them
and call maybeReopen(), and then call get() and gather each
IndexSearcher instance into a new array. Then, make a new
MultiSearcher (opposite of what I said before): while that creates a
small amount of garbage, it'll keep your code simpler (good
tradeoff).
Mike
Amin Mohammed-Coleman wrote:
> sorrry I added
>
> release(multiSearcher);
>
>
> instead of multiSearcher.close();
>
> On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <aminmc@gmail.com
> >wrote:
>
>> Hi
>> I've now done the following:
>>
>> public Summary[] search(final SearchRequest searchRequest)
>> throwsSearchExecutionException {
>>
>> final String searchTerm = searchRequest.getSearchTerm();
>>
>> if (StringUtils.isBlank(searchTerm)) {
>>
>> throw new SearchExecutionException("Search string cannot be empty.
>> There
>> will be too many results to process.");
>>
>> }
>>
>> List<Summary> summaryList = new ArrayList<Summary>();
>>
>> StopWatch stopWatch = new StopWatch("searchStopWatch");
>>
>> stopWatch.start();
>>
>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>
>> try {
>>
>> LOGGER.debug("Ensuring all index readers are up to date...");
>>
>> maybeReopen();
>>
>> LOGGER.debug("All Index Searchers are up to date. No of index
>> searchers '"+ indexSearchers.size() +
>> "'");
>>
>> Query query = queryParser.parse(searchTerm);
>>
>> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
>> query.toString() +"'");
>>
>> Sort sort = null;
>>
>> sort = applySortIfApplicable(searchRequest);
>>
>> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>>
>> ChainedFilter chainedFilter = null;
>>
>> if (filters != null) {
>>
>> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>>
>> }
>>
>> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>>
>> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>>
>> LOGGER.debug("total number of hits for [" + query.toString() + " ]
>> = "+topDocs.
>> totalHits);
>>
>> for (ScoreDoc scoreDoc : scoreDocs) {
>>
>> final Document doc = multiSearcher.doc(scoreDoc.doc);
>>
>> float score = scoreDoc.score;
>>
>> final BaseDocument baseDocument = new BaseDocument(doc, score);
>>
>> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>>
>> summaryList.add(documentSummary);
>>
>> }
>>
>> multiSearcher.close();
>>
>> } catch (Exception e) {
>>
>> throw new IllegalStateException(e);
>>
>> }
>>
>> stopWatch.stop();
>>
>> LOGGER.debug("total time taken for document seach: " +
>> stopWatch.getTotalTimeMillis() + " ms");
>>
>> return summaryList.toArray(new Summary[] {});
>>
>> }
>>
>>
>> And have the following methods:
>>
>> @PostConstruct
>>
>> public void initialiseQueryParser() {
>>
>> PerFieldAnalyzerWrapper analyzerWrapper = new
>> PerFieldAnalyzerWrapper(
>> analyzer);
>>
>> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
>> newKeywordAnalyzer());
>>
>> queryParser =
>> newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
>> analyzerWrapper);
>>
>> try {
>>
>> LOGGER.debug("Initialising multi searcher ....");
>>
>> this.multiSearcher = new
>> MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>>
>> LOGGER.debug("multi searcher initialised");
>>
>> } catch (IOException e) {
>>
>> throw new IllegalStateException(e);
>>
>> }
>>
>> }
>>
>>
>> Initialises mutltisearcher when this class is creared by spring.
>>
>>
>> private synchronized void swapMultiSearcher(MultiSearcher
>> newMultiSearcher) {
>>
>> try {
>>
>> release(multiSearcher);
>>
>> } catch (IOException e) {
>>
>> throw new IllegalStateException(e);
>>
>> }
>>
>> multiSearcher = newMultiSearcher;
>>
>> }
>>
>> public void maybeReopen() throws IOException {
>>
>> MultiSearcher newMultiSeacher = null;
>>
>> boolean refreshMultiSeacher = false;
>>
>> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>>
>> synchronized (searchers) {
>>
>> for (IndexSearcher indexSearcher: searchers) {
>>
>> IndexReader reader = indexSearcher.getIndexReader();
>>
>> reader.incRef();
>>
>> Directory directory = reader.directory();
>>
>> long currentVersion = reader.getVersion();
>>
>> if (IndexReader.getCurrentVersion(directory) != currentVersion) {
>>
>> IndexReader newReader = indexSearcher.getIndexReader().reopen();
>>
>> if (newReader != reader) {
>>
>> reader.decRef();
>>
>> refreshMultiSeacher = true;
>>
>> }
>>
>> reader = newReader;
>>
>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>
>> indexSearchers.add(newSearcher);
>>
>> }
>>
>> }
>>
>> }
>>
>>
>>
>> if (refreshMultiSeacher) {
>>
>> newMultiSeacher = new
>> MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
>>
>> warm(newMultiSeacher);
>>
>> swapMultiSearcher(newMultiSeacher);
>>
>> }
>>
>>
>>
>> }
>>
>>
>> private void warm(MultiSearcher newMultiSeacher) {
>>
>> }
>>
>>
>>
>> private synchronized MultiSearcher get() {
>>
>> for (IndexSearcher indexSearcher: searchers) {
>>
>> indexSearcher.getIndexReader().incRef();
>>
>> }
>>
>> return multiSearcher;
>>
>> }
>>
>> private synchronized void release(MultiSearcher multiSearcher)
>> throwsIOException {
>>
>> for (IndexSearcher indexSearcher: searchers) {
>>
>> indexSearcher.getIndexReader().decRef();
>>
>> }
>>
>> }
>>
>>
>> However I am now getting
>>
>>
>> java.lang.IllegalStateException:
>> org.apache.lucene.store.AlreadyClosedException: this IndexReader is
>> closed
>>
>>
>> on the call:
>>
>>
>> private synchronized MultiSearcher get() {
>>
>> for (IndexSearcher indexSearcher: searchers) {
>>
>> indexSearcher.getIndexReader().incRef();
>>
>> }
>>
>> return multiSearcher;
>>
>> }
>>
>>
>> I'm doing something wrong ..obviously..not sure where though..
>>
>>
>> Cheers
>>
>>
>> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>>
>>> I was wondering the same thing ;)
>>>
>>> It's best to call this method from a single BG "warming" thread,
>>> in which
>>> case it would not need its own synchronization.
>>>
>>> But, to be safe, I'll add internal synchronization to it. You can't
>>> simply put synchronized in front of the method, since you don't
>>> want this to
>>> block searching.
>>>
>>>
>>> Mike
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> just a quick point:
>>>> public void maybeReopen() throws IOException { //D
>>>> long currentVersion =
>>>> currentSearcher.getIndexReader().getVersion();
>>>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>>>> IndexReader newReader =
>>>> currentSearcher.getIndexReader().reopen();
>>>> assert newReader != currentSearcher.getIndexReader();
>>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>>> warm(newSearcher);
>>>> swapSearcher(newSearcher);
>>>> }
>>>> }
>>>>
>>>> should the above be synchronised?
>>>>
>>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <aminmc@gmail.com
>>>>> wrote:
>>>>
>>>> thanks. i will rewrite..in between giving my baby her feed and
>>>> playing
>>>>> with the other child and my wife who wants me to do several other
>>>>> things!
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>>> Thanks for your input. I would like to have a go at doing
>>>>>>> this myself
>>>>>>> first, Solr may be an option.
>>>>>>>
>>>>>>> * You are creating a new Analyzer & QueryParser every time, also
>>>>>>> creating unnecessary garbage; instead, they should be created
>>>>>>> once
>>>>>>> & reused.
>>>>>>>
>>>>>>> -- I can moved the code out so that it is only created once and
>>>>>>> reused.
>>>>>>>
>>>>>>>
>>>>>>> * You always make a new IndexSearcher and a new MultiSearcher
>>>>>>> even
>>>>>>> when nothing has changed. This just generates unnecessary
>>>>>>> garbage
>>>>>>> which GC then must sweep up.
>>>>>>>
>>>>>>> -- This was something I thought about. I could move it out so
>>>>>>> that
>>>>>>> it's
>>>>>>> created once. However I presume inside my code i need to check
>>>>>>> whether
>>>>>>> the
>>>>>>> indexreaders are update to date. This needs to be
>>>>>>> synchronized as
>>>>>>> well I
>>>>>>> guess(?)
>>>>>>>
>>>>>>>
>>>>>> Yes you should synchronize the check for whether the
>>>>>> IndexReader is
>>>>>> current.
>>>>>>
>>>>>> * I don't see any synchronization -- it looks like two search
>>>>>>
>>>>>>> requests are allowed into this method at the same time? Which
>>>>>>> is
>>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>>> readers.
>>>>>>> -- So i need to extract the logic for reopening and provide a
>>>>>>> synchronisation mechanism.
>>>>>>>
>>>>>>>
>>>>>> Yes.
>>>>>>
>>>>>>
>>>>>> Ok. So I have some work to do. I'll refactor the code and see
>>>>>> if I
>>>>>> can
>>>>>>
>>>>>>> get
>>>>>>> inline to your recommendations.
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> On a quick look, I think there are a few problems with the code:
>>>>>>>>
>>>>>>>> * I don't see any synchronization -- it looks like two search
>>>>>>>> requests are allowed into this method at the same time?
>>>>>>>> Which is
>>>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>>>> readers.
>>>>>>>>
>>>>>>>> * You are over-incRef'ing (the reader.incRef inside the loop)
>>>>>>>> -- I
>>>>>>>> don't see a corresponding decRef.
>>>>>>>>
>>>>>>>> * You reopen and warm your searchers "live" (vs with BG
>>>>>>>> thread);
>>>>>>>> meaning the unlucky search request that hits a reopen pays the
>>>>>>>> cost. This might be OK if the index is small enough that
>>>>>>>> reopening & warming takes very little time. But if index gets
>>>>>>>> large, making a random search pay that warming cost is not
>>>>>>>> nice to
>>>>>>>> the end user. It erodes their trust in you.
>>>>>>>>
>>>>>>>> * You always make a new IndexSearcher and a new MultiSearcher
>>>>>>>> even
>>>>>>>> when nothing has changed. This just generates unnecessary
>>>>>>>> garbage
>>>>>>>> which GC then must sweep up.
>>>>>>>>
>>>>>>>> * You are creating a new Analyzer & QueryParser every time,
>>>>>>>> also
>>>>>>>> creating unnecessary garbage; instead, they should be created
>>>>>>>> once
>>>>>>>> & reused.
>>>>>>>>
>>>>>>>> You should consider simply using Solr -- it handles all this
>>>>>>>> logic
>>>>>>>> for
>>>>>>>> you and has been well debugged with time...
>>>>>>>>
>>>>>>>> Mike
>>>>>>>>
>>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>>
>>>>>>>> The reason for the indexreader.reopen is because I have a
>>>>>>>> webapp
>>>>>>>> which
>>>>>>>>
>>>>>>>> enables users to upload files and then search for the
>>>>>>>> documents. If
>>>>>>>>> I
>>>>>>>>> don't
>>>>>>>>> reopen i'm concerned that the facet hit counter won't be
>>>>>>>>> updated.
>>>>>>>>>
>>>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <
>>>>>>>>> aminmc@gmail.com
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> I have been able to get the code working for my scenario,
>>>>>>>>> however I
>>>>>>>>>> have
>>>>>>>>>> a
>>>>>>>>>> question and I was wondering if I could get some help. I
>>>>>>>>>> have a
>>>>>>>>>> list
>>>>>>>>>> of
>>>>>>>>>> IndexSearchers which are used in a MultiSearcher class. I
>>>>>>>>>> use the
>>>>>>>>>> indexsearchers to get each indexreader and put them into a
>>>>>>>>>> MultiIndexReader.
>>>>>>>>>>
>>>>>>>>>> IndexReader[] readers = new IndexReader[searchables.length];
>>>>>>>>>>
>>>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>>>
>>>>>>>>>> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>>>>>>>>>>
>>>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>>>
>>>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>>>
>>>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>>>
>>>>>>>>>> readers[i].close();
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> readers[i] = newReader;
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>>>
>>>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>>>>
>>>>>>>>>> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I then use the indexseacher to do the facet stuff. I end
>>>>>>>>>> the code
>>>>>>>>>> with
>>>>>>>>>> closing the multireader. This is causing problems in another
>>>>>>>>>> method
>>>>>>>>>> where I
>>>>>>>>>> do some other search as the indexreaders are closed. Is it
>>>>>>>>>> ok to
>>>>>>>>>> not
>>>>>>>>>> close
>>>>>>>>>> the multiindexreader or should I do some additional checks
>>>>>>>>>> in the
>>>>>>>>>> other
>>>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
>>>>>>>>>> aminmc@gmail.com
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Thanks just what I needed!
>>>>>>>>>>>
>>>>>>>>>>> Cheers
>>>>>>>>>>> Amin
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <marcelo.ochoa@gmail.com
>>>>>>>>>>> >
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Amin:
>>>>>>>>>>>
>>>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Sorry to re send this email but I was wondering if I
>>>>>>>>>>>>> could get
>>>>>>>>>>>>> some
>>>>>>>>>>>>> advice
>>>>>>>>>>>>> on this.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>
>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am looking at building a faceted search using Lucene.
>>>>>>>>>>>>> I know
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>> Solr
>>>>>>>>>>>>>> comes with this built in, however I would like to try
>>>>>>>>>>>>>> this by
>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>> (something to add to my CV!). I have been looking
>>>>>>>>>>>>>> around and I
>>>>>>>>>>>>>> found
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>> you can use the IndexReader and use TermVectors. This
>>>>>>>>>>>>>> looks ok
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>> not
>>>>>>>>>>>>>> sure how to filter the results so that a particular
>>>>>>>>>>>>>> user can
>>>>>>>>>>>>>> only
>>>>>>>>>>>>>> see
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>> subset of results. The next option I was looking at was
>>>>>>>>>>>>>> something
>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The only problem here is that I have to provide the
>>>>>>>>>>>>>> brand type
>>>>>>>>>>>>>> each
>>>>>>>>>>>>>> time a
>>>>>>>>>>>>>> new brand is created. Again I'm not sure how I can
>>>>>>>>>>>>>> filter the
>>>>>>>>>>>>>> results
>>>>>>>>>>>>>> here.
>>>>>>>>>>>>>> It may be that I'm using the wrong api methods to do
>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> P.S. I am basically trying to do something that
>>>>>>>>>>>>>> displays the
>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>>>> ______________
>>>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
sorrry I added
release(multiSearcher);
instead of multiSearcher.close();
On Sun, Mar 1, 2009 at 2:17 PM, Amin Mohammed-Coleman <am...@gmail.com>wrote:
> Hi
> I've now done the following:
>
> public Summary[] search(final SearchRequest searchRequest) throwsSearchExecutionException {
>
> final String searchTerm = searchRequest.getSearchTerm();
>
> if (StringUtils.isBlank(searchTerm)) {
>
> throw new SearchExecutionException("Search string cannot be empty. There
> will be too many results to process.");
>
> }
>
> List<Summary> summaryList = new ArrayList<Summary>();
>
> StopWatch stopWatch = new StopWatch("searchStopWatch");
>
> stopWatch.start();
>
> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>
> try {
>
> LOGGER.debug("Ensuring all index readers are up to date...");
>
> maybeReopen();
>
> LOGGER.debug("All Index Searchers are up to date. No of index searchers '"+ indexSearchers.size() +
> "'");
>
> Query query = queryParser.parse(searchTerm);
>
> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
> query.toString() +"'");
>
> Sort sort = null;
>
> sort = applySortIfApplicable(searchRequest);
>
> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>
> ChainedFilter chainedFilter = null;
>
> if (filters != null) {
>
> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>
> }
>
> TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
>
> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>
> LOGGER.debug("total number of hits for [" + query.toString() + " ] = "+topDocs.
> totalHits);
>
> for (ScoreDoc scoreDoc : scoreDocs) {
>
> final Document doc = multiSearcher.doc(scoreDoc.doc);
>
> float score = scoreDoc.score;
>
> final BaseDocument baseDocument = new BaseDocument(doc, score);
>
> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>
> summaryList.add(documentSummary);
>
> }
>
> multiSearcher.close();
>
> } catch (Exception e) {
>
> throw new IllegalStateException(e);
>
> }
>
> stopWatch.stop();
>
> LOGGER.debug("total time taken for document seach: " +
> stopWatch.getTotalTimeMillis() + " ms");
>
> return summaryList.toArray(new Summary[] {});
>
> }
>
>
> And have the following methods:
>
> @PostConstruct
>
> public void initialiseQueryParser() {
>
> PerFieldAnalyzerWrapper analyzerWrapper = new PerFieldAnalyzerWrapper(
> analyzer);
>
> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(), newKeywordAnalyzer());
>
> queryParser = newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
> analyzerWrapper);
>
> try {
>
> LOGGER.debug("Initialising multi searcher ....");
>
> this.multiSearcher = new MultiSearcher(searchers.toArray(newIndexSearcher[] {}));
>
> LOGGER.debug("multi searcher initialised");
>
> } catch (IOException e) {
>
> throw new IllegalStateException(e);
>
> }
>
> }
>
>
> Initialises mutltisearcher when this class is creared by spring.
>
>
> private synchronized void swapMultiSearcher(MultiSearcher
> newMultiSearcher) {
>
> try {
>
> release(multiSearcher);
>
> } catch (IOException e) {
>
> throw new IllegalStateException(e);
>
> }
>
> multiSearcher = newMultiSearcher;
>
> }
>
> public void maybeReopen() throws IOException {
>
> MultiSearcher newMultiSeacher = null;
>
> boolean refreshMultiSeacher = false;
>
> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>
> synchronized (searchers) {
>
> for (IndexSearcher indexSearcher: searchers) {
>
> IndexReader reader = indexSearcher.getIndexReader();
>
> reader.incRef();
>
> Directory directory = reader.directory();
>
> long currentVersion = reader.getVersion();
>
> if (IndexReader.getCurrentVersion(directory) != currentVersion) {
>
> IndexReader newReader = indexSearcher.getIndexReader().reopen();
>
> if (newReader != reader) {
>
> reader.decRef();
>
> refreshMultiSeacher = true;
>
> }
>
> reader = newReader;
>
> IndexSearcher newSearcher = new IndexSearcher(newReader);
>
> indexSearchers.add(newSearcher);
>
> }
>
> }
>
> }
>
>
>
> if (refreshMultiSeacher) {
>
> newMultiSeacher = new MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
>
> warm(newMultiSeacher);
>
> swapMultiSearcher(newMultiSeacher);
>
> }
>
>
>
> }
>
>
> private void warm(MultiSearcher newMultiSeacher) {
>
> }
>
>
>
> private synchronized MultiSearcher get() {
>
> for (IndexSearcher indexSearcher: searchers) {
>
> indexSearcher.getIndexReader().incRef();
>
> }
>
> return multiSearcher;
>
> }
>
> private synchronized void release(MultiSearcher multiSearcher) throwsIOException {
>
> for (IndexSearcher indexSearcher: searchers) {
>
> indexSearcher.getIndexReader().decRef();
>
> }
>
> }
>
>
> However I am now getting
>
>
> java.lang.IllegalStateException:
> org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed
>
>
> on the call:
>
>
> private synchronized MultiSearcher get() {
>
> for (IndexSearcher indexSearcher: searchers) {
>
> indexSearcher.getIndexReader().incRef();
>
> }
>
> return multiSearcher;
>
> }
>
>
> I'm doing something wrong ..obviously..not sure where though..
>
>
> Cheers
>
>
> On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>>
>> I was wondering the same thing ;)
>>
>> It's best to call this method from a single BG "warming" thread, in which
>> case it would not need its own synchronization.
>>
>> But, to be safe, I'll add internal synchronization to it. You can't
>> simply put synchronized in front of the method, since you don't want this to
>> block searching.
>>
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> just a quick point:
>>> public void maybeReopen() throws IOException { //D
>>> long currentVersion = currentSearcher.getIndexReader().getVersion();
>>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>>> IndexReader newReader = currentSearcher.getIndexReader().reopen();
>>> assert newReader != currentSearcher.getIndexReader();
>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>> warm(newSearcher);
>>> swapSearcher(newSearcher);
>>> }
>>> }
>>>
>>> should the above be synchronised?
>>>
>>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <aminmc@gmail.com
>>> >wrote:
>>>
>>> thanks. i will rewrite..in between giving my baby her feed and playing
>>>> with the other child and my wife who wants me to do several other
>>>> things!
>>>>
>>>>
>>>>
>>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>>> Thanks for your input. I would like to have a go at doing this myself
>>>>>> first, Solr may be an option.
>>>>>>
>>>>>> * You are creating a new Analyzer & QueryParser every time, also
>>>>>> creating unnecessary garbage; instead, they should be created once
>>>>>> & reused.
>>>>>>
>>>>>> -- I can moved the code out so that it is only created once and
>>>>>> reused.
>>>>>>
>>>>>>
>>>>>> * You always make a new IndexSearcher and a new MultiSearcher even
>>>>>> when nothing has changed. This just generates unnecessary garbage
>>>>>> which GC then must sweep up.
>>>>>>
>>>>>> -- This was something I thought about. I could move it out so that
>>>>>> it's
>>>>>> created once. However I presume inside my code i need to check
>>>>>> whether
>>>>>> the
>>>>>> indexreaders are update to date. This needs to be synchronized as
>>>>>> well I
>>>>>> guess(?)
>>>>>>
>>>>>>
>>>>> Yes you should synchronize the check for whether the IndexReader is
>>>>> current.
>>>>>
>>>>> * I don't see any synchronization -- it looks like two search
>>>>>
>>>>>> requests are allowed into this method at the same time? Which is
>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>> readers.
>>>>>> -- So i need to extract the logic for reopening and provide a
>>>>>> synchronisation mechanism.
>>>>>>
>>>>>>
>>>>> Yes.
>>>>>
>>>>>
>>>>> Ok. So I have some work to do. I'll refactor the code and see if I
>>>>> can
>>>>>
>>>>>> get
>>>>>> inline to your recommendations.
>>>>>>
>>>>>>
>>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>>>> lucene@mikemccandless.com> wrote:
>>>>>>
>>>>>>
>>>>>> On a quick look, I think there are a few problems with the code:
>>>>>>>
>>>>>>> * I don't see any synchronization -- it looks like two search
>>>>>>> requests are allowed into this method at the same time? Which is
>>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>>> readers.
>>>>>>>
>>>>>>> * You are over-incRef'ing (the reader.incRef inside the loop) -- I
>>>>>>> don't see a corresponding decRef.
>>>>>>>
>>>>>>> * You reopen and warm your searchers "live" (vs with BG thread);
>>>>>>> meaning the unlucky search request that hits a reopen pays the
>>>>>>> cost. This might be OK if the index is small enough that
>>>>>>> reopening & warming takes very little time. But if index gets
>>>>>>> large, making a random search pay that warming cost is not nice to
>>>>>>> the end user. It erodes their trust in you.
>>>>>>>
>>>>>>> * You always make a new IndexSearcher and a new MultiSearcher even
>>>>>>> when nothing has changed. This just generates unnecessary garbage
>>>>>>> which GC then must sweep up.
>>>>>>>
>>>>>>> * You are creating a new Analyzer & QueryParser every time, also
>>>>>>> creating unnecessary garbage; instead, they should be created once
>>>>>>> & reused.
>>>>>>>
>>>>>>> You should consider simply using Solr -- it handles all this logic
>>>>>>> for
>>>>>>> you and has been well debugged with time...
>>>>>>>
>>>>>>> Mike
>>>>>>>
>>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>>
>>>>>>> The reason for the indexreader.reopen is because I have a webapp
>>>>>>> which
>>>>>>>
>>>>>>> enables users to upload files and then search for the documents. If
>>>>>>>> I
>>>>>>>> don't
>>>>>>>> reopen i'm concerned that the facet hit counter won't be updated.
>>>>>>>>
>>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <
>>>>>>>> aminmc@gmail.com
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> I have been able to get the code working for my scenario, however I
>>>>>>>>> have
>>>>>>>>> a
>>>>>>>>> question and I was wondering if I could get some help. I have a
>>>>>>>>> list
>>>>>>>>> of
>>>>>>>>> IndexSearchers which are used in a MultiSearcher class. I use the
>>>>>>>>> indexsearchers to get each indexreader and put them into a
>>>>>>>>> MultiIndexReader.
>>>>>>>>>
>>>>>>>>> IndexReader[] readers = new IndexReader[searchables.length];
>>>>>>>>>
>>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>>
>>>>>>>>> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>>>>>>>>>
>>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>>
>>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>>
>>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>>
>>>>>>>>> readers[i].close();
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> readers[i] = newReader;
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>>
>>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>>>
>>>>>>>>> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I then use the indexseacher to do the facet stuff. I end the code
>>>>>>>>> with
>>>>>>>>> closing the multireader. This is causing problems in another
>>>>>>>>> method
>>>>>>>>> where I
>>>>>>>>> do some other search as the indexreaders are closed. Is it ok to
>>>>>>>>> not
>>>>>>>>> close
>>>>>>>>> the multiindexreader or should I do some additional checks in the
>>>>>>>>> other
>>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
>>>>>>>>> aminmc@gmail.com
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Thanks just what I needed!
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>> Amin
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <ma...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Amin:
>>>>>>>>>>
>>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Sorry to re send this email but I was wondering if I could get
>>>>>>>>>>>> some
>>>>>>>>>>>> advice
>>>>>>>>>>>> on this.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers
>>>>>>>>>>>>
>>>>>>>>>>>> Amin
>>>>>>>>>>>>
>>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I am looking at building a faceted search using Lucene. I know
>>>>>>>>>>>>> that
>>>>>>>>>>>>> Solr
>>>>>>>>>>>>> comes with this built in, however I would like to try this by
>>>>>>>>>>>>> myself
>>>>>>>>>>>>> (something to add to my CV!). I have been looking around and I
>>>>>>>>>>>>> found
>>>>>>>>>>>>> that
>>>>>>>>>>>>> you can use the IndexReader and use TermVectors. This looks ok
>>>>>>>>>>>>> but
>>>>>>>>>>>>> I'm
>>>>>>>>>>>>> not
>>>>>>>>>>>>> sure how to filter the results so that a particular user can
>>>>>>>>>>>>> only
>>>>>>>>>>>>> see
>>>>>>>>>>>>> a
>>>>>>>>>>>>> subset of results. The next option I was looking at was
>>>>>>>>>>>>> something
>>>>>>>>>>>>> like
>>>>>>>>>>>>>
>>>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>>
>>>>>>>>>>>>> The only problem here is that I have to provide the brand type
>>>>>>>>>>>>> each
>>>>>>>>>>>>> time a
>>>>>>>>>>>>> new brand is created. Again I'm not sure how I can filter the
>>>>>>>>>>>>> results
>>>>>>>>>>>>> here.
>>>>>>>>>>>>> It may be that I'm using the wrong api methods to do this.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>> Amin
>>>>>>>>>>>>>
>>>>>>>>>>>>> P.S. I am basically trying to do something that displays the
>>>>>>>>>>>>> following
>>>>>>>>>>>>>
>>>>>>>>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>>> ______________
>>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>> java-user-help@lucene.apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
Hi
I've now done the following:
public Summary[] search(final SearchRequest searchRequest)
throwsSearchExecutionException {
final String searchTerm = searchRequest.getSearchTerm();
if (StringUtils.isBlank(searchTerm)) {
throw new SearchExecutionException("Search string cannot be empty. There
will be too many results to process.");
}
List<Summary> summaryList = new ArrayList<Summary>();
StopWatch stopWatch = new StopWatch("searchStopWatch");
stopWatch.start();
List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
try {
LOGGER.debug("Ensuring all index readers are up to date...");
maybeReopen();
LOGGER.debug("All Index Searchers are up to date. No of index searchers '" +
indexSearchers.size() +"'");
Query query = queryParser.parse(searchTerm);
LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
query.toString() +"'");
Sort sort = null;
sort = applySortIfApplicable(searchRequest);
Filter[] filters =applyFiltersIfApplicable(searchRequest);
ChainedFilter chainedFilter = null;
if (filters != null) {
chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
}
TopDocs topDocs = get().search(query,chainedFilter ,100,sort);
ScoreDoc[] scoreDocs = topDocs.scoreDocs;
LOGGER.debug("total number of hits for [" + query.toString() + " ] = "+topDocs.
totalHits);
for (ScoreDoc scoreDoc : scoreDocs) {
final Document doc = multiSearcher.doc(scoreDoc.doc);
float score = scoreDoc.score;
final BaseDocument baseDocument = new BaseDocument(doc, score);
Summary documentSummary = new DocumentSummaryImpl(baseDocument);
summaryList.add(documentSummary);
}
multiSearcher.close();
} catch (Exception e) {
throw new IllegalStateException(e);
}
stopWatch.stop();
LOGGER.debug("total time taken for document seach: " +
stopWatch.getTotalTimeMillis() + " ms");
return summaryList.toArray(new Summary[] {});
}
And have the following methods:
@PostConstruct
public void initialiseQueryParser() {
PerFieldAnalyzerWrapper analyzerWrapper = new PerFieldAnalyzerWrapper(
analyzer);
analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
newKeywordAnalyzer());
queryParser = newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
analyzerWrapper);
try {
LOGGER.debug("Initialising multi searcher ....");
this.multiSearcher = new MultiSearcher(searchers.toArray(new IndexSearcher[]
{}));
LOGGER.debug("multi searcher initialised");
} catch (IOException e) {
throw new IllegalStateException(e);
}
}
Initialises mutltisearcher when this class is creared by spring.
private synchronized void swapMultiSearcher(MultiSearcher newMultiSearcher)
{
try {
release(multiSearcher);
} catch (IOException e) {
throw new IllegalStateException(e);
}
multiSearcher = newMultiSearcher;
}
public void maybeReopen() throws IOException {
MultiSearcher newMultiSeacher = null;
boolean refreshMultiSeacher = false;
List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
synchronized (searchers) {
for (IndexSearcher indexSearcher: searchers) {
IndexReader reader = indexSearcher.getIndexReader();
reader.incRef();
Directory directory = reader.directory();
long currentVersion = reader.getVersion();
if (IndexReader.getCurrentVersion(directory) != currentVersion) {
IndexReader newReader = indexSearcher.getIndexReader().reopen();
if (newReader != reader) {
reader.decRef();
refreshMultiSeacher = true;
}
reader = newReader;
IndexSearcher newSearcher = new IndexSearcher(newReader);
indexSearchers.add(newSearcher);
}
}
}
if (refreshMultiSeacher) {
newMultiSeacher = new
MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
warm(newMultiSeacher);
swapMultiSearcher(newMultiSeacher);
}
}
private void warm(MultiSearcher newMultiSeacher) {
}
private synchronized MultiSearcher get() {
for (IndexSearcher indexSearcher: searchers) {
indexSearcher.getIndexReader().incRef();
}
return multiSearcher;
}
private synchronized void release(MultiSearcher multiSearcher)
throwsIOException {
for (IndexSearcher indexSearcher: searchers) {
indexSearcher.getIndexReader().decRef();
}
}
However I am now getting
java.lang.IllegalStateException:
org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed
on the call:
private synchronized MultiSearcher get() {
for (IndexSearcher indexSearcher: searchers) {
indexSearcher.getIndexReader().incRef();
}
return multiSearcher;
}
I'm doing something wrong ..obviously..not sure where though..
Cheers
On Sun, Mar 1, 2009 at 1:36 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
>
> I was wondering the same thing ;)
>
> It's best to call this method from a single BG "warming" thread, in which
> case it would not need its own synchronization.
>
> But, to be safe, I'll add internal synchronization to it. You can't simply
> put synchronized in front of the method, since you don't want this to block
> searching.
>
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> just a quick point:
>> public void maybeReopen() throws IOException { //D
>> long currentVersion = currentSearcher.getIndexReader().getVersion();
>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>> IndexReader newReader = currentSearcher.getIndexReader().reopen();
>> assert newReader != currentSearcher.getIndexReader();
>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>> warm(newSearcher);
>> swapSearcher(newSearcher);
>> }
>> }
>>
>> should the above be synchronised?
>>
>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <aminmc@gmail.com
>> >wrote:
>>
>> thanks. i will rewrite..in between giving my baby her feed and playing
>>> with the other child and my wife who wants me to do several other things!
>>>
>>>
>>>
>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> Hi
>>>>
>>>>> Thanks for your input. I would like to have a go at doing this myself
>>>>> first, Solr may be an option.
>>>>>
>>>>> * You are creating a new Analyzer & QueryParser every time, also
>>>>> creating unnecessary garbage; instead, they should be created once
>>>>> & reused.
>>>>>
>>>>> -- I can moved the code out so that it is only created once and reused.
>>>>>
>>>>>
>>>>> * You always make a new IndexSearcher and a new MultiSearcher even
>>>>> when nothing has changed. This just generates unnecessary garbage
>>>>> which GC then must sweep up.
>>>>>
>>>>> -- This was something I thought about. I could move it out so that
>>>>> it's
>>>>> created once. However I presume inside my code i need to check whether
>>>>> the
>>>>> indexreaders are update to date. This needs to be synchronized as well
>>>>> I
>>>>> guess(?)
>>>>>
>>>>>
>>>> Yes you should synchronize the check for whether the IndexReader is
>>>> current.
>>>>
>>>> * I don't see any synchronization -- it looks like two search
>>>>
>>>>> requests are allowed into this method at the same time? Which is
>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>> readers.
>>>>> -- So i need to extract the logic for reopening and provide a
>>>>> synchronisation mechanism.
>>>>>
>>>>>
>>>> Yes.
>>>>
>>>>
>>>> Ok. So I have some work to do. I'll refactor the code and see if I can
>>>>
>>>>> get
>>>>> inline to your recommendations.
>>>>>
>>>>>
>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>>
>>>>> On a quick look, I think there are a few problems with the code:
>>>>>>
>>>>>> * I don't see any synchronization -- it looks like two search
>>>>>> requests are allowed into this method at the same time? Which is
>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>> readers.
>>>>>>
>>>>>> * You are over-incRef'ing (the reader.incRef inside the loop) -- I
>>>>>> don't see a corresponding decRef.
>>>>>>
>>>>>> * You reopen and warm your searchers "live" (vs with BG thread);
>>>>>> meaning the unlucky search request that hits a reopen pays the
>>>>>> cost. This might be OK if the index is small enough that
>>>>>> reopening & warming takes very little time. But if index gets
>>>>>> large, making a random search pay that warming cost is not nice to
>>>>>> the end user. It erodes their trust in you.
>>>>>>
>>>>>> * You always make a new IndexSearcher and a new MultiSearcher even
>>>>>> when nothing has changed. This just generates unnecessary garbage
>>>>>> which GC then must sweep up.
>>>>>>
>>>>>> * You are creating a new Analyzer & QueryParser every time, also
>>>>>> creating unnecessary garbage; instead, they should be created once
>>>>>> & reused.
>>>>>>
>>>>>> You should consider simply using Solr -- it handles all this logic for
>>>>>> you and has been well debugged with time...
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>> The reason for the indexreader.reopen is because I have a webapp which
>>>>>>
>>>>>> enables users to upload files and then search for the documents. If
>>>>>>> I
>>>>>>> don't
>>>>>>> reopen i'm concerned that the facet hit counter won't be updated.
>>>>>>>
>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <
>>>>>>> aminmc@gmail.com
>>>>>>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> I have been able to get the code working for my scenario, however I
>>>>>>>> have
>>>>>>>> a
>>>>>>>> question and I was wondering if I could get some help. I have a
>>>>>>>> list
>>>>>>>> of
>>>>>>>> IndexSearchers which are used in a MultiSearcher class. I use the
>>>>>>>> indexsearchers to get each indexreader and put them into a
>>>>>>>> MultiIndexReader.
>>>>>>>>
>>>>>>>> IndexReader[] readers = new IndexReader[searchables.length];
>>>>>>>>
>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>
>>>>>>>> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>>>>>>>>
>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>
>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>
>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>
>>>>>>>> readers[i].close();
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> readers[i] = newReader;
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>
>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>>
>>>>>>>> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>>>>>>>>
>>>>>>>>
>>>>>>>> I then use the indexseacher to do the facet stuff. I end the code
>>>>>>>> with
>>>>>>>> closing the multireader. This is causing problems in another method
>>>>>>>> where I
>>>>>>>> do some other search as the indexreaders are closed. Is it ok to
>>>>>>>> not
>>>>>>>> close
>>>>>>>> the multiindexreader or should I do some additional checks in the
>>>>>>>> other
>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>>
>>>>>>>>
>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
>>>>>>>> aminmc@gmail.com
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks just what I needed!
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>> Amin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <ma...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi Amin:
>>>>>>>>>
>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>>
>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Sorry to re send this email but I was wondering if I could get
>>>>>>>>>>> some
>>>>>>>>>>> advice
>>>>>>>>>>> on this.
>>>>>>>>>>>
>>>>>>>>>>> Cheers
>>>>>>>>>>>
>>>>>>>>>>> Amin
>>>>>>>>>>>
>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I am looking at building a faceted search using Lucene. I know
>>>>>>>>>>>> that
>>>>>>>>>>>> Solr
>>>>>>>>>>>> comes with this built in, however I would like to try this by
>>>>>>>>>>>> myself
>>>>>>>>>>>> (something to add to my CV!). I have been looking around and I
>>>>>>>>>>>> found
>>>>>>>>>>>> that
>>>>>>>>>>>> you can use the IndexReader and use TermVectors. This looks ok
>>>>>>>>>>>> but
>>>>>>>>>>>> I'm
>>>>>>>>>>>> not
>>>>>>>>>>>> sure how to filter the results so that a particular user can
>>>>>>>>>>>> only
>>>>>>>>>>>> see
>>>>>>>>>>>> a
>>>>>>>>>>>> subset of results. The next option I was looking at was
>>>>>>>>>>>> something
>>>>>>>>>>>> like
>>>>>>>>>>>>
>>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>
>>>>>>>>>>>> The only problem here is that I have to provide the brand type
>>>>>>>>>>>> each
>>>>>>>>>>>> time a
>>>>>>>>>>>> new brand is created. Again I'm not sure how I can filter the
>>>>>>>>>>>> results
>>>>>>>>>>>> here.
>>>>>>>>>>>> It may be that I'm using the wrong api methods to do this.
>>>>>>>>>>>>
>>>>>>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers
>>>>>>>>>>>> Amin
>>>>>>>>>>>>
>>>>>>>>>>>> P.S. I am basically trying to do something that displays the
>>>>>>>>>>>> following
>>>>>>>>>>>>
>>>>>>>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>> ______________
>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Faceted Search using Lucene
Posted by Michael McCandless <lu...@mikemccandless.com>.
I was wondering the same thing ;)
It's best to call this method from a single BG "warming" thread, in
which case it would not need its own synchronization.
But, to be safe, I'll add internal synchronization to it. You can't
simply put synchronized in front of the method, since you don't want
this to block searching.
Mike
Amin Mohammed-Coleman wrote:
> just a quick point:
> public void maybeReopen() throws IOException { //D
> long currentVersion = currentSearcher.getIndexReader().getVersion();
> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
> IndexReader newReader = currentSearcher.getIndexReader().reopen();
> assert newReader != currentSearcher.getIndexReader();
> IndexSearcher newSearcher = new IndexSearcher(newReader);
> warm(newSearcher);
> swapSearcher(newSearcher);
> }
> }
>
> should the above be synchronised?
>
> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <aminmc@gmail.com
> >wrote:
>
>> thanks. i will rewrite..in between giving my baby her feed and
>> playing
>> with the other child and my wife who wants me to do several other
>> things!
>>
>>
>>
>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> Hi
>>>> Thanks for your input. I would like to have a go at doing this
>>>> myself
>>>> first, Solr may be an option.
>>>>
>>>> * You are creating a new Analyzer & QueryParser every time, also
>>>> creating unnecessary garbage; instead, they should be created once
>>>> & reused.
>>>>
>>>> -- I can moved the code out so that it is only created once and
>>>> reused.
>>>>
>>>>
>>>> * You always make a new IndexSearcher and a new MultiSearcher even
>>>> when nothing has changed. This just generates unnecessary garbage
>>>> which GC then must sweep up.
>>>>
>>>> -- This was something I thought about. I could move it out so
>>>> that it's
>>>> created once. However I presume inside my code i need to check
>>>> whether
>>>> the
>>>> indexreaders are update to date. This needs to be synchronized
>>>> as well I
>>>> guess(?)
>>>>
>>>
>>> Yes you should synchronize the check for whether the IndexReader is
>>> current.
>>>
>>> * I don't see any synchronization -- it looks like two search
>>>> requests are allowed into this method at the same time? Which is
>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>> readers.
>>>> -- So i need to extract the logic for reopening and provide a
>>>> synchronisation mechanism.
>>>>
>>>
>>> Yes.
>>>
>>>
>>> Ok. So I have some work to do. I'll refactor the code and see if
>>> I can
>>>> get
>>>> inline to your recommendations.
>>>>
>>>>
>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>>
>>>>> On a quick look, I think there are a few problems with the code:
>>>>>
>>>>> * I don't see any synchronization -- it looks like two search
>>>>> requests are allowed into this method at the same time? Which is
>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>> readers.
>>>>>
>>>>> * You are over-incRef'ing (the reader.incRef inside the loop) -- I
>>>>> don't see a corresponding decRef.
>>>>>
>>>>> * You reopen and warm your searchers "live" (vs with BG thread);
>>>>> meaning the unlucky search request that hits a reopen pays the
>>>>> cost. This might be OK if the index is small enough that
>>>>> reopening & warming takes very little time. But if index gets
>>>>> large, making a random search pay that warming cost is not nice to
>>>>> the end user. It erodes their trust in you.
>>>>>
>>>>> * You always make a new IndexSearcher and a new MultiSearcher even
>>>>> when nothing has changed. This just generates unnecessary garbage
>>>>> which GC then must sweep up.
>>>>>
>>>>> * You are creating a new Analyzer & QueryParser every time, also
>>>>> creating unnecessary garbage; instead, they should be created once
>>>>> & reused.
>>>>>
>>>>> You should consider simply using Solr -- it handles all this
>>>>> logic for
>>>>> you and has been well debugged with time...
>>>>>
>>>>> Mike
>>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>> The reason for the indexreader.reopen is because I have a webapp
>>>>> which
>>>>>
>>>>>> enables users to upload files and then search for the
>>>>>> documents. If I
>>>>>> don't
>>>>>> reopen i'm concerned that the facet hit counter won't be updated.
>>>>>>
>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <
>>>>>> aminmc@gmail.com
>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>>> I have been able to get the code working for my scenario,
>>>>>>> however I
>>>>>>> have
>>>>>>> a
>>>>>>> question and I was wondering if I could get some help. I have
>>>>>>> a list
>>>>>>> of
>>>>>>> IndexSearchers which are used in a MultiSearcher class. I use
>>>>>>> the
>>>>>>> indexsearchers to get each indexreader and put them into a
>>>>>>> MultiIndexReader.
>>>>>>>
>>>>>>> IndexReader[] readers = new IndexReader[searchables.length];
>>>>>>>
>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>
>>>>>>> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>>>>>>>
>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>
>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>
>>>>>>> if (newReader != readers[i]) {
>>>>>>>
>>>>>>> readers[i].close();
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> readers[i] = newReader;
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>
>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>
>>>>>>> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>>>>>>>
>>>>>>>
>>>>>>> I then use the indexseacher to do the facet stuff. I end the
>>>>>>> code
>>>>>>> with
>>>>>>> closing the multireader. This is causing problems in another
>>>>>>> method
>>>>>>> where I
>>>>>>> do some other search as the indexreaders are closed. Is it ok
>>>>>>> to not
>>>>>>> close
>>>>>>> the multiindexreader or should I do some additional checks in
>>>>>>> the
>>>>>>> other
>>>>>>> method to see if the indexreader is closed?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>>
>>>>>>> P.S. Hope that made sense...!
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
>>>>>>> aminmc@gmail.com
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks just what I needed!
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>> Amin
>>>>>>>>
>>>>>>>>
>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <marcelo.ochoa@gmail.com
>>>>>>>> >
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi Amin:
>>>>>>>>
>>>>>>>> Please take a look a this blog post:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>
>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>>>>>>>>> aminmc@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Sorry to re send this email but I was wondering if I could
>>>>>>>>>> get some
>>>>>>>>>> advice
>>>>>>>>>> on this.
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>>
>>>>>>>>>> Amin
>>>>>>>>>>
>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <aminmc@gmail.com
>>>>>>>>>> >
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I am looking at building a faceted search using Lucene. I
>>>>>>>>>>> know
>>>>>>>>>>> that
>>>>>>>>>>> Solr
>>>>>>>>>>> comes with this built in, however I would like to try this
>>>>>>>>>>> by
>>>>>>>>>>> myself
>>>>>>>>>>> (something to add to my CV!). I have been looking around
>>>>>>>>>>> and I
>>>>>>>>>>> found
>>>>>>>>>>> that
>>>>>>>>>>> you can use the IndexReader and use TermVectors. This
>>>>>>>>>>> looks ok
>>>>>>>>>>> but
>>>>>>>>>>> I'm
>>>>>>>>>>> not
>>>>>>>>>>> sure how to filter the results so that a particular user
>>>>>>>>>>> can only
>>>>>>>>>>> see
>>>>>>>>>>> a
>>>>>>>>>>> subset of results. The next option I was looking at was
>>>>>>>>>>> something
>>>>>>>>>>> like
>>>>>>>>>>>
>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>
>>>>>>>>>>> The only problem here is that I have to provide the brand
>>>>>>>>>>> type
>>>>>>>>>>> each
>>>>>>>>>>> time a
>>>>>>>>>>> new brand is created. Again I'm not sure how I can filter
>>>>>>>>>>> the
>>>>>>>>>>> results
>>>>>>>>>>> here.
>>>>>>>>>>> It may be that I'm using the wrong api methods to do this.
>>>>>>>>>>>
>>>>>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Cheers
>>>>>>>>>>> Amin
>>>>>>>>>>>
>>>>>>>>>>> P.S. I am basically trying to do something that displays
>>>>>>>>>>> the
>>>>>>>>>>> following
>>>>>>>>>>>
>>>>>>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>> ______________
>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>
>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: java-user-
>>>>>>>>> unsubscribe@lucene.apache.org
>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
Hi again...
Thanks for your patience, I modified the code to do the following:
private void maybeReopen() throws Exception {
startReopen();
try {
MultiSearcher newMultiSeacher = get();
boolean refreshMultiSeacher = false;
List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
synchronized (searchers) {
for (IndexSearcher indexSearcher: searchers) {
IndexReader reader = indexSearcher.getIndexReader();
reader.incRef();
Directory directory = reader.directory();
long currentVersion = reader.getVersion();
if (IndexReader.getCurrentVersion(directory) != currentVersion) {
IndexReader newReader = indexSearcher.getIndexReader().reopen();
if (newReader != reader) {
reader.decRef();
refreshMultiSeacher = true;
}
reader = newReader;
IndexSearcher newSearcher = new IndexSearcher(reader);
indexSearchers.add(newSearcher);
}
}
}
if (refreshMultiSeacher) {
try {
newMultiSeacher = new
MultiSearcher(indexSearchers.toArray(newIndexSearcher[] {}));
warm(newMultiSeacher);
swapMultiSearcher(newMultiSeacher);
}finally {
release(multiSearcher);
}
}
} finally {
doneReopen();
}
}
But I'm still getting an AlreadyCloseException this occurs when I call the
get() method in the main search code.
Cheers
On Sun, Mar 1, 2009 at 2:24 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
>
> OK new version of SearcherManager, that fixes maybeReopen() so that it can
> be called from multiple threads.
>
> NOTE: it's still untested!
>
> Mike
>
> package lia.admin;
>
> import java.io.IOException;
> import java.util.HashMap;
>
> import org.apache.lucene.search.IndexSearcher;
> import org.apache.lucene.index.IndexReader;
> import org.apache.lucene.store.Directory;
>
> /** Utility class to get/refresh searchers when you are
> * using multiple threads. */
>
> public class SearcherManager {
>
> private IndexSearcher currentSearcher; //A
> private Directory dir;
>
> public SearcherManager(Directory dir) throws IOException {
> this.dir = dir;
> currentSearcher = new IndexSearcher(IndexReader.open(dir)); //B
> }
>
> public void warm(IndexSearcher searcher) {} //C
>
> private boolean reopening;
>
> private synchronized void startReopen() //D
> throws InterruptedException {
> while (reopening) {
> wait();
> }
> reopening = true;
> }
>
> private synchronized void doneReopen() { //E
> reopening = false;
> notifyAll();
> }
>
> public void maybeReopen() throws InterruptedException, IOException { //F
>
> startReopen();
>
> try {
> final IndexSearcher searcher = get();
> try {
> long currentVersion = currentSearcher.getIndexReader().getVersion();
> //G
> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
> //G
> IndexReader newReader = currentSearcher.getIndexReader().reopen();
> //G
> assert newReader != currentSearcher.getIndexReader();
> //G
> IndexSearcher newSearcher = new IndexSearcher(newReader);
> //G
> warm(newSearcher);
> //G
> swapSearcher(newSearcher);
> //G
> }
> } finally {
> release(searcher);
> }
> } finally {
> doneReopen();
> }
> }
>
> public synchronized IndexSearcher get() { //H
> currentSearcher.getIndexReader().incRef();
> return currentSearcher;
> }
>
> public synchronized void release(IndexSearcher searcher) //I
> throws IOException {
> searcher.getIndexReader().decRef();
> }
>
> private synchronized void swapSearcher(IndexSearcher newSearcher) //J
> throws IOException {
> release(currentSearcher);
> currentSearcher = newSearcher;
> }
> }
>
> /*
> #A Current IndexSearcher
> #B Create initial searcher
> #C Implement in subclass to warm new searcher
> #D Pauses until no other thread is reopening
> #E Finish reopen and notify other threads
> #F Reopen searcher if there are changes
> #G Check index version and reopen, warm, swap if needed
> #H Returns current searcher
> #I Release searcher
> #J Swaps currentSearcher to new searcher
> */
>
> Mike
>
>
> On Mar 1, 2009, at 8:27 AM, Amin Mohammed-Coleman wrote:
>
> just a quick point:
>> public void maybeReopen() throws IOException { //D
>> long currentVersion = currentSearcher.getIndexReader().getVersion();
>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>> IndexReader newReader = currentSearcher.getIndexReader().reopen();
>> assert newReader != currentSearcher.getIndexReader();
>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>> warm(newSearcher);
>> swapSearcher(newSearcher);
>> }
>> }
>>
>> should the above be synchronised?
>>
>> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <aminmc@gmail.com
>> >wrote:
>>
>> thanks. i will rewrite..in between giving my baby her feed and playing
>>> with the other child and my wife who wants me to do several other things!
>>>
>>>
>>>
>>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> Hi
>>>>
>>>>> Thanks for your input. I would like to have a go at doing this myself
>>>>> first, Solr may be an option.
>>>>>
>>>>> * You are creating a new Analyzer & QueryParser every time, also
>>>>> creating unnecessary garbage; instead, they should be created once
>>>>> & reused.
>>>>>
>>>>> -- I can moved the code out so that it is only created once and reused.
>>>>>
>>>>>
>>>>> * You always make a new IndexSearcher and a new MultiSearcher even
>>>>> when nothing has changed. This just generates unnecessary garbage
>>>>> which GC then must sweep up.
>>>>>
>>>>> -- This was something I thought about. I could move it out so that
>>>>> it's
>>>>> created once. However I presume inside my code i need to check whether
>>>>> the
>>>>> indexreaders are update to date. This needs to be synchronized as well
>>>>> I
>>>>> guess(?)
>>>>>
>>>>>
>>>> Yes you should synchronize the check for whether the IndexReader is
>>>> current.
>>>>
>>>> * I don't see any synchronization -- it looks like two search
>>>>
>>>>> requests are allowed into this method at the same time? Which is
>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>> readers.
>>>>> -- So i need to extract the logic for reopening and provide a
>>>>> synchronisation mechanism.
>>>>>
>>>>>
>>>> Yes.
>>>>
>>>>
>>>> Ok. So I have some work to do. I'll refactor the code and see if I can
>>>>
>>>>> get
>>>>> inline to your recommendations.
>>>>>
>>>>>
>>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>>> lucene@mikemccandless.com> wrote:
>>>>>
>>>>>
>>>>> On a quick look, I think there are a few problems with the code:
>>>>>>
>>>>>> * I don't see any synchronization -- it looks like two search
>>>>>> requests are allowed into this method at the same time? Which is
>>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>>> readers.
>>>>>>
>>>>>> * You are over-incRef'ing (the reader.incRef inside the loop) -- I
>>>>>> don't see a corresponding decRef.
>>>>>>
>>>>>> * You reopen and warm your searchers "live" (vs with BG thread);
>>>>>> meaning the unlucky search request that hits a reopen pays the
>>>>>> cost. This might be OK if the index is small enough that
>>>>>> reopening & warming takes very little time. But if index gets
>>>>>> large, making a random search pay that warming cost is not nice to
>>>>>> the end user. It erodes their trust in you.
>>>>>>
>>>>>> * You always make a new IndexSearcher and a new MultiSearcher even
>>>>>> when nothing has changed. This just generates unnecessary garbage
>>>>>> which GC then must sweep up.
>>>>>>
>>>>>> * You are creating a new Analyzer & QueryParser every time, also
>>>>>> creating unnecessary garbage; instead, they should be created once
>>>>>> & reused.
>>>>>>
>>>>>> You should consider simply using Solr -- it handles all this logic for
>>>>>> you and has been well debugged with time...
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> Amin Mohammed-Coleman wrote:
>>>>>>
>>>>>> The reason for the indexreader.reopen is because I have a webapp which
>>>>>>
>>>>>> enables users to upload files and then search for the documents. If
>>>>>>> I
>>>>>>> don't
>>>>>>> reopen i'm concerned that the facet hit counter won't be updated.
>>>>>>>
>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <
>>>>>>> aminmc@gmail.com
>>>>>>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> I have been able to get the code working for my scenario, however I
>>>>>>>> have
>>>>>>>> a
>>>>>>>> question and I was wondering if I could get some help. I have a
>>>>>>>> list
>>>>>>>> of
>>>>>>>> IndexSearchers which are used in a MultiSearcher class. I use the
>>>>>>>> indexsearchers to get each indexreader and put them into a
>>>>>>>> MultiIndexReader.
>>>>>>>>
>>>>>>>> IndexReader[] readers = new IndexReader[searchables.length];
>>>>>>>>
>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>
>>>>>>>> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>>>>>>>>
>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>
>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>
>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>
>>>>>>>> readers[i].close();
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> readers[i] = newReader;
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>
>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>>
>>>>>>>> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>>>>>>>>
>>>>>>>>
>>>>>>>> I then use the indexseacher to do the facet stuff. I end the code
>>>>>>>> with
>>>>>>>> closing the multireader. This is causing problems in another method
>>>>>>>> where I
>>>>>>>> do some other search as the indexreaders are closed. Is it ok to
>>>>>>>> not
>>>>>>>> close
>>>>>>>> the multiindexreader or should I do some additional checks in the
>>>>>>>> other
>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>>
>>>>>>>>
>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
>>>>>>>> aminmc@gmail.com
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks just what I needed!
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>> Amin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <ma...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi Amin:
>>>>>>>>>
>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>>
>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Sorry to re send this email but I was wondering if I could get
>>>>>>>>>>> some
>>>>>>>>>>> advice
>>>>>>>>>>> on this.
>>>>>>>>>>>
>>>>>>>>>>> Cheers
>>>>>>>>>>>
>>>>>>>>>>> Amin
>>>>>>>>>>>
>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I am looking at building a faceted search using Lucene. I know
>>>>>>>>>>>> that
>>>>>>>>>>>> Solr
>>>>>>>>>>>> comes with this built in, however I would like to try this by
>>>>>>>>>>>> myself
>>>>>>>>>>>> (something to add to my CV!). I have been looking around and I
>>>>>>>>>>>> found
>>>>>>>>>>>> that
>>>>>>>>>>>> you can use the IndexReader and use TermVectors. This looks ok
>>>>>>>>>>>> but
>>>>>>>>>>>> I'm
>>>>>>>>>>>> not
>>>>>>>>>>>> sure how to filter the results so that a particular user can
>>>>>>>>>>>> only
>>>>>>>>>>>> see
>>>>>>>>>>>> a
>>>>>>>>>>>> subset of results. The next option I was looking at was
>>>>>>>>>>>> something
>>>>>>>>>>>> like
>>>>>>>>>>>>
>>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>
>>>>>>>>>>>> The only problem here is that I have to provide the brand type
>>>>>>>>>>>> each
>>>>>>>>>>>> time a
>>>>>>>>>>>> new brand is created. Again I'm not sure how I can filter the
>>>>>>>>>>>> results
>>>>>>>>>>>> here.
>>>>>>>>>>>> It may be that I'm using the wrong api methods to do this.
>>>>>>>>>>>>
>>>>>>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers
>>>>>>>>>>>> Amin
>>>>>>>>>>>>
>>>>>>>>>>>> P.S. I am basically trying to do something that displays the
>>>>>>>>>>>> following
>>>>>>>>>>>>
>>>>>>>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>> ______________
>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Faceted Search using Lucene
Posted by Michael McCandless <lu...@mikemccandless.com>.
OK new version of SearcherManager, that fixes maybeReopen() so that it
can be called from multiple threads.
NOTE: it's still untested!
Mike
package lia.admin;
import java.io.IOException;
import java.util.HashMap;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.store.Directory;
/** Utility class to get/refresh searchers when you are
* using multiple threads. */
public class SearcherManager {
private IndexSearcher currentSearcher; //A
private Directory dir;
public SearcherManager(Directory dir) throws IOException {
this.dir = dir;
currentSearcher = new IndexSearcher(IndexReader.open(dir)); //B
}
public void warm(IndexSearcher searcher) {} //C
private boolean reopening;
private synchronized void startReopen() //D
throws InterruptedException {
while (reopening) {
wait();
}
reopening = true;
}
private synchronized void doneReopen() { //E
reopening = false;
notifyAll();
}
public void maybeReopen() throws InterruptedException, IOException
{ //F
startReopen();
try {
final IndexSearcher searcher = get();
try {
long currentVersion =
currentSearcher.getIndexReader().getVersion(); //G
if (IndexReader.getCurrentVersion(dir) != currentVersion)
{ //G
IndexReader newReader =
currentSearcher.getIndexReader().reopen(); //G
assert newReader !=
currentSearcher.getIndexReader(); //G
IndexSearcher newSearcher = new
IndexSearcher(newReader); //G
warm(newSearcher); //G
swapSearcher(newSearcher); //G
}
} finally {
release(searcher);
}
} finally {
doneReopen();
}
}
public synchronized IndexSearcher get() { //H
currentSearcher.getIndexReader().incRef();
return currentSearcher;
}
public synchronized void release(IndexSearcher searcher) //I
throws IOException {
searcher.getIndexReader().decRef();
}
private synchronized void swapSearcher(IndexSearcher newSearcher) //J
throws IOException {
release(currentSearcher);
currentSearcher = newSearcher;
}
}
/*
#A Current IndexSearcher
#B Create initial searcher
#C Implement in subclass to warm new searcher
#D Pauses until no other thread is reopening
#E Finish reopen and notify other threads
#F Reopen searcher if there are changes
#G Check index version and reopen, warm, swap if needed
#H Returns current searcher
#I Release searcher
#J Swaps currentSearcher to new searcher
*/
Mike
On Mar 1, 2009, at 8:27 AM, Amin Mohammed-Coleman wrote:
> just a quick point:
> public void maybeReopen() throws IOException { //D
> long currentVersion = currentSearcher.getIndexReader().getVersion();
> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
> IndexReader newReader = currentSearcher.getIndexReader().reopen();
> assert newReader != currentSearcher.getIndexReader();
> IndexSearcher newSearcher = new IndexSearcher(newReader);
> warm(newSearcher);
> swapSearcher(newSearcher);
> }
> }
>
> should the above be synchronised?
>
> On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <aminmc@gmail.com
> >wrote:
>
>> thanks. i will rewrite..in between giving my baby her feed and
>> playing
>> with the other child and my wife who wants me to do several other
>> things!
>>
>>
>>
>> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> Hi
>>>> Thanks for your input. I would like to have a go at doing this
>>>> myself
>>>> first, Solr may be an option.
>>>>
>>>> * You are creating a new Analyzer & QueryParser every time, also
>>>> creating unnecessary garbage; instead, they should be created once
>>>> & reused.
>>>>
>>>> -- I can moved the code out so that it is only created once and
>>>> reused.
>>>>
>>>>
>>>> * You always make a new IndexSearcher and a new MultiSearcher even
>>>> when nothing has changed. This just generates unnecessary garbage
>>>> which GC then must sweep up.
>>>>
>>>> -- This was something I thought about. I could move it out so
>>>> that it's
>>>> created once. However I presume inside my code i need to check
>>>> whether
>>>> the
>>>> indexreaders are update to date. This needs to be synchronized
>>>> as well I
>>>> guess(?)
>>>>
>>>
>>> Yes you should synchronize the check for whether the IndexReader is
>>> current.
>>>
>>> * I don't see any synchronization -- it looks like two search
>>>> requests are allowed into this method at the same time? Which is
>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>> readers.
>>>> -- So i need to extract the logic for reopening and provide a
>>>> synchronisation mechanism.
>>>>
>>>
>>> Yes.
>>>
>>>
>>> Ok. So I have some work to do. I'll refactor the code and see if
>>> I can
>>>> get
>>>> inline to your recommendations.
>>>>
>>>>
>>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>>
>>>>> On a quick look, I think there are a few problems with the code:
>>>>>
>>>>> * I don't see any synchronization -- it looks like two search
>>>>> requests are allowed into this method at the same time? Which is
>>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>>> readers.
>>>>>
>>>>> * You are over-incRef'ing (the reader.incRef inside the loop) -- I
>>>>> don't see a corresponding decRef.
>>>>>
>>>>> * You reopen and warm your searchers "live" (vs with BG thread);
>>>>> meaning the unlucky search request that hits a reopen pays the
>>>>> cost. This might be OK if the index is small enough that
>>>>> reopening & warming takes very little time. But if index gets
>>>>> large, making a random search pay that warming cost is not nice to
>>>>> the end user. It erodes their trust in you.
>>>>>
>>>>> * You always make a new IndexSearcher and a new MultiSearcher even
>>>>> when nothing has changed. This just generates unnecessary garbage
>>>>> which GC then must sweep up.
>>>>>
>>>>> * You are creating a new Analyzer & QueryParser every time, also
>>>>> creating unnecessary garbage; instead, they should be created once
>>>>> & reused.
>>>>>
>>>>> You should consider simply using Solr -- it handles all this
>>>>> logic for
>>>>> you and has been well debugged with time...
>>>>>
>>>>> Mike
>>>>>
>>>>> Amin Mohammed-Coleman wrote:
>>>>>
>>>>> The reason for the indexreader.reopen is because I have a webapp
>>>>> which
>>>>>
>>>>>> enables users to upload files and then search for the
>>>>>> documents. If I
>>>>>> don't
>>>>>> reopen i'm concerned that the facet hit counter won't be updated.
>>>>>>
>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <
>>>>>> aminmc@gmail.com
>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>>> I have been able to get the code working for my scenario,
>>>>>>> however I
>>>>>>> have
>>>>>>> a
>>>>>>> question and I was wondering if I could get some help. I have
>>>>>>> a list
>>>>>>> of
>>>>>>> IndexSearchers which are used in a MultiSearcher class. I use
>>>>>>> the
>>>>>>> indexsearchers to get each indexreader and put them into a
>>>>>>> MultiIndexReader.
>>>>>>>
>>>>>>> IndexReader[] readers = new IndexReader[searchables.length];
>>>>>>>
>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>
>>>>>>> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>>>>>>>
>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>
>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>
>>>>>>> if (newReader != readers[i]) {
>>>>>>>
>>>>>>> readers[i].close();
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> readers[i] = newReader;
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>
>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>
>>>>>>> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>>>>>>>
>>>>>>>
>>>>>>> I then use the indexseacher to do the facet stuff. I end the
>>>>>>> code
>>>>>>> with
>>>>>>> closing the multireader. This is causing problems in another
>>>>>>> method
>>>>>>> where I
>>>>>>> do some other search as the indexreaders are closed. Is it ok
>>>>>>> to not
>>>>>>> close
>>>>>>> the multiindexreader or should I do some additional checks in
>>>>>>> the
>>>>>>> other
>>>>>>> method to see if the indexreader is closed?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>>
>>>>>>> P.S. Hope that made sense...!
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
>>>>>>> aminmc@gmail.com
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks just what I needed!
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>> Amin
>>>>>>>>
>>>>>>>>
>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <marcelo.ochoa@gmail.com
>>>>>>>> >
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi Amin:
>>>>>>>>
>>>>>>>> Please take a look a this blog post:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>>> Best regards, Marcelo.
>>>>>>>>>
>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>>>>>>>>> aminmc@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Sorry to re send this email but I was wondering if I could
>>>>>>>>>> get some
>>>>>>>>>> advice
>>>>>>>>>> on this.
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>>
>>>>>>>>>> Amin
>>>>>>>>>>
>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <aminmc@gmail.com
>>>>>>>>>> >
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I am looking at building a faceted search using Lucene. I
>>>>>>>>>>> know
>>>>>>>>>>> that
>>>>>>>>>>> Solr
>>>>>>>>>>> comes with this built in, however I would like to try this
>>>>>>>>>>> by
>>>>>>>>>>> myself
>>>>>>>>>>> (something to add to my CV!). I have been looking around
>>>>>>>>>>> and I
>>>>>>>>>>> found
>>>>>>>>>>> that
>>>>>>>>>>> you can use the IndexReader and use TermVectors. This
>>>>>>>>>>> looks ok
>>>>>>>>>>> but
>>>>>>>>>>> I'm
>>>>>>>>>>> not
>>>>>>>>>>> sure how to filter the results so that a particular user
>>>>>>>>>>> can only
>>>>>>>>>>> see
>>>>>>>>>>> a
>>>>>>>>>>> subset of results. The next option I was looking at was
>>>>>>>>>>> something
>>>>>>>>>>> like
>>>>>>>>>>>
>>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>
>>>>>>>>>>> The only problem here is that I have to provide the brand
>>>>>>>>>>> type
>>>>>>>>>>> each
>>>>>>>>>>> time a
>>>>>>>>>>> new brand is created. Again I'm not sure how I can filter
>>>>>>>>>>> the
>>>>>>>>>>> results
>>>>>>>>>>> here.
>>>>>>>>>>> It may be that I'm using the wrong api methods to do this.
>>>>>>>>>>>
>>>>>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Cheers
>>>>>>>>>>> Amin
>>>>>>>>>>>
>>>>>>>>>>> P.S. I am basically trying to do something that displays
>>>>>>>>>>> the
>>>>>>>>>>> following
>>>>>>>>>>>
>>>>>>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>> ______________
>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>>
>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: java-user-
>>>>>>>>> unsubscribe@lucene.apache.org
>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
just a quick point:
public void maybeReopen() throws IOException { //D
long currentVersion = currentSearcher.getIndexReader().getVersion();
if (IndexReader.getCurrentVersion(dir) != currentVersion) {
IndexReader newReader = currentSearcher.getIndexReader().reopen();
assert newReader != currentSearcher.getIndexReader();
IndexSearcher newSearcher = new IndexSearcher(newReader);
warm(newSearcher);
swapSearcher(newSearcher);
}
}
should the above be synchronised?
On Sun, Mar 1, 2009 at 1:25 PM, Amin Mohammed-Coleman <am...@gmail.com>wrote:
> thanks. i will rewrite..in between giving my baby her feed and playing
> with the other child and my wife who wants me to do several other things!
>
>
>
> On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>>
>> Amin Mohammed-Coleman wrote:
>>
>> Hi
>>> Thanks for your input. I would like to have a go at doing this myself
>>> first, Solr may be an option.
>>>
>>> * You are creating a new Analyzer & QueryParser every time, also
>>> creating unnecessary garbage; instead, they should be created once
>>> & reused.
>>>
>>> -- I can moved the code out so that it is only created once and reused.
>>>
>>>
>>> * You always make a new IndexSearcher and a new MultiSearcher even
>>> when nothing has changed. This just generates unnecessary garbage
>>> which GC then must sweep up.
>>>
>>> -- This was something I thought about. I could move it out so that it's
>>> created once. However I presume inside my code i need to check whether
>>> the
>>> indexreaders are update to date. This needs to be synchronized as well I
>>> guess(?)
>>>
>>
>> Yes you should synchronize the check for whether the IndexReader is
>> current.
>>
>> * I don't see any synchronization -- it looks like two search
>>> requests are allowed into this method at the same time? Which is
>>> dangerous... eg both (or, more) will wastefully reopen the
>>> readers.
>>> -- So i need to extract the logic for reopening and provide a
>>> synchronisation mechanism.
>>>
>>
>> Yes.
>>
>>
>> Ok. So I have some work to do. I'll refactor the code and see if I can
>>> get
>>> inline to your recommendations.
>>>
>>>
>>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>>> On a quick look, I think there are a few problems with the code:
>>>>
>>>> * I don't see any synchronization -- it looks like two search
>>>> requests are allowed into this method at the same time? Which is
>>>> dangerous... eg both (or, more) will wastefully reopen the
>>>> readers.
>>>>
>>>> * You are over-incRef'ing (the reader.incRef inside the loop) -- I
>>>> don't see a corresponding decRef.
>>>>
>>>> * You reopen and warm your searchers "live" (vs with BG thread);
>>>> meaning the unlucky search request that hits a reopen pays the
>>>> cost. This might be OK if the index is small enough that
>>>> reopening & warming takes very little time. But if index gets
>>>> large, making a random search pay that warming cost is not nice to
>>>> the end user. It erodes their trust in you.
>>>>
>>>> * You always make a new IndexSearcher and a new MultiSearcher even
>>>> when nothing has changed. This just generates unnecessary garbage
>>>> which GC then must sweep up.
>>>>
>>>> * You are creating a new Analyzer & QueryParser every time, also
>>>> creating unnecessary garbage; instead, they should be created once
>>>> & reused.
>>>>
>>>> You should consider simply using Solr -- it handles all this logic for
>>>> you and has been well debugged with time...
>>>>
>>>> Mike
>>>>
>>>> Amin Mohammed-Coleman wrote:
>>>>
>>>> The reason for the indexreader.reopen is because I have a webapp which
>>>>
>>>>> enables users to upload files and then search for the documents. If I
>>>>> don't
>>>>> reopen i'm concerned that the facet hit counter won't be updated.
>>>>>
>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <
>>>>> aminmc@gmail.com
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>
>>>>> Hi
>>>>>
>>>>>> I have been able to get the code working for my scenario, however I
>>>>>> have
>>>>>> a
>>>>>> question and I was wondering if I could get some help. I have a list
>>>>>> of
>>>>>> IndexSearchers which are used in a MultiSearcher class. I use the
>>>>>> indexsearchers to get each indexreader and put them into a
>>>>>> MultiIndexReader.
>>>>>>
>>>>>> IndexReader[] readers = new IndexReader[searchables.length];
>>>>>>
>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>
>>>>>> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>>>>>>
>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>
>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>
>>>>>> if (newReader != readers[i]) {
>>>>>>
>>>>>> readers[i].close();
>>>>>>
>>>>>> }
>>>>>>
>>>>>> readers[i] = newReader;
>>>>>>
>>>>>>
>>>>>>
>>>>>> }
>>>>>>
>>>>>> multiReader = new MultiReader(readers);
>>>>>>
>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>
>>>>>> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>>>>>>
>>>>>>
>>>>>> I then use the indexseacher to do the facet stuff. I end the code
>>>>>> with
>>>>>> closing the multireader. This is causing problems in another method
>>>>>> where I
>>>>>> do some other search as the indexreaders are closed. Is it ok to not
>>>>>> close
>>>>>> the multiindexreader or should I do some additional checks in the
>>>>>> other
>>>>>> method to see if the indexreader is closed?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>>>> P.S. Hope that made sense...!
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
>>>>>> aminmc@gmail.com
>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>>>
>>>>>>> Thanks just what I needed!
>>>>>>>
>>>>>>> Cheers
>>>>>>> Amin
>>>>>>>
>>>>>>>
>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <ma...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Amin:
>>>>>>>
>>>>>>> Please take a look a this blog post:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>> Best regards, Marcelo.
>>>>>>>>
>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>>>>>>>> aminmc@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Sorry to re send this email but I was wondering if I could get some
>>>>>>>>> advice
>>>>>>>>> on this.
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>>
>>>>>>>>> Amin
>>>>>>>>>
>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <am...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I am looking at building a faceted search using Lucene. I know
>>>>>>>>>> that
>>>>>>>>>> Solr
>>>>>>>>>> comes with this built in, however I would like to try this by
>>>>>>>>>> myself
>>>>>>>>>> (something to add to my CV!). I have been looking around and I
>>>>>>>>>> found
>>>>>>>>>> that
>>>>>>>>>> you can use the IndexReader and use TermVectors. This looks ok
>>>>>>>>>> but
>>>>>>>>>> I'm
>>>>>>>>>> not
>>>>>>>>>> sure how to filter the results so that a particular user can only
>>>>>>>>>> see
>>>>>>>>>> a
>>>>>>>>>> subset of results. The next option I was looking at was something
>>>>>>>>>> like
>>>>>>>>>>
>>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>>>
>>>>>>>>>> The only problem here is that I have to provide the brand type
>>>>>>>>>> each
>>>>>>>>>> time a
>>>>>>>>>> new brand is created. Again I'm not sure how I can filter the
>>>>>>>>>> results
>>>>>>>>>> here.
>>>>>>>>>> It may be that I'm using the wrong api methods to do this.
>>>>>>>>>>
>>>>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>> Amin
>>>>>>>>>>
>>>>>>>>>> P.S. I am basically trying to do something that displays the
>>>>>>>>>> following
>>>>>>>>>>
>>>>>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Marcelo F. Ochoa
>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>> ______________
>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>
>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
thanks. i will rewrite..in between giving my baby her feed and playing with
the other child and my wife who wants me to do several other things!
On Sun, Mar 1, 2009 at 1:20 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
>
> Amin Mohammed-Coleman wrote:
>
> Hi
>> Thanks for your input. I would like to have a go at doing this myself
>> first, Solr may be an option.
>>
>> * You are creating a new Analyzer & QueryParser every time, also
>> creating unnecessary garbage; instead, they should be created once
>> & reused.
>>
>> -- I can moved the code out so that it is only created once and reused.
>>
>>
>> * You always make a new IndexSearcher and a new MultiSearcher even
>> when nothing has changed. This just generates unnecessary garbage
>> which GC then must sweep up.
>>
>> -- This was something I thought about. I could move it out so that it's
>> created once. However I presume inside my code i need to check whether
>> the
>> indexreaders are update to date. This needs to be synchronized as well I
>> guess(?)
>>
>
> Yes you should synchronize the check for whether the IndexReader is
> current.
>
> * I don't see any synchronization -- it looks like two search
>> requests are allowed into this method at the same time? Which is
>> dangerous... eg both (or, more) will wastefully reopen the
>> readers.
>> -- So i need to extract the logic for reopening and provide a
>> synchronisation mechanism.
>>
>
> Yes.
>
>
> Ok. So I have some work to do. I'll refactor the code and see if I can
>> get
>> inline to your recommendations.
>>
>>
>> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>
>>> On a quick look, I think there are a few problems with the code:
>>>
>>> * I don't see any synchronization -- it looks like two search
>>> requests are allowed into this method at the same time? Which is
>>> dangerous... eg both (or, more) will wastefully reopen the
>>> readers.
>>>
>>> * You are over-incRef'ing (the reader.incRef inside the loop) -- I
>>> don't see a corresponding decRef.
>>>
>>> * You reopen and warm your searchers "live" (vs with BG thread);
>>> meaning the unlucky search request that hits a reopen pays the
>>> cost. This might be OK if the index is small enough that
>>> reopening & warming takes very little time. But if index gets
>>> large, making a random search pay that warming cost is not nice to
>>> the end user. It erodes their trust in you.
>>>
>>> * You always make a new IndexSearcher and a new MultiSearcher even
>>> when nothing has changed. This just generates unnecessary garbage
>>> which GC then must sweep up.
>>>
>>> * You are creating a new Analyzer & QueryParser every time, also
>>> creating unnecessary garbage; instead, they should be created once
>>> & reused.
>>>
>>> You should consider simply using Solr -- it handles all this logic for
>>> you and has been well debugged with time...
>>>
>>> Mike
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> The reason for the indexreader.reopen is because I have a webapp which
>>>
>>>> enables users to upload files and then search for the documents. If I
>>>> don't
>>>> reopen i'm concerned that the facet hit counter won't be updated.
>>>>
>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <
>>>> aminmc@gmail.com
>>>>
>>>>> wrote:
>>>>>
>>>>
>>>> Hi
>>>>
>>>>> I have been able to get the code working for my scenario, however I
>>>>> have
>>>>> a
>>>>> question and I was wondering if I could get some help. I have a list
>>>>> of
>>>>> IndexSearchers which are used in a MultiSearcher class. I use the
>>>>> indexsearchers to get each indexreader and put them into a
>>>>> MultiIndexReader.
>>>>>
>>>>> IndexReader[] readers = new IndexReader[searchables.length];
>>>>>
>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>
>>>>> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>>>>>
>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>
>>>>> IndexReader newReader = readers[i].reopen();
>>>>>
>>>>> if (newReader != readers[i]) {
>>>>>
>>>>> readers[i].close();
>>>>>
>>>>> }
>>>>>
>>>>> readers[i] = newReader;
>>>>>
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>> multiReader = new MultiReader(readers);
>>>>>
>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>> newOpenBitSetFacetHitCounter();
>>>>>
>>>>> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>>>>>
>>>>>
>>>>> I then use the indexseacher to do the facet stuff. I end the code with
>>>>> closing the multireader. This is causing problems in another method
>>>>> where I
>>>>> do some other search as the indexreaders are closed. Is it ok to not
>>>>> close
>>>>> the multiindexreader or should I do some additional checks in the other
>>>>> method to see if the indexreader is closed?
>>>>>
>>>>>
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> P.S. Hope that made sense...!
>>>>>
>>>>>
>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
>>>>> aminmc@gmail.com
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>
>>>>> Hi
>>>>>
>>>>>>
>>>>>> Thanks just what I needed!
>>>>>>
>>>>>> Cheers
>>>>>> Amin
>>>>>>
>>>>>>
>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <ma...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Amin:
>>>>>>
>>>>>> Please take a look a this blog post:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>> Best regards, Marcelo.
>>>>>>>
>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>>>>>>> aminmc@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>>>
>>>>>>>> Sorry to re send this email but I was wondering if I could get some
>>>>>>>> advice
>>>>>>>> on this.
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>>
>>>>>>>> Amin
>>>>>>>>
>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <am...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>>
>>>>>>>>> I am looking at building a faceted search using Lucene. I know
>>>>>>>>> that
>>>>>>>>> Solr
>>>>>>>>> comes with this built in, however I would like to try this by
>>>>>>>>> myself
>>>>>>>>> (something to add to my CV!). I have been looking around and I
>>>>>>>>> found
>>>>>>>>> that
>>>>>>>>> you can use the IndexReader and use TermVectors. This looks ok but
>>>>>>>>> I'm
>>>>>>>>> not
>>>>>>>>> sure how to filter the results so that a particular user can only
>>>>>>>>> see
>>>>>>>>> a
>>>>>>>>> subset of results. The next option I was looking at was something
>>>>>>>>> like
>>>>>>>>>
>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>>
>>>>>>>>> The only problem here is that I have to provide the brand type each
>>>>>>>>> time a
>>>>>>>>> new brand is created. Again I'm not sure how I can filter the
>>>>>>>>> results
>>>>>>>>> here.
>>>>>>>>> It may be that I'm using the wrong api methods to do this.
>>>>>>>>>
>>>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>> Amin
>>>>>>>>>
>>>>>>>>> P.S. I am basically trying to do something that displays the
>>>>>>>>> following
>>>>>>>>>
>>>>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> Marcelo F. Ochoa
>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>> ______________
>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>> Is Oracle 11g REST ready?
>>>>>>>
>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Faceted Search using Lucene
Posted by Michael McCandless <lu...@mikemccandless.com>.
Amin Mohammed-Coleman wrote:
> Hi
> Thanks for your input. I would like to have a go at doing this myself
> first, Solr may be an option.
>
> * You are creating a new Analyzer & QueryParser every time, also
> creating unnecessary garbage; instead, they should be created once
> & reused.
>
> -- I can moved the code out so that it is only created once and
> reused.
>
>
> * You always make a new IndexSearcher and a new MultiSearcher even
> when nothing has changed. This just generates unnecessary garbage
> which GC then must sweep up.
>
> -- This was something I thought about. I could move it out so that
> it's
> created once. However I presume inside my code i need to check
> whether the
> indexreaders are update to date. This needs to be synchronized as
> well I
> guess(?)
Yes you should synchronize the check for whether the IndexReader is
current.
> * I don't see any synchronization -- it looks like two search
> requests are allowed into this method at the same time? Which is
> dangerous... eg both (or, more) will wastefully reopen the
> readers.
> -- So i need to extract the logic for reopening and provide a
> synchronisation mechanism.
Yes.
> Ok. So I have some work to do. I'll refactor the code and see if I
> can get
> inline to your recommendations.
>
>
> On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>>
>> On a quick look, I think there are a few problems with the code:
>>
>> * I don't see any synchronization -- it looks like two search
>> requests are allowed into this method at the same time? Which is
>> dangerous... eg both (or, more) will wastefully reopen the
>> readers.
>>
>> * You are over-incRef'ing (the reader.incRef inside the loop) -- I
>> don't see a corresponding decRef.
>>
>> * You reopen and warm your searchers "live" (vs with BG thread);
>> meaning the unlucky search request that hits a reopen pays the
>> cost. This might be OK if the index is small enough that
>> reopening & warming takes very little time. But if index gets
>> large, making a random search pay that warming cost is not nice to
>> the end user. It erodes their trust in you.
>>
>> * You always make a new IndexSearcher and a new MultiSearcher even
>> when nothing has changed. This just generates unnecessary garbage
>> which GC then must sweep up.
>>
>> * You are creating a new Analyzer & QueryParser every time, also
>> creating unnecessary garbage; instead, they should be created once
>> & reused.
>>
>> You should consider simply using Solr -- it handles all this logic
>> for
>> you and has been well debugged with time...
>>
>> Mike
>>
>> Amin Mohammed-Coleman wrote:
>>
>> The reason for the indexreader.reopen is because I have a webapp
>> which
>>> enables users to upload files and then search for the documents.
>>> If I
>>> don't
>>> reopen i'm concerned that the facet hit counter won't be updated.
>>>
>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <aminmc@gmail.com
>>>> wrote:
>>>
>>> Hi
>>>> I have been able to get the code working for my scenario, however
>>>> I have
>>>> a
>>>> question and I was wondering if I could get some help. I have a
>>>> list of
>>>> IndexSearchers which are used in a MultiSearcher class. I use the
>>>> indexsearchers to get each indexreader and put them into a
>>>> MultiIndexReader.
>>>>
>>>> IndexReader[] readers = new IndexReader[searchables.length];
>>>>
>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>
>>>> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>>>>
>>>> readers[i] = indexSearcher.getIndexReader();
>>>>
>>>> IndexReader newReader = readers[i].reopen();
>>>>
>>>> if (newReader != readers[i]) {
>>>>
>>>> readers[i].close();
>>>>
>>>> }
>>>>
>>>> readers[i] = newReader;
>>>>
>>>>
>>>>
>>>> }
>>>>
>>>> multiReader = new MultiReader(readers);
>>>>
>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>> newOpenBitSetFacetHitCounter();
>>>>
>>>> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>>>>
>>>>
>>>> I then use the indexseacher to do the facet stuff. I end the
>>>> code with
>>>> closing the multireader. This is causing problems in another
>>>> method
>>>> where I
>>>> do some other search as the indexreaders are closed. Is it ok to
>>>> not
>>>> close
>>>> the multiindexreader or should I do some additional checks in the
>>>> other
>>>> method to see if the indexreader is closed?
>>>>
>>>>
>>>>
>>>> Cheers
>>>>
>>>>
>>>> P.S. Hope that made sense...!
>>>>
>>>>
>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <aminmc@gmail.com
>>>>> wrote:
>>>>
>>>> Hi
>>>>>
>>>>> Thanks just what I needed!
>>>>>
>>>>> Cheers
>>>>> Amin
>>>>>
>>>>>
>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <ma...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi Amin:
>>>>>
>>>>>> Please take a look a this blog post:
>>>>>>
>>>>>>
>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>> Best regards, Marcelo.
>>>>>>
>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>>>>>> aminmc@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi
>>>>>>>
>>>>>>> Sorry to re send this email but I was wondering if I could get
>>>>>>> some
>>>>>>> advice
>>>>>>> on this.
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>> Amin
>>>>>>>
>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <aminmc@gmail.com
>>>>>>> >
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>>>
>>>>>>>> I am looking at building a faceted search using Lucene. I
>>>>>>>> know that
>>>>>>>> Solr
>>>>>>>> comes with this built in, however I would like to try this by
>>>>>>>> myself
>>>>>>>> (something to add to my CV!). I have been looking around and
>>>>>>>> I found
>>>>>>>> that
>>>>>>>> you can use the IndexReader and use TermVectors. This looks
>>>>>>>> ok but
>>>>>>>> I'm
>>>>>>>> not
>>>>>>>> sure how to filter the results so that a particular user can
>>>>>>>> only see
>>>>>>>> a
>>>>>>>> subset of results. The next option I was looking at was
>>>>>>>> something
>>>>>>>> like
>>>>>>>>
>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>
>>>>>>>> The only problem here is that I have to provide the brand
>>>>>>>> type each
>>>>>>>> time a
>>>>>>>> new brand is created. Again I'm not sure how I can filter the
>>>>>>>> results
>>>>>>>> here.
>>>>>>>> It may be that I'm using the wrong api methods to do this.
>>>>>>>>
>>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>>
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>> Amin
>>>>>>>>
>>>>>>>> P.S. I am basically trying to do something that displays the
>>>>>>>> following
>>>>>>>>
>>>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Marcelo F. Ochoa
>>>>>> http://marceloochoa.blogspot.com/
>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>> ______________
>>>>>> Want to integrate Lucene and Oracle?
>>>>>>
>>>>>>
>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>> Is Oracle 11g REST ready?
>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
Hi
Thanks for your input. I would like to have a go at doing this myself
first, Solr may be an option.
* You are creating a new Analyzer & QueryParser every time, also
creating unnecessary garbage; instead, they should be created once
& reused.
-- I can moved the code out so that it is only created once and reused.
* You always make a new IndexSearcher and a new MultiSearcher even
when nothing has changed. This just generates unnecessary garbage
which GC then must sweep up.
-- This was something I thought about. I could move it out so that it's
created once. However I presume inside my code i need to check whether the
indexreaders are update to date. This needs to be synchronized as well I
guess(?)
* I don't see any synchronization -- it looks like two search
requests are allowed into this method at the same time? Which is
dangerous... eg both (or, more) will wastefully reopen the
readers.
-- So i need to extract the logic for reopening and provide a
synchronisation mechanism.
Ok. So I have some work to do. I'll refactor the code and see if I can get
inline to your recommendations.
On Sun, Mar 1, 2009 at 12:11 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
>
> On a quick look, I think there are a few problems with the code:
>
> * I don't see any synchronization -- it looks like two search
> requests are allowed into this method at the same time? Which is
> dangerous... eg both (or, more) will wastefully reopen the
> readers.
>
> * You are over-incRef'ing (the reader.incRef inside the loop) -- I
> don't see a corresponding decRef.
>
> * You reopen and warm your searchers "live" (vs with BG thread);
> meaning the unlucky search request that hits a reopen pays the
> cost. This might be OK if the index is small enough that
> reopening & warming takes very little time. But if index gets
> large, making a random search pay that warming cost is not nice to
> the end user. It erodes their trust in you.
>
> * You always make a new IndexSearcher and a new MultiSearcher even
> when nothing has changed. This just generates unnecessary garbage
> which GC then must sweep up.
>
> * You are creating a new Analyzer & QueryParser every time, also
> creating unnecessary garbage; instead, they should be created once
> & reused.
>
> You should consider simply using Solr -- it handles all this logic for
> you and has been well debugged with time...
>
> Mike
>
> Amin Mohammed-Coleman wrote:
>
> The reason for the indexreader.reopen is because I have a webapp which
>> enables users to upload files and then search for the documents. If I
>> don't
>> reopen i'm concerned that the facet hit counter won't be updated.
>>
>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <aminmc@gmail.com
>> >wrote:
>>
>> Hi
>>> I have been able to get the code working for my scenario, however I have
>>> a
>>> question and I was wondering if I could get some help. I have a list of
>>> IndexSearchers which are used in a MultiSearcher class. I use the
>>> indexsearchers to get each indexreader and put them into a
>>> MultiIndexReader.
>>>
>>> IndexReader[] readers = new IndexReader[searchables.length];
>>>
>>> for (int i =0 ; i < searchables.length;i++) {
>>>
>>> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>>>
>>> readers[i] = indexSearcher.getIndexReader();
>>>
>>> IndexReader newReader = readers[i].reopen();
>>>
>>> if (newReader != readers[i]) {
>>>
>>> readers[i].close();
>>>
>>> }
>>>
>>> readers[i] = newReader;
>>>
>>>
>>>
>>> }
>>>
>>> multiReader = new MultiReader(readers);
>>>
>>> OpenBitSetFacetHitCounter facetHitCounter =
>>> newOpenBitSetFacetHitCounter();
>>>
>>> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>>>
>>>
>>> I then use the indexseacher to do the facet stuff. I end the code with
>>> closing the multireader. This is causing problems in another method
>>> where I
>>> do some other search as the indexreaders are closed. Is it ok to not
>>> close
>>> the multiindexreader or should I do some additional checks in the other
>>> method to see if the indexreader is closed?
>>>
>>>
>>>
>>> Cheers
>>>
>>>
>>> P.S. Hope that made sense...!
>>>
>>>
>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <aminmc@gmail.com
>>> >wrote:
>>>
>>> Hi
>>>>
>>>> Thanks just what I needed!
>>>>
>>>> Cheers
>>>> Amin
>>>>
>>>>
>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <ma...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi Amin:
>>>>
>>>>> Please take a look a this blog post:
>>>>>
>>>>>
>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>> Best regards, Marcelo.
>>>>>
>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>>>>> aminmc@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi
>>>>>>
>>>>>> Sorry to re send this email but I was wondering if I could get some
>>>>>> advice
>>>>>> on this.
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> Amin
>>>>>>
>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <am...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>>>
>>>>>>> I am looking at building a faceted search using Lucene. I know that
>>>>>>> Solr
>>>>>>> comes with this built in, however I would like to try this by myself
>>>>>>> (something to add to my CV!). I have been looking around and I found
>>>>>>> that
>>>>>>> you can use the IndexReader and use TermVectors. This looks ok but
>>>>>>> I'm
>>>>>>> not
>>>>>>> sure how to filter the results so that a particular user can only see
>>>>>>> a
>>>>>>> subset of results. The next option I was looking at was something
>>>>>>> like
>>>>>>>
>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>
>>>>>>> The only problem here is that I have to provide the brand type each
>>>>>>> time a
>>>>>>> new brand is created. Again I'm not sure how I can filter the
>>>>>>> results
>>>>>>> here.
>>>>>>> It may be that I'm using the wrong api methods to do this.
>>>>>>>
>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>
>>>>>>>
>>>>>>> Cheers
>>>>>>> Amin
>>>>>>>
>>>>>>> P.S. I am basically trying to do something that displays the
>>>>>>> following
>>>>>>>
>>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Marcelo F. Ochoa
>>>>> http://marceloochoa.blogspot.com/
>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>> ______________
>>>>> Want to integrate Lucene and Oracle?
>>>>>
>>>>>
>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>> Is Oracle 11g REST ready?
>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Faceted Search using Lucene
Posted by Michael McCandless <lu...@mikemccandless.com>.
On a quick look, I think there are a few problems with the code:
* I don't see any synchronization -- it looks like two search
requests are allowed into this method at the same time? Which is
dangerous... eg both (or, more) will wastefully reopen the
readers.
* You are over-incRef'ing (the reader.incRef inside the loop) -- I
don't see a corresponding decRef.
* You reopen and warm your searchers "live" (vs with BG thread);
meaning the unlucky search request that hits a reopen pays the
cost. This might be OK if the index is small enough that
reopening & warming takes very little time. But if index gets
large, making a random search pay that warming cost is not nice to
the end user. It erodes their trust in you.
* You always make a new IndexSearcher and a new MultiSearcher even
when nothing has changed. This just generates unnecessary garbage
which GC then must sweep up.
* You are creating a new Analyzer & QueryParser every time, also
creating unnecessary garbage; instead, they should be created once
& reused.
You should consider simply using Solr -- it handles all this logic for
you and has been well debugged with time...
Mike
Amin Mohammed-Coleman wrote:
> The reason for the indexreader.reopen is because I have a webapp which
> enables users to upload files and then search for the documents. If
> I don't
> reopen i'm concerned that the facet hit counter won't be updated.
>
> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <aminmc@gmail.com
> >wrote:
>
>> Hi
>> I have been able to get the code working for my scenario, however I
>> have a
>> question and I was wondering if I could get some help. I have a
>> list of
>> IndexSearchers which are used in a MultiSearcher class. I use the
>> indexsearchers to get each indexreader and put them into a
>> MultiIndexReader.
>>
>> IndexReader[] readers = new IndexReader[searchables.length];
>>
>> for (int i =0 ; i < searchables.length;i++) {
>>
>> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>>
>> readers[i] = indexSearcher.getIndexReader();
>>
>> IndexReader newReader = readers[i].reopen();
>>
>> if (newReader != readers[i]) {
>>
>> readers[i].close();
>>
>> }
>>
>> readers[i] = newReader;
>>
>>
>>
>> }
>>
>> multiReader = new MultiReader(readers);
>>
>> OpenBitSetFacetHitCounter facetHitCounter =
>> newOpenBitSetFacetHitCounter();
>>
>> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>>
>>
>> I then use the indexseacher to do the facet stuff. I end the code
>> with
>> closing the multireader. This is causing problems in another
>> method where I
>> do some other search as the indexreaders are closed. Is it ok to
>> not close
>> the multiindexreader or should I do some additional checks in the
>> other
>> method to see if the indexreader is closed?
>>
>>
>>
>> Cheers
>>
>>
>> P.S. Hope that made sense...!
>>
>>
>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <aminmc@gmail.com
>> >wrote:
>>
>>> Hi
>>>
>>> Thanks just what I needed!
>>>
>>> Cheers
>>> Amin
>>>
>>>
>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <ma...@gmail.com>
>>> wrote:
>>>
>>> Hi Amin:
>>>> Please take a look a this blog post:
>>>>
>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>> Best regards, Marcelo.
>>>>
>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <aminmc@gmail.com
>>>> >
>>>> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> Sorry to re send this email but I was wondering if I could get
>>>>> some
>>>>> advice
>>>>> on this.
>>>>>
>>>>> Cheers
>>>>>
>>>>> Amin
>>>>>
>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <am...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi
>>>>>>
>>>>>> I am looking at building a faceted search using Lucene. I know
>>>>>> that
>>>>>> Solr
>>>>>> comes with this built in, however I would like to try this by
>>>>>> myself
>>>>>> (something to add to my CV!). I have been looking around and I
>>>>>> found
>>>>>> that
>>>>>> you can use the IndexReader and use TermVectors. This looks ok
>>>>>> but I'm
>>>>>> not
>>>>>> sure how to filter the results so that a particular user can
>>>>>> only see a
>>>>>> subset of results. The next option I was looking at was
>>>>>> something like
>>>>>>
>>>>>> Term term1 = new Term("brand", "ford");
>>>>>> Term term2 = new Term("brand", "vw");
>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>
>>>>>> The only problem here is that I have to provide the brand type
>>>>>> each
>>>>>> time a
>>>>>> new brand is created. Again I'm not sure how I can filter the
>>>>>> results
>>>>>> here.
>>>>>> It may be that I'm using the wrong api methods to do this.
>>>>>>
>>>>>> I would be grateful if I could get some advice on this.
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>> Amin
>>>>>>
>>>>>> P.S. I am basically trying to do something that displays the
>>>>>> following
>>>>>>
>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Marcelo F. Ochoa
>>>> http://marceloochoa.blogspot.com/
>>>> http://marcelo.ochoa.googlepages.com/home
>>>> ______________
>>>> Want to integrate Lucene and Oracle?
>>>>
>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>> Is Oracle 11g REST ready?
>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
Hi
Thanks for your reply. I have modified the code to the following:
public Map<String, Long> getFacetHitCount(String searchTerm) {
QueryParser queryParser =
newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
analyzer);
Query baseQuery = null;
try {
if (!StringUtils.isBlank(searchTerm)) {
baseQuery = queryParser.parse(searchTerm);
LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
baseQuery.toString() +"'");
} else {
LOGGER.debug("No base query. Using default, which is going to check for all
documents of every type.");
}
} catch (ParseException e1) {
throw new RuntimeException(e1);
}
Map<String, Query> subQueries = constructDocTypeSubQueriesMap();
Map<String,Long> facetHitCount = new HashMap<String, Long>();
MultiReader multiReader = null;
try {
Searchable[] searchables = this.searchers.toArray(new Searchable[]
{}).clone();
IndexReader[] readers = new IndexReader[searchables.length];
for (int i =0 ; i < searchables.length;i++) {
IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
readers[i] = indexSearcher.getIndexReader();
Directory directory = readers[i].directory();
IndexReader indexReader = IndexReader.open(directory);
readers[i] = indexReader;
}
multiReader = new MultiReader(readers);
OpenBitSetFacetHitCounter facetHitCounter = new OpenBitSetFacetHitCounter();
IndexSearcher indexSearcher = new IndexSearcher(multiReader);
if (baseQuery != null) {
facetHitCounter.setBaseQuery(baseQuery);
}
facetHitCounter.setSearcher(indexSearcher);
facetHitCounter.setSubQueries(subQueries);
facetHitCount= facetHitCounter.getFacetHitCounts();
LOGGER.debug("Document Type Facet Hit Count '" + facetHitCount + "'");
} catch (Exception e) {
throw new IllegalStateException(e);
} finally {
try {
multiReader.close();
LOGGER.debug("Closed multi reader.");
} catch (IOException e) {
throw new IllegalStateException(e);
}
}
return facetHitCount;
}
Does this make sense? I am new to lucene and working on a complete search
solution so I would be grateful for any advice on what is best practice.
Cheers
On Thu, Feb 26, 2009 at 7:55 AM, Michael Stoppelman <st...@gmail.com>wrote:
> If another thread is executing a query with the handle to one of readers[i]
> you're going to kill it since the IndexReader is now closed.
> Just don't call the IndexReader#close() method. If nothing is pointing at
> the readers they should be garbage collected. Also, you might
> want to warm up your new IndexSearcher before you switch to it, meaning run
> a few queries on it before you swap the old one out.
>
> M
>
>
>
> On Tue, Feb 24, 2009 at 12:48 PM, Amin Mohammed-Coleman <aminmc@gmail.com
> >wrote:
>
> > The reason for the indexreader.reopen is because I have a webapp which
> > enables users to upload files and then search for the documents. If I
> > don't
> > reopen i'm concerned that the facet hit counter won't be updated.
> >
> > On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <aminmc@gmail.com
> > >wrote:
> >
> > > Hi
> > > I have been able to get the code working for my scenario, however I
> have
> > a
> > > question and I was wondering if I could get some help. I have a list
> of
> > > IndexSearchers which are used in a MultiSearcher class. I use the
> > > indexsearchers to get each indexreader and put them into a
> > MultiIndexReader.
> > >
> > > IndexReader[] readers = new IndexReader[searchables.length];
> > >
> > > for (int i =0 ; i < searchables.length;i++) {
> > >
> > > IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
> > >
> > > readers[i] = indexSearcher.getIndexReader();
> > >
> > > IndexReader newReader = readers[i].reopen();
> > >
> > > if (newReader != readers[i]) {
> > >
> > > readers[i].close();
> > >
> > > }
> > >
> > > readers[i] = newReader;
> > >
> > >
> > >
> > > }
> > >
> > > multiReader = new MultiReader(readers);
> > >
> > > OpenBitSetFacetHitCounter facetHitCounter =
> > newOpenBitSetFacetHitCounter();
> > >
> > > IndexSearcher indexSearcher = new IndexSearcher(multiReader);
> > >
> > >
> > > I then use the indexseacher to do the facet stuff. I end the code with
> > > closing the multireader. This is causing problems in another method
> > where I
> > > do some other search as the indexreaders are closed. Is it ok to not
> > close
> > > the multiindexreader or should I do some additional checks in the other
> > > method to see if the indexreader is closed?
> > >
> > >
> > >
> > > Cheers
> > >
> > >
> > > P.S. Hope that made sense...!
> > >
> > >
> > > On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
> aminmc@gmail.com
> > >wrote:
> > >
> > >> Hi
> > >>
> > >> Thanks just what I needed!
> > >>
> > >> Cheers
> > >> Amin
> > >>
> > >>
> > >> On 22 Feb 2009, at 16:11, Marcelo Ochoa <ma...@gmail.com>
> > wrote:
> > >>
> > >> Hi Amin:
> > >>> Please take a look a this blog post:
> > >>>
> > >>>
> >
> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
> > >>> Best regards, Marcelo.
> > >>>
> > >>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
> > aminmc@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Hi
> > >>>>
> > >>>> Sorry to re send this email but I was wondering if I could get some
> > >>>> advice
> > >>>> on this.
> > >>>>
> > >>>> Cheers
> > >>>>
> > >>>> Amin
> > >>>>
> > >>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <am...@gmail.com>
> > >>>> wrote:
> > >>>>
> > >>>> Hi
> > >>>>>
> > >>>>> I am looking at building a faceted search using Lucene. I know
> that
> > >>>>> Solr
> > >>>>> comes with this built in, however I would like to try this by
> myself
> > >>>>> (something to add to my CV!). I have been looking around and I
> found
> > >>>>> that
> > >>>>> you can use the IndexReader and use TermVectors. This looks ok but
> > I'm
> > >>>>> not
> > >>>>> sure how to filter the results so that a particular user can only
> see
> > a
> > >>>>> subset of results. The next option I was looking at was something
> > like
> > >>>>>
> > >>>>> Term term1 = new Term("brand", "ford");
> > >>>>> Term term2 = new Term("brand", "vw");
> > >>>>> Term[] termsArray = new Term[] { term1, term2 };un
> > >>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
> > >>>>>
> > >>>>> The only problem here is that I have to provide the brand type each
> > >>>>> time a
> > >>>>> new brand is created. Again I'm not sure how I can filter the
> > results
> > >>>>> here.
> > >>>>> It may be that I'm using the wrong api methods to do this.
> > >>>>>
> > >>>>> I would be grateful if I could get some advice on this.
> > >>>>>
> > >>>>>
> > >>>>> Cheers
> > >>>>> Amin
> > >>>>>
> > >>>>> P.S. I am basically trying to do something that displays the
> > following
> > >>>>>
> > >>>>> Personal Contact (23) Business Contact (45) and so on..
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>> --
> > >>> Marcelo F. Ochoa
> > >>> http://marceloochoa.blogspot.com/
> > >>> http://marcelo.ochoa.googlepages.com/home
> > >>> ______________
> > >>> Want to integrate Lucene and Oracle?
> > >>>
> > >>>
> >
> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
> > >>> Is Oracle 11g REST ready?
> > >>>
> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > >>> For additional commands, e-mail: java-user-help@lucene.apache.org
> > >>>
> > >>>
> > >
> >
>
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
Forgot to mention that the previous code that i sent was related to facet
search. This is a general search method I have implemented (they can
probably be combined...).
On Thu, Feb 26, 2009 at 8:21 PM, Amin Mohammed-Coleman <am...@gmail.com>wrote:
> Hi
> I have modified my search code. Here is the following:
> [code]
>
> public Summary[] search(SearchRequest searchRequest) throwsSearchExecutionException {
>
> String searchTerm = searchRequest.getSearchTerm();
>
> if (StringUtils.isBlank(searchTerm)) {
>
> throw new SearchExecutionException("Search string cannot be empty. There
> will be too many results to process.");
>
> }
>
> List<Summary> summaryList = new ArrayList<Summary>();
>
> StopWatch stopWatch = new StopWatch("searchStopWatch");
>
> stopWatch.start();
>
> MultiSearcher multiSearcher = null;
>
> List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
>
> boolean refreshSearchers = false;
>
> try {
>
> LOGGER.debug("Ensuring all index readers are up to date...");
>
> for (IndexSearcher indexSearcher: searchers) {
>
> IndexReader reader = indexSearcher.getIndexReader();
>
> reader.incRef();
>
> Directory directory = reader.directory();
>
>
>
> long currentVersion = reader.getVersion();
>
> if (IndexReader.getCurrentVersion(directory) != currentVersion) {
>
> IndexReader newReader = reader.reopen();
>
> if (newReader != reader) {
>
> reader.decRef();
>
> refreshSearchers = true;
>
> }
>
> reader = newReader;
>
> }
>
> IndexSearcher indexSearch = new IndexSearcher(reader);
>
> indexSearchers.add(indexSearch);
>
> }
>
> if (refreshSearchers) {
>
> searchers.clear();
>
> searchers = new ArrayList<IndexSearcher>(indexSearchers);
>
> }
>
> LOGGER.debug("All Index Searchers are up to date. No of index searchers '"+ indexSearchers.size() +
> "'");
>
> multiSearcher = new MultiSearcher(searchers.toArray(new IndexSearcher[]
> {}));
>
> PerFieldAnalyzerWrapper analyzerWrapper = new PerFieldAnalyzerWrapper(
> analyzer);
>
> analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(), newKeywordAnalyzer());
>
> QueryParser queryParser = newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
> analyzerWrapper);
>
> Query query = queryParser.parse(searchTerm);
>
> LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
> query.toString() +"'");
>
> Sort sort = null;
>
> sort = applySortIfApplicable(searchRequest);
>
> Filter[] filters =applyFiltersIfApplicable(searchRequest);
>
> ChainedFilter chainedFilter = null;
>
> if (filters != null) {
>
> chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
>
> }
>
> TopDocs topDocs = multiSearcher.search(query,chainedFilter ,100,sort);
>
> ScoreDoc[] scoreDocs = topDocs.scoreDocs;
>
> LOGGER.debug("total number of hits for [" + query.toString() + " ] = "+topDocs.
> totalHits);
>
> for (ScoreDoc scoreDoc : scoreDocs) {
>
> final Document doc = multiSearcher.doc(scoreDoc.doc);
>
> float score = scoreDoc.score;
>
> final BaseDocument baseDocument = new BaseDocument(doc, score);
>
> Summary documentSummary = new DocumentSummaryImpl(baseDocument);
>
> summaryList.add(documentSummary);
>
> }
>
> multiSearcher.close();
>
> } catch (Exception e) {
>
> throw new IllegalStateException(e);
>
> }
>
> stopWatch.stop();
>
> LOGGER.debug("total time taken for document seach: " +
> stopWatch.getTotalTimeMillis() + " ms");
>
> return summaryList.toArray(new Summary[] {});
>
> }
>
> [/code]
>
> Just some background:
>
> There is a list of indexsearchers that are injected via Spring. These
> searchers are configured again by Spring. As you can see the multisearcher
> is a local variable. I then have a variable that checks if a indexreader is
> not up to date. When this is set to true the indexsearchers are refreshed.
>
> I would be grateful on your thoughts.
>
>
> On Thu, Feb 26, 2009 at 1:35 PM, Amin Mohammed-Coleman <am...@gmail.com>wrote:
>
>> Hi
>>
>> Thanks for your help. I will modify my facet search and my other code to
>> use the recommendations. Would it be ok to get a review of the completed
>> code? I just want to make sure that I'm not doing anything that may cause
>> any problems (threading, memory).
>>
>> Cheers
>>
>>
>> On Thu, Feb 26, 2009 at 1:10 PM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>>
>>> See below -- this is an excerpt from the upcoming Lucene in Action
>>> revision (chapter 10).
>>>
>>> It's a simple class. Use it like this for searching:
>>>
>>> IndexSearcher searcher = manager.get();
>>> try {
>>> searcher.search(...).
>>> ...render results...
>>> } finally {
>>> manager.release(searcher);
>>> searcher = null;
>>> }
>>>
>>> When you want to reopen (application dependent), call maybeReopen.
>>> Subclass and define the warm() method if needed.
>>>
>>> NOTE: this hasn't yet been heavily tested (I just quickly revised it to
>>> use
>>> incRef/decRef).
>>>
>>> Mike
>>>
>>> import java.io.IOException;
>>> import java.util.HashMap;
>>>
>>> import org.apache.lucene.search.IndexSearcher;
>>> import org.apache.lucene.index.IndexReader;
>>> import org.apache.lucene.store.Directory;
>>>
>>> /** Utility class to get/refresh searchers when you are
>>> * using multiple threads. */
>>>
>>> public class SearcherManager {
>>>
>>> private IndexSearcher currentSearcher; //A
>>> private Directory dir;
>>>
>>> public SearcherManager(Directory dir) throws IOException {
>>> this.dir = dir;
>>> currentSearcher = new IndexSearcher(IndexReader.open(dir)); //B
>>> }
>>>
>>> public void warm(IndexSearcher searcher) {} //C
>>>
>>> public void maybeReopen() throws IOException { //D
>>> long currentVersion = currentSearcher.getIndexReader().getVersion();
>>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>>> IndexReader newReader = currentSearcher.getIndexReader().reopen();
>>> assert newReader != currentSearcher.getIndexReader();
>>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>>> warm(newSearcher);
>>> swapSearcher(newSearcher);
>>> }
>>> }
>>>
>>> public synchronized IndexSearcher get() { //E
>>> currentSearcher.getIndexReader().incRef();
>>> return currentSearcher;
>>> }
>>>
>>> public synchronized void release(IndexSearcher searcher) //F
>>> throws IOException {
>>> searcher.getIndexReader().decRef();
>>> }
>>>
>>> private synchronized void swapSearcher(IndexSearcher newSearcher) //G
>>> throws IOException {
>>> release(currentSearcher);
>>> currentSearcher = newSearcher;
>>> }
>>> }
>>>
>>> /*
>>> #A Current IndexSearcher
>>> #B Create initial searcher
>>> #C Implement in subclass to warm new searcher
>>> #D Call this to reopen searcher if index changed
>>> #E Returns current searcher
>>> #F Release searcher
>>> #G Swaps currentSearcher to new searcher
>>> */
>>>
>>> Mike
>>>
>>>
>>> Amin Mohammed-Coleman wrote:
>>>
>>> Hi
>>>>
>>>> Thanks for your reply. Without sound completely ...silly...how do i go
>>>> abouts using the methods you mentioned...
>>>>
>>>> Cheers
>>>> Amin
>>>>
>>>> On Thu, Feb 26, 2009 at 10:24 AM, Michael McCandless <
>>>> lucene@mikemccandless.com> wrote:
>>>>
>>>>
>>>>> Actually, it's best to use IndexReader.incRef/decRef to track the
>>>>> IndexReader.
>>>>>
>>>>> You should not rely on GC to close your IndexReader since this can
>>>>> easily
>>>>> tie up resources (eg open file descriptors) for too long.
>>>>>
>>>>> Mike
>>>>>
>>>>>
>>>>> Michael Stoppelman wrote:
>>>>>
>>>>> If another thread is executing a query with the handle to one of
>>>>>
>>>>>> readers[i]
>>>>>> you're going to kill it since the IndexReader is now closed.
>>>>>> Just don't call the IndexReader#close() method. If nothing is pointing
>>>>>> at
>>>>>> the readers they should be garbage collected. Also, you might
>>>>>> want to warm up your new IndexSearcher before you switch to it,
>>>>>> meaning
>>>>>> run
>>>>>> a few queries on it before you swap the old one out.
>>>>>>
>>>>>> M
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Feb 24, 2009 at 12:48 PM, Amin Mohammed-Coleman <
>>>>>> aminmc@gmail.com
>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>
>>>>>> The reason for the indexreader.reopen is because I have a webapp which
>>>>>>
>>>>>>> enables users to upload files and then search for the documents. If
>>>>>>> I
>>>>>>> don't
>>>>>>> reopen i'm concerned that the facet hit counter won't be updated.
>>>>>>>
>>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <
>>>>>>> aminmc@gmail.com
>>>>>>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>>> I have been able to get the code working for my scenario, however I
>>>>>>>> have
>>>>>>>>
>>>>>>>> a
>>>>>>>
>>>>>>> question and I was wondering if I could get some help. I have a
>>>>>>>> list of
>>>>>>>> IndexSearchers which are used in a MultiSearcher class. I use the
>>>>>>>> indexsearchers to get each indexreader and put them into a
>>>>>>>>
>>>>>>>> MultiIndexReader.
>>>>>>>
>>>>>>>
>>>>>>>> IndexReader[] readers = new IndexReader[searchables.length];
>>>>>>>>
>>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>>
>>>>>>>> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>>>>>>>>
>>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>>
>>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>>
>>>>>>>> if (newReader != readers[i]) {
>>>>>>>>
>>>>>>>> readers[i].close();
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> readers[i] = newReader;
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>>
>>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>>
>>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>>
>>>>>>>
>>>>>>>> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>>>>>>>>
>>>>>>>>
>>>>>>>> I then use the indexseacher to do the facet stuff. I end the code
>>>>>>>> with
>>>>>>>> closing the multireader. This is causing problems in another method
>>>>>>>>
>>>>>>>> where I
>>>>>>>
>>>>>>> do some other search as the indexreaders are closed. Is it ok to
>>>>>>>> not
>>>>>>>>
>>>>>>>> close
>>>>>>>
>>>>>>> the multiindexreader or should I do some additional checks in the
>>>>>>>> other
>>>>>>>> method to see if the indexreader is closed?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>>
>>>>>>>>
>>>>>>>> P.S. Hope that made sense...!
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
>>>>>>>> aminmc@gmail.com
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks just what I needed!
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>> Amin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <ma...@gmail.com>
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Hi Amin:
>>>>>>>>>
>>>>>>>>> Please take a look a this blog post:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>>
>>>>>>> Best regards, Marcelo.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>>>>>>>>>>
>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>
>>>>>>>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Sorry to re send this email but I was wondering if I could get
>>>>>>>>>>> some
>>>>>>>>>>> advice
>>>>>>>>>>> on this.
>>>>>>>>>>>
>>>>>>>>>>> Cheers
>>>>>>>>>>>
>>>>>>>>>>> Amin
>>>>>>>>>>>
>>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <
>>>>>>>>>>> aminmc@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> I am looking at building a faceted search using Lucene. I know
>>>>>>>>>>>> that
>>>>>>>>>>>> Solr
>>>>>>>>>>>> comes with this built in, however I would like to try this by
>>>>>>>>>>>> myself
>>>>>>>>>>>> (something to add to my CV!). I have been looking around and I
>>>>>>>>>>>> found
>>>>>>>>>>>> that
>>>>>>>>>>>> you can use the IndexReader and use TermVectors. This looks ok
>>>>>>>>>>>> but
>>>>>>>>>>>>
>>>>>>>>>>>> I'm
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>> not
>>>>>>>>
>>>>>>>>> sure how to filter the results so that a particular user can only
>>>>>>>>>>>> see
>>>>>>>>>>>>
>>>>>>>>>>>> a
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>> subset of results. The next option I was looking at was something
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>> like
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>>
>>>>>>>>>>>> The only problem here is that I have to provide the brand type
>>>>>>>>>>>> each
>>>>>>>>>>>> time a
>>>>>>>>>>>> new brand is created. Again I'm not sure how I can filter the
>>>>>>>>>>>>
>>>>>>>>>>>> results
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>> here.
>>>>>>>>
>>>>>>>>> It may be that I'm using the wrong api methods to do this.
>>>>>>>>>>>>
>>>>>>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers
>>>>>>>>>>>> Amin
>>>>>>>>>>>>
>>>>>>>>>>>> P.S. I am basically trying to do something that displays the
>>>>>>>>>>>>
>>>>>>>>>>>> following
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>>> ______________
>>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>>
>>>>>>> Is Oracle 11g REST ready?
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
Hi
I have modified my search code. Here is the following:
[code]
public Summary[] search(SearchRequest searchRequest)
throwsSearchExecutionException {
String searchTerm = searchRequest.getSearchTerm();
if (StringUtils.isBlank(searchTerm)) {
throw new SearchExecutionException("Search string cannot be empty. There
will be too many results to process.");
}
List<Summary> summaryList = new ArrayList<Summary>();
StopWatch stopWatch = new StopWatch("searchStopWatch");
stopWatch.start();
MultiSearcher multiSearcher = null;
List<IndexSearcher> indexSearchers = new ArrayList<IndexSearcher>();
boolean refreshSearchers = false;
try {
LOGGER.debug("Ensuring all index readers are up to date...");
for (IndexSearcher indexSearcher: searchers) {
IndexReader reader = indexSearcher.getIndexReader();
reader.incRef();
Directory directory = reader.directory();
long currentVersion = reader.getVersion();
if (IndexReader.getCurrentVersion(directory) != currentVersion) {
IndexReader newReader = reader.reopen();
if (newReader != reader) {
reader.decRef();
refreshSearchers = true;
}
reader = newReader;
}
IndexSearcher indexSearch = new IndexSearcher(reader);
indexSearchers.add(indexSearch);
}
if (refreshSearchers) {
searchers.clear();
searchers = new ArrayList<IndexSearcher>(indexSearchers);
}
LOGGER.debug("All Index Searchers are up to date. No of index searchers '" +
indexSearchers.size() +"'");
multiSearcher = new MultiSearcher(searchers.toArray(new IndexSearcher[]
{}));
PerFieldAnalyzerWrapper analyzerWrapper = new PerFieldAnalyzerWrapper(
analyzer);
analyzerWrapper.addAnalyzer(FieldNameEnum.TYPE.getDescription(),
newKeywordAnalyzer());
QueryParser queryParser =
newMultiFieldQueryParser(FieldNameEnum.fieldNameDescriptions(),
analyzerWrapper);
Query query = queryParser.parse(searchTerm);
LOGGER.debug("Search Term '" + searchTerm +"' ----> Lucene Query '" +
query.toString() +"'");
Sort sort = null;
sort = applySortIfApplicable(searchRequest);
Filter[] filters =applyFiltersIfApplicable(searchRequest);
ChainedFilter chainedFilter = null;
if (filters != null) {
chainedFilter = new ChainedFilter(filters, ChainedFilter.OR);
}
TopDocs topDocs = multiSearcher.search(query,chainedFilter ,100,sort);
ScoreDoc[] scoreDocs = topDocs.scoreDocs;
LOGGER.debug("total number of hits for [" + query.toString() + " ] = "+topDocs.
totalHits);
for (ScoreDoc scoreDoc : scoreDocs) {
final Document doc = multiSearcher.doc(scoreDoc.doc);
float score = scoreDoc.score;
final BaseDocument baseDocument = new BaseDocument(doc, score);
Summary documentSummary = new DocumentSummaryImpl(baseDocument);
summaryList.add(documentSummary);
}
multiSearcher.close();
} catch (Exception e) {
throw new IllegalStateException(e);
}
stopWatch.stop();
LOGGER.debug("total time taken for document seach: " +
stopWatch.getTotalTimeMillis() + " ms");
return summaryList.toArray(new Summary[] {});
}
[/code]
Just some background:
There is a list of indexsearchers that are injected via Spring. These
searchers are configured again by Spring. As you can see the multisearcher
is a local variable. I then have a variable that checks if a indexreader is
not up to date. When this is set to true the indexsearchers are refreshed.
I would be grateful on your thoughts.
On Thu, Feb 26, 2009 at 1:35 PM, Amin Mohammed-Coleman <am...@gmail.com>wrote:
> Hi
>
> Thanks for your help. I will modify my facet search and my other code to
> use the recommendations. Would it be ok to get a review of the completed
> code? I just want to make sure that I'm not doing anything that may cause
> any problems (threading, memory).
>
> Cheers
>
>
> On Thu, Feb 26, 2009 at 1:10 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>>
>> See below -- this is an excerpt from the upcoming Lucene in Action
>> revision (chapter 10).
>>
>> It's a simple class. Use it like this for searching:
>>
>> IndexSearcher searcher = manager.get();
>> try {
>> searcher.search(...).
>> ...render results...
>> } finally {
>> manager.release(searcher);
>> searcher = null;
>> }
>>
>> When you want to reopen (application dependent), call maybeReopen.
>> Subclass and define the warm() method if needed.
>>
>> NOTE: this hasn't yet been heavily tested (I just quickly revised it to
>> use
>> incRef/decRef).
>>
>> Mike
>>
>> import java.io.IOException;
>> import java.util.HashMap;
>>
>> import org.apache.lucene.search.IndexSearcher;
>> import org.apache.lucene.index.IndexReader;
>> import org.apache.lucene.store.Directory;
>>
>> /** Utility class to get/refresh searchers when you are
>> * using multiple threads. */
>>
>> public class SearcherManager {
>>
>> private IndexSearcher currentSearcher; //A
>> private Directory dir;
>>
>> public SearcherManager(Directory dir) throws IOException {
>> this.dir = dir;
>> currentSearcher = new IndexSearcher(IndexReader.open(dir)); //B
>> }
>>
>> public void warm(IndexSearcher searcher) {} //C
>>
>> public void maybeReopen() throws IOException { //D
>> long currentVersion = currentSearcher.getIndexReader().getVersion();
>> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
>> IndexReader newReader = currentSearcher.getIndexReader().reopen();
>> assert newReader != currentSearcher.getIndexReader();
>> IndexSearcher newSearcher = new IndexSearcher(newReader);
>> warm(newSearcher);
>> swapSearcher(newSearcher);
>> }
>> }
>>
>> public synchronized IndexSearcher get() { //E
>> currentSearcher.getIndexReader().incRef();
>> return currentSearcher;
>> }
>>
>> public synchronized void release(IndexSearcher searcher) //F
>> throws IOException {
>> searcher.getIndexReader().decRef();
>> }
>>
>> private synchronized void swapSearcher(IndexSearcher newSearcher) //G
>> throws IOException {
>> release(currentSearcher);
>> currentSearcher = newSearcher;
>> }
>> }
>>
>> /*
>> #A Current IndexSearcher
>> #B Create initial searcher
>> #C Implement in subclass to warm new searcher
>> #D Call this to reopen searcher if index changed
>> #E Returns current searcher
>> #F Release searcher
>> #G Swaps currentSearcher to new searcher
>> */
>>
>> Mike
>>
>>
>> Amin Mohammed-Coleman wrote:
>>
>> Hi
>>>
>>> Thanks for your reply. Without sound completely ...silly...how do i go
>>> abouts using the methods you mentioned...
>>>
>>> Cheers
>>> Amin
>>>
>>> On Thu, Feb 26, 2009 at 10:24 AM, Michael McCandless <
>>> lucene@mikemccandless.com> wrote:
>>>
>>>
>>>> Actually, it's best to use IndexReader.incRef/decRef to track the
>>>> IndexReader.
>>>>
>>>> You should not rely on GC to close your IndexReader since this can
>>>> easily
>>>> tie up resources (eg open file descriptors) for too long.
>>>>
>>>> Mike
>>>>
>>>>
>>>> Michael Stoppelman wrote:
>>>>
>>>> If another thread is executing a query with the handle to one of
>>>>
>>>>> readers[i]
>>>>> you're going to kill it since the IndexReader is now closed.
>>>>> Just don't call the IndexReader#close() method. If nothing is pointing
>>>>> at
>>>>> the readers they should be garbage collected. Also, you might
>>>>> want to warm up your new IndexSearcher before you switch to it, meaning
>>>>> run
>>>>> a few queries on it before you swap the old one out.
>>>>>
>>>>> M
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Feb 24, 2009 at 12:48 PM, Amin Mohammed-Coleman <
>>>>> aminmc@gmail.com
>>>>>
>>>>>> wrote:
>>>>>>
>>>>>
>>>>> The reason for the indexreader.reopen is because I have a webapp which
>>>>>
>>>>>> enables users to upload files and then search for the documents. If I
>>>>>> don't
>>>>>> reopen i'm concerned that the facet hit counter won't be updated.
>>>>>>
>>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <
>>>>>> aminmc@gmail.com
>>>>>>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>> Hi
>>>>>>
>>>>>>> I have been able to get the code working for my scenario, however I
>>>>>>> have
>>>>>>>
>>>>>>> a
>>>>>>
>>>>>> question and I was wondering if I could get some help. I have a list
>>>>>>> of
>>>>>>> IndexSearchers which are used in a MultiSearcher class. I use the
>>>>>>> indexsearchers to get each indexreader and put them into a
>>>>>>>
>>>>>>> MultiIndexReader.
>>>>>>
>>>>>>
>>>>>>> IndexReader[] readers = new IndexReader[searchables.length];
>>>>>>>
>>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>>
>>>>>>> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>>>>>>>
>>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>>
>>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>>
>>>>>>> if (newReader != readers[i]) {
>>>>>>>
>>>>>>> readers[i].close();
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> readers[i] = newReader;
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> multiReader = new MultiReader(readers);
>>>>>>>
>>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>>
>>>>>>> newOpenBitSetFacetHitCounter();
>>>>>>
>>>>>>
>>>>>>> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>>>>>>>
>>>>>>>
>>>>>>> I then use the indexseacher to do the facet stuff. I end the code
>>>>>>> with
>>>>>>> closing the multireader. This is causing problems in another method
>>>>>>>
>>>>>>> where I
>>>>>>
>>>>>> do some other search as the indexreaders are closed. Is it ok to not
>>>>>>>
>>>>>>> close
>>>>>>
>>>>>> the multiindexreader or should I do some additional checks in the
>>>>>>> other
>>>>>>> method to see if the indexreader is closed?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>>
>>>>>>> P.S. Hope that made sense...!
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
>>>>>>> aminmc@gmail.com
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks just what I needed!
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>> Amin
>>>>>>>>
>>>>>>>>
>>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <ma...@gmail.com>
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>
>>>>>>
>>>>>>
>>>>>>> Hi Amin:
>>>>>>>>
>>>>>>>> Please take a look a this blog post:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>>
>>>>>> Best regards, Marcelo.
>>>>>>>
>>>>>>>>
>>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>>>>>>>>>
>>>>>>>>> aminmc@gmail.com>
>>>>>>>>
>>>>>>>
>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Sorry to re send this email but I was wondering if I could get
>>>>>>>>>> some
>>>>>>>>>> advice
>>>>>>>>>> on this.
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>>
>>>>>>>>>> Amin
>>>>>>>>>>
>>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <aminmc@gmail.com
>>>>>>>>>> >
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I am looking at building a faceted search using Lucene. I know
>>>>>>>>>>> that
>>>>>>>>>>> Solr
>>>>>>>>>>> comes with this built in, however I would like to try this by
>>>>>>>>>>> myself
>>>>>>>>>>> (something to add to my CV!). I have been looking around and I
>>>>>>>>>>> found
>>>>>>>>>>> that
>>>>>>>>>>> you can use the IndexReader and use TermVectors. This looks ok
>>>>>>>>>>> but
>>>>>>>>>>>
>>>>>>>>>>> I'm
>>>>>>>>>>
>>>>>>>>>
>>>>>> not
>>>>>>>
>>>>>>>> sure how to filter the results so that a particular user can only
>>>>>>>>>>> see
>>>>>>>>>>>
>>>>>>>>>>> a
>>>>>>>>>>
>>>>>>>>>
>>>>>> subset of results. The next option I was looking at was something
>>>>>>>
>>>>>>>>
>>>>>>>>>>> like
>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>>>>
>>>>>>>>>>> The only problem here is that I have to provide the brand type
>>>>>>>>>>> each
>>>>>>>>>>> time a
>>>>>>>>>>> new brand is created. Again I'm not sure how I can filter the
>>>>>>>>>>>
>>>>>>>>>>> results
>>>>>>>>>>
>>>>>>>>>
>>>>>> here.
>>>>>>>
>>>>>>>> It may be that I'm using the wrong api methods to do this.
>>>>>>>>>>>
>>>>>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Cheers
>>>>>>>>>>> Amin
>>>>>>>>>>>
>>>>>>>>>>> P.S. I am basically trying to do something that displays the
>>>>>>>>>>>
>>>>>>>>>>> following
>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Marcelo F. Ochoa
>>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>>> ______________
>>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>>
>>>>>> Is Oracle 11g REST ready?
>>>>>>>
>>>>>>>>
>>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
Hi
Thanks for your help. I will modify my facet search and my other code to
use the recommendations. Would it be ok to get a review of the completed
code? I just want to make sure that I'm not doing anything that may cause
any problems (threading, memory).
Cheers
On Thu, Feb 26, 2009 at 1:10 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
>
> See below -- this is an excerpt from the upcoming Lucene in Action
> revision (chapter 10).
>
> It's a simple class. Use it like this for searching:
>
> IndexSearcher searcher = manager.get();
> try {
> searcher.search(...).
> ...render results...
> } finally {
> manager.release(searcher);
> searcher = null;
> }
>
> When you want to reopen (application dependent), call maybeReopen.
> Subclass and define the warm() method if needed.
>
> NOTE: this hasn't yet been heavily tested (I just quickly revised it to use
> incRef/decRef).
>
> Mike
>
> import java.io.IOException;
> import java.util.HashMap;
>
> import org.apache.lucene.search.IndexSearcher;
> import org.apache.lucene.index.IndexReader;
> import org.apache.lucene.store.Directory;
>
> /** Utility class to get/refresh searchers when you are
> * using multiple threads. */
>
> public class SearcherManager {
>
> private IndexSearcher currentSearcher; //A
> private Directory dir;
>
> public SearcherManager(Directory dir) throws IOException {
> this.dir = dir;
> currentSearcher = new IndexSearcher(IndexReader.open(dir)); //B
> }
>
> public void warm(IndexSearcher searcher) {} //C
>
> public void maybeReopen() throws IOException { //D
> long currentVersion = currentSearcher.getIndexReader().getVersion();
> if (IndexReader.getCurrentVersion(dir) != currentVersion) {
> IndexReader newReader = currentSearcher.getIndexReader().reopen();
> assert newReader != currentSearcher.getIndexReader();
> IndexSearcher newSearcher = new IndexSearcher(newReader);
> warm(newSearcher);
> swapSearcher(newSearcher);
> }
> }
>
> public synchronized IndexSearcher get() { //E
> currentSearcher.getIndexReader().incRef();
> return currentSearcher;
> }
>
> public synchronized void release(IndexSearcher searcher) //F
> throws IOException {
> searcher.getIndexReader().decRef();
> }
>
> private synchronized void swapSearcher(IndexSearcher newSearcher) //G
> throws IOException {
> release(currentSearcher);
> currentSearcher = newSearcher;
> }
> }
>
> /*
> #A Current IndexSearcher
> #B Create initial searcher
> #C Implement in subclass to warm new searcher
> #D Call this to reopen searcher if index changed
> #E Returns current searcher
> #F Release searcher
> #G Swaps currentSearcher to new searcher
> */
>
> Mike
>
>
> Amin Mohammed-Coleman wrote:
>
> Hi
>>
>> Thanks for your reply. Without sound completely ...silly...how do i go
>> abouts using the methods you mentioned...
>>
>> Cheers
>> Amin
>>
>> On Thu, Feb 26, 2009 at 10:24 AM, Michael McCandless <
>> lucene@mikemccandless.com> wrote:
>>
>>
>>> Actually, it's best to use IndexReader.incRef/decRef to track the
>>> IndexReader.
>>>
>>> You should not rely on GC to close your IndexReader since this can easily
>>> tie up resources (eg open file descriptors) for too long.
>>>
>>> Mike
>>>
>>>
>>> Michael Stoppelman wrote:
>>>
>>> If another thread is executing a query with the handle to one of
>>>
>>>> readers[i]
>>>> you're going to kill it since the IndexReader is now closed.
>>>> Just don't call the IndexReader#close() method. If nothing is pointing
>>>> at
>>>> the readers they should be garbage collected. Also, you might
>>>> want to warm up your new IndexSearcher before you switch to it, meaning
>>>> run
>>>> a few queries on it before you swap the old one out.
>>>>
>>>> M
>>>>
>>>>
>>>>
>>>> On Tue, Feb 24, 2009 at 12:48 PM, Amin Mohammed-Coleman <
>>>> aminmc@gmail.com
>>>>
>>>>> wrote:
>>>>>
>>>>
>>>> The reason for the indexreader.reopen is because I have a webapp which
>>>>
>>>>> enables users to upload files and then search for the documents. If I
>>>>> don't
>>>>> reopen i'm concerned that the facet hit counter won't be updated.
>>>>>
>>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <
>>>>> aminmc@gmail.com
>>>>>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>> Hi
>>>>>
>>>>>> I have been able to get the code working for my scenario, however I
>>>>>> have
>>>>>>
>>>>>> a
>>>>>
>>>>> question and I was wondering if I could get some help. I have a list
>>>>>> of
>>>>>> IndexSearchers which are used in a MultiSearcher class. I use the
>>>>>> indexsearchers to get each indexreader and put them into a
>>>>>>
>>>>>> MultiIndexReader.
>>>>>
>>>>>
>>>>>> IndexReader[] readers = new IndexReader[searchables.length];
>>>>>>
>>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>>
>>>>>> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>>>>>>
>>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>>
>>>>>> IndexReader newReader = readers[i].reopen();
>>>>>>
>>>>>> if (newReader != readers[i]) {
>>>>>>
>>>>>> readers[i].close();
>>>>>>
>>>>>> }
>>>>>>
>>>>>> readers[i] = newReader;
>>>>>>
>>>>>>
>>>>>>
>>>>>> }
>>>>>>
>>>>>> multiReader = new MultiReader(readers);
>>>>>>
>>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>>
>>>>>> newOpenBitSetFacetHitCounter();
>>>>>
>>>>>
>>>>>> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>>>>>>
>>>>>>
>>>>>> I then use the indexseacher to do the facet stuff. I end the code
>>>>>> with
>>>>>> closing the multireader. This is causing problems in another method
>>>>>>
>>>>>> where I
>>>>>
>>>>> do some other search as the indexreaders are closed. Is it ok to not
>>>>>>
>>>>>> close
>>>>>
>>>>> the multiindexreader or should I do some additional checks in the
>>>>>> other
>>>>>> method to see if the indexreader is closed?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>>>> P.S. Hope that made sense...!
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
>>>>>> aminmc@gmail.com
>>>>>> wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>>>
>>>>>>> Thanks just what I needed!
>>>>>>>
>>>>>>> Cheers
>>>>>>> Amin
>>>>>>>
>>>>>>>
>>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <ma...@gmail.com>
>>>>>>>
>>>>>>> wrote:
>>>>>>
>>>>>
>>>>>
>>>>>> Hi Amin:
>>>>>>>
>>>>>>> Please take a look a this blog post:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>>
>>>>> Best regards, Marcelo.
>>>>>>
>>>>>>>
>>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>>>>>>>>
>>>>>>>> aminmc@gmail.com>
>>>>>>>
>>>>>>
>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Sorry to re send this email but I was wondering if I could get some
>>>>>>>>> advice
>>>>>>>>> on this.
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>>
>>>>>>>>> Amin
>>>>>>>>>
>>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <am...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I am looking at building a faceted search using Lucene. I know
>>>>>>>>>> that
>>>>>>>>>> Solr
>>>>>>>>>> comes with this built in, however I would like to try this by
>>>>>>>>>> myself
>>>>>>>>>> (something to add to my CV!). I have been looking around and I
>>>>>>>>>> found
>>>>>>>>>> that
>>>>>>>>>> you can use the IndexReader and use TermVectors. This looks ok
>>>>>>>>>> but
>>>>>>>>>>
>>>>>>>>>> I'm
>>>>>>>>>
>>>>>>>>
>>>>> not
>>>>>>
>>>>>>> sure how to filter the results so that a particular user can only
>>>>>>>>>> see
>>>>>>>>>>
>>>>>>>>>> a
>>>>>>>>>
>>>>>>>>
>>>>> subset of results. The next option I was looking at was something
>>>>>>
>>>>>>>
>>>>>>>>>> like
>>>>>>>>>
>>>>>>>>
>>>>>
>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>>>
>>>>>>>>>> The only problem here is that I have to provide the brand type
>>>>>>>>>> each
>>>>>>>>>> time a
>>>>>>>>>> new brand is created. Again I'm not sure how I can filter the
>>>>>>>>>>
>>>>>>>>>> results
>>>>>>>>>
>>>>>>>>
>>>>> here.
>>>>>>
>>>>>>> It may be that I'm using the wrong api methods to do this.
>>>>>>>>>>
>>>>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>> Amin
>>>>>>>>>>
>>>>>>>>>> P.S. I am basically trying to do something that displays the
>>>>>>>>>>
>>>>>>>>>> following
>>>>>>>>>
>>>>>>>>
>>>>>
>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Marcelo F. Ochoa
>>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>>> ______________
>>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>>
>>>>> Is Oracle 11g REST ready?
>>>>>>
>>>>>>>
>>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Faceted Search using Lucene
Posted by Michael McCandless <lu...@mikemccandless.com>.
See below -- this is an excerpt from the upcoming Lucene in Action
revision (chapter 10).
It's a simple class. Use it like this for searching:
IndexSearcher searcher = manager.get();
try {
searcher.search(...).
...render results...
} finally {
manager.release(searcher);
searcher = null;
}
When you want to reopen (application dependent), call maybeReopen.
Subclass and define the warm() method if needed.
NOTE: this hasn't yet been heavily tested (I just quickly revised it
to use
incRef/decRef).
Mike
import java.io.IOException;
import java.util.HashMap;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.store.Directory;
/** Utility class to get/refresh searchers when you are
* using multiple threads. */
public class SearcherManager {
private IndexSearcher currentSearcher; //A
private Directory dir;
public SearcherManager(Directory dir) throws IOException {
this.dir = dir;
currentSearcher = new IndexSearcher(IndexReader.open(dir)); //B
}
public void warm(IndexSearcher searcher) {} //C
public void maybeReopen() throws IOException { //D
long currentVersion =
currentSearcher.getIndexReader().getVersion();
if (IndexReader.getCurrentVersion(dir) != currentVersion) {
IndexReader newReader =
currentSearcher.getIndexReader().reopen();
assert newReader != currentSearcher.getIndexReader();
IndexSearcher newSearcher = new IndexSearcher(newReader);
warm(newSearcher);
swapSearcher(newSearcher);
}
}
public synchronized IndexSearcher get() { //E
currentSearcher.getIndexReader().incRef();
return currentSearcher;
}
public synchronized void release(IndexSearcher searcher) //F
throws IOException {
searcher.getIndexReader().decRef();
}
private synchronized void swapSearcher(IndexSearcher newSearcher) //G
throws IOException {
release(currentSearcher);
currentSearcher = newSearcher;
}
}
/*
#A Current IndexSearcher
#B Create initial searcher
#C Implement in subclass to warm new searcher
#D Call this to reopen searcher if index changed
#E Returns current searcher
#F Release searcher
#G Swaps currentSearcher to new searcher
*/
Mike
Amin Mohammed-Coleman wrote:
> Hi
>
> Thanks for your reply. Without sound completely ...silly...how do i
> go
> abouts using the methods you mentioned...
>
> Cheers
> Amin
>
> On Thu, Feb 26, 2009 at 10:24 AM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>>
>> Actually, it's best to use IndexReader.incRef/decRef to track the
>> IndexReader.
>>
>> You should not rely on GC to close your IndexReader since this can
>> easily
>> tie up resources (eg open file descriptors) for too long.
>>
>> Mike
>>
>>
>> Michael Stoppelman wrote:
>>
>> If another thread is executing a query with the handle to one of
>>> readers[i]
>>> you're going to kill it since the IndexReader is now closed.
>>> Just don't call the IndexReader#close() method. If nothing is
>>> pointing at
>>> the readers they should be garbage collected. Also, you might
>>> want to warm up your new IndexSearcher before you switch to it,
>>> meaning
>>> run
>>> a few queries on it before you swap the old one out.
>>>
>>> M
>>>
>>>
>>>
>>> On Tue, Feb 24, 2009 at 12:48 PM, Amin Mohammed-Coleman <aminmc@gmail.com
>>>> wrote:
>>>
>>> The reason for the indexreader.reopen is because I have a webapp
>>> which
>>>> enables users to upload files and then search for the documents.
>>>> If I
>>>> don't
>>>> reopen i'm concerned that the facet hit counter won't be updated.
>>>>
>>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <aminmc@gmail.com
>>>>
>>>>> wrote:
>>>>>
>>>>
>>>> Hi
>>>>> I have been able to get the code working for my scenario,
>>>>> however I have
>>>>>
>>>> a
>>>>
>>>>> question and I was wondering if I could get some help. I have a
>>>>> list of
>>>>> IndexSearchers which are used in a MultiSearcher class. I use the
>>>>> indexsearchers to get each indexreader and put them into a
>>>>>
>>>> MultiIndexReader.
>>>>
>>>>>
>>>>> IndexReader[] readers = new IndexReader[searchables.length];
>>>>>
>>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>>
>>>>> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>>>>>
>>>>> readers[i] = indexSearcher.getIndexReader();
>>>>>
>>>>> IndexReader newReader = readers[i].reopen();
>>>>>
>>>>> if (newReader != readers[i]) {
>>>>>
>>>>> readers[i].close();
>>>>>
>>>>> }
>>>>>
>>>>> readers[i] = newReader;
>>>>>
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>> multiReader = new MultiReader(readers);
>>>>>
>>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>>
>>>> newOpenBitSetFacetHitCounter();
>>>>
>>>>>
>>>>> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>>>>>
>>>>>
>>>>> I then use the indexseacher to do the facet stuff. I end the
>>>>> code with
>>>>> closing the multireader. This is causing problems in another
>>>>> method
>>>>>
>>>> where I
>>>>
>>>>> do some other search as the indexreaders are closed. Is it ok
>>>>> to not
>>>>>
>>>> close
>>>>
>>>>> the multiindexreader or should I do some additional checks in
>>>>> the other
>>>>> method to see if the indexreader is closed?
>>>>>
>>>>>
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> P.S. Hope that made sense...!
>>>>>
>>>>>
>>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
>>>>> aminmc@gmail.com
>>>>> wrote:
>>>>>
>>>>> Hi
>>>>>>
>>>>>> Thanks just what I needed!
>>>>>>
>>>>>> Cheers
>>>>>> Amin
>>>>>>
>>>>>>
>>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <ma...@gmail.com>
>>>>>>
>>>>> wrote:
>>>>
>>>>>
>>>>>> Hi Amin:
>>>>>>
>>>>>>> Please take a look a this blog post:
>>>>>>>
>>>>>>>
>>>>>>>
>>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>
>>>>> Best regards, Marcelo.
>>>>>>>
>>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>>>>>>>
>>>>>> aminmc@gmail.com>
>>>>
>>>>> wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>>>
>>>>>>>> Sorry to re send this email but I was wondering if I could
>>>>>>>> get some
>>>>>>>> advice
>>>>>>>> on this.
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>>
>>>>>>>> Amin
>>>>>>>>
>>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <aminmc@gmail.com
>>>>>>>> >
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I am looking at building a faceted search using Lucene. I
>>>>>>>>> know that
>>>>>>>>> Solr
>>>>>>>>> comes with this built in, however I would like to try this
>>>>>>>>> by myself
>>>>>>>>> (something to add to my CV!). I have been looking around
>>>>>>>>> and I
>>>>>>>>> found
>>>>>>>>> that
>>>>>>>>> you can use the IndexReader and use TermVectors. This looks
>>>>>>>>> ok but
>>>>>>>>>
>>>>>>>> I'm
>>>>
>>>>> not
>>>>>>>>> sure how to filter the results so that a particular user can
>>>>>>>>> only
>>>>>>>>> see
>>>>>>>>>
>>>>>>>> a
>>>>
>>>>> subset of results. The next option I was looking at was something
>>>>>>>>>
>>>>>>>> like
>>>>
>>>>>
>>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>>
>>>>>>>>> The only problem here is that I have to provide the brand
>>>>>>>>> type each
>>>>>>>>> time a
>>>>>>>>> new brand is created. Again I'm not sure how I can filter the
>>>>>>>>>
>>>>>>>> results
>>>>
>>>>> here.
>>>>>>>>> It may be that I'm using the wrong api methods to do this.
>>>>>>>>>
>>>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>> Amin
>>>>>>>>>
>>>>>>>>> P.S. I am basically trying to do something that displays the
>>>>>>>>>
>>>>>>>> following
>>>>
>>>>>
>>>>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Marcelo F. Ochoa
>>>>>>> http://marceloochoa.blogspot.com/
>>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>>> ______________
>>>>>>> Want to integrate Lucene and Oracle?
>>>>>>>
>>>>>>>
>>>>>>>
>>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>
>>>>> Is Oracle 11g REST ready?
>>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: java-user-
>>>>>>> help@lucene.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
Hi
Thanks for your reply. Without sound completely ...silly...how do i go
abouts using the methods you mentioned...
Cheers
Amin
On Thu, Feb 26, 2009 at 10:24 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:
>
> Actually, it's best to use IndexReader.incRef/decRef to track the
> IndexReader.
>
> You should not rely on GC to close your IndexReader since this can easily
> tie up resources (eg open file descriptors) for too long.
>
> Mike
>
>
> Michael Stoppelman wrote:
>
> If another thread is executing a query with the handle to one of
>> readers[i]
>> you're going to kill it since the IndexReader is now closed.
>> Just don't call the IndexReader#close() method. If nothing is pointing at
>> the readers they should be garbage collected. Also, you might
>> want to warm up your new IndexSearcher before you switch to it, meaning
>> run
>> a few queries on it before you swap the old one out.
>>
>> M
>>
>>
>>
>> On Tue, Feb 24, 2009 at 12:48 PM, Amin Mohammed-Coleman <aminmc@gmail.com
>> >wrote:
>>
>> The reason for the indexreader.reopen is because I have a webapp which
>>> enables users to upload files and then search for the documents. If I
>>> don't
>>> reopen i'm concerned that the facet hit counter won't be updated.
>>>
>>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <aminmc@gmail.com
>>>
>>>> wrote:
>>>>
>>>
>>> Hi
>>>> I have been able to get the code working for my scenario, however I have
>>>>
>>> a
>>>
>>>> question and I was wondering if I could get some help. I have a list of
>>>> IndexSearchers which are used in a MultiSearcher class. I use the
>>>> indexsearchers to get each indexreader and put them into a
>>>>
>>> MultiIndexReader.
>>>
>>>>
>>>> IndexReader[] readers = new IndexReader[searchables.length];
>>>>
>>>> for (int i =0 ; i < searchables.length;i++) {
>>>>
>>>> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>>>>
>>>> readers[i] = indexSearcher.getIndexReader();
>>>>
>>>> IndexReader newReader = readers[i].reopen();
>>>>
>>>> if (newReader != readers[i]) {
>>>>
>>>> readers[i].close();
>>>>
>>>> }
>>>>
>>>> readers[i] = newReader;
>>>>
>>>>
>>>>
>>>> }
>>>>
>>>> multiReader = new MultiReader(readers);
>>>>
>>>> OpenBitSetFacetHitCounter facetHitCounter =
>>>>
>>> newOpenBitSetFacetHitCounter();
>>>
>>>>
>>>> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>>>>
>>>>
>>>> I then use the indexseacher to do the facet stuff. I end the code with
>>>> closing the multireader. This is causing problems in another method
>>>>
>>> where I
>>>
>>>> do some other search as the indexreaders are closed. Is it ok to not
>>>>
>>> close
>>>
>>>> the multiindexreader or should I do some additional checks in the other
>>>> method to see if the indexreader is closed?
>>>>
>>>>
>>>>
>>>> Cheers
>>>>
>>>>
>>>> P.S. Hope that made sense...!
>>>>
>>>>
>>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <
>>>> aminmc@gmail.com
>>>> wrote:
>>>>
>>>> Hi
>>>>>
>>>>> Thanks just what I needed!
>>>>>
>>>>> Cheers
>>>>> Amin
>>>>>
>>>>>
>>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <ma...@gmail.com>
>>>>>
>>>> wrote:
>>>
>>>>
>>>>> Hi Amin:
>>>>>
>>>>>> Please take a look a this blog post:
>>>>>>
>>>>>>
>>>>>>
>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>
>>>> Best regards, Marcelo.
>>>>>>
>>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>>>>>>
>>>>> aminmc@gmail.com>
>>>
>>>> wrote:
>>>>>>
>>>>>> Hi
>>>>>>>
>>>>>>> Sorry to re send this email but I was wondering if I could get some
>>>>>>> advice
>>>>>>> on this.
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>> Amin
>>>>>>>
>>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <am...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>>>
>>>>>>>> I am looking at building a faceted search using Lucene. I know that
>>>>>>>> Solr
>>>>>>>> comes with this built in, however I would like to try this by myself
>>>>>>>> (something to add to my CV!). I have been looking around and I
>>>>>>>> found
>>>>>>>> that
>>>>>>>> you can use the IndexReader and use TermVectors. This looks ok but
>>>>>>>>
>>>>>>> I'm
>>>
>>>> not
>>>>>>>> sure how to filter the results so that a particular user can only
>>>>>>>> see
>>>>>>>>
>>>>>>> a
>>>
>>>> subset of results. The next option I was looking at was something
>>>>>>>>
>>>>>>> like
>>>
>>>>
>>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>>
>>>>>>>> The only problem here is that I have to provide the brand type each
>>>>>>>> time a
>>>>>>>> new brand is created. Again I'm not sure how I can filter the
>>>>>>>>
>>>>>>> results
>>>
>>>> here.
>>>>>>>> It may be that I'm using the wrong api methods to do this.
>>>>>>>>
>>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>>
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>> Amin
>>>>>>>>
>>>>>>>> P.S. I am basically trying to do something that displays the
>>>>>>>>
>>>>>>> following
>>>
>>>>
>>>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Marcelo F. Ochoa
>>>>>> http://marceloochoa.blogspot.com/
>>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>>> ______________
>>>>>> Want to integrate Lucene and Oracle?
>>>>>>
>>>>>>
>>>>>>
>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>
>>>> Is Oracle 11g REST ready?
>>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Faceted Search using Lucene
Posted by Michael McCandless <lu...@mikemccandless.com>.
Actually, it's best to use IndexReader.incRef/decRef to track the
IndexReader.
You should not rely on GC to close your IndexReader since this can
easily tie up resources (eg open file descriptors) for too long.
Mike
Michael Stoppelman wrote:
> If another thread is executing a query with the handle to one of
> readers[i]
> you're going to kill it since the IndexReader is now closed.
> Just don't call the IndexReader#close() method. If nothing is
> pointing at
> the readers they should be garbage collected. Also, you might
> want to warm up your new IndexSearcher before you switch to it,
> meaning run
> a few queries on it before you swap the old one out.
>
> M
>
>
>
> On Tue, Feb 24, 2009 at 12:48 PM, Amin Mohammed-Coleman <aminmc@gmail.com
> >wrote:
>
>> The reason for the indexreader.reopen is because I have a webapp
>> which
>> enables users to upload files and then search for the documents.
>> If I
>> don't
>> reopen i'm concerned that the facet hit counter won't be updated.
>>
>> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <aminmc@gmail.com
>>> wrote:
>>
>>> Hi
>>> I have been able to get the code working for my scenario, however
>>> I have
>> a
>>> question and I was wondering if I could get some help. I have a
>>> list of
>>> IndexSearchers which are used in a MultiSearcher class. I use the
>>> indexsearchers to get each indexreader and put them into a
>> MultiIndexReader.
>>>
>>> IndexReader[] readers = new IndexReader[searchables.length];
>>>
>>> for (int i =0 ; i < searchables.length;i++) {
>>>
>>> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>>>
>>> readers[i] = indexSearcher.getIndexReader();
>>>
>>> IndexReader newReader = readers[i].reopen();
>>>
>>> if (newReader != readers[i]) {
>>>
>>> readers[i].close();
>>>
>>> }
>>>
>>> readers[i] = newReader;
>>>
>>>
>>>
>>> }
>>>
>>> multiReader = new MultiReader(readers);
>>>
>>> OpenBitSetFacetHitCounter facetHitCounter =
>> newOpenBitSetFacetHitCounter();
>>>
>>> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>>>
>>>
>>> I then use the indexseacher to do the facet stuff. I end the code
>>> with
>>> closing the multireader. This is causing problems in another method
>> where I
>>> do some other search as the indexreaders are closed. Is it ok to
>>> not
>> close
>>> the multiindexreader or should I do some additional checks in the
>>> other
>>> method to see if the indexreader is closed?
>>>
>>>
>>>
>>> Cheers
>>>
>>>
>>> P.S. Hope that made sense...!
>>>
>>>
>>> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <aminmc@gmail.com
>>> wrote:
>>>
>>>> Hi
>>>>
>>>> Thanks just what I needed!
>>>>
>>>> Cheers
>>>> Amin
>>>>
>>>>
>>>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <ma...@gmail.com>
>> wrote:
>>>>
>>>> Hi Amin:
>>>>> Please take a look a this blog post:
>>>>>
>>>>>
>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>>>> Best regards, Marcelo.
>>>>>
>>>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
>> aminmc@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> Sorry to re send this email but I was wondering if I could get
>>>>>> some
>>>>>> advice
>>>>>> on this.
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> Amin
>>>>>>
>>>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman
>>>>>> <am...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi
>>>>>>>
>>>>>>> I am looking at building a faceted search using Lucene. I
>>>>>>> know that
>>>>>>> Solr
>>>>>>> comes with this built in, however I would like to try this by
>>>>>>> myself
>>>>>>> (something to add to my CV!). I have been looking around and
>>>>>>> I found
>>>>>>> that
>>>>>>> you can use the IndexReader and use TermVectors. This looks
>>>>>>> ok but
>> I'm
>>>>>>> not
>>>>>>> sure how to filter the results so that a particular user can
>>>>>>> only see
>> a
>>>>>>> subset of results. The next option I was looking at was
>>>>>>> something
>> like
>>>>>>>
>>>>>>> Term term1 = new Term("brand", "ford");
>>>>>>> Term term2 = new Term("brand", "vw");
>>>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>>>
>>>>>>> The only problem here is that I have to provide the brand type
>>>>>>> each
>>>>>>> time a
>>>>>>> new brand is created. Again I'm not sure how I can filter the
>> results
>>>>>>> here.
>>>>>>> It may be that I'm using the wrong api methods to do this.
>>>>>>>
>>>>>>> I would be grateful if I could get some advice on this.
>>>>>>>
>>>>>>>
>>>>>>> Cheers
>>>>>>> Amin
>>>>>>>
>>>>>>> P.S. I am basically trying to do something that displays the
>> following
>>>>>>>
>>>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Marcelo F. Ochoa
>>>>> http://marceloochoa.blogspot.com/
>>>>> http://marcelo.ochoa.googlepages.com/home
>>>>> ______________
>>>>> Want to integrate Lucene and Oracle?
>>>>>
>>>>>
>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>>>> Is Oracle 11g REST ready?
>>>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Faceted Search using Lucene
Posted by Michael Stoppelman <st...@gmail.com>.
If another thread is executing a query with the handle to one of readers[i]
you're going to kill it since the IndexReader is now closed.
Just don't call the IndexReader#close() method. If nothing is pointing at
the readers they should be garbage collected. Also, you might
want to warm up your new IndexSearcher before you switch to it, meaning run
a few queries on it before you swap the old one out.
M
On Tue, Feb 24, 2009 at 12:48 PM, Amin Mohammed-Coleman <am...@gmail.com>wrote:
> The reason for the indexreader.reopen is because I have a webapp which
> enables users to upload files and then search for the documents. If I
> don't
> reopen i'm concerned that the facet hit counter won't be updated.
>
> On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <aminmc@gmail.com
> >wrote:
>
> > Hi
> > I have been able to get the code working for my scenario, however I have
> a
> > question and I was wondering if I could get some help. I have a list of
> > IndexSearchers which are used in a MultiSearcher class. I use the
> > indexsearchers to get each indexreader and put them into a
> MultiIndexReader.
> >
> > IndexReader[] readers = new IndexReader[searchables.length];
> >
> > for (int i =0 ; i < searchables.length;i++) {
> >
> > IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
> >
> > readers[i] = indexSearcher.getIndexReader();
> >
> > IndexReader newReader = readers[i].reopen();
> >
> > if (newReader != readers[i]) {
> >
> > readers[i].close();
> >
> > }
> >
> > readers[i] = newReader;
> >
> >
> >
> > }
> >
> > multiReader = new MultiReader(readers);
> >
> > OpenBitSetFacetHitCounter facetHitCounter =
> newOpenBitSetFacetHitCounter();
> >
> > IndexSearcher indexSearcher = new IndexSearcher(multiReader);
> >
> >
> > I then use the indexseacher to do the facet stuff. I end the code with
> > closing the multireader. This is causing problems in another method
> where I
> > do some other search as the indexreaders are closed. Is it ok to not
> close
> > the multiindexreader or should I do some additional checks in the other
> > method to see if the indexreader is closed?
> >
> >
> >
> > Cheers
> >
> >
> > P.S. Hope that made sense...!
> >
> >
> > On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <aminmc@gmail.com
> >wrote:
> >
> >> Hi
> >>
> >> Thanks just what I needed!
> >>
> >> Cheers
> >> Amin
> >>
> >>
> >> On 22 Feb 2009, at 16:11, Marcelo Ochoa <ma...@gmail.com>
> wrote:
> >>
> >> Hi Amin:
> >>> Please take a look a this blog post:
> >>>
> >>>
> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
> >>> Best regards, Marcelo.
> >>>
> >>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <
> aminmc@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi
> >>>>
> >>>> Sorry to re send this email but I was wondering if I could get some
> >>>> advice
> >>>> on this.
> >>>>
> >>>> Cheers
> >>>>
> >>>> Amin
> >>>>
> >>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <am...@gmail.com>
> >>>> wrote:
> >>>>
> >>>> Hi
> >>>>>
> >>>>> I am looking at building a faceted search using Lucene. I know that
> >>>>> Solr
> >>>>> comes with this built in, however I would like to try this by myself
> >>>>> (something to add to my CV!). I have been looking around and I found
> >>>>> that
> >>>>> you can use the IndexReader and use TermVectors. This looks ok but
> I'm
> >>>>> not
> >>>>> sure how to filter the results so that a particular user can only see
> a
> >>>>> subset of results. The next option I was looking at was something
> like
> >>>>>
> >>>>> Term term1 = new Term("brand", "ford");
> >>>>> Term term2 = new Term("brand", "vw");
> >>>>> Term[] termsArray = new Term[] { term1, term2 };un
> >>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
> >>>>>
> >>>>> The only problem here is that I have to provide the brand type each
> >>>>> time a
> >>>>> new brand is created. Again I'm not sure how I can filter the
> results
> >>>>> here.
> >>>>> It may be that I'm using the wrong api methods to do this.
> >>>>>
> >>>>> I would be grateful if I could get some advice on this.
> >>>>>
> >>>>>
> >>>>> Cheers
> >>>>> Amin
> >>>>>
> >>>>> P.S. I am basically trying to do something that displays the
> following
> >>>>>
> >>>>> Personal Contact (23) Business Contact (45) and so on..
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Marcelo F. Ochoa
> >>> http://marceloochoa.blogspot.com/
> >>> http://marcelo.ochoa.googlepages.com/home
> >>> ______________
> >>> Want to integrate Lucene and Oracle?
> >>>
> >>>
> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
> >>> Is Oracle 11g REST ready?
> >>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>
> >>>
> >
>
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
The reason for the indexreader.reopen is because I have a webapp which
enables users to upload files and then search for the documents. If I don't
reopen i'm concerned that the facet hit counter won't be updated.
On Tue, Feb 24, 2009 at 8:32 PM, Amin Mohammed-Coleman <am...@gmail.com>wrote:
> Hi
> I have been able to get the code working for my scenario, however I have a
> question and I was wondering if I could get some help. I have a list of
> IndexSearchers which are used in a MultiSearcher class. I use the
> indexsearchers to get each indexreader and put them into a MultiIndexReader.
>
> IndexReader[] readers = new IndexReader[searchables.length];
>
> for (int i =0 ; i < searchables.length;i++) {
>
> IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
>
> readers[i] = indexSearcher.getIndexReader();
>
> IndexReader newReader = readers[i].reopen();
>
> if (newReader != readers[i]) {
>
> readers[i].close();
>
> }
>
> readers[i] = newReader;
>
>
>
> }
>
> multiReader = new MultiReader(readers);
>
> OpenBitSetFacetHitCounter facetHitCounter = newOpenBitSetFacetHitCounter();
>
> IndexSearcher indexSearcher = new IndexSearcher(multiReader);
>
>
> I then use the indexseacher to do the facet stuff. I end the code with
> closing the multireader. This is causing problems in another method where I
> do some other search as the indexreaders are closed. Is it ok to not close
> the multiindexreader or should I do some additional checks in the other
> method to see if the indexreader is closed?
>
>
>
> Cheers
>
>
> P.S. Hope that made sense...!
>
>
> On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <am...@gmail.com>wrote:
>
>> Hi
>>
>> Thanks just what I needed!
>>
>> Cheers
>> Amin
>>
>>
>> On 22 Feb 2009, at 16:11, Marcelo Ochoa <ma...@gmail.com> wrote:
>>
>> Hi Amin:
>>> Please take a look a this blog post:
>>>
>>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>>> Best regards, Marcelo.
>>>
>>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <am...@gmail.com>
>>> wrote:
>>>
>>>> Hi
>>>>
>>>> Sorry to re send this email but I was wondering if I could get some
>>>> advice
>>>> on this.
>>>>
>>>> Cheers
>>>>
>>>> Amin
>>>>
>>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <am...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi
>>>>>
>>>>> I am looking at building a faceted search using Lucene. I know that
>>>>> Solr
>>>>> comes with this built in, however I would like to try this by myself
>>>>> (something to add to my CV!). I have been looking around and I found
>>>>> that
>>>>> you can use the IndexReader and use TermVectors. This looks ok but I'm
>>>>> not
>>>>> sure how to filter the results so that a particular user can only see a
>>>>> subset of results. The next option I was looking at was something like
>>>>>
>>>>> Term term1 = new Term("brand", "ford");
>>>>> Term term2 = new Term("brand", "vw");
>>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>>
>>>>> The only problem here is that I have to provide the brand type each
>>>>> time a
>>>>> new brand is created. Again I'm not sure how I can filter the results
>>>>> here.
>>>>> It may be that I'm using the wrong api methods to do this.
>>>>>
>>>>> I would be grateful if I could get some advice on this.
>>>>>
>>>>>
>>>>> Cheers
>>>>> Amin
>>>>>
>>>>> P.S. I am basically trying to do something that displays the following
>>>>>
>>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Marcelo F. Ochoa
>>> http://marceloochoa.blogspot.com/
>>> http://marcelo.ochoa.googlepages.com/home
>>> ______________
>>> Want to integrate Lucene and Oracle?
>>>
>>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>>> Is Oracle 11g REST ready?
>>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
Hi
I have been able to get the code working for my scenario, however I have a
question and I was wondering if I could get some help. I have a list of
IndexSearchers which are used in a MultiSearcher class. I use the
indexsearchers to get each indexreader and put them into a MultiIndexReader.
IndexReader[] readers = new IndexReader[searchables.length];
for (int i =0 ; i < searchables.length;i++) {
IndexSearcher indexSearcher = (IndexSearcher)searchables[i];
readers[i] = indexSearcher.getIndexReader();
IndexReader newReader = readers[i].reopen();
if (newReader != readers[i]) {
readers[i].close();
}
readers[i] = newReader;
}
multiReader = new MultiReader(readers);
OpenBitSetFacetHitCounter facetHitCounter = new OpenBitSetFacetHitCounter();
IndexSearcher indexSearcher = new IndexSearcher(multiReader);
I then use the indexseacher to do the facet stuff. I end the code with
closing the multireader. This is causing problems in another method where I
do some other search as the indexreaders are closed. Is it ok to not close
the multiindexreader or should I do some additional checks in the other
method to see if the indexreader is closed?
Cheers
P.S. Hope that made sense...!
On Mon, Feb 23, 2009 at 7:20 AM, Amin Mohammed-Coleman <am...@gmail.com>wrote:
> Hi
>
> Thanks just what I needed!
>
> Cheers
> Amin
>
>
> On 22 Feb 2009, at 16:11, Marcelo Ochoa <ma...@gmail.com> wrote:
>
> Hi Amin:
>> Please take a look a this blog post:
>> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
>> Best regards, Marcelo.
>>
>> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <am...@gmail.com>
>> wrote:
>>
>>> Hi
>>>
>>> Sorry to re send this email but I was wondering if I could get some
>>> advice
>>> on this.
>>>
>>> Cheers
>>>
>>> Amin
>>>
>>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <am...@gmail.com>
>>> wrote:
>>>
>>> Hi
>>>>
>>>> I am looking at building a faceted search using Lucene. I know that
>>>> Solr
>>>> comes with this built in, however I would like to try this by myself
>>>> (something to add to my CV!). I have been looking around and I found
>>>> that
>>>> you can use the IndexReader and use TermVectors. This looks ok but I'm
>>>> not
>>>> sure how to filter the results so that a particular user can only see a
>>>> subset of results. The next option I was looking at was something like
>>>>
>>>> Term term1 = new Term("brand", "ford");
>>>> Term term2 = new Term("brand", "vw");
>>>> Term[] termsArray = new Term[] { term1, term2 };un
>>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>>
>>>> The only problem here is that I have to provide the brand type each time
>>>> a
>>>> new brand is created. Again I'm not sure how I can filter the results
>>>> here.
>>>> It may be that I'm using the wrong api methods to do this.
>>>>
>>>> I would be grateful if I could get some advice on this.
>>>>
>>>>
>>>> Cheers
>>>> Amin
>>>>
>>>> P.S. I am basically trying to do something that displays the following
>>>>
>>>> Personal Contact (23) Business Contact (45) and so on..
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Marcelo F. Ochoa
>> http://marceloochoa.blogspot.com/
>> http://marcelo.ochoa.googlepages.com/home
>> ______________
>> Want to integrate Lucene and Oracle?
>>
>> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
>> Is Oracle 11g REST ready?
>> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
Hi
Thanks just what I needed!
Cheers
Amin
On 22 Feb 2009, at 16:11, Marcelo Ochoa <ma...@gmail.com> wrote:
> Hi Amin:
> Please take a look a this blog post:
> http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
> Best regards, Marcelo.
>
> On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <aminmc@gmail.com
> > wrote:
>> Hi
>>
>> Sorry to re send this email but I was wondering if I could get some
>> advice
>> on this.
>>
>> Cheers
>>
>> Amin
>>
>> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <am...@gmail.com>
>> wrote:
>>
>>> Hi
>>>
>>> I am looking at building a faceted search using Lucene. I know
>>> that Solr
>>> comes with this built in, however I would like to try this by myself
>>> (something to add to my CV!). I have been looking around and I
>>> found that
>>> you can use the IndexReader and use TermVectors. This looks ok
>>> but I'm not
>>> sure how to filter the results so that a particular user can only
>>> see a
>>> subset of results. The next option I was looking at was something
>>> like
>>>
>>> Term term1 = new Term("brand", "ford");
>>> Term term2 = new Term("brand", "vw");
>>> Term[] termsArray = new Term[] { term1, term2 };un
>>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>>
>>> The only problem here is that I have to provide the brand type
>>> each time a
>>> new brand is created. Again I'm not sure how I can filter the
>>> results here.
>>> It may be that I'm using the wrong api methods to do this.
>>>
>>> I would be grateful if I could get some advice on this.
>>>
>>>
>>> Cheers
>>> Amin
>>>
>>> P.S. I am basically trying to do something that displays the
>>> following
>>>
>>> Personal Contact (23) Business Contact (45) and so on..
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
>
>
> --
> Marcelo F. Ochoa
> http://marceloochoa.blogspot.com/
> http://marcelo.ochoa.googlepages.com/home
> ______________
> Want to integrate Lucene and Oracle?
> http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
> Is Oracle 11g REST ready?
> http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Faceted Search using Lucene
Posted by Marcelo Ochoa <ma...@gmail.com>.
Hi Amin:
Please take a look a this blog post:
http://sujitpal.blogspot.com/2007/04/lucene-search-within-search-with.html
Best regards, Marcelo.
On Sun, Feb 22, 2009 at 1:18 PM, Amin Mohammed-Coleman <am...@gmail.com> wrote:
> Hi
>
> Sorry to re send this email but I was wondering if I could get some advice
> on this.
>
> Cheers
>
> Amin
>
> On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <am...@gmail.com> wrote:
>
>> Hi
>>
>> I am looking at building a faceted search using Lucene. I know that Solr
>> comes with this built in, however I would like to try this by myself
>> (something to add to my CV!). I have been looking around and I found that
>> you can use the IndexReader and use TermVectors. This looks ok but I'm not
>> sure how to filter the results so that a particular user can only see a
>> subset of results. The next option I was looking at was something like
>>
>> Term term1 = new Term("brand", "ford");
>> Term term2 = new Term("brand", "vw");
>> Term[] termsArray = new Term[] { term1, term2 };un
>> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>>
>> The only problem here is that I have to provide the brand type each time a
>> new brand is created. Again I'm not sure how I can filter the results here.
>> It may be that I'm using the wrong api methods to do this.
>>
>> I would be grateful if I could get some advice on this.
>>
>>
>> Cheers
>> Amin
>>
>> P.S. I am basically trying to do something that displays the following
>>
>> Personal Contact (23) Business Contact (45) and so on..
>>
>>
>>
>>
>>
>>
>
--
Marcelo F. Ochoa
http://marceloochoa.blogspot.com/
http://marcelo.ochoa.googlepages.com/home
______________
Want to integrate Lucene and Oracle?
http://marceloochoa.blogspot.com/2007/09/running-lucene-inside-your-oracle-jvm.html
Is Oracle 11g REST ready?
http://marceloochoa.blogspot.com/2008/02/is-oracle-11g-rest-ready.html
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Faceted Search using Lucene
Posted by Amin Mohammed-Coleman <am...@gmail.com>.
Hi
Sorry to re send this email but I was wondering if I could get some
advice on this.
Cheers
Amin
On 16 Feb 2009, at 20:37, Amin Mohammed-Coleman <am...@gmail.com>
wrote:
> Hi
>
> I am looking at building a faceted search using Lucene. I know that
> Solr comes with this built in, however I would like to try this by
> myself (something to add to my CV!). I have been looking around and
> I found that you can use the IndexReader and use TermVectors. This
> looks ok but I'm not sure how to filter the results so that a
> particular user can only see a subset of results. The next option I
> was looking at was something like
>
> Term term1 = new Term("brand", "ford");
> Term term2 = new Term("brand", "vw");
> Term[] termsArray = new Term[] { term1, term2 };un
> int[] docFreqs = indexSearcher.docFreqs(termsArray);
>
> The only problem here is that I have to provide the brand type each
> time a new brand is created. Again I'm not sure how I can filter
> the results here. It may be that I'm using the wrong api methods to
> do this.
>
> I would be grateful if I could get some advice on this.
>
>
> Cheers
> Amin
>
> P.S. I am basically trying to do something that displays the
> following
>
> Personal Contact (23) Business Contact (45) and so on..
>
>
>
>
>
>