Index merging and optimizing

[sp...@gmx.eu]

Hi,

are there any ready to use tools out there which I can use for merging and
optimzing?

I have seen that Luke can optimize, but not merge?

Or do I have to write my own utility?

Thank you


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

RE: Index merging and optimizing

[sp...@gmx.eu]

> But it also seems that the parallel/not parallel decision is
> something you control on the back end, so I'm not sure the user
> is involved in the merge question at all. In other words, you could
> easily split the indexing task up amongst several machines and/or
> processes and combine all the results after all the sub-indexes
> were build, thus making your question basically irrelevant.

I'm just writing the tool. The customer (IT staff, not end user) uses it.
Now I have to find out the best stategy which allows fast indexing and
searching.

> But you still haven't explained what the user is getting from all
> this flexibility. 

I'm not asking about flexibility, only about indexing performance.
I know that I have to index in parallel (initial data is about 150 million
pages with more than 1 TByte).
Now I'm thinking about what I have to take into consideration when building
the merge tool.
Then I saw the code of IndexMergeTool I was astonished about the hardcoded
use of SimpleAnalyzer and why one uses an anylzer at all. Therefore my
questions.

> I have a hard time understanding the use-case
> you're trying to support. If you're trying to build a generic 
> front-end
> to allow parameterized Lucene index building, have you looked at
> SOLR, which uses XML configuration files to drive the indexing
> and searching process? (which I haven't used, but I'm tracking
> the user's group list.....).

No the frontend just provides a textbox where the user can type in a text
like "foo*".

The query is then executed against some un_tokenized fields (fixed
parameters for the current user like company etc.) and against this single
tokenized field.
I want to filter on these fixed fields too, to reduce the number of hits for
the fulltext query. I think this is the right approach to achieve a better
search performance. Isn't it?

Thank you!


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Index merging and optimizing

[Erick Erickson <er...@gmail.com>]

OK, I think I'm getting a better handle here. I can't imagine how
it would work to combine indexes that use *different* analyzers
on the *same* field. Regardless of what Lucene did, you
simply could NOT explain this to a user. To take a simple example,
index part of your data for field1 with KeywordAnalyzer and
part with WhiteSpaceAnalyzer. Now you index the same
data in two different documents, i.e. field1 has
"some data" in two docs.

Now one doc has the tokens "some" and "data", and the other
has "some data". Depending on the analyzer you use at
query time, searching for "some" would return one doc. Searching
for "some data" would return the other doc. Or would return the
first doc depending upon the query-time analyzer. And no
matter what analyzer you used, searching for "some" would never
return doc 2.

Even assuming Lucene allows this
how in the world could you explain what the "correct" results
were to a user? Even assuming that Lucene does something
reasonable, it'll still be wrong enough of the time to make the
system unusable.

And we're not even into stopwords, cases, different languages,
accent folding.

Not to mention how in the world you'd be able to explain what
query analyzer to use <G>....

That said, you *can* use *different* analyzers on *different* fields
in the same index. See PerFieldAnalyzerWrapper. And there's
no restriction that all documents have the *same* fields. So as
long as all your field names were disjoint, it could work. *But* I
have no real idea whether this is supported, so you'll have to
experiment if you still think it's a good thing.

But it also seems that the parallel/not parallel decision is
something you control on the back end, so I'm not sure the user
is involved in the merge question at all. In other words, you could
easily split the indexing task up amongst several machines and/or
processes and combine all the results after all the sub-indexes
were build, thus making your question basically irrelevant.

But you still haven't explained what the user is getting from all
this flexibility. I have a hard time understanding the use-case
you're trying to support. If you're trying to build a generic front-end
to allow parameterized Lucene index building, have you looked at
SOLR, which uses XML configuration files to drive the indexing
and searching process? (which I haven't used, but I'm tracking
the user's group list.....).

Best
Erick


On Jan 14, 2008 10:35 AM, <sp...@gmx.eu> wrote:

> > Then why would you want to combine them?
> >
> > I really think you need to explain what you're trying to accomplish
> > rather then obsess on the details.
>
> I have to create indexes in parallel because the amount of data is very
> high.
> Then I want to merge them into bigger indexes an move them to the search
> server.
>
> (See therad "Max size of index (FSDirectory )" too.)
>
> My question now is, what will happen if one merges indexes which were
> created with different analyzer (the customer can confige the analyzer
> depending on the data which is indexed)?
>
> I think this will produce unpredicted results when searching.
> If so I have to document that only indexes created with the same analyzer
> are allowed to merge.
>
> Thank you
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

RE: Index merging and optimizing

[sp...@gmx.eu]

> Then why would you want to combine them?
> 
> I really think you need to explain what you're trying to accomplish
> rather then obsess on the details.

I have to create indexes in parallel because the amount of data is very
high.
Then I want to merge them into bigger indexes an move them to the search
server.

(See therad "Max size of index (FSDirectory )" too.)

My question now is, what will happen if one merges indexes which were
created with different analyzer (the customer can confige the analyzer
depending on the data which is indexed)?

I think this will produce unpredicted results when searching.
If so I have to document that only indexes created with the same analyzer
are allowed to merge.

Thank you


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Index merging and optimizing

[Erick Erickson <er...@gmail.com>]

Then why would you want to combine them?

I really think you need to explain what you're trying to accomplish
rather then obsess on the details.

Erick

On Jan 14, 2008 10:17 AM, <sp...@gmx.eu> wrote:

> > I admit I've never used IndexMergeTool, I've always used
> > IndexWriter.AddIndexex and then execute
> > IndexWriter.optimize().
> >
> > And I've seen no problems. That call takes no
> > analyzer.
>
> So you take the first index an add a remaining indexes via addIndexes?
> What happens if the indexes were created with different analyzers?
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

RE: Index merging and optimizing

[sp...@gmx.eu]

> I admit I've never used IndexMergeTool, I've always used
> IndexWriter.AddIndexex and then execute
> IndexWriter.optimize().
> 
> And I've seen no problems. That call takes no
> analyzer.

So you take the first index an add a remaining indexes via addIndexes?
What happens if the indexes were created with different analyzers?


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Index merging and optimizing

[Erick Erickson <er...@gmail.com>]

I admit I've never used IndexMergeTool, I've always used
IndexWriter.AddIndexex and then execute
IndexWriter.optimize().

And I've seen no problems. That call takes no
analyzer.

Erick

On Jan 14, 2008 6:12 AM, <sp...@gmx.eu> wrote:

> > See org.apache.lucene.misc.IndexMergeTool
>
> Thank you.
>
> But this uses a hardcoded analyzer and deprecated API-Calls.
>
> How does the used analyzer effect the merge process?
> Is everything reindexed with this new analyzer again? Does this make
> sense?
> What if the sources indexes had other analyzers used?
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

RE: Index merging and optimizing

[sp...@gmx.eu]

> See org.apache.lucene.misc.IndexMergeTool

Thank you. 

But this uses a hardcoded analyzer and deprecated API-Calls.

How does the used analyzer effect the merge process?
Is everything reindexed with this new analyzer again? Does this make sense?
What if the sources indexes had other analyzers used?



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Index merging and optimizing

[Erick Erickson <er...@gmail.com>]

See IndexWriter.AddIndexes.
See org.apache.lucene.misc.IndexMergeTool

Erick

On Jan 13, 2008 12:10 PM, <sp...@gmx.eu> wrote:

> Hi,
>
> are there any ready to use tools out there which I can use for merging and
> optimzing?
>
> I have seen that Luke can optimize, but not merge?
>
> Or do I have to write my own utility?
>
> Thank you
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>