You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by sp...@gmx.eu on 2008/01/09 22:39:32 UTC
Design questions
Hi,
I have to index (tokenized) documents which may have very much pages, up to 10.000.
I also have to know on which pages the search phrase occurs.
I have to update some stored index fields for my document.
The content is never changed.
Thus I think I have to add one lucene document with the index fields and one lucene document per page.
Mapping
=======
MyDocument
-ID
-Field 1-N
-Page 1-N
Lucene
-Lucene Document with ID, page number 0 and Field1 - N (stored fields)
-Lucene Document 1 with ID, page number 1 and tokenized content of Page 1
...
-Lucene Document N with ID, page number N and tokenized content of Page N
Delete of MyDocument -> IndexWriter#deleteDocuments(Term:ID=foo)
Update of stored index fields -> IndexWriter#updateDocument(Term: ID=foo, page number = 0)
Search with index and content.
Step 1: Search on stored index fields -> List of IDs
Step 2: Search on ID field (list from above OR'ed together) and content -> List of IDs and page numbers
Does this work?
What drawbacks has this approch?
Is there another way to achieve what I want?
Thank you.
P.S.
There are millions of documents with a page range from 1 to 10.000.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Design questions
Posted by sp...@gmx.eu.
> Why not just use $$$$$$$$?
Because nearly every analyzer removes it (SimpleAnalyzer, German, Russian,
French...)
Just tested it with luke in the search dialog.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Design questions
Posted by sp...@gmx.eu.
> You need to watch both the positionincrementgap
> (which, as I remember, gets added for each new field of the
> same name you add to the document). Make it 0 rather than
> whatever it is currently. You may have to create a new analyzer
> by subclassing your favorite analyzer and overriding the
> getPositionIncrementGap (?)
Well. I'm using GermanAnalyzer and this does not override
getPositionIncrementGap in Analyzer.
And in Analyzer getPositionIncrementGap returns 0.
> Also, I'm not sure whether the term increment (see
> get/setPositionIncrement)
> needs to be taken into account. See the SynonymAnalyzer in
> Lucene in Action.
I cannot find the source of SynonymAnalyzer.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Design questions
Posted by Erick Erickson <er...@gmail.com>.
You need to watch both the positionincrementgap
(which, as I remember, gets added for each new field of the
same name you add to the document). Make it 0 rather than
whatever it is currently. You may have to create a new analyzer
by subclassing your favorite analyzer and overriding the
getPositionIncrementGap (?)
Also, I'm not sure whether the term increment (see get/setPositionIncrement)
needs to be taken into account. See the SynonymAnalyzer in
Lucene in Action.
On Fri, Feb 15, 2008 at 8:37 AM, <sp...@gmx.eu> wrote:
> > > Document doc = new Document()
> > > for (int i = 0; i < pages.length; i++) {
> > > doc.add(new Field("text", pages[i], Field.Store.NO,
> > > Field.Index.TOKENIZED));
> > > doc.add(new Field("text", "$$", Field.Store.NO,
> > > Field.Index.UN_TOKENIZED));
> > > }
> >
> > UN_TOKENIZED. Nice idea!
> > I will check this out.
>
>
> Hm... when I try this, something strange happens with my offsets.
>
> When I use
> doc.add(new Field("text", pages[i] +
> "012345678901234567890123456789012345678901234567890123456789",
> Field.Store.NO, Field.Index.TOKENIZED))
> everything is fine. Offsets are as I expect.
>
> But when I use
> doc.add(new Field("text", pages[i], Field.Store.NO, Field.Index.TOKENIZED
> ))
> doc.add(new Field("text",
> "012345678901234567890123456789012345678901234567890123456789",
> Field.Store.NO, Field.Index.UN_TOKENIZED))
>
> the offsets of my terms are to high.
>
> What is the difference?
>
> Thank you.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Design questions
Posted by sp...@gmx.eu.
> > Document doc = new Document()
> > for (int i = 0; i < pages.length; i++) {
> > doc.add(new Field("text", pages[i], Field.Store.NO,
> > Field.Index.TOKENIZED));
> > doc.add(new Field("text", "$$", Field.Store.NO,
> > Field.Index.UN_TOKENIZED));
> > }
>
> UN_TOKENIZED. Nice idea!
> I will check this out.
Hm... when I try this, something strange happens with my offsets.
When I use
doc.add(new Field("text", pages[i] +
"012345678901234567890123456789012345678901234567890123456789",
Field.Store.NO, Field.Index.TOKENIZED))
everything is fine. Offsets are as I expect.
But when I use
doc.add(new Field("text", pages[i], Field.Store.NO, Field.Index.TOKENIZED))
doc.add(new Field("text",
"012345678901234567890123456789012345678901234567890123456789",
Field.Store.NO, Field.Index.UN_TOKENIZED))
the offsets of my terms are to high.
What is the difference?
Thank you.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Design questions
Posted by sp...@gmx.eu.
> Document doc = new Document()
> for (int i = 0; i < pages.length; i++) {
> doc.add(new Field("text", pages[i], Field.Store.NO,
> Field.Index.TOKENIZED));
> doc.add(new Field("text", "$$", Field.Store.NO,
> Field.Index.UN_TOKENIZED));
> }
UN_TOKENIZED. Nice idea!
I will check this out.
> 2) if your goal is just to be able to make sure you can query
> for phrases
> without crossing page boundaries, it's a lot simpler just to use are
> really big positionIncimentGap with your analyzer (and add
> each page as a
> seperate Field instance). boundary tokens like these are relaly only
> neccessary if you want more complex queries (like "find X and Y on
> the same page but not in the same sentence")
Hm. This is what Erik already recommended.
I had to store the field with TermVector.WITH_POSITIONS, right?
But I do not know the maximum number of terms per page and I do not know the
maximum number of pages.
I already had documents with more than 50.000 pages (A4) and documents with
1 page but 100 MB data.
How many terms can 100 MB have? Hm...
Since positions are stored as int I could have a maximum of 40.000 terms per
page (50.000 pages * 40.000 term -> nearly Integer.MAX_VALUE).
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Design questions
Posted by sp...@gmx.eu.
Well, it seems that this may be a solution for me too.
But I'm afraid that someone one day will change this string. And then my app
will not work anymore...
> -----Original Message-----
> From: Adrian Smith [mailto:adrian.m.smith@gmail.com]
> Sent: Freitag, 15. Februar 2008 13:02
> To: java-user@lucene.apache.org
> Subject: Re: Design questions
>
> Hi,
>
> I have a similar sitaution. I also considered using $. But
> for the sake of
> not running into (potential) problems with Tokenisers, I just
> defined a
> string in a config file which for sure is never going to
> occur in a document
> and will never be searched for, e.g.
>
> dfgjkjrkruigduhfkdgjrugr
>
> Cheers, Adrian
> --
> Java Software Developer
> http://www.databasesandlife.com/
>
>
>
> On 15/02/2008, Chris Hostetter <ho...@fucit.org> wrote:
> >
> >
> > I haven't really been following this thread that closely, but...
> >
> > : Why not just use $$$$$$$$? Check to insure that it makes
> >
> > : it through whatever analyzer you choose though. For instance,
> > : LetterTokenizer will remove it...
> >
> >
> > 1) i'm 99% sure you can do something like this...
> >
> > Document doc = new Document()
> > for (int i = 0; i < pages.length; i++) {
> > doc.add(new Field("text", pages[i], Field.Store.NO,
> > Field.Index.TOKENIZED));
> > doc.add(new Field("text", "$$", Field.Store.NO,
> > Field.Index.UN_TOKENIZED));
> > }
> >
> > ...and you'll get your magic token regardless of whether it
> would normally
> > make it through your analyzer. In fact: you want it to be
> something your
> > analyzer could never produce, even if it appears in the
> orriginal text, so
> > you don't get false boundaries (ie: if you use an Analzeer
> that lowercases
> > everything, then "A" makes a perfectly fine boundary token.
> >
> > 2) if your goal is just to be able to make sure you can
> query for phrases
> > without crossing page boundaries, it's a lot simpler just to use are
> > really big positionIncimentGap with your analyzer (and add
> each page as a
> > seperate Field instance). boundary tokens like these are
> relaly only
> > neccessary if you want more complex queries (like "find X and Y on
> > the same page but not in the same sentence")
> >
> >
> >
> >
> > -Hoss
> >
> >
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Design questions
Posted by Adrian Smith <ad...@gmail.com>.
Hi,
I have a similar sitaution. I also considered using $. But for the sake of
not running into (potential) problems with Tokenisers, I just defined a
string in a config file which for sure is never going to occur in a document
and will never be searched for, e.g.
dfgjkjrkruigduhfkdgjrugr
Cheers, Adrian
--
Java Software Developer
http://www.databasesandlife.com/
On 15/02/2008, Chris Hostetter <ho...@fucit.org> wrote:
>
>
> I haven't really been following this thread that closely, but...
>
> : Why not just use $$$$$$$$? Check to insure that it makes
>
> : it through whatever analyzer you choose though. For instance,
> : LetterTokenizer will remove it...
>
>
> 1) i'm 99% sure you can do something like this...
>
> Document doc = new Document()
> for (int i = 0; i < pages.length; i++) {
> doc.add(new Field("text", pages[i], Field.Store.NO,
> Field.Index.TOKENIZED));
> doc.add(new Field("text", "$$", Field.Store.NO,
> Field.Index.UN_TOKENIZED));
> }
>
> ...and you'll get your magic token regardless of whether it would normally
> make it through your analyzer. In fact: you want it to be something your
> analyzer could never produce, even if it appears in the orriginal text, so
> you don't get false boundaries (ie: if you use an Analzeer that lowercases
> everything, then "A" makes a perfectly fine boundary token.
>
> 2) if your goal is just to be able to make sure you can query for phrases
> without crossing page boundaries, it's a lot simpler just to use are
> really big positionIncimentGap with your analyzer (and add each page as a
> seperate Field instance). boundary tokens like these are relaly only
> neccessary if you want more complex queries (like "find X and Y on
> the same page but not in the same sentence")
>
>
>
>
> -Hoss
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Design questions
Posted by Chris Hostetter <ho...@fucit.org>.
I haven't really been following this thread that closely, but...
: Why not just use $$$$$$$$? Check to insure that it makes
: it through whatever analyzer you choose though. For instance,
: LetterTokenizer will remove it...
1) i'm 99% sure you can do something like this...
Document doc = new Document()
for (int i = 0; i < pages.length; i++) {
doc.add(new Field("text", pages[i], Field.Store.NO, Field.Index.TOKENIZED));
doc.add(new Field("text", "$$", Field.Store.NO, Field.Index.UN_TOKENIZED));
}
...and you'll get your magic token regardless of whether it would normally
make it through your analyzer. In fact: you want it to be something your
analyzer could never produce, even if it appears in the orriginal text, so
you don't get false boundaries (ie: if you use an Analzeer that lowercases
everything, then "A" makes a perfectly fine boundary token.
2) if your goal is just to be able to make sure you can query for phrases
without crossing page boundaries, it's a lot simpler just to use are
really big positionIncimentGap with your analyzer (and add each page as a
seperate Field instance). boundary tokens like these are relaly only
neccessary if you want more complex queries (like "find X and Y on
the same page but not in the same sentence")
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Design questions
Posted by Erick Erickson <er...@gmail.com>.
Why not just use $$$$$$$$? Check to insure that it makes
it through whatever analyzer you choose though. For instance,
LetterTokenizer will remove it...
Erick
On Thu, Feb 14, 2008 at 4:41 PM, <sp...@gmx.eu> wrote:
> > Rather than index one doc per page, you could index a special
> > token between pages. Say you index $$$$$$$$$ as the special
> > token.
>
> I have decided to use this version, but...
>
> What token can I use? It must be a token which gets never removed by an
> analyzer or altered in a way that it not unique in the resulting
> tokenstream.
>
> Is something like $0123456789$ the way to go?
>
> Thank you.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Design questions
Posted by sp...@gmx.eu.
> Rather than index one doc per page, you could index a special
> token between pages. Say you index $$$$$$$$$ as the special
> token.
I have decided to use this version, but...
What token can I use? It must be a token which gets never removed by an
analyzer or altered in a way that it not unique in the resulting
tokenstream.
Is something like $0123456789$ the way to go?
Thank you.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Design questions
Posted by sp...@gmx.eu.
OK, I will give this a try.
Now I have the problem that I do not know how to get the offsets (or
positions? What is the difference?) back from the searched document...
There is a IndexReader#termPositions (Term t) - but this returns the
positions for the whole index, not a single document.
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Donnerstag, 24. Januar 2008 20:56
> To: java-user@lucene.apache.org
> Subject: Re: Design questions
>
> I think you'll have to implement your own Analyzer and count.
> That is, every call to next() that returns a token will have to
> also increment some counter by 1.
>
> To use this, you must have some way of knowing when a page
> ends, and at that point you call your instance of your custom
> analyzer to see what the count is. Or your analyzer maintains
> the list and you can call for it after you've added all the pages.
>
> Analyzer.getPositionIncrementGap is called every time you
> call document.add("field".....
>
> So, you have something like this
> while (more pages for doc) {
> string pagedata = getPageText();
> doc.add("text", pagedata);
> }
>
> Under the covers, your custom analyzer adds the current offset
> (which you've kept track of) to, say, an ArrayList. And after the
> last page is added, you get this arraylist and add it to your
> document.
>
> Or, you could just do things twice. That is, send your text through
> a TokenStream, then call next() and count. Then send it all
> through doc.add().
>
> There are probably cleverer ways, but that should do for a start.
>
> Best
> Erick
>
> On Jan 24, 2008 2:33 PM, <sp...@gmx.eu> wrote:
>
> > > -----Original Message-----
> > > From: Erick Erickson [mailto:erickerickson@gmail.com]
> > > Sent: Freitag, 11. Januar 2008 16:16
> > > To: java-user@lucene.apache.org
> > > Subject: Re: Design questions
> >
> > > But you could also vary this scheme by simply storing in
> your document
> > > the offsets for the beginning of each page.
> >
> > Well, this is the best for my app I think, but...
> >
> > How do I find out these offsets?
> >
> > I'm adding the content field with:
> >
> > IndexWriter#add(new Field("content", myContentReader));
> >
> > I have no clue how find out the offsets in this reader.
> Must be something
> > with an analyzer and a TokenStream?
> >
> > Thank you
> >
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Design questions
Posted by sp...@gmx.eu.
> Or, you could just do things twice. That is, send your text through
> a TokenStream, then call next() and count. Then send it all
> through doc.add().
Hm.
This means read the content twice, doesn't matter using an own analyzer oder
overriding/wrapping the main analyzer.
Is there anywhere a hook where I can grap the last token when I call
Document#add?
Thank you.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Design questions
Posted by Erick Erickson <er...@gmail.com>.
I think you'll have to implement your own Analyzer and count.
That is, every call to next() that returns a token will have to
also increment some counter by 1.
To use this, you must have some way of knowing when a page
ends, and at that point you call your instance of your custom
analyzer to see what the count is. Or your analyzer maintains
the list and you can call for it after you've added all the pages.
Analyzer.getPositionIncrementGap is called every time you
call document.add("field".....
So, you have something like this
while (more pages for doc) {
string pagedata = getPageText();
doc.add("text", pagedata);
}
Under the covers, your custom analyzer adds the current offset
(which you've kept track of) to, say, an ArrayList. And after the
last page is added, you get this arraylist and add it to your
document.
Or, you could just do things twice. That is, send your text through
a TokenStream, then call next() and count. Then send it all
through doc.add().
There are probably cleverer ways, but that should do for a start.
Best
Erick
On Jan 24, 2008 2:33 PM, <sp...@gmx.eu> wrote:
> > -----Original Message-----
> > From: Erick Erickson [mailto:erickerickson@gmail.com]
> > Sent: Freitag, 11. Januar 2008 16:16
> > To: java-user@lucene.apache.org
> > Subject: Re: Design questions
>
> > But you could also vary this scheme by simply storing in your document
> > the offsets for the beginning of each page.
>
> Well, this is the best for my app I think, but...
>
> How do I find out these offsets?
>
> I'm adding the content field with:
>
> IndexWriter#add(new Field("content", myContentReader));
>
> I have no clue how find out the offsets in this reader. Must be something
> with an analyzer and a TokenStream?
>
> Thank you
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Design questions
Posted by sp...@gmx.eu.
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Freitag, 11. Januar 2008 16:16
> To: java-user@lucene.apache.org
> Subject: Re: Design questions
> But you could also vary this scheme by simply storing in your document
> the offsets for the beginning of each page.
Well, this is the best for my app I think, but...
How do I find out these offsets?
I'm adding the content field with:
IndexWriter#add(new Field("content", myContentReader));
I have no clue how find out the offsets in this reader. Must be something
with an analyzer and a TokenStream?
Thank you
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Design questions
Posted by sp...@gmx.eu.
OK, thank you! I will try this out.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Design questions
Posted by Erick Erickson <er...@gmail.com>.
See below
On Jan 11, 2008 9:36 AM, <sp...@gmx.eu> wrote:
> Hi,
>
>
> > You could even store all of the page offsets in your
> > meta-data document
> > in a special field if you wanted, then lazy-load that field
> > rather than
> > dynamically counting.
>
> How can I lazy load a field?
>
See FieldSelector. Basically, it allows you to determine
which fields are loaded when you fetch a doc. You can still
load the other fields explicitly.
>
> > You'd have to be careful that your offsets
> > corresponded to the data *after* it was analyzed, but that shouldn't
> > be too hard.
>
> The TermPosition is the position after analyzing?
>
Yes. The PositionIncrementGap directly affects this by changing
the positions of two consecutive terms by something other than 1.
To see why it must be this way, consider stop words. Let's say a
stop word is "are". Then indexing "democrats are running for office"
must position "democrats" right next to running, because searching
for, say, span queries with a slop of 0 of "democrats running" and
"democrats are running" should both match. But they wouldn't
if the position of "democrats" was 1 and "running" was 3.
>
> > You'd have to read this field before deleting the doc
> > and make sure it was stored with the replacement.....
>
> Don't understand...
>
If you are deleting a document but NOT reindexing all the text,
you want to preserve any special fields you have stored with the
offsets (if this field is stored in the Lucene doc being replaced)
and put it back into the replacement. Unless you're willing to
re-tokenize all the text to recreate the field.
>
> > And, since I'm getting random ideas anyway, here's another.
> > The PositionIncrementGap is the "distance" (measured in
> > terms) between two tokens. Let's claim that you have no
> > page with more than 10,000 (or whatever) tokens. Just
> > bump the positionincrementgap to the next 10,000 at the
> > start of each page. So, the first term on the first page
> > has an offset of 0. the first term on the second page
> > has an offset of 10,000. The first term on the third
> > page has an offset of 20,000.
> >
> > Now, with the SpanNearQuery trick from above, your
> > term position modulo 10,000 is also your page. This would
> > also NOT match across pages. Hmmmm, I kind of like that
> > idea.
>
> But I have to know, how many tokens each page has!?
No, you only need to know the maximum number of tokens on any page
you're going to index.
Make (that number * some slop factor) * (page number - 1) be the starting
term of each page. I'm assuming here that parsing a number of pages
in your data set and counting will give you a reasonable upper limit.
I suspect that you'll find that the actual number of tokens on a page is
*very* much lower than 10,000. More in the 1,500 range at most. But
only examining your dataset will tell. But you must be sure that whatever
number you use times the most pages you'll index (10,000 in your example?)
is smaller than an int.
And, of course, build in something that'll tell you if this expectation is
violated
during actual indexing of the dataset.
But you could also vary this scheme by simply storing in your document
the offsets for the beginning of each page. You'd have to load the offsets
field whenever you wanted to correlate positions to pages, but this may be
acceptable for your problem space, I haven't a clue since I don't know the
characteristics of your app.
Hope this helps
Erick
>
> Thank you!
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Design questions
Posted by sp...@gmx.eu.
Hi,
> You could even store all of the page offsets in your
> meta-data document
> in a special field if you wanted, then lazy-load that field
> rather than
> dynamically counting.
How can I lazy load a field?
> You'd have to be careful that your offsets
> corresponded to the data *after* it was analyzed, but that shouldn't
> be too hard.
The TermPosition is the position after analyzing?
> You'd have to read this field before deleting the doc
> and make sure it was stored with the replacement.....
Don't understand...
> And, since I'm getting random ideas anyway, here's another.
> The PositionIncrementGap is the "distance" (measured in
> terms) between two tokens. Let's claim that you have no
> page with more than 10,000 (or whatever) tokens. Just
> bump the positionincrementgap to the next 10,000 at the
> start of each page. So, the first term on the first page
> has an offset of 0. the first term on the second page
> has an offset of 10,000. The first term on the third
> page has an offset of 20,000.
>
> Now, with the SpanNearQuery trick from above, your
> term position modulo 10,000 is also your page. This would
> also NOT match across pages. Hmmmm, I kind of like that
> idea.
But I have to know, how many tokens each page has!?
Thank you!
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Design questions
Posted by Erick Erickson <er...@gmail.com>.
You can do several things:
Rather than index one doc per page, you could index a special
token between pages. Say you index $$$$$$$$$ as the special
token. So your index looks like this:
last of page 1 $$$$$$$$ first of page 2.... last of page 2 $$$$$$$$ first of
page 3
and so on. Now, if you used SpanNearQuery with a slop of 0, you would never
match across pages.
Now, you can call SpanNear.getSpans() to get the offsets of all your
matches.
You can then correlate these to pages by using TermPositions (?) or similar
interface and determine what pages you matched on.
This is not as expensive as it sounds, since you're not reading the
document,
just the indexes.
This is a possibility, I'd think that it would be easier to keep track of if
there's
a 1-to-1 correspondence between your documents in the two indexes.
As an aside, note that you don't *require* two separate indexes. There's no
requirement that all documents in an index have the same fields. So you
could
index your meta-data with an ID of, say, "meta_doc_id" and your page text
with "text_id" where these are your unique (external to Lucene) IDs. Then
you could delete with a term delete on "meta_doc_id"........
So a meta-doc looks something like:
meta_doc_id:453
field1:
field2:
field3:
and the text doc (the one and only) would be
text_id:543
text: (all 10,000 pages with page delimiters, maybe (see below)).
You could even store all of the page offsets in your meta-data document
in a special field if you wanted, then lazy-load that field rather than
dynamically counting. You'd have to be careful that your offsets
corresponded to the data *after* it was analyzed, but that shouldn't
be too hard. You'd have to read this field before deleting the doc
and make sure it was stored with the replacement.....
One caution: Lucene by default only stores the first 10,000 tokens for
a field in a document. So be sure to bump this limit with
IndexWrite.setMaxFieldLength
If you stored all the offsets of page breaks, you wouldn't have to store the
special
token since you'd have no reason to have to count them later. Be aware that
you'd get a match for a phrase that spanned the last word of one page and
the first word of the next. Which may be good, but you'll have to decide
that. You
should be able to do this pretty easily with a custom Analyzer.
One more point: I once determined that the following two actions are
identical:
1> create one really big string with all the page data concatenated together
and then add it to a document
and
2> just add successive fragments to the same document. That is,
Document doc;
doc.add(new Field("text", <all the text in all the pages>
is just like
Document doc;
while (more pages) {
doc.add(new Field("text", <text for this page>
}
********I like this variant better********
And, since I'm getting random ideas anyway, here's another.
The PositionIncrementGap is the "distance" (measured in
terms) between two tokens. Let's claim that you have no
page with more than 10,000 (or whatever) tokens. Just
bump the positionincrementgap to the next 10,000 at the
start of each page. So, the first term on the first page
has an offset of 0. the first term on the second page
has an offset of 10,000. The first term on the third
page has an offset of 20,000.
Now, with the SpanNearQuery trick from above, your
term position modulo 10,000 is also your page. This would
also NOT match across pages. Hmmmm, I kind of like that
idea.
I guess my last question is "How often will a document change"?
The added complexity of keeping two documents per unique ID
may be unnecessary if your documents don't change all that often.....
Anyway, all FWIW
Best
Erick
On Jan 9, 2008 4:39 PM, <sp...@gmx.eu> wrote:
> Hi,
>
> I have to index (tokenized) documents which may have very much pages, up
> to 10.000.
> I also have to know on which pages the search phrase occurs.
> I have to update some stored index fields for my document.
> The content is never changed.
>
> Thus I think I have to add one lucene document with the index fields and
> one lucene document per page.
>
> Mapping
> =======
>
> MyDocument
> -ID
> -Field 1-N
> -Page 1-N
>
>
> Lucene
> -Lucene Document with ID, page number 0 and Field1 - N (stored fields)
> -Lucene Document 1 with ID, page number 1 and tokenized content of Page 1
> ...
> -Lucene Document N with ID, page number N and tokenized content of Page N
>
> Delete of MyDocument -> IndexWriter#deleteDocuments(Term:ID=foo)
>
> Update of stored index fields -> IndexWriter#updateDocument(Term: ID=foo,
> page number = 0)
>
> Search with index and content.
>
> Step 1: Search on stored index fields -> List of IDs
> Step 2: Search on ID field (list from above OR'ed together) and content ->
> List of IDs and page numbers
>
> Does this work?
>
> What drawbacks has this approch?
> Is there another way to achieve what I want?
>
> Thank you.
>
> P.S.
>
> There are millions of documents with a page range from 1 to 10.000.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>