You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by "Kevin A. Burton" <bu...@newsmonster.org> on 2005/02/24 09:00:54 UTC
Possible to mix/match indexes with diff TermInfosWriter.INDEX_INTERVAL
??
I finally had some time to take Doug's advice and reburn our indexes
with a larger TermInfosWriter.INDEX_INTERVAL value.
The default is 128 but I increased it to 256 and then burned our indexes
again and was lucky enough to notice that our memory usage dropped in 1/2.
This introduced a bug however where when we try to load our pages before
and after we're missing 99% of documents from our index. What happens
is that we have a term -> key mapping so that we can pull out documents
based on essentially a primary key. The key is just hte URL of the
document. With the default value it works fine but when I change it to
256 it cant' find the majority of the documents. In fact its only able
to find one.
Is this setting incompatible with older indexes burned with the lower value?
Kevin
--
Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an
invite! Also see irc.freenode.net #rojo if you want to chat.
Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html
If you're interested in RSS, Weblogs, Social Networking, etc... then you
should work for Rojo! If you recommend someone and we hire them you'll
get a free iPod!
Kevin A. Burton, Location - San Francisco, CA
AIM/YIM - sfburtonator, Web - http://peerfear.org/
GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Possible to mix/match indexes with diff TermInfosWriter.INDEX_INTERVAL
??
Posted by "Kevin A. Burton" <bu...@newsmonster.org>.
Doug Cutting wrote:
> Not without hacking things. If your 1.3 indexes were generated with
> 256 then you can modify your version of Lucene 1.4+ to use 256 instead
> of 128 when reading a Lucene 1.3 format index (SegmentTermEnum.java:54
> today).
>
> Prior to 1.4 this was a constant, hardwired into the index format. In
> 1.4 and later each index segment stores this value as a parameter. So
> once 1.4 has re-written your index you'll no longer need a modified
> version.
Thanks for the feedback doug.
This makes more sense now. I didn't understand why the website
documented the fact that the .tii file was soring the index interval.
I think I'm going to investigate just moving to 1.4 ... I need to do it
anyway. Might as well bite the bullet now.
Kevin
--
Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an
invite! Also see irc.freenode.net #rojo if you want to chat.
Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html
If you're interested in RSS, Weblogs, Social Networking, etc... then you
should work for Rojo! If you recommend someone and we hire them you'll
get a free iPod!
Kevin A. Burton, Location - San Francisco, CA
AIM/YIM - sfburtonator, Web - http://peerfear.org/
GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Possible to mix/match indexes with diff TermInfosWriter.INDEX_INTERVAL
??
Posted by Doug Cutting <cu...@apache.org>.
Kevin A. Burton wrote:
>>> Is this setting incompatible with older indexes burned with the lower
>>> value?
>>
>> Prior to 1.4, yes. After 1.4, no.
>>
> What happens after 1.4? Can I take indexes burned with 256 (a greater
> value) in 1.3 and open them up correctly with 1.4?
Not without hacking things. If your 1.3 indexes were generated with 256
then you can modify your version of Lucene 1.4+ to use 256 instead of
128 when reading a Lucene 1.3 format index (SegmentTermEnum.java:54 today).
Prior to 1.4 this was a constant, hardwired into the index format. In
1.4 and later each index segment stores this value as a parameter. So
once 1.4 has re-written your index you'll no longer need a modified version.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Possible to mix/match indexes with diff TermInfosWriter.INDEX_INTERVAL
??
Posted by "Kevin A. Burton" <bu...@newsmonster.org>.
Doug Cutting wrote:
> Kevin A. Burton wrote:
>
>> I finally had some time to take Doug's advice and reburn our indexes
>> with a larger TermInfosWriter.INDEX_INTERVAL value.
>
>
> It looks like you're using a pre-1.4 version of Lucene. Since 1.4
> this is no longer called TermInfosWriter.INDEX_INTERVAL, but rather
> TermInfosWriter.indexInterval.
Yes... we're trying to be conservative and haven't migrated yet. Though
doing so might be required for this move I think...
>> Is this setting incompatible with older indexes burned with the lower
>> value?
>
>
> Prior to 1.4, yes. After 1.4, no.
>
What happens after 1.4? Can I take indexes burned with 256 (a greater
value) in 1.3 and open them up correctly with 1.4?
Kevin
PS. Once I get this working I'm going to create a wiki page documenting
this process.
Kevin
--
Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an
invite! Also see irc.freenode.net #rojo if you want to chat.
Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html
If you're interested in RSS, Weblogs, Social Networking, etc... then you
should work for Rojo! If you recommend someone and we hire them you'll
get a free iPod!
Kevin A. Burton, Location - San Francisco, CA
AIM/YIM - sfburtonator, Web - http://peerfear.org/
GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Possible to mix/match indexes with diff TermInfosWriter.INDEX_INTERVAL
??
Posted by Doug Cutting <cu...@apache.org>.
Kevin A. Burton wrote:
> I finally had some time to take Doug's advice and reburn our indexes
> with a larger TermInfosWriter.INDEX_INTERVAL value.
It looks like you're using a pre-1.4 version of Lucene. Since 1.4 this
is no longer called TermInfosWriter.INDEX_INTERVAL, but rather
TermInfosWriter.indexInterval.
> Is this setting incompatible with older indexes burned with the lower
> value?
Prior to 1.4, yes. After 1.4, no.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Not entire document being indexed?
Posted by Andrzej Bialecki <ab...@getopt.org>.
amigo@max3d.com wrote:
> Anyone else has any ideas why wouldn't the whole documents be indexed as
> described below?
>
> Or perhaps someone can enlighten me on how to use Luke to find out if
> the whole document was indexed or not.
> I have not used Luke in such capacity before so not sure what to do or
> look for?
Well, you could try to use the "Reconstruct & Edit" function - this will
give you an idea what tokens ended up in the index, and which was the
last one. In Luke 0.6, if the field is stored then you will see two tabs
- one is for stored content, the other displays tokenized content where
tokens are separated by commas. If the field was un-stored, then the
only tab you will get will be the reconstructed content. In any case,
just scroll down and check what are the last tokens.
You could also look for presence of some special terms that occur only
at the end of that document, and check if they are present in the index.
There are really only few reasons why this might be happening:
* your extractor has a bug, or
* the max token limit is wrongly set, or
* the indexing process doesn't close the IndexWriter properly.
--
Best regards,
Andrzej Bialecki
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Not entire document being indexed?
Posted by Andrzej Bialecki <ab...@getopt.org>.
Pasha Bizhan wrote:
> Also, 230Kb is not equal 20.000. Try to set writer.maxFieldLength to 250
> 000.
maxFieldLength's unit is a token, not a character.
--
Best regards,
Andrzej Bialecki
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Not entire document being indexed?
Posted by "amigo@max3d.com" <am...@max3d.com>.
Thanks Andrzej and Pasha for your prompt replies and suggestions.
I will try everything you have suggested and report back on the findings!
regards
-pedja
Pasha Bizhan said the following on 2/25/2005 6:32 PM:
>Hi,
>
>whole document was indexed or not.
>
>Luke can help you to give an answer the question: does my index contain a
>correct data?
>
>Let do the following steps:
>- run Luke
>- open the index
>- find the specified document (document tab)
>- click "reconstruct and edit" button
>- select the field and look the original stored content of this field
>reconstructed from index
>
>Does this reconstructed content contain your last 2-3 paragraphs?
>
>Also, 230Kb is not equal 20.000. Try to set writer.maxFieldLength to 250
>000.
>
>Pasha Bizhan
>http://lucenedotnet.com
>
RE: Not entire document being indexed?
Posted by Pasha Bizhan <fc...@ok.ru>.
Hi,
> From: amigo@max3d.com [mailto:amigo@max3d.com]
> Or perhaps someone can enlighten me on how to use Luke to find out if the
whole document was indexed or not.
Luke can help you to give an answer the question: does my index contain a
correct data?
Let do the following steps:
- run Luke
- open the index
- find the specified document (document tab)
- click "reconstruct and edit" button
- select the field and look the original stored content of this field
reconstructed from index
Does this reconstructed content contain your last 2-3 paragraphs?
Also, 230Kb is not equal 20.000. Try to set writer.maxFieldLength to 250
000.
Pasha Bizhan
http://lucenedotnet.com
> > For example one document which is about 230KB in size when
> converted
> > to plain text, when indexed and later searched for a pharse in the
> > last 2-3 paragraphs returns no hits, yet searching anything above
> > those paragraphs works just fine. WordExtractor does convert the
> > entire document to text, I've checked that.
> >
> > I've tried increasing the number of terms per field from default
> > 10,000 to 20,000 with writer.maxFieldLength but that didnt make any
> > difference, still cant find phrases from the last 2-3 paragraphs.
> >
> > Any ideas as to why this could be happening and how I could
> rectify it?
> >
> >
> > thanks,
> >
> > -pedja
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Not entire document being indexed?
Posted by "amigo@max3d.com" <am...@max3d.com>.
Anyone else has any ideas why wouldn't the whole documents be indexed as
described below?
Or perhaps someone can enlighten me on how to use Luke to find out if
the whole document was indexed or not.
I have not used Luke in such capacity before so not sure what to do or
look for?
thanks
-pedja
amigo@max3d.com said the following on 2/24/2005 2:08 PM:
> Hi everyone
>
> I'm having a bizzare problem with a few of the documents here that do
> not seem to get indexed entirely.
>
> I use textmining WordExtractor to convert M$ Word to plain text and
> then index that text.
> For example one document which is about 230KB in size when converted
> to plain text, when indexed and
> later searched for a pharse in the last 2-3 paragraphs returns no
> hits, yet searching anything above those
> paragraphs works just fine. WordExtractor does convert the entire
> document to text, I've checked that.
>
> I've tried increasing the number of terms per field from default
> 10,000 to 20,000 with writer.maxFieldLength
> but that didnt make any difference, still cant find phrases from the
> last 2-3 paragraphs.
>
> Any ideas as to why this could be happening and how I could rectify it?
>
>
> thanks,
>
> -pedja
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Not entire document being indexed?
Posted by "amigo@max3d.com" <am...@max3d.com>.
Hi Otis
Thanks for the reply, what exactly should I be looking for with Luke?
What would setting the max value to maxInteger do? Is this some
arbitrary value or...?
-pedja
Otis Gospodnetic said the following on 2/24/2005 2:24 PM:
>Use Luke to peek in your index and find out what really got indexed.
>You could also try the extreme case and set that max value to the max
>Integer.
>
>Otis
>
>--- "amigo@max3d.com" <am...@max3d.com> wrote:
>
>
>
>>Hi everyone
>>
>>I'm having a bizzare problem with a few of the documents here that do
>>
>>not seem to get indexed entirely.
>>
>>I use textmining WordExtractor to convert M$ Word to plain text and
>>then
>>index that text.
>>For example one document which is about 230KB in size when converted
>>to
>>plain text, when indexed and
>>later searched for a pharse in the last 2-3 paragraphs returns no
>>hits,
>>yet searching anything above those
>>paragraphs works just fine. WordExtractor does convert the entire
>>document to text, I've checked that.
>>
>>I've tried increasing the number of terms per field from default
>>10,000
>>to 20,000 with writer.maxFieldLength
>>but that didnt make any difference, still cant find phrases from the
>>last 2-3 paragraphs.
>>
>>Any ideas as to why this could be happening and how I could rectify
>>it?
>>
>>
>>thanks,
>>
>>-pedja
>>
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>>For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>>
>>
>>
>>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
>
>
>
Re: Not entire document being indexed?
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Use Luke to peek in your index and find out what really got indexed.
You could also try the extreme case and set that max value to the max
Integer.
Otis
--- "amigo@max3d.com" <am...@max3d.com> wrote:
> Hi everyone
>
> I'm having a bizzare problem with a few of the documents here that do
>
> not seem to get indexed entirely.
>
> I use textmining WordExtractor to convert M$ Word to plain text and
> then
> index that text.
> For example one document which is about 230KB in size when converted
> to
> plain text, when indexed and
> later searched for a pharse in the last 2-3 paragraphs returns no
> hits,
> yet searching anything above those
> paragraphs works just fine. WordExtractor does convert the entire
> document to text, I've checked that.
>
> I've tried increasing the number of terms per field from default
> 10,000
> to 20,000 with writer.maxFieldLength
> but that didnt make any difference, still cant find phrases from the
> last 2-3 paragraphs.
>
> Any ideas as to why this could be happening and how I could rectify
> it?
>
>
> thanks,
>
> -pedja
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Not entire document being indexed?
Posted by "amigo@max3d.com" <am...@max3d.com>.
Hi everyone
I'm having a bizzare problem with a few of the documents here that do
not seem to get indexed entirely.
I use textmining WordExtractor to convert M$ Word to plain text and then
index that text.
For example one document which is about 230KB in size when converted to
plain text, when indexed and
later searched for a pharse in the last 2-3 paragraphs returns no hits,
yet searching anything above those
paragraphs works just fine. WordExtractor does convert the entire
document to text, I've checked that.
I've tried increasing the number of terms per field from default 10,000
to 20,000 with writer.maxFieldLength
but that didnt make any difference, still cant find phrases from the
last 2-3 paragraphs.
Any ideas as to why this could be happening and how I could rectify it?
thanks,
-pedja
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: ngramj
Posted by "Kevin A. Burton" <bu...@newsmonster.org>.
petite_abeille wrote:
>
> On Feb 24, 2005, at 14:50, Gusenbauer Stefan wrote:
>
>> Does anyone know a good tutorial or the javadoc for ngramj because i
>> need it for guessing the language of the documents which should be
>> indexed?
>
>
> http://cvs.sourceforge.net/viewcvs.py/nutch/nutch/src/plugin/
> languageidentifier/
Wow.. interesting! Where'd this come from?
I actually wrote an implementation of NGram language categorization a
while back. I'll have to check this out. I'm willing to bet mine's
better though ;)
I was going to put it in Jakarta Commons...
Kevin
--
Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an
invite! Also see irc.freenode.net #rojo if you want to chat.
Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html
If you're interested in RSS, Weblogs, Social Networking, etc... then you
should work for Rojo! If you recommend someone and we hire them you'll
get a free iPod!
Kevin A. Burton, Location - San Francisco, CA
AIM/YIM - sfburtonator, Web - http://peerfear.org/
GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: ngramj
Posted by petite_abeille <pe...@mac.com>.
On Feb 24, 2005, at 14:50, Gusenbauer Stefan wrote:
> Does anyone know a good tutorial or the javadoc for ngramj because i
> need it for guessing the language of the documents which should be
> indexed?
http://cvs.sourceforge.net/viewcvs.py/nutch/nutch/src/plugin/
languageidentifier/
Cheers
--
PA, Onnay Equitursay
http://alt.textdrive.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
ngramj
Posted by Gusenbauer Stefan <gu...@eduhi.at>.
Does anyone know a good tutorial or the javadoc for ngramj because i
need it for guessing the language of the documents which should be indexed?
thx
stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Custom filters & document numbers
Posted by Stanislav Jordanov <st...@sirma.bg>.
Given an IndexReader a custom filter is supposed to create a bit set, that
maps each document numbers to {'visible', 'invisible'}
On the other hand, it is stated that Lucene is allowed to change document
numbers.
Is it guaranteed that this BitSet's view of document numbers won't change
while the BitSet is still in use (or perhaps the corresponding IndexReader
is still opened) ?
And another (more low-level) question.
When Lucene may change document numbers?
Is it only when the index is optimized after there has been a delete
operation?
Regards: StJ
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Possible to mix/match indexes with diff TermInfosWriter.INDEX_INTERVAL
??
Posted by "Kevin A. Burton" <bu...@newsmonster.org>.
Kevin A. Burton wrote:
> Kevin A. Burton wrote:
>
>> I finally had some time to take Doug's advice and reburn our indexes
>> with a larger TermInfosWriter.INDEX_INTERVAL value.
>
>
> You know... it looks like the problem is that TermInfosReader uses
> INDEX_INTERVAL during seeks and is probably just jumping RIGHT past
> the offsets that I need.
I guess I'm thinking out loud here...
Looks like the only thing written to the tii index for metainfo is the
"size" of the index. Its an int and is the int of the stream (which is
reserved).
Now I'm curious if there's any other way I can infer this value........
Kevin
--
Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an
invite! Also see irc.freenode.net #rojo if you want to chat.
Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html
If you're interested in RSS, Weblogs, Social Networking, etc... then you
should work for Rojo! If you recommend someone and we hire them you'll
get a free iPod!
Kevin A. Burton, Location - San Francisco, CA
AIM/YIM - sfburtonator, Web - http://peerfear.org/
GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Possible to mix/match indexes with diff TermInfosWriter.INDEX_INTERVAL
??
Posted by "Kevin A. Burton" <bu...@newsmonster.org>.
Kevin A. Burton wrote:
> I finally had some time to take Doug's advice and reburn our indexes
> with a larger TermInfosWriter.INDEX_INTERVAL value.
You know... it looks like the problem is that TermInfosReader uses
INDEX_INTERVAL during seeks and is probably just jumping RIGHT past the
offsets that I need.
If this is going to be a practical way of reducing Lucene memory
footprint for HUGE indexes then its going to need a way to change this
value based on the current index thats being opened.
Is there anyway to determine the INDEX_INTERVAL from the file? It
looks according to:
http://jakarta.apache.org/lucene/docs/fileformats.html
That the .tis file (which according to the docs the .tii file "is very
similar to the .tis file" ) should have this data:
So according to this:
TermInfoFile (.tis)--> TIVersion, TermCount, IndexInterval,
SkipInterval, TermInfos
The only problem is that the .tii and .tis files I have on disk don't
have a constant preamble and doesnt' look like there's an index interval
here...
Kevin
--
Use Rojo (RSS/Atom aggregator). Visit http://rojo.com. Ask me for an
invite! Also see irc.freenode.net #rojo if you want to chat.
Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html
If you're interested in RSS, Weblogs, Social Networking, etc... then you
should work for Rojo! If you recommend someone and we hire them you'll
get a free iPod!
Kevin A. Burton, Location - San Francisco, CA
AIM/YIM - sfburtonator, Web - http://peerfear.org/
GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org