You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@lucenenet.apache.org by Igor Kalders <ig...@wefocus.com> on 2008/03/04 12:40:30 UTC

Find unstored content

I have a lucene index that basically contains parsed IIS logs. I use this to
calculate some custom web statistics.

 

Currently, my lucene index contains only indexed fields - nothing is stored.
E.g. I can query "cs-uri-stem:/index.htm and date:20080304" to check how
often index.htm was requested today, but I cannot subsequently read
cs-uri-stem or date from the returned documents.

 

Now, this approach works pretty well with one disadvantage: I cannot figure
out the date of my last stored item. I can search all documents and sort on
date, but I cannot tell the dates of my returned documents. I need this for
only one purpose: figuring out the date of the last added document.

 

Does anyone have an idea if I would be able to tackle this _without_
indexing all data and _without_ implementing a custom checkpoint file? If I
can get this done, it would save me more than half of the index disk space.

Igor

 


RE: Find unstored content

Posted by Igor Kalders <ig...@wefocus.com>.
Good idea. In meantime however, I 've dealt with it in another way. Whenever
I detect there might be entries in the index that are about to be stored in
duplicate, I perform a search for it first. If the hit count is not 0, I do
not index. Else I do. If that makes sense... :)

Igor

-----Oorspronkelijk bericht-----
Van: Jokin Cuadrado [mailto:jokin.c@gmail.com] 
Verzonden: dinsdag 4 maart 2008 13:01
Aan: lucene-net-user@incubator.apache.org
Onderwerp: Re: Find unstored content

you could use a term enumerator to get the term list of a field and get
the last of them. If i remember well they are sorted alphabetically. 

-- 
jokin 

Jatorrizko mezua: ar., 2008-03-04 12:40 +0100, egilea: Igor Kalders
> 
> 
> Does anyone have an idea if I would be able to tackle this _without_
> indexing all data and _without_ implementing a custom checkpoint file?
> If I
> can get this done, it would save me more than half of the index disk
> space.


Re: Find unstored content

Posted by Jokin Cuadrado <jo...@gmail.com>.
you could use a term enumerator to get the term list of a field and get
the last of them. If i remember well they are sorted alphabetically. 

-- 
jokin 

Jatorrizko mezua: ar., 2008-03-04 12:40 +0100, egilea: Igor Kalders
> 
> 
> Does anyone have an idea if I would be able to tackle this _without_
> indexing all data and _without_ implementing a custom checkpoint file?
> If I
> can get this done, it would save me more than half of the index disk
> space.


RE: Find unstored content

Posted by Igor Kalders <ig...@wefocus.com>.
Thought of it, but then I need to make sure not to do optimizations on the
index.

For now, I just made it so that I detect whether de indexer is prone to
duplicates. In that case, I switch to double-check mode and search for the
document before adding it.

There's one flaw with this design: I'm storing time fields with precisions
of a second. So if the same document gets requested twice within a second,
double-check mode would only register one request. I can live with that,
because double-check mode will almost never occur and if so, the effect on
log statistics would be minimal for most use cases.

Igor

-----Oorspronkelijk bericht-----
Van: Digy [mailto:digydigy@gmail.com] 
Verzonden: dinsdag 4 maart 2008 17:52
Aan: lucene-net-user@incubator.apache.org
Onderwerp: RE: Find unstored content

> I need this for only one purpose: figuring out the date of the last added
document.

Modification Time of the Index Directory?

DIGY



RE: Find unstored content

Posted by Digy <di...@gmail.com>.
> I need this for only one purpose: figuring out the date of the last added
document.

Modification Time of the Index Directory?

DIGY