You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Sriram Sankar <sa...@gmail.com> on 2013/07/05 22:16:36 UTC
posting list strings
It looks like Lucene stores the string names of the posting lists in the
index. How compact is this storage (when there may be a very large number
of posting lists, and the string lengths may be large - for example, I may
have an entry that looks like "Cn:4593846->8957363,485". I've seen other
systems hash the string, use the hash and discard the string. Does Lucene
have this option?
Thanks,
Sriram.
Re: posting list strings
Posted by SachinMB <sa...@gmail.com>.
I wanted to know whether the terms are hashed before putting them in the term
dictionary in Lucene. Also, does Earlybird use Hashes to retrieve the
documents which contain the searched term.
The Earlybird Paper [1] does talk about indexing the terms but can you tell
me where exactly is the hashing comparison implemented. How do i find a PHF?
Please reply soon..
--
View this message in context: http://lucene.472066.n3.nabble.com/posting-list-strings-tp4075816p4091028.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: posting list strings
Posted by Sriram Sankar <sa...@gmail.com>.
Thanks!
On Tue, Jul 9, 2013 at 2:34 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Hi,
>
> You can replace the term by their hash directly in the analyzer chain.
> Just write a custom TermToBytesRef attribute that hashes the term to a
> constant-length byte[] (using a AttributeFactory)! :-) This would give you
> all features of hashed, constant length terms, but you would lose prefix
> and wildcard queries. In fact, NumericTokenStream is doing this for numeric!
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Adrien Grand [mailto:jpountz@gmail.com]
> > Sent: Tuesday, July 09, 2013 11:25 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: posting list strings
> >
> > Hi,
> >
> > Lucene stores the string because it may need it to run prefix or range
> > queries. We don't have a hash-based terms dictionary right now but I know
> > some people wrote one since they don't need support for these queries,
> see
> > for instance the Earlybird paper[1]. Then if you can find a perfect
> hashing
> > function, you can just replace your terms by their hash.
> >
> > [1]
> > http://www.umiacs.umd.edu/~jimmylin/publications/Busch_etal_ICDE2012.
> > pdf
> >
> > --
> > Adrien
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: posting list strings
Posted by Sriram Sankar <sa...@gmail.com>.
The large majority of terms in my index are not text terms. For example, I
have connection terms. Suppose user 543 and user 664 are connected. Then
the doc corresponding to user 543 will have a term connection:664 indexed.
It is not useful to do prefix matching on this - and ideally I'd not want
to pay the overhead for that.
How exactly is prefix matching done on Lucene? Is there any doc describing
details?
Thanks,
Sriram.
On Sun, Jul 14, 2013 at 7:48 PM, Lance Norskog <go...@gmail.com> wrote:
> Is there a Trie-based term index? Seems like this would be smaller, and
> very fast on non-leading wildcards.
>
>
> On 07/09/2013 02:34 PM, Uwe Schindler wrote:
>
>> Hi,
>>
>> You can replace the term by their hash directly in the analyzer chain.
>> Just write a custom TermToBytesRef attribute that hashes the term to a
>> constant-length byte[] (using a AttributeFactory)! :-) This would give you
>> all features of hashed, constant length terms, but you would lose prefix
>> and wildcard queries. In fact, NumericTokenStream is doing this for numeric!
>>
>> Uwe
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>>
>> -----Original Message-----
>>> From: Adrien Grand [mailto:jpountz@gmail.com]
>>> Sent: Tuesday, July 09, 2013 11:25 PM
>>> To: java-user@lucene.apache.org
>>> Subject: Re: posting list strings
>>>
>>> Hi,
>>>
>>> Lucene stores the string because it may need it to run prefix or range
>>> queries. We don't have a hash-based terms dictionary right now but I know
>>> some people wrote one since they don't need support for these queries,
>>> see
>>> for instance the Earlybird paper[1]. Then if you can find a perfect
>>> hashing
>>> function, you can just replace your terms by their hash.
>>>
>>> [1]
>>> http://www.umiacs.umd.edu/~**jimmylin/publications/Busch_**etal_ICDE2012<http://www.umiacs.umd.edu/~jimmylin/publications/Busch_etal_ICDE2012>
>>> .
>>> pdf
>>>
>>> --
>>> Adrien
>>>
>>> ------------------------------**------------------------------**
>>> ---------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<ja...@lucene.apache.org>
>>> For additional commands, e-mail: java-user-help@lucene.apache.**org<ja...@lucene.apache.org>
>>>
>>
>> ------------------------------**------------------------------**---------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<ja...@lucene.apache.org>
>> For additional commands, e-mail: java-user-help@lucene.apache.**org<ja...@lucene.apache.org>
>>
>>
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<ja...@lucene.apache.org>
> For additional commands, e-mail: java-user-help@lucene.apache.**org<ja...@lucene.apache.org>
>
>
Re: posting list strings
Posted by Lance Norskog <go...@gmail.com>.
Is there a Trie-based term index? Seems like this would be smaller, and
very fast on non-leading wildcards.
On 07/09/2013 02:34 PM, Uwe Schindler wrote:
> Hi,
>
> You can replace the term by their hash directly in the analyzer chain. Just write a custom TermToBytesRef attribute that hashes the term to a constant-length byte[] (using a AttributeFactory)! :-) This would give you all features of hashed, constant length terms, but you would lose prefix and wildcard queries. In fact, NumericTokenStream is doing this for numeric!
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: Adrien Grand [mailto:jpountz@gmail.com]
>> Sent: Tuesday, July 09, 2013 11:25 PM
>> To: java-user@lucene.apache.org
>> Subject: Re: posting list strings
>>
>> Hi,
>>
>> Lucene stores the string because it may need it to run prefix or range
>> queries. We don't have a hash-based terms dictionary right now but I know
>> some people wrote one since they don't need support for these queries, see
>> for instance the Earlybird paper[1]. Then if you can find a perfect hashing
>> function, you can just replace your terms by their hash.
>>
>> [1]
>> http://www.umiacs.umd.edu/~jimmylin/publications/Busch_etal_ICDE2012.
>> pdf
>>
>> --
>> Adrien
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: posting list strings
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
You can replace the term by their hash directly in the analyzer chain. Just write a custom TermToBytesRef attribute that hashes the term to a constant-length byte[] (using a AttributeFactory)! :-) This would give you all features of hashed, constant length terms, but you would lose prefix and wildcard queries. In fact, NumericTokenStream is doing this for numeric!
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Adrien Grand [mailto:jpountz@gmail.com]
> Sent: Tuesday, July 09, 2013 11:25 PM
> To: java-user@lucene.apache.org
> Subject: Re: posting list strings
>
> Hi,
>
> Lucene stores the string because it may need it to run prefix or range
> queries. We don't have a hash-based terms dictionary right now but I know
> some people wrote one since they don't need support for these queries, see
> for instance the Earlybird paper[1]. Then if you can find a perfect hashing
> function, you can just replace your terms by their hash.
>
> [1]
> http://www.umiacs.umd.edu/~jimmylin/publications/Busch_etal_ICDE2012.
> pdf
>
> --
> Adrien
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: posting list strings
Posted by Adrien Grand <jp...@gmail.com>.
Hi,
Lucene stores the string because it may need it to run prefix or range
queries. We don't have a hash-based terms dictionary right now but I
know some people wrote one since they don't need support for these
queries, see for instance the Earlybird paper[1]. Then if you can find
a perfect hashing function, you can just replace your terms by their
hash.
[1] http://www.umiacs.umd.edu/~jimmylin/publications/Busch_etal_ICDE2012.pdf
--
Adrien
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org