You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Antony Bowesman <ad...@teamware.com> on 2008/08/18 06:41:19 UTC
Fields with the same name?? - Was Re: Payloads and tokenizers
> I assume you already know this but just to make sure what I meant was clear
> - on tokenization but still indexing just means that the entire field's text
> becomes a single unchanged token. I believe this is exactly what
> SingleTokenTokenStream can buy you - a single token, for which you can pre
> set a payload.
Yes, I was with you :)
> It is. Field maintains its value and it is either string/stream/etc. Once
> you set it to tokenStream the string value is lost and there's no way to
> store it.
Thanks for that - I delved a little further into FieldsWriter and see what you
mean.
> How about adding this field in two parts, one part for indexing with the
> payload and the other part for storing, i.e. something like this:
>
> Token token = new Token(...);
> token.setPayload(...);
> SingleTokenTokenStream ts = new SingleTokenTokenStream(token);
>
> Field f1 = new Field("f","some-stored-content",Store.YES,Index.NO);
> Field f2 = new Field("f", ts);
Now that got me thinking and I have exposed a rather large misconception in my
understanding of the Lucene internals when consider fields of the same name.
Your idea above looked like a good one. However, I realise I am probably trying
to use payloads wrongly. I have the following information to store for a single
Document
contentId - 1 instance
ownerId 1..n instances
accessId 1..n instances
One ownerId has a corresponding accessId for the contentId.
My search criteria are ownerId:XXX + user criteria. When there is a hit, I need
the contentId and the corresponding accessId (for the owner) back. So, I wanted
to store the accessId as a payload to the ownerId.
This is where I came unstuck. For 'n=3' above, I used the
SingleTokenTokenStream as you suggested with the accessId as the payload for
ownerId. However, at the Document level, I cannot get the payloads from the
field so, in trying to understand fields with the same name, I discovered that
there is a big difference between
(a)
Field f = new Field("ownerId", "OID1", Store.YES, Index.NO_NORMS);
f = new Field("ownerId", "OID2", Store.YES, Index.NO_NORMS);
f = new Field("ownerId", "OID3", Store.YES, Index.NO_NORMS);
and (b)
Field f = new Field("ownerId", "OID1 OID2 OID3", Store.YES, Index.NO_NORMS);
as Document.getFields("ownerId") for (a) will be 3 and for (b) it will be 1.
My question then is, if I do
for (int i = 0; i < owners; i++)
{
f = new Field("ownerId", oid[i], Store.YES, Index.NO_NORMS);
doc.add(f);
f = new Field("accessId", aid[i], Store.YES, Index.NO_NORMS);
doc.add(f);
}
then will the array elements for the corresponding Field arrays returned by
Document.getFields("ownerId")
Document.getFields("accessId")
**guarantee** that the array element order is the same as the order they were added?
Antony
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Fields with the same name?? - Was Re: Payloads and tokenizers
Posted by Doron Cohen <cd...@gmail.com>.
On Tue, Aug 19, 2008 at 2:15 AM, Antony Bowesman <ad...@teamware.com> wrote:
>
> Thanks for you time and I appreciate your valuable insight Doron.
> Antony
>
I'm glad I could help!
Doron
Re: Fields with the same name?? - Was Re: Payloads and tokenizers
Posted by Antony Bowesman <ad...@teamware.com>.
Doron Cohen wrote:
> The API definitely doesn't promise this.
> AFAIK implementation wise it happens to be like this but I can be wrong and
> plus it might change in the future. It would make me nervous to rely on
> this.
I made some tests and it 'seems' to work, but I agree, it also makes me nervous
to rely on empirical evidence for the design rather than a clearly documented API!
> Anyhow, for your need I can think of two options:
>
> Option 1: just index the owenerID, do not store it, do not index or store
> accessID (unless you wish to search by it, in this case just index it). In
> addition store a dedicated mapping field that maps from ownerID to accessID.
> E.g. with serialization of HashMap or something thinner. At runtime retrieve
> this map from the document and it has all that information.
Hey that's an interesting idea! I'd not considered storing the mapping, only
re-creating it from fields at runtime. I'll explore this.
> Option 2: as you describe above, just index the ownerID with accessID as
> payload, and then for the hitting docid of interest use termPositions to get
> the payload, i.e. something like:
> TermPositions tp = reader.termPositions();
> tp.seek(new Term("ownerID",oid));
> tp.skipTo(docid);
> tp.nextPosition();
> if (tp.isPayloadAvailable()) {
> byte [] accessIDBytes = tp.getPayload(...);
> ...
Yes, I was playing with this technique yesterday. It's not easy to determine
the performance implications of this method. I will be using caches, but my
volumes are potentially so large that I may never be able to cache everything
(perhaps 500M Docs), so this has to be very quick.
I'll play with both approaches and see which works best.
Thanks for you time and I appreciate your valuable insight Doron.
Antony
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Fields with the same name?? - Was Re: Payloads and tokenizers
Posted by Doron Cohen <cd...@gmail.com>.
>
> payload and the other part for storing, i.e. something like this:
>>
>> Token token = new Token(...);
>> token.setPayload(...);
>> SingleTokenTokenStream ts = new SingleTokenTokenStream(token);
>>
>> Field f1 = new Field("f","some-stored-content",Store.YES,Index.NO);
>> Field f2 = new Field("f", ts);
>>
> How about adding this field in two parts, one part for indexing with the
>
> Now that got me thinking and I have exposed a rather large misconception in
> my understanding of the Lucene internals when consider fields of the same
> name.
>
> Your idea above looked like a good one. However, I realise I am probably
> trying to use payloads wrongly. I have the following information to store
> for a single Document
>
> contentId - 1 instance
> ownerId 1..n instances
> accessId 1..n instances
>
> One ownerId has a corresponding accessId for the contentId.
>
> My search criteria are ownerId:XXX + user criteria. When there is a hit, I
> need the contentId and the corresponding accessId (for the owner) back. So,
> I wanted to store the accessId as a payload to the ownerId.
>
> This is where I came unstuck. For 'n=3' above, I used the
> SingleTokenTokenStream as you suggested with the accessId as the payload for
> ownerId. However, at the Document level, I cannot get the payloads from the
> field so, in trying to understand fields with the same name, I discovered
> that there is a big difference between
>
> (a)
> Field f = new Field("ownerId", "OID1", Store.YES, Index.NO_NORMS);
> f = new Field("ownerId", "OID2", Store.YES, Index.NO_NORMS);
> f = new Field("ownerId", "OID3", Store.YES, Index.NO_NORMS);
>
> and (b)
> Field f = new Field("ownerId", "OID1 OID2 OID3", Store.YES,
> Index.NO_NORMS);
>
> as Document.getFields("ownerId") for (a) will be 3 and for (b) it will be
> 1.
>
> My question then is, if I do
>
> for (int i = 0; i < owners; i++)
> {
> f = new Field("ownerId", oid[i], Store.YES, Index.NO_NORMS);
> doc.add(f);
> f = new Field("accessId", aid[i], Store.YES, Index.NO_NORMS);
> doc.add(f);
> }
>
> then will the array elements for the corresponding Field arrays returned by
>
> Document.getFields("ownerId")
> Document.getFields("accessId")
>
> **guarantee** that the array element order is the same as the order they
> were added?
>
The API definitely doesn't promise this.
AFAIK implementation wise it happens to be like this but I can be wrong and
plus it might change in the future. It would make me nervous to rely on
this.
The difficulty stems from that any specific information on the actual
matching token is digested at scoring and not reaching the hit collector in
effect. It somewhat reminds me the situation with highlighting, where
positions might have been considered for scoring, yet for a certain matching
doc of interest that is being displayed with highlighting, positions (and
offsets) need to be found again.
Anyhow, for your need I can think of two options:
Option 1: just index the owenerID, do not store it, do not index or store
accessID (unless you wish to search by it, in this case just index it). In
addition store a dedicated mapping field that maps from ownerID to accessID.
E.g. with serialization of HashMap or something thinner. At runtime retrieve
this map from the document and it has all that information.
Option 2: as you describe above, just index the ownerID with accessID as
payload, and then for the hitting docid of interest use termPositions to get
the payload, i.e. something like:
TermPositions tp = reader.termPositions();
tp.seek(new Term("ownerID",oid));
tp.skipTo(docid);
tp.nextPosition();
if (tp.isPayloadAvailable()) {
byte [] accessIDBytes = tp.getPayload(...);
...
Each has its overhead but I think both should work...
Doron