You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by hellorsanjeev <sa...@3pillarglobal.com> on 2012/12/18 08:59:06 UTC
"order" question on solr multi value field
Hi - We have been using solr since over an year. Some of our fields are multi
valued. We are saving and retrieving data from those multi valued fields
perfectly fine assuming the order in which we save, we will get the values
in the same order.
The question is - Is it documented somewhere that Solr maintains the order
of values inserted into multi-valued field OR is it just by luck we are
getting them in insertion order? By my experience so far, it has always
maintained.
Thanks in advance.
--
View this message in context: http://lucene.472066.n3.nabble.com/order-question-on-solr-multi-value-field-tp4027695.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: "order" question on solr multi value field
Posted by Upayavira <uv...@odoko.co.uk>.
This is not a definitive answer, but what you are retrieving is the
stored versions of those fields. When the document is indexed, it will
also store the multi-valued values of that field. I see no reason why at
index time it should re-order them - it just writes them to the index in
the order they are found. Solr (or Lucene rather) once a document has
been written, will never rewrite it, so it seems perfectly reasonable to
expect stored multi-valued fields to return in the same order.
When it comes to indexed terms, that's a different matter.
Upayavira
On Tue, Dec 18, 2012, at 07:59 AM, hellorsanjeev wrote:
> Hi - We have been using solr since over an year. Some of our fields are
> multi
> valued. We are saving and retrieving data from those multi valued fields
> perfectly fine assuming the order in which we save, we will get the
> values
> in the same order.
>
> The question is - Is it documented somewhere that Solr maintains the
> order
> of values inserted into multi-valued field OR is it just by luck we are
> getting them in insertion order? By my experience so far, it has always
> maintained.
>
> Thanks in advance.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/order-question-on-solr-multi-value-field-tp4027695.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: "order" question on solr multi value field
Posted by Otis Gospodnetic <ot...@gmail.com>.
Stick a javadoc patch in JIRA Jack?
Otis
--
SOLR Performance Monitoring - http://sematext.com/spm
On Dec 18, 2012 2:23 PM, "Jack Krupansky" <ja...@basetechnology.com> wrote:
> Yeah, and this is why I want it to be explicit.
>
> -- Jack Krupansky
>
> -----Original Message----- From: Dyer, James
> Sent: Tuesday, December 18, 2012 1:54 PM
> To: solr-user@lucene.apache.org
> Subject: RE: "order" question on solr multi value field
>
> I agree with you the documentation should be more explicit. I just don't
> want to give new users the impression that stored fields won't return in
> the order they are added. This is the behavior and I think a lot of us
> rely on that today.
>
> James Dyer
> E-Commerce Systems
> Ingram Content Group
> (615) 213-4311
>
>
> -----Original Message-----
> From: Jack Krupansky [mailto:jack@basetechnology.**com<ja...@basetechnology.com>
> ]
> Sent: Tuesday, December 18, 2012 11:35 AM
> To: solr-user@lucene.apache.org
> Subject: Re: "order" question on solr multi value field
>
> "A multi-valued field is the same as a single-valued field with the
> positions artifically incremented"
>
> Yes, for the INDEX. But the concept of Lucene-level "positions" is not
> relevant to STORED values.
>
> I'm not sure why you are confusing the two.
>
> A single string has guaranteed order of the characters in the string -
> nobody was disputing that!
>
> Sure, you can argue that the contract is semi-sort-of-implied, but the
> point
> is that we should make it explicit. If I have time later today I'll file
> the
> Jira.
>
> -- Jack Krupansky
>
> -----Original Message----- From: Dyer, James
> Sent: Tuesday, December 18, 2012 12:08 PM
> To: solr-user@lucene.apache.org
> Subject: RE: "order" question on solr multi value field
>
> I suppose using this same logic you can not guarantee the tokens on a
> stored, single-value field would be stored in the order they arrive either,
> can you? A multi-valued field is the same as a single-valued field with
> the
> positions artifically incremented, so what's the difference?
>
> James Dyer
> E-Commerce Systems
> Ingram Content Group
> (615) 213-4311
>
>
> -----Original Message-----
> From: Jack Krupansky [mailto:jack@basetechnology.**com<ja...@basetechnology.com>
> ]
> Sent: Tuesday, December 18, 2012 10:16 AM
> To: solr-user@lucene.apache.org
> Subject: Re: "order" question on solr multi value field
>
> Ah, good points, but only for the INDEXED data, not the STORED data! I
> believe the concern is the stored values that are returned from a query.
> Although the question does also apply to how a span query would work for a
> multi-valued field.
>
> -- Jack Krupansky
>
> -----Original Message----- From: Dyer, James
> Sent: Tuesday, December 18, 2012 10:54 AM
> To: solr-user@lucene.apache.org
> Subject: RE: "order" question on solr multi value field
>
> I would say such a guarantee is implied by the javadoc to
> Analyzer#**getPositionIncrementGap . It says this value is an
> "increment" to
> be "added to the next token emitted from tokenStream."
>
> http://lucene.apache.org/core/**4_0_0-ALPHA/core/org/apache/**
> lucene/analysis/Analyzer.html#**getPositionIncrementGap%**
> 28java.lang.String%29<http://lucene.apache.org/core/4_0_0-ALPHA/core/org/apache/lucene/analysis/Analyzer.html#getPositionIncrementGap%28java.lang.String%29>
>
> Also compare unofficial documentation such as Lucene In Action 2nd ed,
> section 4.7.1: "Lucene logically appends the tokens...sequentially."
>
> Having multi-valued fields stay in the order in which they were added to
> the
> Document is a guarantee that many many users depend on.
>
> James Dyer
> E-Commerce Systems
> Ingram Content Group
> (615) 213-4311
>
>
> -----Original Message-----
> From: Jack Krupansky [mailto:jack@basetechnology.**com<ja...@basetechnology.com>
> ]
> Sent: Tuesday, December 18, 2012 9:30 AM
> To: solr-user@lucene.apache.org
> Subject: Re: "order" question on solr multi value field
>
> If there is no "official" guarantee in the Javadoc for the code then there
> is no official guarantee. Period. If somebody wants an official,
> contractual
> guarantee, a Jira should be filed to do so. To put it simple, are the
> values
> a "list" or a "set"?
>
> -- Jack Krupansky
>
> -----Original Message----- From: Erik Hatcher
> Sent: Tuesday, December 18, 2012 9:40 AM
> To: solr-user@lucene.apache.org
> Cc: solr-user@lucene.apache.org
> Subject: Re: "order" question on solr multi value field
>
> I don't know of an "official" guarantee of maintaining order but it's
> definitely guaranteed an relied upon to retain order. Many will scream if
> this changed.
>
> Indexed doesn't matter here because what you get back are the stored values
> no matter if the field is indexed or not.
>
> Erik
>
> On Dec 18, 2012, at 3:04, hellorsanjeev <sanjeev.dhiman@3pillarglobal.**
> com <sa...@3pillarglobal.com>>
> wrote:
>
> thank you for quick response :)
>>
>> I also have the same observation and I too believe that there is no reason
>> for Solr to reorder a multi value field.
>>
>> But would you stay firm on your conclusion if I say that my multi value
>> field was indexed?
>>
>> Please note - as per my one year experience with Solr, it always returned
>> the values in the insertions order irrespective of the fact that field was
>> indexed or not.
>>
>> My main concern is because I couldn't find it documented anywhere, it
>> might
>> happen that in Solr 4.0 or later, they start reordering them. If they do
>> then there will be a big problem for us :)
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.**nabble.com/order-question-on-**
>> solr-multi-value-field-**tp4027695p4027713.html<http://lucene.472066.n3.nabble.com/order-question-on-solr-multi-value-field-tp4027695p4027713.html>
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>
>
>
>
>
Re: "order" question on solr multi value field
Posted by Jack Krupansky <ja...@basetechnology.com>.
Yeah, and this is why I want it to be explicit.
-- Jack Krupansky
-----Original Message-----
From: Dyer, James
Sent: Tuesday, December 18, 2012 1:54 PM
To: solr-user@lucene.apache.org
Subject: RE: "order" question on solr multi value field
I agree with you the documentation should be more explicit. I just don't
want to give new users the impression that stored fields won't return in the
order they are added. This is the behavior and I think a lot of us rely on
that today.
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com]
Sent: Tuesday, December 18, 2012 11:35 AM
To: solr-user@lucene.apache.org
Subject: Re: "order" question on solr multi value field
"A multi-valued field is the same as a single-valued field with the
positions artifically incremented"
Yes, for the INDEX. But the concept of Lucene-level "positions" is not
relevant to STORED values.
I'm not sure why you are confusing the two.
A single string has guaranteed order of the characters in the string -
nobody was disputing that!
Sure, you can argue that the contract is semi-sort-of-implied, but the point
is that we should make it explicit. If I have time later today I'll file the
Jira.
-- Jack Krupansky
-----Original Message-----
From: Dyer, James
Sent: Tuesday, December 18, 2012 12:08 PM
To: solr-user@lucene.apache.org
Subject: RE: "order" question on solr multi value field
I suppose using this same logic you can not guarantee the tokens on a
stored, single-value field would be stored in the order they arrive either,
can you? A multi-valued field is the same as a single-valued field with the
positions artifically incremented, so what's the difference?
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com]
Sent: Tuesday, December 18, 2012 10:16 AM
To: solr-user@lucene.apache.org
Subject: Re: "order" question on solr multi value field
Ah, good points, but only for the INDEXED data, not the STORED data! I
believe the concern is the stored values that are returned from a query.
Although the question does also apply to how a span query would work for a
multi-valued field.
-- Jack Krupansky
-----Original Message-----
From: Dyer, James
Sent: Tuesday, December 18, 2012 10:54 AM
To: solr-user@lucene.apache.org
Subject: RE: "order" question on solr multi value field
I would say such a guarantee is implied by the javadoc to
Analyzer#getPositionIncrementGap . It says this value is an "increment" to
be "added to the next token emitted from tokenStream."
http://lucene.apache.org/core/4_0_0-ALPHA/core/org/apache/lucene/analysis/Analyzer.html#getPositionIncrementGap%28java.lang.String%29
Also compare unofficial documentation such as Lucene In Action 2nd ed,
section 4.7.1: "Lucene logically appends the tokens...sequentially."
Having multi-valued fields stay in the order in which they were added to the
Document is a guarantee that many many users depend on.
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com]
Sent: Tuesday, December 18, 2012 9:30 AM
To: solr-user@lucene.apache.org
Subject: Re: "order" question on solr multi value field
If there is no "official" guarantee in the Javadoc for the code then there
is no official guarantee. Period. If somebody wants an official, contractual
guarantee, a Jira should be filed to do so. To put it simple, are the values
a "list" or a "set"?
-- Jack Krupansky
-----Original Message-----
From: Erik Hatcher
Sent: Tuesday, December 18, 2012 9:40 AM
To: solr-user@lucene.apache.org
Cc: solr-user@lucene.apache.org
Subject: Re: "order" question on solr multi value field
I don't know of an "official" guarantee of maintaining order but it's
definitely guaranteed an relied upon to retain order. Many will scream if
this changed.
Indexed doesn't matter here because what you get back are the stored values
no matter if the field is indexed or not.
Erik
On Dec 18, 2012, at 3:04, hellorsanjeev <sa...@3pillarglobal.com>
wrote:
> thank you for quick response :)
>
> I also have the same observation and I too believe that there is no reason
> for Solr to reorder a multi value field.
>
> But would you stay firm on your conclusion if I say that my multi value
> field was indexed?
>
> Please note - as per my one year experience with Solr, it always returned
> the values in the insertions order irrespective of the fact that field was
> indexed or not.
>
> My main concern is because I couldn't find it documented anywhere, it
> might
> happen that in Solr 4.0 or later, they start reordering them. If they do
> then there will be a big problem for us :)
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/order-question-on-solr-multi-value-field-tp4027695p4027713.html
> Sent from the Solr - User mailing list archive at Nabble.com.
RE: "order" question on solr multi value field
Posted by "Dyer, James" <Ja...@ingramcontent.com>.
I agree with you the documentation should be more explicit. I just don't want to give new users the impression that stored fields won't return in the order they are added. This is the behavior and I think a lot of us rely on that today.
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com]
Sent: Tuesday, December 18, 2012 11:35 AM
To: solr-user@lucene.apache.org
Subject: Re: "order" question on solr multi value field
"A multi-valued field is the same as a single-valued field with the
positions artifically incremented"
Yes, for the INDEX. But the concept of Lucene-level "positions" is not
relevant to STORED values.
I'm not sure why you are confusing the two.
A single string has guaranteed order of the characters in the string -
nobody was disputing that!
Sure, you can argue that the contract is semi-sort-of-implied, but the point
is that we should make it explicit. If I have time later today I'll file the
Jira.
-- Jack Krupansky
-----Original Message-----
From: Dyer, James
Sent: Tuesday, December 18, 2012 12:08 PM
To: solr-user@lucene.apache.org
Subject: RE: "order" question on solr multi value field
I suppose using this same logic you can not guarantee the tokens on a
stored, single-value field would be stored in the order they arrive either,
can you? A multi-valued field is the same as a single-valued field with the
positions artifically incremented, so what's the difference?
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com]
Sent: Tuesday, December 18, 2012 10:16 AM
To: solr-user@lucene.apache.org
Subject: Re: "order" question on solr multi value field
Ah, good points, but only for the INDEXED data, not the STORED data! I
believe the concern is the stored values that are returned from a query.
Although the question does also apply to how a span query would work for a
multi-valued field.
-- Jack Krupansky
-----Original Message-----
From: Dyer, James
Sent: Tuesday, December 18, 2012 10:54 AM
To: solr-user@lucene.apache.org
Subject: RE: "order" question on solr multi value field
I would say such a guarantee is implied by the javadoc to
Analyzer#getPositionIncrementGap . It says this value is an "increment" to
be "added to the next token emitted from tokenStream."
http://lucene.apache.org/core/4_0_0-ALPHA/core/org/apache/lucene/analysis/Analyzer.html#getPositionIncrementGap%28java.lang.String%29
Also compare unofficial documentation such as Lucene In Action 2nd ed,
section 4.7.1: "Lucene logically appends the tokens...sequentially."
Having multi-valued fields stay in the order in which they were added to the
Document is a guarantee that many many users depend on.
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com]
Sent: Tuesday, December 18, 2012 9:30 AM
To: solr-user@lucene.apache.org
Subject: Re: "order" question on solr multi value field
If there is no "official" guarantee in the Javadoc for the code then there
is no official guarantee. Period. If somebody wants an official, contractual
guarantee, a Jira should be filed to do so. To put it simple, are the values
a "list" or a "set"?
-- Jack Krupansky
-----Original Message-----
From: Erik Hatcher
Sent: Tuesday, December 18, 2012 9:40 AM
To: solr-user@lucene.apache.org
Cc: solr-user@lucene.apache.org
Subject: Re: "order" question on solr multi value field
I don't know of an "official" guarantee of maintaining order but it's
definitely guaranteed an relied upon to retain order. Many will scream if
this changed.
Indexed doesn't matter here because what you get back are the stored values
no matter if the field is indexed or not.
Erik
On Dec 18, 2012, at 3:04, hellorsanjeev <sa...@3pillarglobal.com>
wrote:
> thank you for quick response :)
>
> I also have the same observation and I too believe that there is no reason
> for Solr to reorder a multi value field.
>
> But would you stay firm on your conclusion if I say that my multi value
> field was indexed?
>
> Please note - as per my one year experience with Solr, it always returned
> the values in the insertions order irrespective of the fact that field was
> indexed or not.
>
> My main concern is because I couldn't find it documented anywhere, it
> might
> happen that in Solr 4.0 or later, they start reordering them. If they do
> then there will be a big problem for us :)
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/order-question-on-solr-multi-value-field-tp4027695p4027713.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: "order" question on solr multi value field
Posted by Jack Krupansky <ja...@basetechnology.com>.
"A multi-valued field is the same as a single-valued field with the
positions artifically incremented"
Yes, for the INDEX. But the concept of Lucene-level "positions" is not
relevant to STORED values.
I'm not sure why you are confusing the two.
A single string has guaranteed order of the characters in the string -
nobody was disputing that!
Sure, you can argue that the contract is semi-sort-of-implied, but the point
is that we should make it explicit. If I have time later today I'll file the
Jira.
-- Jack Krupansky
-----Original Message-----
From: Dyer, James
Sent: Tuesday, December 18, 2012 12:08 PM
To: solr-user@lucene.apache.org
Subject: RE: "order" question on solr multi value field
I suppose using this same logic you can not guarantee the tokens on a
stored, single-value field would be stored in the order they arrive either,
can you? A multi-valued field is the same as a single-valued field with the
positions artifically incremented, so what's the difference?
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com]
Sent: Tuesday, December 18, 2012 10:16 AM
To: solr-user@lucene.apache.org
Subject: Re: "order" question on solr multi value field
Ah, good points, but only for the INDEXED data, not the STORED data! I
believe the concern is the stored values that are returned from a query.
Although the question does also apply to how a span query would work for a
multi-valued field.
-- Jack Krupansky
-----Original Message-----
From: Dyer, James
Sent: Tuesday, December 18, 2012 10:54 AM
To: solr-user@lucene.apache.org
Subject: RE: "order" question on solr multi value field
I would say such a guarantee is implied by the javadoc to
Analyzer#getPositionIncrementGap . It says this value is an "increment" to
be "added to the next token emitted from tokenStream."
http://lucene.apache.org/core/4_0_0-ALPHA/core/org/apache/lucene/analysis/Analyzer.html#getPositionIncrementGap%28java.lang.String%29
Also compare unofficial documentation such as Lucene In Action 2nd ed,
section 4.7.1: "Lucene logically appends the tokens...sequentially."
Having multi-valued fields stay in the order in which they were added to the
Document is a guarantee that many many users depend on.
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com]
Sent: Tuesday, December 18, 2012 9:30 AM
To: solr-user@lucene.apache.org
Subject: Re: "order" question on solr multi value field
If there is no "official" guarantee in the Javadoc for the code then there
is no official guarantee. Period. If somebody wants an official, contractual
guarantee, a Jira should be filed to do so. To put it simple, are the values
a "list" or a "set"?
-- Jack Krupansky
-----Original Message-----
From: Erik Hatcher
Sent: Tuesday, December 18, 2012 9:40 AM
To: solr-user@lucene.apache.org
Cc: solr-user@lucene.apache.org
Subject: Re: "order" question on solr multi value field
I don't know of an "official" guarantee of maintaining order but it's
definitely guaranteed an relied upon to retain order. Many will scream if
this changed.
Indexed doesn't matter here because what you get back are the stored values
no matter if the field is indexed or not.
Erik
On Dec 18, 2012, at 3:04, hellorsanjeev <sa...@3pillarglobal.com>
wrote:
> thank you for quick response :)
>
> I also have the same observation and I too believe that there is no reason
> for Solr to reorder a multi value field.
>
> But would you stay firm on your conclusion if I say that my multi value
> field was indexed?
>
> Please note - as per my one year experience with Solr, it always returned
> the values in the insertions order irrespective of the fact that field was
> indexed or not.
>
> My main concern is because I couldn't find it documented anywhere, it
> might
> happen that in Solr 4.0 or later, they start reordering them. If they do
> then there will be a big problem for us :)
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/order-question-on-solr-multi-value-field-tp4027695p4027713.html
> Sent from the Solr - User mailing list archive at Nabble.com.
RE: "order" question on solr multi value field
Posted by "Dyer, James" <Ja...@ingramcontent.com>.
I suppose using this same logic you can not guarantee the tokens on a stored, single-value field would be stored in the order they arrive either, can you? A multi-valued field is the same as a single-valued field with the positions artifically incremented, so what's the difference?
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com]
Sent: Tuesday, December 18, 2012 10:16 AM
To: solr-user@lucene.apache.org
Subject: Re: "order" question on solr multi value field
Ah, good points, but only for the INDEXED data, not the STORED data! I
believe the concern is the stored values that are returned from a query.
Although the question does also apply to how a span query would work for a
multi-valued field.
-- Jack Krupansky
-----Original Message-----
From: Dyer, James
Sent: Tuesday, December 18, 2012 10:54 AM
To: solr-user@lucene.apache.org
Subject: RE: "order" question on solr multi value field
I would say such a guarantee is implied by the javadoc to
Analyzer#getPositionIncrementGap . It says this value is an "increment" to
be "added to the next token emitted from tokenStream."
http://lucene.apache.org/core/4_0_0-ALPHA/core/org/apache/lucene/analysis/Analyzer.html#getPositionIncrementGap%28java.lang.String%29
Also compare unofficial documentation such as Lucene In Action 2nd ed,
section 4.7.1: "Lucene logically appends the tokens...sequentially."
Having multi-valued fields stay in the order in which they were added to the
Document is a guarantee that many many users depend on.
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com]
Sent: Tuesday, December 18, 2012 9:30 AM
To: solr-user@lucene.apache.org
Subject: Re: "order" question on solr multi value field
If there is no "official" guarantee in the Javadoc for the code then there
is no official guarantee. Period. If somebody wants an official, contractual
guarantee, a Jira should be filed to do so. To put it simple, are the values
a "list" or a "set"?
-- Jack Krupansky
-----Original Message-----
From: Erik Hatcher
Sent: Tuesday, December 18, 2012 9:40 AM
To: solr-user@lucene.apache.org
Cc: solr-user@lucene.apache.org
Subject: Re: "order" question on solr multi value field
I don't know of an "official" guarantee of maintaining order but it's
definitely guaranteed an relied upon to retain order. Many will scream if
this changed.
Indexed doesn't matter here because what you get back are the stored values
no matter if the field is indexed or not.
Erik
On Dec 18, 2012, at 3:04, hellorsanjeev <sa...@3pillarglobal.com>
wrote:
> thank you for quick response :)
>
> I also have the same observation and I too believe that there is no reason
> for Solr to reorder a multi value field.
>
> But would you stay firm on your conclusion if I say that my multi value
> field was indexed?
>
> Please note - as per my one year experience with Solr, it always returned
> the values in the insertions order irrespective of the fact that field was
> indexed or not.
>
> My main concern is because I couldn't find it documented anywhere, it
> might
> happen that in Solr 4.0 or later, they start reordering them. If they do
> then there will be a big problem for us :)
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/order-question-on-solr-multi-value-field-tp4027695p4027713.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: "order" question on solr multi value field
Posted by Jack Krupansky <ja...@basetechnology.com>.
Ah, good points, but only for the INDEXED data, not the STORED data! I
believe the concern is the stored values that are returned from a query.
Although the question does also apply to how a span query would work for a
multi-valued field.
-- Jack Krupansky
-----Original Message-----
From: Dyer, James
Sent: Tuesday, December 18, 2012 10:54 AM
To: solr-user@lucene.apache.org
Subject: RE: "order" question on solr multi value field
I would say such a guarantee is implied by the javadoc to
Analyzer#getPositionIncrementGap . It says this value is an "increment" to
be "added to the next token emitted from tokenStream."
http://lucene.apache.org/core/4_0_0-ALPHA/core/org/apache/lucene/analysis/Analyzer.html#getPositionIncrementGap%28java.lang.String%29
Also compare unofficial documentation such as Lucene In Action 2nd ed,
section 4.7.1: "Lucene logically appends the tokens...sequentially."
Having multi-valued fields stay in the order in which they were added to the
Document is a guarantee that many many users depend on.
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com]
Sent: Tuesday, December 18, 2012 9:30 AM
To: solr-user@lucene.apache.org
Subject: Re: "order" question on solr multi value field
If there is no "official" guarantee in the Javadoc for the code then there
is no official guarantee. Period. If somebody wants an official, contractual
guarantee, a Jira should be filed to do so. To put it simple, are the values
a "list" or a "set"?
-- Jack Krupansky
-----Original Message-----
From: Erik Hatcher
Sent: Tuesday, December 18, 2012 9:40 AM
To: solr-user@lucene.apache.org
Cc: solr-user@lucene.apache.org
Subject: Re: "order" question on solr multi value field
I don't know of an "official" guarantee of maintaining order but it's
definitely guaranteed an relied upon to retain order. Many will scream if
this changed.
Indexed doesn't matter here because what you get back are the stored values
no matter if the field is indexed or not.
Erik
On Dec 18, 2012, at 3:04, hellorsanjeev <sa...@3pillarglobal.com>
wrote:
> thank you for quick response :)
>
> I also have the same observation and I too believe that there is no reason
> for Solr to reorder a multi value field.
>
> But would you stay firm on your conclusion if I say that my multi value
> field was indexed?
>
> Please note - as per my one year experience with Solr, it always returned
> the values in the insertions order irrespective of the fact that field was
> indexed or not.
>
> My main concern is because I couldn't find it documented anywhere, it
> might
> happen that in Solr 4.0 or later, they start reordering them. If they do
> then there will be a big problem for us :)
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/order-question-on-solr-multi-value-field-tp4027695p4027713.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: "order" question on solr multi value field
Posted by Yonik Seeley <yo...@lucidworks.com>.
On Tue, Dec 18, 2012 at 8:24 PM, Robert Muir <rc...@gmail.com> wrote:
> I agree with James. Actually lucene tests will fail if a codec violates this.
>
> Actually it goes much deeper than this.
>
> From the lucene apis, when you call IndexReader.document() with your
> storedfieldVisitor, it must visit the fields in the original order
> added.
Yeah, Solr has a weaker guarantee.
Order is guaranteed to be maintained for values in a multi-valued field.
Order of different fields is not maintained.
-Yonik
http://lucidworks.com
RE: "order" question on solr multi value field
Posted by "Dyer, James" <Ja...@ingramcontent.com>.
I was noticing when I was looking this up in LIA yesterday a mention was made that when using a field list to return just some of the stored fields, that it still takes time to seek over the fields you don't want. So, if it has this strict guarantee about order, could a user potentially optimize this by always adding short metadata fields to a document first and then longer content fields at the end?
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: Robert Muir [mailto:rcmuir@gmail.com]
Sent: Tuesday, December 18, 2012 7:25 PM
To: solr-user@lucene.apache.org
Subject: Re: "order" question on solr multi value field
I agree with James. Actually lucene tests will fail if a codec violates this.
Actually it goes much deeper than this.
From the lucene apis, when you call IndexReader.document() with your
storedfieldVisitor, it must visit the fields in the original order
added.
so even if you do:
add("title", "title value 1");
add("body", "body value");
add("title", "title value 2");
Currently stored fields must be returned in exactly this order:
title1, then body, then title2.
This is pretty annoying :) I don't think its truly necessary to
maintain that crazy guarantee, but I'm pretty sure something tests it
somewhere. In my opinion its too restrictive and prevents useful
optimizations.
But in my opinion, title1 should always come back before title2 in the
order you added them just like today.
On Tue, Dec 18, 2012 at 10:54 AM, Dyer, James
<Ja...@ingramcontent.com> wrote:
> I would say such a guarantee is implied by the javadoc to Analyzer#getPositionIncrementGap . It says this value is an "increment" to be "added to the next token emitted from tokenStream."
>
> http://lucene.apache.org/core/4_0_0-ALPHA/core/org/apache/lucene/analysis/Analyzer.html#getPositionIncrementGap%28java.lang.String%29
>
> Also compare unofficial documentation such as Lucene In Action 2nd ed, section 4.7.1: "Lucene logically appends the tokens...sequentially."
>
> Having multi-valued fields stay in the order in which they were added to the Document is a guarantee that many many users depend on.
>
> James Dyer
> E-Commerce Systems
> Ingram Content Group
> (615) 213-4311
>
>
> -----Original Message-----
> From: Jack Krupansky [mailto:jack@basetechnology.com]
> Sent: Tuesday, December 18, 2012 9:30 AM
> To: solr-user@lucene.apache.org
> Subject: Re: "order" question on solr multi value field
>
> If there is no "official" guarantee in the Javadoc for the code then there
> is no official guarantee. Period. If somebody wants an official, contractual
> guarantee, a Jira should be filed to do so. To put it simple, are the values
> a "list" or a "set"?
>
> -- Jack Krupansky
>
> -----Original Message-----
> From: Erik Hatcher
> Sent: Tuesday, December 18, 2012 9:40 AM
> To: solr-user@lucene.apache.org
> Cc: solr-user@lucene.apache.org
> Subject: Re: "order" question on solr multi value field
>
> I don't know of an "official" guarantee of maintaining order but it's
> definitely guaranteed an relied upon to retain order. Many will scream if
> this changed.
>
> Indexed doesn't matter here because what you get back are the stored values
> no matter if the field is indexed or not.
>
> Erik
>
> On Dec 18, 2012, at 3:04, hellorsanjeev <sa...@3pillarglobal.com>
> wrote:
>
>> thank you for quick response :)
>>
>> I also have the same observation and I too believe that there is no reason
>> for Solr to reorder a multi value field.
>>
>> But would you stay firm on your conclusion if I say that my multi value
>> field was indexed?
>>
>> Please note - as per my one year experience with Solr, it always returned
>> the values in the insertions order irrespective of the fact that field was
>> indexed or not.
>>
>> My main concern is because I couldn't find it documented anywhere, it
>> might
>> happen that in Solr 4.0 or later, they start reordering them. If they do
>> then there will be a big problem for us :)
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/order-question-on-solr-multi-value-field-tp4027695p4027713.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
Re: "order" question on solr multi value field
Posted by Robert Muir <rc...@gmail.com>.
I agree with James. Actually lucene tests will fail if a codec violates this.
Actually it goes much deeper than this.
>From the lucene apis, when you call IndexReader.document() with your
storedfieldVisitor, it must visit the fields in the original order
added.
so even if you do:
add("title", "title value 1");
add("body", "body value");
add("title", "title value 2");
Currently stored fields must be returned in exactly this order:
title1, then body, then title2.
This is pretty annoying :) I don't think its truly necessary to
maintain that crazy guarantee, but I'm pretty sure something tests it
somewhere. In my opinion its too restrictive and prevents useful
optimizations.
But in my opinion, title1 should always come back before title2 in the
order you added them just like today.
On Tue, Dec 18, 2012 at 10:54 AM, Dyer, James
<Ja...@ingramcontent.com> wrote:
> I would say such a guarantee is implied by the javadoc to Analyzer#getPositionIncrementGap . It says this value is an "increment" to be "added to the next token emitted from tokenStream."
>
> http://lucene.apache.org/core/4_0_0-ALPHA/core/org/apache/lucene/analysis/Analyzer.html#getPositionIncrementGap%28java.lang.String%29
>
> Also compare unofficial documentation such as Lucene In Action 2nd ed, section 4.7.1: "Lucene logically appends the tokens...sequentially."
>
> Having multi-valued fields stay in the order in which they were added to the Document is a guarantee that many many users depend on.
>
> James Dyer
> E-Commerce Systems
> Ingram Content Group
> (615) 213-4311
>
>
> -----Original Message-----
> From: Jack Krupansky [mailto:jack@basetechnology.com]
> Sent: Tuesday, December 18, 2012 9:30 AM
> To: solr-user@lucene.apache.org
> Subject: Re: "order" question on solr multi value field
>
> If there is no "official" guarantee in the Javadoc for the code then there
> is no official guarantee. Period. If somebody wants an official, contractual
> guarantee, a Jira should be filed to do so. To put it simple, are the values
> a "list" or a "set"?
>
> -- Jack Krupansky
>
> -----Original Message-----
> From: Erik Hatcher
> Sent: Tuesday, December 18, 2012 9:40 AM
> To: solr-user@lucene.apache.org
> Cc: solr-user@lucene.apache.org
> Subject: Re: "order" question on solr multi value field
>
> I don't know of an "official" guarantee of maintaining order but it's
> definitely guaranteed an relied upon to retain order. Many will scream if
> this changed.
>
> Indexed doesn't matter here because what you get back are the stored values
> no matter if the field is indexed or not.
>
> Erik
>
> On Dec 18, 2012, at 3:04, hellorsanjeev <sa...@3pillarglobal.com>
> wrote:
>
>> thank you for quick response :)
>>
>> I also have the same observation and I too believe that there is no reason
>> for Solr to reorder a multi value field.
>>
>> But would you stay firm on your conclusion if I say that my multi value
>> field was indexed?
>>
>> Please note - as per my one year experience with Solr, it always returned
>> the values in the insertions order irrespective of the fact that field was
>> indexed or not.
>>
>> My main concern is because I couldn't find it documented anywhere, it
>> might
>> happen that in Solr 4.0 or later, they start reordering them. If they do
>> then there will be a big problem for us :)
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/order-question-on-solr-multi-value-field-tp4027695p4027713.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
RE: "order" question on solr multi value field
Posted by "Dyer, James" <Ja...@ingramcontent.com>.
I would say such a guarantee is implied by the javadoc to Analyzer#getPositionIncrementGap . It says this value is an "increment" to be "added to the next token emitted from tokenStream."
http://lucene.apache.org/core/4_0_0-ALPHA/core/org/apache/lucene/analysis/Analyzer.html#getPositionIncrementGap%28java.lang.String%29
Also compare unofficial documentation such as Lucene In Action 2nd ed, section 4.7.1: "Lucene logically appends the tokens...sequentially."
Having multi-valued fields stay in the order in which they were added to the Document is a guarantee that many many users depend on.
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: Jack Krupansky [mailto:jack@basetechnology.com]
Sent: Tuesday, December 18, 2012 9:30 AM
To: solr-user@lucene.apache.org
Subject: Re: "order" question on solr multi value field
If there is no "official" guarantee in the Javadoc for the code then there
is no official guarantee. Period. If somebody wants an official, contractual
guarantee, a Jira should be filed to do so. To put it simple, are the values
a "list" or a "set"?
-- Jack Krupansky
-----Original Message-----
From: Erik Hatcher
Sent: Tuesday, December 18, 2012 9:40 AM
To: solr-user@lucene.apache.org
Cc: solr-user@lucene.apache.org
Subject: Re: "order" question on solr multi value field
I don't know of an "official" guarantee of maintaining order but it's
definitely guaranteed an relied upon to retain order. Many will scream if
this changed.
Indexed doesn't matter here because what you get back are the stored values
no matter if the field is indexed or not.
Erik
On Dec 18, 2012, at 3:04, hellorsanjeev <sa...@3pillarglobal.com>
wrote:
> thank you for quick response :)
>
> I also have the same observation and I too believe that there is no reason
> for Solr to reorder a multi value field.
>
> But would you stay firm on your conclusion if I say that my multi value
> field was indexed?
>
> Please note - as per my one year experience with Solr, it always returned
> the values in the insertions order irrespective of the fact that field was
> indexed or not.
>
> My main concern is because I couldn't find it documented anywhere, it
> might
> happen that in Solr 4.0 or later, they start reordering them. If they do
> then there will be a big problem for us :)
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/order-question-on-solr-multi-value-field-tp4027695p4027713.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: "order" question on solr multi value field
Posted by Jack Krupansky <ja...@basetechnology.com>.
If there is no "official" guarantee in the Javadoc for the code then there
is no official guarantee. Period. If somebody wants an official, contractual
guarantee, a Jira should be filed to do so. To put it simple, are the values
a "list" or a "set"?
-- Jack Krupansky
-----Original Message-----
From: Erik Hatcher
Sent: Tuesday, December 18, 2012 9:40 AM
To: solr-user@lucene.apache.org
Cc: solr-user@lucene.apache.org
Subject: Re: "order" question on solr multi value field
I don't know of an "official" guarantee of maintaining order but it's
definitely guaranteed an relied upon to retain order. Many will scream if
this changed.
Indexed doesn't matter here because what you get back are the stored values
no matter if the field is indexed or not.
Erik
On Dec 18, 2012, at 3:04, hellorsanjeev <sa...@3pillarglobal.com>
wrote:
> thank you for quick response :)
>
> I also have the same observation and I too believe that there is no reason
> for Solr to reorder a multi value field.
>
> But would you stay firm on your conclusion if I say that my multi value
> field was indexed?
>
> Please note - as per my one year experience with Solr, it always returned
> the values in the insertions order irrespective of the fact that field was
> indexed or not.
>
> My main concern is because I couldn't find it documented anywhere, it
> might
> happen that in Solr 4.0 or later, they start reordering them. If they do
> then there will be a big problem for us :)
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/order-question-on-solr-multi-value-field-tp4027695p4027713.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: "order" question on solr multi value field
Posted by Erik Hatcher <er...@gmail.com>.
I don't know of an "official" guarantee of maintaining order but it's definitely guaranteed an relied upon to retain order. Many will scream if this changed.
Indexed doesn't matter here because what you get back are the stored values no matter if the field is indexed or not.
Erik
On Dec 18, 2012, at 3:04, hellorsanjeev <sa...@3pillarglobal.com> wrote:
> thank you for quick response :)
>
> I also have the same observation and I too believe that there is no reason
> for Solr to reorder a multi value field.
>
> But would you stay firm on your conclusion if I say that my multi value
> field was indexed?
>
> Please note - as per my one year experience with Solr, it always returned
> the values in the insertions order irrespective of the fact that field was
> indexed or not.
>
> My main concern is because I couldn't find it documented anywhere, it might
> happen that in Solr 4.0 or later, they start reordering them. If they do
> then there will be a big problem for us :)
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/order-question-on-solr-multi-value-field-tp4027695p4027713.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: "order" question on solr multi value field
Posted by hellorsanjeev <sa...@3pillarglobal.com>.
thank you for quick response :)
I also have the same observation and I too believe that there is no reason
for Solr to reorder a multi value field.
But would you stay firm on your conclusion if I say that my multi value
field was indexed?
Please note - as per my one year experience with Solr, it always returned
the values in the insertions order irrespective of the fact that field was
indexed or not.
My main concern is because I couldn't find it documented anywhere, it might
happen that in Solr 4.0 or later, they start reordering them. If they do
then there will be a big problem for us :)
--
View this message in context: http://lucene.472066.n3.nabble.com/order-question-on-solr-multi-value-field-tp4027695p4027713.html
Sent from the Solr - User mailing list archive at Nabble.com.