You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Chris Ulicny <cu...@iq.media> on 2017/04/25 19:40:02 UTC
Atomic Updates
Hello all,
Suppose I have the following fields in a document and populate all 4 fields
for every document.
id: uniqueKey, indexed and stored
integer_field: indexed and stored
text_field: indexed and stored
othertext_field: indexed but not stored
No default values, multivalues, docvalues, copyfields, or any other
properties set.
If I make an atomic update to a document like the following:
{"id":"1", "integer_field":{set: "1000"}}
what should we expect to happen with the othertext_field?
In a few tests, it seemed like the original indexed values of the record
were preserved.
I know to use atomic updates all fields should be stored since the document
is read and reindexed internally, but was curious if there was any
consistency or expected results for the state of othertext_field after an
atomic update.
Thanks,
Chris
Re: Atomic Updates
Posted by Erick Erickson <er...@gmail.com>.
Been there, done that, got the t-shirt. Thanks for closing it out!
Erick
On Thu, Apr 27, 2017 at 10:29 AM, Chris Ulicny <cu...@iq.media> wrote:
> While recreating it with a fresh schema, I realized that this was a case of
> a very, very stupid user error during configuring the cores.
>
> I setup the testing cores with the wrong configset, and then proceeded to
> edit the schema in the right configset. So, the field was actually stored
> by default, but I wasn't attempting to retrieve it so I never realized it
> was being stored since I ended up looking at the wrong schema.
>
> I switched the config sets and everything works as expected. Any atomic
> updates clear out the indexed values for the non-stored field.
>
> Thanks for bearing with me.
> Chris
>
>
> On Thu, Apr 27, 2017 at 11:23 AM Chris Ulicny <cu...@iq.media> wrote:
>
>> I'm sending commit=true with every update while testing. I'll write up the
>> tests and see if someone else can reproduce it.
>>
>> On Thu, Apr 27, 2017 at 10:54 AM Erick Erickson <er...@gmail.com>
>> wrote:
>>
>>> bq: but is there any possibility that the values stick around until
>>> there is a segment merge for some strange reason
>>>
>>> There better not be or it's a bug. Things will stick around until
>>> you issue a commit, is there any chance that's the problem?
>>>
>>> If you can document the exact steps, maybe we can reproduce
>>> the issue and raise a JIRA.
>>>
>>> Best,
>>> Erick
>>>
>>> On Thu, Apr 27, 2017 at 6:03 AM, Chris Ulicny <cu...@iq.media> wrote:
>>> > Yeah, something's not quite right somewhere. We never even considered
>>> > in-place updates an option since it requires the fields to be
>>> non-indexed
>>> > and non-stored. Our schemas never have any field that satisfies those
>>> two
>>> > conditions let alone the other necessary ones.
>>> >
>>> > I went ahead and tested the atomic updates on different textfields, and
>>> I
>>> > still can't get the indexed but not-stored othertext_field to
>>> disappear. So
>>> > far set, add, and remove updates do not change it regardless of what the
>>> > fields are in the atomic update.
>>> >
>>> > It would be extraordinarily useful if this update behavior is now
>>> expected
>>> > (but not currently documented) functionality.
>>> >
>>> > I'm not too familiar with the nitty-gritty details of merging segment
>>> > files, but is there any possibility that the values stick around until
>>> > there is a segment merge for some strange reason?
>>> >
>>> > On Thu, Apr 27, 2017 at 12:59 AM Dorian Hoxha <do...@gmail.com>
>>> > wrote:
>>> >
>>> >> @Chris,
>>> >> According to doc-link-above, only INC,SET are in-place-updates. And
>>> only
>>> >> when they're not indexed/stored, while your 'integer-field' is. So
>>> still
>>> >> shenanigans in there somewhere (docs,your-code,your-test,solr-code).
>>> >>
>>> >> On Thu, Apr 27, 2017 at 2:04 AM, Chris Ulicny <cu...@iq.media>
>>> wrote:
>>> >>
>>> >> > That's probably it then. None of the atomic updates that I've tried
>>> have
>>> >> > been on TextFields. I'll give the TextField atomic update to verify
>>> that
>>> >> it
>>> >> > will clear the other field.
>>> >> >
>>> >> > Has this functionality been consistent since atomic updates were
>>> >> > introduced, or is this a side effect of some other change? It'd be
>>> very
>>> >> > convenient for us to use this functionality as it currently works,
>>> but if
>>> >> > it's something that prevents us from upgrading versions in the
>>> future, we
>>> >> > should probably avoid expecting it to work.
>>> >> >
>>> >> > On Wed, Apr 26, 2017 at 7:36 PM Ishan Chattopadhyaya <
>>> >> > ichattopadhyaya@gmail.com> wrote:
>>> >> >
>>> >> > > > Hmm, interesting. I can imagine that as long as you're updating
>>> >> > > > docValues fields, the other_text field would be there. But the
>>> >> instant
>>> >> > > > you updated a non-docValues field (text_field in your example)
>>> the
>>> >> > > > other_text field would disappear
>>> >> > >
>>> >> > > I can confirm this. When in-place updates to DV fields are done,
>>> the
>>> >> rest
>>> >> > > of the fields remain as they were.
>>> >> > >
>>> >> > > On Thu, Apr 27, 2017 at 4:33 AM, Erick Erickson <
>>> >> erickerickson@gmail.com
>>> >> > >
>>> >> > > wrote:
>>> >> > >
>>> >> > > > Hmm, interesting. I can imagine that as long as you're updating
>>> >> > > > docValues fields, the other_text field would be there. But the
>>> >> instant
>>> >> > > > you updated a non-docValues field (text_field in your example)
>>> the
>>> >> > > > other_text field would disappear.
>>> >> > > >
>>> >> > > > I DO NOT KNOW this for a fact, but I'm asking people who do.
>>> >> > > >
>>> >> > > > On Wed, Apr 26, 2017 at 2:13 PM, Dorian Hoxha <
>>> >> dorian.hoxha@gmail.com>
>>> >> > > > wrote:
>>> >> > > > > There are In Place Updates, but according to docs they stll
>>> >> shouldn't
>>> >> > > > work
>>> >> > > > > in your case:
>>> >> > > > > https://cwiki.apache.org/confluence/display/solr/
>>> >> > > > Updating+Parts+of+Documents
>>> >> > > > >
>>> >> > > > > On Wed, Apr 26, 2017 at 10:36 PM, Chris Ulicny
>>> <cu...@iq.media>
>>> >> > > wrote:
>>> >> > > > >
>>> >> > > > >> That's the thing I'm curious about though. As I mentioned in
>>> the
>>> >> > first
>>> >> > > > >> post, I've already tried a few tests, and the value seems to
>>> still
>>> >> > be
>>> >> > > > >> present after an atomic update.
>>> >> > > > >>
>>> >> > > > >> I haven't exhausted all possible atomic updates, but 'set' and
>>> >> 'add'
>>> >> > > > seem
>>> >> > > > >> to preserve the non-stored text field.
>>> >> > > > >>
>>> >> > > > >> Thanks,
>>> >> > > > >> Chris
>>> >> > > > >>
>>> >> > > > >> On Wed, Apr 26, 2017 at 4:07 PM Dorian Hoxha <
>>> >> > dorian.hoxha@gmail.com>
>>> >> > > > >> wrote:
>>> >> > > > >>
>>> >> > > > >> > You'll lose the data in that field. Try doing a commit and
>>> it
>>> >> > should
>>> >> > > > >> > happen.
>>> >> > > > >> >
>>> >> > > > >> > On Wed, Apr 26, 2017 at 9:50 PM, Chris Ulicny
>>> <culicny@iq.media
>>> >> >
>>> >> > > > wrote:
>>> >> > > > >> >
>>> >> > > > >> > > Thanks Shawn, I didn't realize docValues were enabled by
>>> >> default
>>> >> > > > now.
>>> >> > > > >> > > That's very convenient and probably makes a lot of the
>>> schemas
>>> >> > > we've
>>> >> > > > >> been
>>> >> > > > >> > > making excessively verbose.
>>> >> > > > >> > >
>>> >> > > > >> > > This is on 6.3.0. Do you know what the first version was
>>> that
>>> >> > they
>>> >> > > > >> added
>>> >> > > > >> > > the docValues by default for non-Text field?
>>> >> > > > >> > >
>>> >> > > > >> > > However, that shouldn't apply to this since I'm concerned
>>> >> with a
>>> >> > > > >> > non-stored
>>> >> > > > >> > > TextField without docValues enabled.
>>> >> > > > >> > >
>>> >> > > > >> > > Best,
>>> >> > > > >> > > Chris
>>> >> > > > >> > >
>>> >> > > > >> > > On Wed, Apr 26, 2017 at 3:36 PM Shawn Heisey <
>>> >> > apache@elyograg.org
>>> >> > > >
>>> >> > > > >> > wrote:
>>> >> > > > >> > >
>>> >> > > > >> > > > On 4/25/2017 1:40 PM, Chris Ulicny wrote:
>>> >> > > > >> > > > > Hello all,
>>> >> > > > >> > > > >
>>> >> > > > >> > > > > Suppose I have the following fields in a document and
>>> >> > populate
>>> >> > > > all
>>> >> > > > >> 4
>>> >> > > > >> > > > fields
>>> >> > > > >> > > > > for every document.
>>> >> > > > >> > > > >
>>> >> > > > >> > > > > id: uniqueKey, indexed and stored
>>> >> > > > >> > > > > integer_field: indexed and stored
>>> >> > > > >> > > > > text_field: indexed and stored
>>> >> > > > >> > > > > othertext_field: indexed but not stored
>>> >> > > > >> > > > >
>>> >> > > > >> > > > > No default values, multivalues, docvalues,
>>> copyfields, or
>>> >> > any
>>> >> > > > other
>>> >> > > > >> > > > > properties set.
>>> >> > > > >> > > >
>>> >> > > > >> > > > You didn't indicate the Solr version. In recent Solr
>>> >> > versions,
>>> >> > > > most
>>> >> > > > >> > > > field classes other than TextField have docValues
>>> enabled by
>>> >> > > > default,
>>> >> > > > >> > > > even if the config is not mentioned on the field, and in
>>> >> those
>>> >> > > > >> > versions,
>>> >> > > > >> > > > docValues will take the place of stored if stored is
>>> false.
>>> >> > > > >> > > >
>>> >> > > > >> > > > Thanks,
>>> >> > > > >> > > > Shawn
>>> >> > > > >> > > >
>>> >> > > > >> > > >
>>> >> > > > >> > >
>>> >> > > > >> >
>>> >> > > > >>
>>> >> > > >
>>> >> > >
>>> >> >
>>> >>
>>>
>>
Re: Atomic Updates
Posted by Chris Ulicny <cu...@iq.media>.
While recreating it with a fresh schema, I realized that this was a case of
a very, very stupid user error during configuring the cores.
I setup the testing cores with the wrong configset, and then proceeded to
edit the schema in the right configset. So, the field was actually stored
by default, but I wasn't attempting to retrieve it so I never realized it
was being stored since I ended up looking at the wrong schema.
I switched the config sets and everything works as expected. Any atomic
updates clear out the indexed values for the non-stored field.
Thanks for bearing with me.
Chris
On Thu, Apr 27, 2017 at 11:23 AM Chris Ulicny <cu...@iq.media> wrote:
> I'm sending commit=true with every update while testing. I'll write up the
> tests and see if someone else can reproduce it.
>
> On Thu, Apr 27, 2017 at 10:54 AM Erick Erickson <er...@gmail.com>
> wrote:
>
>> bq: but is there any possibility that the values stick around until
>> there is a segment merge for some strange reason
>>
>> There better not be or it's a bug. Things will stick around until
>> you issue a commit, is there any chance that's the problem?
>>
>> If you can document the exact steps, maybe we can reproduce
>> the issue and raise a JIRA.
>>
>> Best,
>> Erick
>>
>> On Thu, Apr 27, 2017 at 6:03 AM, Chris Ulicny <cu...@iq.media> wrote:
>> > Yeah, something's not quite right somewhere. We never even considered
>> > in-place updates an option since it requires the fields to be
>> non-indexed
>> > and non-stored. Our schemas never have any field that satisfies those
>> two
>> > conditions let alone the other necessary ones.
>> >
>> > I went ahead and tested the atomic updates on different textfields, and
>> I
>> > still can't get the indexed but not-stored othertext_field to
>> disappear. So
>> > far set, add, and remove updates do not change it regardless of what the
>> > fields are in the atomic update.
>> >
>> > It would be extraordinarily useful if this update behavior is now
>> expected
>> > (but not currently documented) functionality.
>> >
>> > I'm not too familiar with the nitty-gritty details of merging segment
>> > files, but is there any possibility that the values stick around until
>> > there is a segment merge for some strange reason?
>> >
>> > On Thu, Apr 27, 2017 at 12:59 AM Dorian Hoxha <do...@gmail.com>
>> > wrote:
>> >
>> >> @Chris,
>> >> According to doc-link-above, only INC,SET are in-place-updates. And
>> only
>> >> when they're not indexed/stored, while your 'integer-field' is. So
>> still
>> >> shenanigans in there somewhere (docs,your-code,your-test,solr-code).
>> >>
>> >> On Thu, Apr 27, 2017 at 2:04 AM, Chris Ulicny <cu...@iq.media>
>> wrote:
>> >>
>> >> > That's probably it then. None of the atomic updates that I've tried
>> have
>> >> > been on TextFields. I'll give the TextField atomic update to verify
>> that
>> >> it
>> >> > will clear the other field.
>> >> >
>> >> > Has this functionality been consistent since atomic updates were
>> >> > introduced, or is this a side effect of some other change? It'd be
>> very
>> >> > convenient for us to use this functionality as it currently works,
>> but if
>> >> > it's something that prevents us from upgrading versions in the
>> future, we
>> >> > should probably avoid expecting it to work.
>> >> >
>> >> > On Wed, Apr 26, 2017 at 7:36 PM Ishan Chattopadhyaya <
>> >> > ichattopadhyaya@gmail.com> wrote:
>> >> >
>> >> > > > Hmm, interesting. I can imagine that as long as you're updating
>> >> > > > docValues fields, the other_text field would be there. But the
>> >> instant
>> >> > > > you updated a non-docValues field (text_field in your example)
>> the
>> >> > > > other_text field would disappear
>> >> > >
>> >> > > I can confirm this. When in-place updates to DV fields are done,
>> the
>> >> rest
>> >> > > of the fields remain as they were.
>> >> > >
>> >> > > On Thu, Apr 27, 2017 at 4:33 AM, Erick Erickson <
>> >> erickerickson@gmail.com
>> >> > >
>> >> > > wrote:
>> >> > >
>> >> > > > Hmm, interesting. I can imagine that as long as you're updating
>> >> > > > docValues fields, the other_text field would be there. But the
>> >> instant
>> >> > > > you updated a non-docValues field (text_field in your example)
>> the
>> >> > > > other_text field would disappear.
>> >> > > >
>> >> > > > I DO NOT KNOW this for a fact, but I'm asking people who do.
>> >> > > >
>> >> > > > On Wed, Apr 26, 2017 at 2:13 PM, Dorian Hoxha <
>> >> dorian.hoxha@gmail.com>
>> >> > > > wrote:
>> >> > > > > There are In Place Updates, but according to docs they stll
>> >> shouldn't
>> >> > > > work
>> >> > > > > in your case:
>> >> > > > > https://cwiki.apache.org/confluence/display/solr/
>> >> > > > Updating+Parts+of+Documents
>> >> > > > >
>> >> > > > > On Wed, Apr 26, 2017 at 10:36 PM, Chris Ulicny
>> <cu...@iq.media>
>> >> > > wrote:
>> >> > > > >
>> >> > > > >> That's the thing I'm curious about though. As I mentioned in
>> the
>> >> > first
>> >> > > > >> post, I've already tried a few tests, and the value seems to
>> still
>> >> > be
>> >> > > > >> present after an atomic update.
>> >> > > > >>
>> >> > > > >> I haven't exhausted all possible atomic updates, but 'set' and
>> >> 'add'
>> >> > > > seem
>> >> > > > >> to preserve the non-stored text field.
>> >> > > > >>
>> >> > > > >> Thanks,
>> >> > > > >> Chris
>> >> > > > >>
>> >> > > > >> On Wed, Apr 26, 2017 at 4:07 PM Dorian Hoxha <
>> >> > dorian.hoxha@gmail.com>
>> >> > > > >> wrote:
>> >> > > > >>
>> >> > > > >> > You'll lose the data in that field. Try doing a commit and
>> it
>> >> > should
>> >> > > > >> > happen.
>> >> > > > >> >
>> >> > > > >> > On Wed, Apr 26, 2017 at 9:50 PM, Chris Ulicny
>> <culicny@iq.media
>> >> >
>> >> > > > wrote:
>> >> > > > >> >
>> >> > > > >> > > Thanks Shawn, I didn't realize docValues were enabled by
>> >> default
>> >> > > > now.
>> >> > > > >> > > That's very convenient and probably makes a lot of the
>> schemas
>> >> > > we've
>> >> > > > >> been
>> >> > > > >> > > making excessively verbose.
>> >> > > > >> > >
>> >> > > > >> > > This is on 6.3.0. Do you know what the first version was
>> that
>> >> > they
>> >> > > > >> added
>> >> > > > >> > > the docValues by default for non-Text field?
>> >> > > > >> > >
>> >> > > > >> > > However, that shouldn't apply to this since I'm concerned
>> >> with a
>> >> > > > >> > non-stored
>> >> > > > >> > > TextField without docValues enabled.
>> >> > > > >> > >
>> >> > > > >> > > Best,
>> >> > > > >> > > Chris
>> >> > > > >> > >
>> >> > > > >> > > On Wed, Apr 26, 2017 at 3:36 PM Shawn Heisey <
>> >> > apache@elyograg.org
>> >> > > >
>> >> > > > >> > wrote:
>> >> > > > >> > >
>> >> > > > >> > > > On 4/25/2017 1:40 PM, Chris Ulicny wrote:
>> >> > > > >> > > > > Hello all,
>> >> > > > >> > > > >
>> >> > > > >> > > > > Suppose I have the following fields in a document and
>> >> > populate
>> >> > > > all
>> >> > > > >> 4
>> >> > > > >> > > > fields
>> >> > > > >> > > > > for every document.
>> >> > > > >> > > > >
>> >> > > > >> > > > > id: uniqueKey, indexed and stored
>> >> > > > >> > > > > integer_field: indexed and stored
>> >> > > > >> > > > > text_field: indexed and stored
>> >> > > > >> > > > > othertext_field: indexed but not stored
>> >> > > > >> > > > >
>> >> > > > >> > > > > No default values, multivalues, docvalues,
>> copyfields, or
>> >> > any
>> >> > > > other
>> >> > > > >> > > > > properties set.
>> >> > > > >> > > >
>> >> > > > >> > > > You didn't indicate the Solr version. In recent Solr
>> >> > versions,
>> >> > > > most
>> >> > > > >> > > > field classes other than TextField have docValues
>> enabled by
>> >> > > > default,
>> >> > > > >> > > > even if the config is not mentioned on the field, and in
>> >> those
>> >> > > > >> > versions,
>> >> > > > >> > > > docValues will take the place of stored if stored is
>> false.
>> >> > > > >> > > >
>> >> > > > >> > > > Thanks,
>> >> > > > >> > > > Shawn
>> >> > > > >> > > >
>> >> > > > >> > > >
>> >> > > > >> > >
>> >> > > > >> >
>> >> > > > >>
>> >> > > >
>> >> > >
>> >> >
>> >>
>>
>
Re: Atomic Updates
Posted by Chris Ulicny <cu...@iq.media>.
I'm sending commit=true with every update while testing. I'll write up the
tests and see if someone else can reproduce it.
On Thu, Apr 27, 2017 at 10:54 AM Erick Erickson <er...@gmail.com>
wrote:
> bq: but is there any possibility that the values stick around until
> there is a segment merge for some strange reason
>
> There better not be or it's a bug. Things will stick around until
> you issue a commit, is there any chance that's the problem?
>
> If you can document the exact steps, maybe we can reproduce
> the issue and raise a JIRA.
>
> Best,
> Erick
>
> On Thu, Apr 27, 2017 at 6:03 AM, Chris Ulicny <cu...@iq.media> wrote:
> > Yeah, something's not quite right somewhere. We never even considered
> > in-place updates an option since it requires the fields to be non-indexed
> > and non-stored. Our schemas never have any field that satisfies those two
> > conditions let alone the other necessary ones.
> >
> > I went ahead and tested the atomic updates on different textfields, and I
> > still can't get the indexed but not-stored othertext_field to disappear.
> So
> > far set, add, and remove updates do not change it regardless of what the
> > fields are in the atomic update.
> >
> > It would be extraordinarily useful if this update behavior is now
> expected
> > (but not currently documented) functionality.
> >
> > I'm not too familiar with the nitty-gritty details of merging segment
> > files, but is there any possibility that the values stick around until
> > there is a segment merge for some strange reason?
> >
> > On Thu, Apr 27, 2017 at 12:59 AM Dorian Hoxha <do...@gmail.com>
> > wrote:
> >
> >> @Chris,
> >> According to doc-link-above, only INC,SET are in-place-updates. And only
> >> when they're not indexed/stored, while your 'integer-field' is. So still
> >> shenanigans in there somewhere (docs,your-code,your-test,solr-code).
> >>
> >> On Thu, Apr 27, 2017 at 2:04 AM, Chris Ulicny <cu...@iq.media> wrote:
> >>
> >> > That's probably it then. None of the atomic updates that I've tried
> have
> >> > been on TextFields. I'll give the TextField atomic update to verify
> that
> >> it
> >> > will clear the other field.
> >> >
> >> > Has this functionality been consistent since atomic updates were
> >> > introduced, or is this a side effect of some other change? It'd be
> very
> >> > convenient for us to use this functionality as it currently works,
> but if
> >> > it's something that prevents us from upgrading versions in the
> future, we
> >> > should probably avoid expecting it to work.
> >> >
> >> > On Wed, Apr 26, 2017 at 7:36 PM Ishan Chattopadhyaya <
> >> > ichattopadhyaya@gmail.com> wrote:
> >> >
> >> > > > Hmm, interesting. I can imagine that as long as you're updating
> >> > > > docValues fields, the other_text field would be there. But the
> >> instant
> >> > > > you updated a non-docValues field (text_field in your example) the
> >> > > > other_text field would disappear
> >> > >
> >> > > I can confirm this. When in-place updates to DV fields are done, the
> >> rest
> >> > > of the fields remain as they were.
> >> > >
> >> > > On Thu, Apr 27, 2017 at 4:33 AM, Erick Erickson <
> >> erickerickson@gmail.com
> >> > >
> >> > > wrote:
> >> > >
> >> > > > Hmm, interesting. I can imagine that as long as you're updating
> >> > > > docValues fields, the other_text field would be there. But the
> >> instant
> >> > > > you updated a non-docValues field (text_field in your example) the
> >> > > > other_text field would disappear.
> >> > > >
> >> > > > I DO NOT KNOW this for a fact, but I'm asking people who do.
> >> > > >
> >> > > > On Wed, Apr 26, 2017 at 2:13 PM, Dorian Hoxha <
> >> dorian.hoxha@gmail.com>
> >> > > > wrote:
> >> > > > > There are In Place Updates, but according to docs they stll
> >> shouldn't
> >> > > > work
> >> > > > > in your case:
> >> > > > > https://cwiki.apache.org/confluence/display/solr/
> >> > > > Updating+Parts+of+Documents
> >> > > > >
> >> > > > > On Wed, Apr 26, 2017 at 10:36 PM, Chris Ulicny <culicny@iq.media
> >
> >> > > wrote:
> >> > > > >
> >> > > > >> That's the thing I'm curious about though. As I mentioned in
> the
> >> > first
> >> > > > >> post, I've already tried a few tests, and the value seems to
> still
> >> > be
> >> > > > >> present after an atomic update.
> >> > > > >>
> >> > > > >> I haven't exhausted all possible atomic updates, but 'set' and
> >> 'add'
> >> > > > seem
> >> > > > >> to preserve the non-stored text field.
> >> > > > >>
> >> > > > >> Thanks,
> >> > > > >> Chris
> >> > > > >>
> >> > > > >> On Wed, Apr 26, 2017 at 4:07 PM Dorian Hoxha <
> >> > dorian.hoxha@gmail.com>
> >> > > > >> wrote:
> >> > > > >>
> >> > > > >> > You'll lose the data in that field. Try doing a commit and it
> >> > should
> >> > > > >> > happen.
> >> > > > >> >
> >> > > > >> > On Wed, Apr 26, 2017 at 9:50 PM, Chris Ulicny
> <culicny@iq.media
> >> >
> >> > > > wrote:
> >> > > > >> >
> >> > > > >> > > Thanks Shawn, I didn't realize docValues were enabled by
> >> default
> >> > > > now.
> >> > > > >> > > That's very convenient and probably makes a lot of the
> schemas
> >> > > we've
> >> > > > >> been
> >> > > > >> > > making excessively verbose.
> >> > > > >> > >
> >> > > > >> > > This is on 6.3.0. Do you know what the first version was
> that
> >> > they
> >> > > > >> added
> >> > > > >> > > the docValues by default for non-Text field?
> >> > > > >> > >
> >> > > > >> > > However, that shouldn't apply to this since I'm concerned
> >> with a
> >> > > > >> > non-stored
> >> > > > >> > > TextField without docValues enabled.
> >> > > > >> > >
> >> > > > >> > > Best,
> >> > > > >> > > Chris
> >> > > > >> > >
> >> > > > >> > > On Wed, Apr 26, 2017 at 3:36 PM Shawn Heisey <
> >> > apache@elyograg.org
> >> > > >
> >> > > > >> > wrote:
> >> > > > >> > >
> >> > > > >> > > > On 4/25/2017 1:40 PM, Chris Ulicny wrote:
> >> > > > >> > > > > Hello all,
> >> > > > >> > > > >
> >> > > > >> > > > > Suppose I have the following fields in a document and
> >> > populate
> >> > > > all
> >> > > > >> 4
> >> > > > >> > > > fields
> >> > > > >> > > > > for every document.
> >> > > > >> > > > >
> >> > > > >> > > > > id: uniqueKey, indexed and stored
> >> > > > >> > > > > integer_field: indexed and stored
> >> > > > >> > > > > text_field: indexed and stored
> >> > > > >> > > > > othertext_field: indexed but not stored
> >> > > > >> > > > >
> >> > > > >> > > > > No default values, multivalues, docvalues, copyfields,
> or
> >> > any
> >> > > > other
> >> > > > >> > > > > properties set.
> >> > > > >> > > >
> >> > > > >> > > > You didn't indicate the Solr version. In recent Solr
> >> > versions,
> >> > > > most
> >> > > > >> > > > field classes other than TextField have docValues
> enabled by
> >> > > > default,
> >> > > > >> > > > even if the config is not mentioned on the field, and in
> >> those
> >> > > > >> > versions,
> >> > > > >> > > > docValues will take the place of stored if stored is
> false.
> >> > > > >> > > >
> >> > > > >> > > > Thanks,
> >> > > > >> > > > Shawn
> >> > > > >> > > >
> >> > > > >> > > >
> >> > > > >> > >
> >> > > > >> >
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >>
>
Re: Atomic Updates
Posted by Erick Erickson <er...@gmail.com>.
bq: but is there any possibility that the values stick around until
there is a segment merge for some strange reason
There better not be or it's a bug. Things will stick around until
you issue a commit, is there any chance that's the problem?
If you can document the exact steps, maybe we can reproduce
the issue and raise a JIRA.
Best,
Erick
On Thu, Apr 27, 2017 at 6:03 AM, Chris Ulicny <cu...@iq.media> wrote:
> Yeah, something's not quite right somewhere. We never even considered
> in-place updates an option since it requires the fields to be non-indexed
> and non-stored. Our schemas never have any field that satisfies those two
> conditions let alone the other necessary ones.
>
> I went ahead and tested the atomic updates on different textfields, and I
> still can't get the indexed but not-stored othertext_field to disappear. So
> far set, add, and remove updates do not change it regardless of what the
> fields are in the atomic update.
>
> It would be extraordinarily useful if this update behavior is now expected
> (but not currently documented) functionality.
>
> I'm not too familiar with the nitty-gritty details of merging segment
> files, but is there any possibility that the values stick around until
> there is a segment merge for some strange reason?
>
> On Thu, Apr 27, 2017 at 12:59 AM Dorian Hoxha <do...@gmail.com>
> wrote:
>
>> @Chris,
>> According to doc-link-above, only INC,SET are in-place-updates. And only
>> when they're not indexed/stored, while your 'integer-field' is. So still
>> shenanigans in there somewhere (docs,your-code,your-test,solr-code).
>>
>> On Thu, Apr 27, 2017 at 2:04 AM, Chris Ulicny <cu...@iq.media> wrote:
>>
>> > That's probably it then. None of the atomic updates that I've tried have
>> > been on TextFields. I'll give the TextField atomic update to verify that
>> it
>> > will clear the other field.
>> >
>> > Has this functionality been consistent since atomic updates were
>> > introduced, or is this a side effect of some other change? It'd be very
>> > convenient for us to use this functionality as it currently works, but if
>> > it's something that prevents us from upgrading versions in the future, we
>> > should probably avoid expecting it to work.
>> >
>> > On Wed, Apr 26, 2017 at 7:36 PM Ishan Chattopadhyaya <
>> > ichattopadhyaya@gmail.com> wrote:
>> >
>> > > > Hmm, interesting. I can imagine that as long as you're updating
>> > > > docValues fields, the other_text field would be there. But the
>> instant
>> > > > you updated a non-docValues field (text_field in your example) the
>> > > > other_text field would disappear
>> > >
>> > > I can confirm this. When in-place updates to DV fields are done, the
>> rest
>> > > of the fields remain as they were.
>> > >
>> > > On Thu, Apr 27, 2017 at 4:33 AM, Erick Erickson <
>> erickerickson@gmail.com
>> > >
>> > > wrote:
>> > >
>> > > > Hmm, interesting. I can imagine that as long as you're updating
>> > > > docValues fields, the other_text field would be there. But the
>> instant
>> > > > you updated a non-docValues field (text_field in your example) the
>> > > > other_text field would disappear.
>> > > >
>> > > > I DO NOT KNOW this for a fact, but I'm asking people who do.
>> > > >
>> > > > On Wed, Apr 26, 2017 at 2:13 PM, Dorian Hoxha <
>> dorian.hoxha@gmail.com>
>> > > > wrote:
>> > > > > There are In Place Updates, but according to docs they stll
>> shouldn't
>> > > > work
>> > > > > in your case:
>> > > > > https://cwiki.apache.org/confluence/display/solr/
>> > > > Updating+Parts+of+Documents
>> > > > >
>> > > > > On Wed, Apr 26, 2017 at 10:36 PM, Chris Ulicny <cu...@iq.media>
>> > > wrote:
>> > > > >
>> > > > >> That's the thing I'm curious about though. As I mentioned in the
>> > first
>> > > > >> post, I've already tried a few tests, and the value seems to still
>> > be
>> > > > >> present after an atomic update.
>> > > > >>
>> > > > >> I haven't exhausted all possible atomic updates, but 'set' and
>> 'add'
>> > > > seem
>> > > > >> to preserve the non-stored text field.
>> > > > >>
>> > > > >> Thanks,
>> > > > >> Chris
>> > > > >>
>> > > > >> On Wed, Apr 26, 2017 at 4:07 PM Dorian Hoxha <
>> > dorian.hoxha@gmail.com>
>> > > > >> wrote:
>> > > > >>
>> > > > >> > You'll lose the data in that field. Try doing a commit and it
>> > should
>> > > > >> > happen.
>> > > > >> >
>> > > > >> > On Wed, Apr 26, 2017 at 9:50 PM, Chris Ulicny <culicny@iq.media
>> >
>> > > > wrote:
>> > > > >> >
>> > > > >> > > Thanks Shawn, I didn't realize docValues were enabled by
>> default
>> > > > now.
>> > > > >> > > That's very convenient and probably makes a lot of the schemas
>> > > we've
>> > > > >> been
>> > > > >> > > making excessively verbose.
>> > > > >> > >
>> > > > >> > > This is on 6.3.0. Do you know what the first version was that
>> > they
>> > > > >> added
>> > > > >> > > the docValues by default for non-Text field?
>> > > > >> > >
>> > > > >> > > However, that shouldn't apply to this since I'm concerned
>> with a
>> > > > >> > non-stored
>> > > > >> > > TextField without docValues enabled.
>> > > > >> > >
>> > > > >> > > Best,
>> > > > >> > > Chris
>> > > > >> > >
>> > > > >> > > On Wed, Apr 26, 2017 at 3:36 PM Shawn Heisey <
>> > apache@elyograg.org
>> > > >
>> > > > >> > wrote:
>> > > > >> > >
>> > > > >> > > > On 4/25/2017 1:40 PM, Chris Ulicny wrote:
>> > > > >> > > > > Hello all,
>> > > > >> > > > >
>> > > > >> > > > > Suppose I have the following fields in a document and
>> > populate
>> > > > all
>> > > > >> 4
>> > > > >> > > > fields
>> > > > >> > > > > for every document.
>> > > > >> > > > >
>> > > > >> > > > > id: uniqueKey, indexed and stored
>> > > > >> > > > > integer_field: indexed and stored
>> > > > >> > > > > text_field: indexed and stored
>> > > > >> > > > > othertext_field: indexed but not stored
>> > > > >> > > > >
>> > > > >> > > > > No default values, multivalues, docvalues, copyfields, or
>> > any
>> > > > other
>> > > > >> > > > > properties set.
>> > > > >> > > >
>> > > > >> > > > You didn't indicate the Solr version. In recent Solr
>> > versions,
>> > > > most
>> > > > >> > > > field classes other than TextField have docValues enabled by
>> > > > default,
>> > > > >> > > > even if the config is not mentioned on the field, and in
>> those
>> > > > >> > versions,
>> > > > >> > > > docValues will take the place of stored if stored is false.
>> > > > >> > > >
>> > > > >> > > > Thanks,
>> > > > >> > > > Shawn
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>>
Re: Atomic Updates
Posted by Chris Ulicny <cu...@iq.media>.
Yeah, something's not quite right somewhere. We never even considered
in-place updates an option since it requires the fields to be non-indexed
and non-stored. Our schemas never have any field that satisfies those two
conditions let alone the other necessary ones.
I went ahead and tested the atomic updates on different textfields, and I
still can't get the indexed but not-stored othertext_field to disappear. So
far set, add, and remove updates do not change it regardless of what the
fields are in the atomic update.
It would be extraordinarily useful if this update behavior is now expected
(but not currently documented) functionality.
I'm not too familiar with the nitty-gritty details of merging segment
files, but is there any possibility that the values stick around until
there is a segment merge for some strange reason?
On Thu, Apr 27, 2017 at 12:59 AM Dorian Hoxha <do...@gmail.com>
wrote:
> @Chris,
> According to doc-link-above, only INC,SET are in-place-updates. And only
> when they're not indexed/stored, while your 'integer-field' is. So still
> shenanigans in there somewhere (docs,your-code,your-test,solr-code).
>
> On Thu, Apr 27, 2017 at 2:04 AM, Chris Ulicny <cu...@iq.media> wrote:
>
> > That's probably it then. None of the atomic updates that I've tried have
> > been on TextFields. I'll give the TextField atomic update to verify that
> it
> > will clear the other field.
> >
> > Has this functionality been consistent since atomic updates were
> > introduced, or is this a side effect of some other change? It'd be very
> > convenient for us to use this functionality as it currently works, but if
> > it's something that prevents us from upgrading versions in the future, we
> > should probably avoid expecting it to work.
> >
> > On Wed, Apr 26, 2017 at 7:36 PM Ishan Chattopadhyaya <
> > ichattopadhyaya@gmail.com> wrote:
> >
> > > > Hmm, interesting. I can imagine that as long as you're updating
> > > > docValues fields, the other_text field would be there. But the
> instant
> > > > you updated a non-docValues field (text_field in your example) the
> > > > other_text field would disappear
> > >
> > > I can confirm this. When in-place updates to DV fields are done, the
> rest
> > > of the fields remain as they were.
> > >
> > > On Thu, Apr 27, 2017 at 4:33 AM, Erick Erickson <
> erickerickson@gmail.com
> > >
> > > wrote:
> > >
> > > > Hmm, interesting. I can imagine that as long as you're updating
> > > > docValues fields, the other_text field would be there. But the
> instant
> > > > you updated a non-docValues field (text_field in your example) the
> > > > other_text field would disappear.
> > > >
> > > > I DO NOT KNOW this for a fact, but I'm asking people who do.
> > > >
> > > > On Wed, Apr 26, 2017 at 2:13 PM, Dorian Hoxha <
> dorian.hoxha@gmail.com>
> > > > wrote:
> > > > > There are In Place Updates, but according to docs they stll
> shouldn't
> > > > work
> > > > > in your case:
> > > > > https://cwiki.apache.org/confluence/display/solr/
> > > > Updating+Parts+of+Documents
> > > > >
> > > > > On Wed, Apr 26, 2017 at 10:36 PM, Chris Ulicny <cu...@iq.media>
> > > wrote:
> > > > >
> > > > >> That's the thing I'm curious about though. As I mentioned in the
> > first
> > > > >> post, I've already tried a few tests, and the value seems to still
> > be
> > > > >> present after an atomic update.
> > > > >>
> > > > >> I haven't exhausted all possible atomic updates, but 'set' and
> 'add'
> > > > seem
> > > > >> to preserve the non-stored text field.
> > > > >>
> > > > >> Thanks,
> > > > >> Chris
> > > > >>
> > > > >> On Wed, Apr 26, 2017 at 4:07 PM Dorian Hoxha <
> > dorian.hoxha@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > You'll lose the data in that field. Try doing a commit and it
> > should
> > > > >> > happen.
> > > > >> >
> > > > >> > On Wed, Apr 26, 2017 at 9:50 PM, Chris Ulicny <culicny@iq.media
> >
> > > > wrote:
> > > > >> >
> > > > >> > > Thanks Shawn, I didn't realize docValues were enabled by
> default
> > > > now.
> > > > >> > > That's very convenient and probably makes a lot of the schemas
> > > we've
> > > > >> been
> > > > >> > > making excessively verbose.
> > > > >> > >
> > > > >> > > This is on 6.3.0. Do you know what the first version was that
> > they
> > > > >> added
> > > > >> > > the docValues by default for non-Text field?
> > > > >> > >
> > > > >> > > However, that shouldn't apply to this since I'm concerned
> with a
> > > > >> > non-stored
> > > > >> > > TextField without docValues enabled.
> > > > >> > >
> > > > >> > > Best,
> > > > >> > > Chris
> > > > >> > >
> > > > >> > > On Wed, Apr 26, 2017 at 3:36 PM Shawn Heisey <
> > apache@elyograg.org
> > > >
> > > > >> > wrote:
> > > > >> > >
> > > > >> > > > On 4/25/2017 1:40 PM, Chris Ulicny wrote:
> > > > >> > > > > Hello all,
> > > > >> > > > >
> > > > >> > > > > Suppose I have the following fields in a document and
> > populate
> > > > all
> > > > >> 4
> > > > >> > > > fields
> > > > >> > > > > for every document.
> > > > >> > > > >
> > > > >> > > > > id: uniqueKey, indexed and stored
> > > > >> > > > > integer_field: indexed and stored
> > > > >> > > > > text_field: indexed and stored
> > > > >> > > > > othertext_field: indexed but not stored
> > > > >> > > > >
> > > > >> > > > > No default values, multivalues, docvalues, copyfields, or
> > any
> > > > other
> > > > >> > > > > properties set.
> > > > >> > > >
> > > > >> > > > You didn't indicate the Solr version. In recent Solr
> > versions,
> > > > most
> > > > >> > > > field classes other than TextField have docValues enabled by
> > > > default,
> > > > >> > > > even if the config is not mentioned on the field, and in
> those
> > > > >> > versions,
> > > > >> > > > docValues will take the place of stored if stored is false.
> > > > >> > > >
> > > > >> > > > Thanks,
> > > > >> > > > Shawn
> > > > >> > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
Re: Atomic Updates
Posted by Dorian Hoxha <do...@gmail.com>.
@Chris,
According to doc-link-above, only INC,SET are in-place-updates. And only
when they're not indexed/stored, while your 'integer-field' is. So still
shenanigans in there somewhere (docs,your-code,your-test,solr-code).
On Thu, Apr 27, 2017 at 2:04 AM, Chris Ulicny <cu...@iq.media> wrote:
> That's probably it then. None of the atomic updates that I've tried have
> been on TextFields. I'll give the TextField atomic update to verify that it
> will clear the other field.
>
> Has this functionality been consistent since atomic updates were
> introduced, or is this a side effect of some other change? It'd be very
> convenient for us to use this functionality as it currently works, but if
> it's something that prevents us from upgrading versions in the future, we
> should probably avoid expecting it to work.
>
> On Wed, Apr 26, 2017 at 7:36 PM Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
>
> > > Hmm, interesting. I can imagine that as long as you're updating
> > > docValues fields, the other_text field would be there. But the instant
> > > you updated a non-docValues field (text_field in your example) the
> > > other_text field would disappear
> >
> > I can confirm this. When in-place updates to DV fields are done, the rest
> > of the fields remain as they were.
> >
> > On Thu, Apr 27, 2017 at 4:33 AM, Erick Erickson <erickerickson@gmail.com
> >
> > wrote:
> >
> > > Hmm, interesting. I can imagine that as long as you're updating
> > > docValues fields, the other_text field would be there. But the instant
> > > you updated a non-docValues field (text_field in your example) the
> > > other_text field would disappear.
> > >
> > > I DO NOT KNOW this for a fact, but I'm asking people who do.
> > >
> > > On Wed, Apr 26, 2017 at 2:13 PM, Dorian Hoxha <do...@gmail.com>
> > > wrote:
> > > > There are In Place Updates, but according to docs they stll shouldn't
> > > work
> > > > in your case:
> > > > https://cwiki.apache.org/confluence/display/solr/
> > > Updating+Parts+of+Documents
> > > >
> > > > On Wed, Apr 26, 2017 at 10:36 PM, Chris Ulicny <cu...@iq.media>
> > wrote:
> > > >
> > > >> That's the thing I'm curious about though. As I mentioned in the
> first
> > > >> post, I've already tried a few tests, and the value seems to still
> be
> > > >> present after an atomic update.
> > > >>
> > > >> I haven't exhausted all possible atomic updates, but 'set' and 'add'
> > > seem
> > > >> to preserve the non-stored text field.
> > > >>
> > > >> Thanks,
> > > >> Chris
> > > >>
> > > >> On Wed, Apr 26, 2017 at 4:07 PM Dorian Hoxha <
> dorian.hoxha@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > You'll lose the data in that field. Try doing a commit and it
> should
> > > >> > happen.
> > > >> >
> > > >> > On Wed, Apr 26, 2017 at 9:50 PM, Chris Ulicny <cu...@iq.media>
> > > wrote:
> > > >> >
> > > >> > > Thanks Shawn, I didn't realize docValues were enabled by default
> > > now.
> > > >> > > That's very convenient and probably makes a lot of the schemas
> > we've
> > > >> been
> > > >> > > making excessively verbose.
> > > >> > >
> > > >> > > This is on 6.3.0. Do you know what the first version was that
> they
> > > >> added
> > > >> > > the docValues by default for non-Text field?
> > > >> > >
> > > >> > > However, that shouldn't apply to this since I'm concerned with a
> > > >> > non-stored
> > > >> > > TextField without docValues enabled.
> > > >> > >
> > > >> > > Best,
> > > >> > > Chris
> > > >> > >
> > > >> > > On Wed, Apr 26, 2017 at 3:36 PM Shawn Heisey <
> apache@elyograg.org
> > >
> > > >> > wrote:
> > > >> > >
> > > >> > > > On 4/25/2017 1:40 PM, Chris Ulicny wrote:
> > > >> > > > > Hello all,
> > > >> > > > >
> > > >> > > > > Suppose I have the following fields in a document and
> populate
> > > all
> > > >> 4
> > > >> > > > fields
> > > >> > > > > for every document.
> > > >> > > > >
> > > >> > > > > id: uniqueKey, indexed and stored
> > > >> > > > > integer_field: indexed and stored
> > > >> > > > > text_field: indexed and stored
> > > >> > > > > othertext_field: indexed but not stored
> > > >> > > > >
> > > >> > > > > No default values, multivalues, docvalues, copyfields, or
> any
> > > other
> > > >> > > > > properties set.
> > > >> > > >
> > > >> > > > You didn't indicate the Solr version. In recent Solr
> versions,
> > > most
> > > >> > > > field classes other than TextField have docValues enabled by
> > > default,
> > > >> > > > even if the config is not mentioned on the field, and in those
> > > >> > versions,
> > > >> > > > docValues will take the place of stored if stored is false.
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Shawn
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
Re: Atomic Updates
Posted by Chris Ulicny <cu...@iq.media>.
That's probably it then. None of the atomic updates that I've tried have
been on TextFields. I'll give the TextField atomic update to verify that it
will clear the other field.
Has this functionality been consistent since atomic updates were
introduced, or is this a side effect of some other change? It'd be very
convenient for us to use this functionality as it currently works, but if
it's something that prevents us from upgrading versions in the future, we
should probably avoid expecting it to work.
On Wed, Apr 26, 2017 at 7:36 PM Ishan Chattopadhyaya <
ichattopadhyaya@gmail.com> wrote:
> > Hmm, interesting. I can imagine that as long as you're updating
> > docValues fields, the other_text field would be there. But the instant
> > you updated a non-docValues field (text_field in your example) the
> > other_text field would disappear
>
> I can confirm this. When in-place updates to DV fields are done, the rest
> of the fields remain as they were.
>
> On Thu, Apr 27, 2017 at 4:33 AM, Erick Erickson <er...@gmail.com>
> wrote:
>
> > Hmm, interesting. I can imagine that as long as you're updating
> > docValues fields, the other_text field would be there. But the instant
> > you updated a non-docValues field (text_field in your example) the
> > other_text field would disappear.
> >
> > I DO NOT KNOW this for a fact, but I'm asking people who do.
> >
> > On Wed, Apr 26, 2017 at 2:13 PM, Dorian Hoxha <do...@gmail.com>
> > wrote:
> > > There are In Place Updates, but according to docs they stll shouldn't
> > work
> > > in your case:
> > > https://cwiki.apache.org/confluence/display/solr/
> > Updating+Parts+of+Documents
> > >
> > > On Wed, Apr 26, 2017 at 10:36 PM, Chris Ulicny <cu...@iq.media>
> wrote:
> > >
> > >> That's the thing I'm curious about though. As I mentioned in the first
> > >> post, I've already tried a few tests, and the value seems to still be
> > >> present after an atomic update.
> > >>
> > >> I haven't exhausted all possible atomic updates, but 'set' and 'add'
> > seem
> > >> to preserve the non-stored text field.
> > >>
> > >> Thanks,
> > >> Chris
> > >>
> > >> On Wed, Apr 26, 2017 at 4:07 PM Dorian Hoxha <do...@gmail.com>
> > >> wrote:
> > >>
> > >> > You'll lose the data in that field. Try doing a commit and it should
> > >> > happen.
> > >> >
> > >> > On Wed, Apr 26, 2017 at 9:50 PM, Chris Ulicny <cu...@iq.media>
> > wrote:
> > >> >
> > >> > > Thanks Shawn, I didn't realize docValues were enabled by default
> > now.
> > >> > > That's very convenient and probably makes a lot of the schemas
> we've
> > >> been
> > >> > > making excessively verbose.
> > >> > >
> > >> > > This is on 6.3.0. Do you know what the first version was that they
> > >> added
> > >> > > the docValues by default for non-Text field?
> > >> > >
> > >> > > However, that shouldn't apply to this since I'm concerned with a
> > >> > non-stored
> > >> > > TextField without docValues enabled.
> > >> > >
> > >> > > Best,
> > >> > > Chris
> > >> > >
> > >> > > On Wed, Apr 26, 2017 at 3:36 PM Shawn Heisey <apache@elyograg.org
> >
> > >> > wrote:
> > >> > >
> > >> > > > On 4/25/2017 1:40 PM, Chris Ulicny wrote:
> > >> > > > > Hello all,
> > >> > > > >
> > >> > > > > Suppose I have the following fields in a document and populate
> > all
> > >> 4
> > >> > > > fields
> > >> > > > > for every document.
> > >> > > > >
> > >> > > > > id: uniqueKey, indexed and stored
> > >> > > > > integer_field: indexed and stored
> > >> > > > > text_field: indexed and stored
> > >> > > > > othertext_field: indexed but not stored
> > >> > > > >
> > >> > > > > No default values, multivalues, docvalues, copyfields, or any
> > other
> > >> > > > > properties set.
> > >> > > >
> > >> > > > You didn't indicate the Solr version. In recent Solr versions,
> > most
> > >> > > > field classes other than TextField have docValues enabled by
> > default,
> > >> > > > even if the config is not mentioned on the field, and in those
> > >> > versions,
> > >> > > > docValues will take the place of stored if stored is false.
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Shawn
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
Re: Atomic Updates
Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
> Hmm, interesting. I can imagine that as long as you're updating
> docValues fields, the other_text field would be there. But the instant
> you updated a non-docValues field (text_field in your example) the
> other_text field would disappear
I can confirm this. When in-place updates to DV fields are done, the rest
of the fields remain as they were.
On Thu, Apr 27, 2017 at 4:33 AM, Erick Erickson <er...@gmail.com>
wrote:
> Hmm, interesting. I can imagine that as long as you're updating
> docValues fields, the other_text field would be there. But the instant
> you updated a non-docValues field (text_field in your example) the
> other_text field would disappear.
>
> I DO NOT KNOW this for a fact, but I'm asking people who do.
>
> On Wed, Apr 26, 2017 at 2:13 PM, Dorian Hoxha <do...@gmail.com>
> wrote:
> > There are In Place Updates, but according to docs they stll shouldn't
> work
> > in your case:
> > https://cwiki.apache.org/confluence/display/solr/
> Updating+Parts+of+Documents
> >
> > On Wed, Apr 26, 2017 at 10:36 PM, Chris Ulicny <cu...@iq.media> wrote:
> >
> >> That's the thing I'm curious about though. As I mentioned in the first
> >> post, I've already tried a few tests, and the value seems to still be
> >> present after an atomic update.
> >>
> >> I haven't exhausted all possible atomic updates, but 'set' and 'add'
> seem
> >> to preserve the non-stored text field.
> >>
> >> Thanks,
> >> Chris
> >>
> >> On Wed, Apr 26, 2017 at 4:07 PM Dorian Hoxha <do...@gmail.com>
> >> wrote:
> >>
> >> > You'll lose the data in that field. Try doing a commit and it should
> >> > happen.
> >> >
> >> > On Wed, Apr 26, 2017 at 9:50 PM, Chris Ulicny <cu...@iq.media>
> wrote:
> >> >
> >> > > Thanks Shawn, I didn't realize docValues were enabled by default
> now.
> >> > > That's very convenient and probably makes a lot of the schemas we've
> >> been
> >> > > making excessively verbose.
> >> > >
> >> > > This is on 6.3.0. Do you know what the first version was that they
> >> added
> >> > > the docValues by default for non-Text field?
> >> > >
> >> > > However, that shouldn't apply to this since I'm concerned with a
> >> > non-stored
> >> > > TextField without docValues enabled.
> >> > >
> >> > > Best,
> >> > > Chris
> >> > >
> >> > > On Wed, Apr 26, 2017 at 3:36 PM Shawn Heisey <ap...@elyograg.org>
> >> > wrote:
> >> > >
> >> > > > On 4/25/2017 1:40 PM, Chris Ulicny wrote:
> >> > > > > Hello all,
> >> > > > >
> >> > > > > Suppose I have the following fields in a document and populate
> all
> >> 4
> >> > > > fields
> >> > > > > for every document.
> >> > > > >
> >> > > > > id: uniqueKey, indexed and stored
> >> > > > > integer_field: indexed and stored
> >> > > > > text_field: indexed and stored
> >> > > > > othertext_field: indexed but not stored
> >> > > > >
> >> > > > > No default values, multivalues, docvalues, copyfields, or any
> other
> >> > > > > properties set.
> >> > > >
> >> > > > You didn't indicate the Solr version. In recent Solr versions,
> most
> >> > > > field classes other than TextField have docValues enabled by
> default,
> >> > > > even if the config is not mentioned on the field, and in those
> >> > versions,
> >> > > > docValues will take the place of stored if stored is false.
> >> > > >
> >> > > > Thanks,
> >> > > > Shawn
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
>
Re: Atomic Updates
Posted by Erick Erickson <er...@gmail.com>.
Hmm, interesting. I can imagine that as long as you're updating
docValues fields, the other_text field would be there. But the instant
you updated a non-docValues field (text_field in your example) the
other_text field would disappear.
I DO NOT KNOW this for a fact, but I'm asking people who do.
On Wed, Apr 26, 2017 at 2:13 PM, Dorian Hoxha <do...@gmail.com> wrote:
> There are In Place Updates, but according to docs they stll shouldn't work
> in your case:
> https://cwiki.apache.org/confluence/display/solr/Updating+Parts+of+Documents
>
> On Wed, Apr 26, 2017 at 10:36 PM, Chris Ulicny <cu...@iq.media> wrote:
>
>> That's the thing I'm curious about though. As I mentioned in the first
>> post, I've already tried a few tests, and the value seems to still be
>> present after an atomic update.
>>
>> I haven't exhausted all possible atomic updates, but 'set' and 'add' seem
>> to preserve the non-stored text field.
>>
>> Thanks,
>> Chris
>>
>> On Wed, Apr 26, 2017 at 4:07 PM Dorian Hoxha <do...@gmail.com>
>> wrote:
>>
>> > You'll lose the data in that field. Try doing a commit and it should
>> > happen.
>> >
>> > On Wed, Apr 26, 2017 at 9:50 PM, Chris Ulicny <cu...@iq.media> wrote:
>> >
>> > > Thanks Shawn, I didn't realize docValues were enabled by default now.
>> > > That's very convenient and probably makes a lot of the schemas we've
>> been
>> > > making excessively verbose.
>> > >
>> > > This is on 6.3.0. Do you know what the first version was that they
>> added
>> > > the docValues by default for non-Text field?
>> > >
>> > > However, that shouldn't apply to this since I'm concerned with a
>> > non-stored
>> > > TextField without docValues enabled.
>> > >
>> > > Best,
>> > > Chris
>> > >
>> > > On Wed, Apr 26, 2017 at 3:36 PM Shawn Heisey <ap...@elyograg.org>
>> > wrote:
>> > >
>> > > > On 4/25/2017 1:40 PM, Chris Ulicny wrote:
>> > > > > Hello all,
>> > > > >
>> > > > > Suppose I have the following fields in a document and populate all
>> 4
>> > > > fields
>> > > > > for every document.
>> > > > >
>> > > > > id: uniqueKey, indexed and stored
>> > > > > integer_field: indexed and stored
>> > > > > text_field: indexed and stored
>> > > > > othertext_field: indexed but not stored
>> > > > >
>> > > > > No default values, multivalues, docvalues, copyfields, or any other
>> > > > > properties set.
>> > > >
>> > > > You didn't indicate the Solr version. In recent Solr versions, most
>> > > > field classes other than TextField have docValues enabled by default,
>> > > > even if the config is not mentioned on the field, and in those
>> > versions,
>> > > > docValues will take the place of stored if stored is false.
>> > > >
>> > > > Thanks,
>> > > > Shawn
>> > > >
>> > > >
>> > >
>> >
>>
Re: Atomic Updates
Posted by Dorian Hoxha <do...@gmail.com>.
There are In Place Updates, but according to docs they stll shouldn't work
in your case:
https://cwiki.apache.org/confluence/display/solr/Updating+Parts+of+Documents
On Wed, Apr 26, 2017 at 10:36 PM, Chris Ulicny <cu...@iq.media> wrote:
> That's the thing I'm curious about though. As I mentioned in the first
> post, I've already tried a few tests, and the value seems to still be
> present after an atomic update.
>
> I haven't exhausted all possible atomic updates, but 'set' and 'add' seem
> to preserve the non-stored text field.
>
> Thanks,
> Chris
>
> On Wed, Apr 26, 2017 at 4:07 PM Dorian Hoxha <do...@gmail.com>
> wrote:
>
> > You'll lose the data in that field. Try doing a commit and it should
> > happen.
> >
> > On Wed, Apr 26, 2017 at 9:50 PM, Chris Ulicny <cu...@iq.media> wrote:
> >
> > > Thanks Shawn, I didn't realize docValues were enabled by default now.
> > > That's very convenient and probably makes a lot of the schemas we've
> been
> > > making excessively verbose.
> > >
> > > This is on 6.3.0. Do you know what the first version was that they
> added
> > > the docValues by default for non-Text field?
> > >
> > > However, that shouldn't apply to this since I'm concerned with a
> > non-stored
> > > TextField without docValues enabled.
> > >
> > > Best,
> > > Chris
> > >
> > > On Wed, Apr 26, 2017 at 3:36 PM Shawn Heisey <ap...@elyograg.org>
> > wrote:
> > >
> > > > On 4/25/2017 1:40 PM, Chris Ulicny wrote:
> > > > > Hello all,
> > > > >
> > > > > Suppose I have the following fields in a document and populate all
> 4
> > > > fields
> > > > > for every document.
> > > > >
> > > > > id: uniqueKey, indexed and stored
> > > > > integer_field: indexed and stored
> > > > > text_field: indexed and stored
> > > > > othertext_field: indexed but not stored
> > > > >
> > > > > No default values, multivalues, docvalues, copyfields, or any other
> > > > > properties set.
> > > >
> > > > You didn't indicate the Solr version. In recent Solr versions, most
> > > > field classes other than TextField have docValues enabled by default,
> > > > even if the config is not mentioned on the field, and in those
> > versions,
> > > > docValues will take the place of stored if stored is false.
> > > >
> > > > Thanks,
> > > > Shawn
> > > >
> > > >
> > >
> >
>
Re: Atomic Updates
Posted by Chris Ulicny <cu...@iq.media>.
That's the thing I'm curious about though. As I mentioned in the first
post, I've already tried a few tests, and the value seems to still be
present after an atomic update.
I haven't exhausted all possible atomic updates, but 'set' and 'add' seem
to preserve the non-stored text field.
Thanks,
Chris
On Wed, Apr 26, 2017 at 4:07 PM Dorian Hoxha <do...@gmail.com> wrote:
> You'll lose the data in that field. Try doing a commit and it should
> happen.
>
> On Wed, Apr 26, 2017 at 9:50 PM, Chris Ulicny <cu...@iq.media> wrote:
>
> > Thanks Shawn, I didn't realize docValues were enabled by default now.
> > That's very convenient and probably makes a lot of the schemas we've been
> > making excessively verbose.
> >
> > This is on 6.3.0. Do you know what the first version was that they added
> > the docValues by default for non-Text field?
> >
> > However, that shouldn't apply to this since I'm concerned with a
> non-stored
> > TextField without docValues enabled.
> >
> > Best,
> > Chris
> >
> > On Wed, Apr 26, 2017 at 3:36 PM Shawn Heisey <ap...@elyograg.org>
> wrote:
> >
> > > On 4/25/2017 1:40 PM, Chris Ulicny wrote:
> > > > Hello all,
> > > >
> > > > Suppose I have the following fields in a document and populate all 4
> > > fields
> > > > for every document.
> > > >
> > > > id: uniqueKey, indexed and stored
> > > > integer_field: indexed and stored
> > > > text_field: indexed and stored
> > > > othertext_field: indexed but not stored
> > > >
> > > > No default values, multivalues, docvalues, copyfields, or any other
> > > > properties set.
> > >
> > > You didn't indicate the Solr version. In recent Solr versions, most
> > > field classes other than TextField have docValues enabled by default,
> > > even if the config is not mentioned on the field, and in those
> versions,
> > > docValues will take the place of stored if stored is false.
> > >
> > > Thanks,
> > > Shawn
> > >
> > >
> >
>
Re: Atomic Updates
Posted by Dorian Hoxha <do...@gmail.com>.
You'll lose the data in that field. Try doing a commit and it should happen.
On Wed, Apr 26, 2017 at 9:50 PM, Chris Ulicny <cu...@iq.media> wrote:
> Thanks Shawn, I didn't realize docValues were enabled by default now.
> That's very convenient and probably makes a lot of the schemas we've been
> making excessively verbose.
>
> This is on 6.3.0. Do you know what the first version was that they added
> the docValues by default for non-Text field?
>
> However, that shouldn't apply to this since I'm concerned with a non-stored
> TextField without docValues enabled.
>
> Best,
> Chris
>
> On Wed, Apr 26, 2017 at 3:36 PM Shawn Heisey <ap...@elyograg.org> wrote:
>
> > On 4/25/2017 1:40 PM, Chris Ulicny wrote:
> > > Hello all,
> > >
> > > Suppose I have the following fields in a document and populate all 4
> > fields
> > > for every document.
> > >
> > > id: uniqueKey, indexed and stored
> > > integer_field: indexed and stored
> > > text_field: indexed and stored
> > > othertext_field: indexed but not stored
> > >
> > > No default values, multivalues, docvalues, copyfields, or any other
> > > properties set.
> >
> > You didn't indicate the Solr version. In recent Solr versions, most
> > field classes other than TextField have docValues enabled by default,
> > even if the config is not mentioned on the field, and in those versions,
> > docValues will take the place of stored if stored is false.
> >
> > Thanks,
> > Shawn
> >
> >
>
Re: Atomic Updates
Posted by Chris Ulicny <cu...@iq.media>.
Thanks Shawn, I didn't realize docValues were enabled by default now.
That's very convenient and probably makes a lot of the schemas we've been
making excessively verbose.
This is on 6.3.0. Do you know what the first version was that they added
the docValues by default for non-Text field?
However, that shouldn't apply to this since I'm concerned with a non-stored
TextField without docValues enabled.
Best,
Chris
On Wed, Apr 26, 2017 at 3:36 PM Shawn Heisey <ap...@elyograg.org> wrote:
> On 4/25/2017 1:40 PM, Chris Ulicny wrote:
> > Hello all,
> >
> > Suppose I have the following fields in a document and populate all 4
> fields
> > for every document.
> >
> > id: uniqueKey, indexed and stored
> > integer_field: indexed and stored
> > text_field: indexed and stored
> > othertext_field: indexed but not stored
> >
> > No default values, multivalues, docvalues, copyfields, or any other
> > properties set.
>
> You didn't indicate the Solr version. In recent Solr versions, most
> field classes other than TextField have docValues enabled by default,
> even if the config is not mentioned on the field, and in those versions,
> docValues will take the place of stored if stored is false.
>
> Thanks,
> Shawn
>
>
Re: Atomic Updates
Posted by Shawn Heisey <ap...@elyograg.org>.
On 4/25/2017 1:40 PM, Chris Ulicny wrote:
> Hello all,
>
> Suppose I have the following fields in a document and populate all 4 fields
> for every document.
>
> id: uniqueKey, indexed and stored
> integer_field: indexed and stored
> text_field: indexed and stored
> othertext_field: indexed but not stored
>
> No default values, multivalues, docvalues, copyfields, or any other
> properties set.
You didn't indicate the Solr version. In recent Solr versions, most
field classes other than TextField have docValues enabled by default,
even if the config is not mentioned on the field, and in those versions,
docValues will take the place of stored if stored is false.
Thanks,
Shawn
Re: Atomic Updates
Posted by Chris Ulicny <cu...@iq.media>.
All fields are being explicitly populated on the initial document load
without copyFields, and the atomic updates come after.
This situation actually came up while we were planning removing copyField
properties from one of the fields in a schema for a new collection.
On Tue, Apr 25, 2017 at 3:54 PM Erick Erickson <er...@gmail.com>
wrote:
> How is "otherText" getting values in the first place? If
> it's the destination of a copyField directive, it'll be repopulated
> if the source of the copyField is stored=true.
>
> Best,
> Erick
>
> On Tue, Apr 25, 2017 at 12:40 PM, Chris Ulicny <cu...@iq.media> wrote:
> > Hello all,
> >
> > Suppose I have the following fields in a document and populate all 4
> fields
> > for every document.
> >
> > id: uniqueKey, indexed and stored
> > integer_field: indexed and stored
> > text_field: indexed and stored
> > othertext_field: indexed but not stored
> >
> > No default values, multivalues, docvalues, copyfields, or any other
> > properties set.
> >
> > If I make an atomic update to a document like the following:
> > {"id":"1", "integer_field":{set: "1000"}}
> >
> > what should we expect to happen with the othertext_field?
> >
> > In a few tests, it seemed like the original indexed values of the record
> > were preserved.
> >
> > I know to use atomic updates all fields should be stored since the
> document
> > is read and reindexed internally, but was curious if there was any
> > consistency or expected results for the state of othertext_field after an
> > atomic update.
> >
> > Thanks,
> > Chris
>
Re: Atomic Updates
Posted by Erick Erickson <er...@gmail.com>.
How is "otherText" getting values in the first place? If
it's the destination of a copyField directive, it'll be repopulated
if the source of the copyField is stored=true.
Best,
Erick
On Tue, Apr 25, 2017 at 12:40 PM, Chris Ulicny <cu...@iq.media> wrote:
> Hello all,
>
> Suppose I have the following fields in a document and populate all 4 fields
> for every document.
>
> id: uniqueKey, indexed and stored
> integer_field: indexed and stored
> text_field: indexed and stored
> othertext_field: indexed but not stored
>
> No default values, multivalues, docvalues, copyfields, or any other
> properties set.
>
> If I make an atomic update to a document like the following:
> {"id":"1", "integer_field":{set: "1000"}}
>
> what should we expect to happen with the othertext_field?
>
> In a few tests, it seemed like the original indexed values of the record
> were preserved.
>
> I know to use atomic updates all fields should be stored since the document
> is read and reindexed internally, but was curious if there was any
> consistency or expected results for the state of othertext_field after an
> atomic update.
>
> Thanks,
> Chris