Few questions on updatable DocValues

[Ravikumar Govindarajan <ra...@gmail.com>]

Hi,

I have few questions related to updatable DocValues API... It would be
great if I can get help.

1. Is it possible to provide updateNumericDocValue(Term term,
Map<String,Long>), incase I wish to update multiple-fields and it's
doc-values?

2. Instead of a "Term" based update, is it possible to extend it to using a
"Query"? What are the obvious problems in doing so?

3. TrackingIndexWriter does not have updateNumericDocValue exposed. Any
reason of not doing so?

4. Is it possible to update a DocValue other than long, like lets say a
BinaryDV?

--
Ravi

Re: Few questions on updatable DocValues

[Shai Erera <se...@gmail.com>]

Double fields can be implemented today over NumericDVField and therefore
already support updates.

String can be implemented on Sorted/SortedSetDVField, but not updates for
them yet. I hope that once I'm done w/ LUCENE-5513, adding update support
for Sorted/SortedSet will be even easier.

Shai

On Fri, Mar 14, 2014 at 6:22 PM, Gopal Patwa <go...@gmail.com> wrote:

> r lot;s of use case where you have muc

Re: Few questions on updatable DocValues

[Gopal Patwa <go...@gmail.com>]

Hi Shai, Thanks for implementing this feature. I was wondering if we could
have another these 2 types for updatable docvalues 1) Double and 2) String
this could cover lot;s of use case where you have much higher update rate
than normal document update.





On Fri, Mar 14, 2014 at 4:20 AM, Shai Erera <se...@gmail.com> wrote:

> Hi
>
> 1. Is it possible to provide updateNumericDocValue(Term term,
> > Map<String,Long>), incase I wish to update multiple-fields and it's
> > doc-values?
> >
>
> For now you can call updateNDV multiple times, each time w/ a new field.
> Under the covers, we currently process each update separately anyway.
> I think in order to change it we'd need to change the API such that it
> allows you to define an update in many ways (e.g. Query, see below). Then,
> an update by a single Term to multiple fields is "atomic". I don't want
> though to add many updateNDV variants to IW, especially as we'd like to add
> more DV update capabilities. Want to open an issue to explore that?
>
> 2. Instead of a "Term" based update, is it possible to extend it to using a
> > "Query"? What are the obvious problems in doing so?
> >
>
> Technically yes, but currently it's not exposed. At the lowest level we
> pull a DocsEnum and iterate on docs to apply the update. So Term/Query
> would work the same. I think we can explore generalizing the API such that
> you can define your own update following some well thought of API, and that
> way you have the flexibility in one hand, yet we don't need to maintain all
> options in the Lucene source code. We can explore that on an issue.
>
> 3. TrackingIndexWriter does not have updateNumericDocValue exposed. Any
> > reason of not doing so?
> >
>
> No reason in particular :). Can you open an issue (separate from the API)?
>
> 4. Is it possible to update a DocValue other than long, like lets say a
> > BinaryDV?
> >
>
> This is something I currently do on LUCENE-5513, so hopefully very soon you
> will be able to do that. If I'm fast enough, maybe even in 4.8 :).
>
> Shai
>
>
> On Fri, Mar 14, 2014 at 12:14 PM, Ravikumar Govindarajan <
> ravikumar.govindarajan@gmail.com> wrote:
>
> > Hi,
> >
> > I have few questions related to updatable DocValues API... It would be
> > great if I can get help.
> >
> > 1. Is it possible to provide updateNumericDocValue(Term term,
> > Map<String,Long>), incase I wish to update multiple-fields and it's
> > doc-values?
> >
> > 2. Instead of a "Term" based update, is it possible to extend it to
> using a
> > "Query"? What are the obvious problems in doing so?
> >
> > 3. TrackingIndexWriter does not have updateNumericDocValue exposed. Any
> > reason of not doing so?
> >
> > 4. Is it possible to update a DocValue other than long, like lets say a
> > BinaryDV?
> >
> > --
> > Ravi
> >
>

Re: Few questions on updatable DocValues

[Shai Erera <se...@gmail.com>]

Hi

1. Is it possible to provide updateNumericDocValue(Term term,
> Map<String,Long>), incase I wish to update multiple-fields and it's
> doc-values?
>

For now you can call updateNDV multiple times, each time w/ a new field.
Under the covers, we currently process each update separately anyway.
I think in order to change it we'd need to change the API such that it
allows you to define an update in many ways (e.g. Query, see below). Then,
an update by a single Term to multiple fields is "atomic". I don't want
though to add many updateNDV variants to IW, especially as we'd like to add
more DV update capabilities. Want to open an issue to explore that?

2. Instead of a "Term" based update, is it possible to extend it to using a
> "Query"? What are the obvious problems in doing so?
>

Technically yes, but currently it's not exposed. At the lowest level we
pull a DocsEnum and iterate on docs to apply the update. So Term/Query
would work the same. I think we can explore generalizing the API such that
you can define your own update following some well thought of API, and that
way you have the flexibility in one hand, yet we don't need to maintain all
options in the Lucene source code. We can explore that on an issue.

3. TrackingIndexWriter does not have updateNumericDocValue exposed. Any
> reason of not doing so?
>

No reason in particular :). Can you open an issue (separate from the API)?

4. Is it possible to update a DocValue other than long, like lets say a
> BinaryDV?
>

This is something I currently do on LUCENE-5513, so hopefully very soon you
will be able to do that. If I'm fast enough, maybe even in 4.8 :).

Shai


On Fri, Mar 14, 2014 at 12:14 PM, Ravikumar Govindarajan <
ravikumar.govindarajan@gmail.com> wrote:

> Hi,
>
> I have few questions related to updatable DocValues API... It would be
> great if I can get help.
>
> 1. Is it possible to provide updateNumericDocValue(Term term,
> Map<String,Long>), incase I wish to update multiple-fields and it's
> doc-values?
>
> 2. Instead of a "Term" based update, is it possible to extend it to using a
> "Query"? What are the obvious problems in doing so?
>
> 3. TrackingIndexWriter does not have updateNumericDocValue exposed. Any
> reason of not doing so?
>
> 4. Is it possible to update a DocValue other than long, like lets say a
> BinaryDV?
>
> --
> Ravi
>