You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Yago Riveiro <ya...@gmail.com> on 2017/01/21 21:08:08 UTC

Streams return default values for fields that doesn't exist in the document

I'm trying to use the streaming API to reindex data from one collection to
another.

I have a lot of dynamic fields on my documents and not every document has
the same fields, therefore, to fetch the list if fields that exists in the
collection, I need to run a luke query to fetch all of them.

I ran the stream with the fl with all the fields returned by the luke query
an all documents returned has the same fields, so far so good.

The main problem is that the field doesn't exist in the returned document is
filled with the default value of the field type, not ok. 

If the field doesn't exist in the document the return value can't be the
default value, should be something that we can identify as "this field
doesn't exists in this document"

Right now I have a docs with 2 integer fields with value 0, one indeed
belongs to the document and was indexed with 0 as the correct value, the
other doesn't exists in the source document.

Why not return the value as null? when indexing a field with null value is
ignored, the reverse operation should returns the same ...



-----
Best regards

/Yago
--
View this message in context: http://lucene.472066.n3.nabble.com/Streams-return-default-values-for-fields-that-doesn-t-exist-in-the-document-tp4315229.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Streams return default values for fields that doesn't exist in the document

Posted by Joel Bernstein <jo...@gmail.com>.
Also take a look at:
http://joelsolr.blogspot.com/2016/10/solr-63-batch-jobs-parallel-etl-and.html

It describes a very flexible approach to do batch re-indexing jobs.

Joel Bernstein
http://joelsolr.blogspot.com/

On Sat, Jan 21, 2017 at 6:44 PM, Yago Riveiro <ya...@gmail.com>
wrote:

> 6.3.0
>
> I will try again with 6.4.0
>
> Thank Erick
>
> --
>
> /Yago Riveiro
>
> On 21 Jan 2017, 21:23 +0000, Erick Erickson <er...@gmail.com>,
> wrote:
> > What version of Solr? See: https://issues.apache.org/
> jira/browse/SOLR-9166
> >
> > Best,
> > Erick
> >
> > On Sat, Jan 21, 2017 at 1:08 PM, Yago Riveiro <ya...@gmail.com>
> wrote:
> > > I'm trying to use the streaming API to reindex data from one
> collection to
> > > another.
> > >
> > > I have a lot of dynamic fields on my documents and not every document
> has
> > > the same fields, therefore, to fetch the list if fields that exists in
> the
> > > collection, I need to run a luke query to fetch all of them.
> > >
> > > I ran the stream with the fl with all the fields returned by the luke
> query
> > > an all documents returned has the same fields, so far so good.
> > >
> > > The main problem is that the field doesn't exist in the returned
> document is
> > > filled with the default value of the field type, not ok.
> > >
> > > If the field doesn't exist in the document the return value can't be
> the
> > > default value, should be something that we can identify as "this field
> > > doesn't exists in this document"
> > >
> > > Right now I have a docs with 2 integer fields with value 0, one indeed
> > > belongs to the document and was indexed with 0 as the correct value,
> the
> > > other doesn't exists in the source document.
> > >
> > > Why not return the value as null? when indexing a field with null
> value is
> > > ignored, the reverse operation should returns the same ...
> > >
> > >
> > >
> > > -----
> > > Best regards
> > >
> > > /Yago
> > > --
> > > View this message in context: http://lucene.472066.n3.
> nabble.com/Streams-return-default-values-for-fields-
> that-doesn-t-exist-in-the-document-tp4315229.html
> > > Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Streams return default values for fields that doesn't exist in the document

Posted by Yago Riveiro <ya...@gmail.com>.
6.3.0

I will try again with 6.4.0

Thank Erick

--

/Yago Riveiro

On 21 Jan 2017, 21:23 +0000, Erick Erickson <er...@gmail.com>, wrote:
> What version of Solr? See: https://issues.apache.org/jira/browse/SOLR-9166
>
> Best,
> Erick
>
> On Sat, Jan 21, 2017 at 1:08 PM, Yago Riveiro <ya...@gmail.com> wrote:
> > I'm trying to use the streaming API to reindex data from one collection to
> > another.
> >
> > I have a lot of dynamic fields on my documents and not every document has
> > the same fields, therefore, to fetch the list if fields that exists in the
> > collection, I need to run a luke query to fetch all of them.
> >
> > I ran the stream with the fl with all the fields returned by the luke query
> > an all documents returned has the same fields, so far so good.
> >
> > The main problem is that the field doesn't exist in the returned document is
> > filled with the default value of the field type, not ok.
> >
> > If the field doesn't exist in the document the return value can't be the
> > default value, should be something that we can identify as "this field
> > doesn't exists in this document"
> >
> > Right now I have a docs with 2 integer fields with value 0, one indeed
> > belongs to the document and was indexed with 0 as the correct value, the
> > other doesn't exists in the source document.
> >
> > Why not return the value as null? when indexing a field with null value is
> > ignored, the reverse operation should returns the same ...
> >
> >
> >
> > -----
> > Best regards
> >
> > /Yago
> > --
> > View this message in context: http://lucene.472066.n3.nabble.com/Streams-return-default-values-for-fields-that-doesn-t-exist-in-the-document-tp4315229.html
> > Sent from the Solr - User mailing list archive at Nabble.com.

Re: Streams return default values for fields that doesn't exist in the document

Posted by Erick Erickson <er...@gmail.com>.
What version of Solr? See: https://issues.apache.org/jira/browse/SOLR-9166

Best,
Erick

On Sat, Jan 21, 2017 at 1:08 PM, Yago Riveiro <ya...@gmail.com> wrote:
> I'm trying to use the streaming API to reindex data from one collection to
> another.
>
> I have a lot of dynamic fields on my documents and not every document has
> the same fields, therefore, to fetch the list if fields that exists in the
> collection, I need to run a luke query to fetch all of them.
>
> I ran the stream with the fl with all the fields returned by the luke query
> an all documents returned has the same fields, so far so good.
>
> The main problem is that the field doesn't exist in the returned document is
> filled with the default value of the field type, not ok.
>
> If the field doesn't exist in the document the return value can't be the
> default value, should be something that we can identify as "this field
> doesn't exists in this document"
>
> Right now I have a docs with 2 integer fields with value 0, one indeed
> belongs to the document and was indexed with 0 as the correct value, the
> other doesn't exists in the source document.
>
> Why not return the value as null? when indexing a field with null value is
> ignored, the reverse operation should returns the same ...
>
>
>
> -----
> Best regards
>
> /Yago
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Streams-return-default-values-for-fields-that-doesn-t-exist-in-the-document-tp4315229.html
> Sent from the Solr - User mailing list archive at Nabble.com.