You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by AJ Lemke <aj...@securitylabs.com> on 2017/04/25 13:21:50 UTC
DIH Issues
Hey all,
We are using 6.3.0 and we have issues with DIH throwing errors. We are seeing an intermittent issue where on a full index a single error will be thrown. The error is always "missing required field: fieldname".
Our SQL database always has data in the field that comes up with the error. Most of the errors are coming on fields that SQL has marked as required.
Would anyone have any hints or ideas where to look to remedy this situation.
As always if you need more information let me know.
Thanks
AJ
Re: DIH Issues
Posted by Sales <in...@smallbusinessconsultingexperts.com>.
> On Apr 25, 2017, at 10:28 AM, AJ Lemke <aj...@securitylabs.com> wrote:
>
> Thanks for the thought Alex!
> The fields that have this happen most often are numeric and boolean fields. These fields have real data (id numbers, true/false, etc.)
>
> AJ
>
We had an identical problem a few months ago, and there was no question that the field was populated in all MySQL records. We figured out how to use another field in the schema to do the same query, so, ended up deleting the troublesome field. Never did discover why, all ideas failed. In our case, the same data populated 2 different fields, one worked, one did not, but, never found a good reason for that. I’d love to know if you figure it out, as, it could be the reason why ours did the same thing. Our is a much older version though. We figured it’s some sort of rare bug. We played around for several weeks. Hope you can find it.
Steve
Re: DIH Issues
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
I wonder if it is possible to write a component/URP/something that
will intercept exceptions like these and dump out full record.
Regards,
Alex.
----
http://www.solr-start.com/ - Resources for Solr users, new and experienced
On 25 April 2017 at 12:19, Erick Erickson <er...@gmail.com> wrote:
> You say your SQL database always has the values, but does the output
> from the SQL query you actually use have them? I've been fooled before
> by the query I form "somehow" doesn't have a value for all fields I
> expect.
>
> You could also crank the logging level up enough to see the docs that
> are indexed, although that would probably only confirm that the fields
> weren't in the docs which you know already, not tell you why they are
> missing. Pull the SQL out and run it independently perhaps?
>
> I sound a bit like a broken record, but this is why I like SolrJ, I
> can actually debug that:
> https://lucidworks.com/2012/02/14/indexing-with-solrj/
>
> Best,
> Erick
>
> On Tue, Apr 25, 2017 at 8:28 AM, AJ Lemke <aj...@securitylabs.com> wrote:
>> Thanks for the thought Alex!
>> The fields that have this happen most often are numeric and boolean fields. These fields have real data (id numbers, true/false, etc.)
>>
>> AJ
>>
>> -----Original Message-----
>> From: Alexandre Rafalovitch [mailto:arafalov@gmail.com]
>> Sent: Tuesday, April 25, 2017 8:27 AM
>> To: solr-user <so...@lucene.apache.org>
>> Subject: Re: DIH Issues
>>
>> Maybe the content gets simplified away between the database and the Solr schema. For example if your field contains just spaces and you have UpdateRequestProcessors to do trim and removal of empty fields?
>>
>> Schemaless mode will remove empty fields, but will not trim for example.
>>
>> Regards,
>> Alex.
>> ----
>> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>>
>>
>> On 25 April 2017 at 09:21, AJ Lemke <aj...@securitylabs.com> wrote:
>>> Hey all,
>>>
>>> We are using 6.3.0 and we have issues with DIH throwing errors. We are seeing an intermittent issue where on a full index a single error will be thrown. The error is always "missing required field: fieldname".
>>> Our SQL database always has data in the field that comes up with the error. Most of the errors are coming on fields that SQL has marked as required.
>>>
>>> Would anyone have any hints or ideas where to look to remedy this situation.
>>>
>>> As always if you need more information let me know.
>>>
>>> Thanks
>>> AJ
Re: DIH Issues
Posted by Erick Erickson <er...@gmail.com>.
You say your SQL database always has the values, but does the output
from the SQL query you actually use have them? I've been fooled before
by the query I form "somehow" doesn't have a value for all fields I
expect.
You could also crank the logging level up enough to see the docs that
are indexed, although that would probably only confirm that the fields
weren't in the docs which you know already, not tell you why they are
missing. Pull the SQL out and run it independently perhaps?
I sound a bit like a broken record, but this is why I like SolrJ, I
can actually debug that:
https://lucidworks.com/2012/02/14/indexing-with-solrj/
Best,
Erick
On Tue, Apr 25, 2017 at 8:28 AM, AJ Lemke <aj...@securitylabs.com> wrote:
> Thanks for the thought Alex!
> The fields that have this happen most often are numeric and boolean fields. These fields have real data (id numbers, true/false, etc.)
>
> AJ
>
> -----Original Message-----
> From: Alexandre Rafalovitch [mailto:arafalov@gmail.com]
> Sent: Tuesday, April 25, 2017 8:27 AM
> To: solr-user <so...@lucene.apache.org>
> Subject: Re: DIH Issues
>
> Maybe the content gets simplified away between the database and the Solr schema. For example if your field contains just spaces and you have UpdateRequestProcessors to do trim and removal of empty fields?
>
> Schemaless mode will remove empty fields, but will not trim for example.
>
> Regards,
> Alex.
> ----
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 25 April 2017 at 09:21, AJ Lemke <aj...@securitylabs.com> wrote:
>> Hey all,
>>
>> We are using 6.3.0 and we have issues with DIH throwing errors. We are seeing an intermittent issue where on a full index a single error will be thrown. The error is always "missing required field: fieldname".
>> Our SQL database always has data in the field that comes up with the error. Most of the errors are coming on fields that SQL has marked as required.
>>
>> Would anyone have any hints or ideas where to look to remedy this situation.
>>
>> As always if you need more information let me know.
>>
>> Thanks
>> AJ
RE: DIH Issues
Posted by AJ Lemke <aj...@securitylabs.com>.
Thanks for the thought Alex!
The fields that have this happen most often are numeric and boolean fields. These fields have real data (id numbers, true/false, etc.)
AJ
-----Original Message-----
From: Alexandre Rafalovitch [mailto:arafalov@gmail.com]
Sent: Tuesday, April 25, 2017 8:27 AM
To: solr-user <so...@lucene.apache.org>
Subject: Re: DIH Issues
Maybe the content gets simplified away between the database and the Solr schema. For example if your field contains just spaces and you have UpdateRequestProcessors to do trim and removal of empty fields?
Schemaless mode will remove empty fields, but will not trim for example.
Regards,
Alex.
----
http://www.solr-start.com/ - Resources for Solr users, new and experienced
On 25 April 2017 at 09:21, AJ Lemke <aj...@securitylabs.com> wrote:
> Hey all,
>
> We are using 6.3.0 and we have issues with DIH throwing errors. We are seeing an intermittent issue where on a full index a single error will be thrown. The error is always "missing required field: fieldname".
> Our SQL database always has data in the field that comes up with the error. Most of the errors are coming on fields that SQL has marked as required.
>
> Would anyone have any hints or ideas where to look to remedy this situation.
>
> As always if you need more information let me know.
>
> Thanks
> AJ
Re: DIH Issues
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Maybe the content gets simplified away between the database and the
Solr schema. For example if your field contains just spaces and you
have UpdateRequestProcessors to do trim and removal of empty fields?
Schemaless mode will remove empty fields, but will not trim for example.
Regards,
Alex.
----
http://www.solr-start.com/ - Resources for Solr users, new and experienced
On 25 April 2017 at 09:21, AJ Lemke <aj...@securitylabs.com> wrote:
> Hey all,
>
> We are using 6.3.0 and we have issues with DIH throwing errors. We are seeing an intermittent issue where on a full index a single error will be thrown. The error is always "missing required field: fieldname".
> Our SQL database always has data in the field that comes up with the error. Most of the errors are coming on fields that SQL has marked as required.
>
> Would anyone have any hints or ideas where to look to remedy this situation.
>
> As always if you need more information let me know.
>
> Thanks
> AJ