You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Daniel Alheiros <Da...@bbc.co.uk> on 2007/11/28 14:41:45 UTC
SOLR 1.2 - Updates sent containing fields that are not on the
Schema fail silently
Hi
I experienced a very unpleasant problem recently, when my search indexing
adaptor was changed to add some new fields. The problem is my schema didn't
follow those changes (new fields added), and after that SOLR was silently
ignoring all documents I sent.
Neither SOLR Java client or SOLR server returned me an error code or log
message. In the server side, nothing was logged and the client received a
standard success return.
Why didn't my documents got indexed and this new fields were just ignored?
That is what I think it was supposed to do.
Please let me know your thoughts.
Regards,
Daniel
http://www.bbc.co.uk/
This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.
Re: SOLR 1.2 - Updates sent containing fields that are not on the Schema fail silently
Posted by Ravish Bhagdev <ra...@gmail.com>.
Yup, I do remember that happening to me before.
Is this intentionally so?
Ravish
On Nov 28, 2007 1:41 PM, Daniel Alheiros <Da...@bbc.co.uk> wrote:
> Hi
>
> I experienced a very unpleasant problem recently, when my search indexing
> adaptor was changed to add some new fields. The problem is my schema didn't
> follow those changes (new fields added), and after that SOLR was silently
> ignoring all documents I sent.
>
> Neither SOLR Java client or SOLR server returned me an error code or log
> message. In the server side, nothing was logged and the client received a
> standard success return.
>
> Why didn't my documents got indexed and this new fields were just ignored?
> That is what I think it was supposed to do.
>
> Please let me know your thoughts.
>
> Regards,
> Daniel
>
>
> http://www.bbc.co.uk/
> This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
> If you have received it in error, please delete it from your system.
> Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
> Please note that the BBC monitors e-mails sent or received.
> Further communication will signify your consent to this.
>
>
Re: SOLR 1.2 - Updates sent containing fields that are not on the
Schema fail silently
Posted by Ryan McKinley <ry...@gmail.com>.
Daniel Alheiros wrote:
> Hi Hoss.
>
> I'm using Solr 1.2 and a SolrJ client built from the trunk some time ago
> (21st of June 2007).
>
One thing I just thought of.... do you have a request handler defined at:
<requestHandler name="/update" class="solr.XmlUpdateRequestHandler" >
If not, it uses a legacy 1.1 format that returns a 200 response code and
puts the error in the response body. Solrj does not handle this legacy
format.
Take a look at the example solrconfig.xml
http://svn.apache.org/repos/asf/lucene/solr/tags/release-1.2.0/example/solr/conf/solrconfig.xml
If that does not fix things, can you try sending a document with post.sh
or post.jar? Does this get an error?
I am confident that solrj propagates errors to the client, but for
debugging this will be easier if we can find the simplest possible test
case.
ryan
Re: SOLR 1.2 - Updates sent containing fields that are not on the
Schema fail silently
Posted by Daniel Alheiros <Da...@bbc.co.uk>.
Hi Hoss.
I'm using Solr 1.2 and a SolrJ client built from the trunk some time ago
(21st of June 2007).
When a document is indexed I can see that INFO message on my logs showing
exactly what you said, but nothing is logged in this situation I've
described initially.
I'm using this logging conf:
handlers=java.util.logging.FileHandler
.level=INFO
java.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter
java.util.logging.FileHandler.pattern=/tmp/log/solr.log
java.util.logging.FileHandler.limit=10000000
java.util.logging.FileHandler.count=10
java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter
Thanks for your help.
Regards,
Daniel
On 30/11/07 02:13, "Chris Hostetter" <ho...@fucit.org> wrote:
>
> : Exactly, you can try it really easily, just remove one of your fields on the
> : example schema config and try to add content using the Java client API...
> : Well I'm using SOLRJ and it returns no error code for me. But anyway don't
> : you think the server should also have some logging informing that documents
> : are being discarded?
>
> As someone who is not very familiar with SolrJ, I can imagine that perhaps
> it has a bug where it might not return an error code in situations like
> this (it would suprise me, but i can imagine it) however I'm really
> confused by your comment that the server isn't logging that documents are
> being discarded. If you try to index a document with a field SOlr doesn't
> recognize, it logs quite a big exception. This is easily reproducable
> using post.jar and the example schema (unchanged). Running this
> command...
>
> java -Ddata=args -jar post.jar '<add><doc><field
> name="hoss">hoss</field></doc></add>'
>
> ...triggers this log messages in Solr...
>
> Nov 29, 2007 6:09:28 PM org.apache.solr.common.SolrException log
> SEVERE: org.apache.solr.common.SolrException: ERROR:unknown field 'hoss'
> at
> org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:245)
> at
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProces
> sorFactory.java:66)
> at
> org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate(XmlUpdateRequest
> Handler.java:196)
> ...
>
>
> ...which leads me to suspect there's something wonky with your setup.
>
> exactly which version of Solr are you using, what does your SolrJ code
> look like, and what log messages do you see when a document is
> *successfully* indexed? you should see somehting like...
>
> INFO: {add=[SOLR1000]} 0 102
>
> ...where the uniqueKey of your doc is in the []. If you don't see those
> messages, then you aren't looking in the right place for Solr's log
> messages.
>
>
>
>
>
> -Hoss
>
http://www.bbc.co.uk/
This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.
Re: SOLR 1.2 - Updates sent containing fields that are not on the
Schema fail silently
Posted by Chris Hostetter <ho...@fucit.org>.
: Exactly, you can try it really easily, just remove one of your fields on the
: example schema config and try to add content using the Java client API...
: Well I'm using SOLRJ and it returns no error code for me. But anyway don't
: you think the server should also have some logging informing that documents
: are being discarded?
As someone who is not very familiar with SolrJ, I can imagine that perhaps
it has a bug where it might not return an error code in situations like
this (it would suprise me, but i can imagine it) however I'm really
confused by your comment that the server isn't logging that documents are
being discarded. If you try to index a document with a field SOlr doesn't
recognize, it logs quite a big exception. This is easily reproducable
using post.jar and the example schema (unchanged). Running this
command...
java -Ddata=args -jar post.jar '<add><doc><field name="hoss">hoss</field></doc></add>'
...triggers this log messages in Solr...
Nov 29, 2007 6:09:28 PM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: ERROR:unknown field 'hoss'
at org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:245)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:66)
at org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate(XmlUpdateRequestHandler.java:196)
...
...which leads me to suspect there's something wonky with your setup.
exactly which version of Solr are you using, what does your SolrJ code
look like, and what log messages do you see when a document is
*successfully* indexed? you should see somehting like...
INFO: {add=[SOLR1000]} 0 102
...where the uniqueKey of your doc is in the []. If you don't see those
messages, then you aren't looking in the right place for Solr's log
messages.
-Hoss
Re: SOLR 1.2 - Updates sent containing fields that are not on the
Schema fail silently
Posted by Ryan McKinley <ry...@gmail.com>.
To be clear: solr *should* fail with an error if you send an unknown
field.
I just tested this with a clean checkout of 1.3-dev and 1.2 and in both
cases I get an error 400 "unknown field 'asgasdgasgd'"
The suggestion to look at the "ignore option" is to make sure you don't
have one -- this should be the only to add an arbitrary unknown field
without an error.
From a clean 1.2/1.3-dev install, how can you reproduce the error?
I tried:
$ ant example
$ cd example/
$ java -jar start.jar
another terminal:
edit mem.xml to add: <field name="asgasdgasgd">5</field>
$ cd example/exampledocs
$ ./post.sh mem.xml
this gives:
<body><h2>HTTP ERROR: 400</h2><pre>ERROR:unknown field 'asgasdgasgd'</pre>
running either 1.2 or 1.3
ryan
Daniel Alheiros wrote:
> Hi Hoss.
>
> Well I'll enable this ignore options for fields that aren't declared in my
> schema. Thanks.
>
> Exactly, you can try it really easily, just remove one of your fields on the
> example schema config and try to add content using the Java client API...
> Well I'm using SOLRJ and it returns no error code for me. But anyway don't
> you think the server should also have some logging informing that documents
> are being discarded?
>
> Cheers,
> Daniel
>
>
> On 28/11/07 19:25, "Chris Hostetter" <ho...@fucit.org> wrote:
>
>> : I didn't know that trick.
>>
>> erik is refering to this in the example schema.xml...
>>
>> <!-- uncomment the following to ignore any fields that don't already match
>> an existing
>> field name or dynamic field, rather than reporting them as an error.
>> alternately, change the type="ignored" to some other type e.g. "text"
>> if you want
>> unknown fields indexed and/or stored by default -->
>> <!--dynamicField name="*" type="ignored" /-->
>>
>> ...but it sounds like you are having some other problem ... you said that
>> when you POST your documents with "extra" fields you get a 200
>> response but the documents aren't getting indexed at all correct?
>>
>> that is not suppose to happen, Solr should be generating an error. can
>> you give us more info on your setup: what does your schema.xml look like,
>> what does your update code look like (you said you were using SolrJ i
>> believe?) what does Solr log when these updates happen, etc...
>>
>>
>>
>> -Hoss
>>
>
>
> http://www.bbc.co.uk/
> This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
> If you have received it in error, please delete it from your system.
> Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
> Please note that the BBC monitors e-mails sent or received.
> Further communication will signify your consent to this.
>
>
Re: SOLR 1.2 - Updates sent containing fields that are not on the
Schema fail silently
Posted by Daniel Alheiros <Da...@bbc.co.uk>.
Hi Hoss.
Well I'll enable this ignore options for fields that aren't declared in my
schema. Thanks.
Exactly, you can try it really easily, just remove one of your fields on the
example schema config and try to add content using the Java client API...
Well I'm using SOLRJ and it returns no error code for me. But anyway don't
you think the server should also have some logging informing that documents
are being discarded?
Cheers,
Daniel
On 28/11/07 19:25, "Chris Hostetter" <ho...@fucit.org> wrote:
>
> : I didn't know that trick.
>
> erik is refering to this in the example schema.xml...
>
> <!-- uncomment the following to ignore any fields that don't already match
> an existing
> field name or dynamic field, rather than reporting them as an error.
> alternately, change the type="ignored" to some other type e.g. "text"
> if you want
> unknown fields indexed and/or stored by default -->
> <!--dynamicField name="*" type="ignored" /-->
>
> ...but it sounds like you are having some other problem ... you said that
> when you POST your documents with "extra" fields you get a 200
> response but the documents aren't getting indexed at all correct?
>
> that is not suppose to happen, Solr should be generating an error. can
> you give us more info on your setup: what does your schema.xml look like,
> what does your update code look like (you said you were using SolrJ i
> believe?) what does Solr log when these updates happen, etc...
>
>
>
> -Hoss
>
http://www.bbc.co.uk/
This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.
Re: SOLR 1.2 - Updates sent containing fields that are not on the
Schema fail silently
Posted by Chris Hostetter <ho...@fucit.org>.
: I didn't know that trick.
erik is refering to this in the example schema.xml...
<!-- uncomment the following to ignore any fields that don't already match an existing
field name or dynamic field, rather than reporting them as an error.
alternately, change the type="ignored" to some other type e.g. "text" if you want
unknown fields indexed and/or stored by default -->
<!--dynamicField name="*" type="ignored" /-->
...but it sounds like you are having some other problem ... you said that
when you POST your documents with "extra" fields you get a 200
response but the documents aren't getting indexed at all correct?
that is not suppose to happen, Solr should be generating an error. can
you give us more info on your setup: what does your schema.xml look like,
what does your update code look like (you said you were using SolrJ i
believe?) what does Solr log when these updates happen, etc...
-Hoss
Re: SOLR 1.2 - Updates sent containing fields that are not on the
Schema fail silently
Posted by Daniel Alheiros <Da...@bbc.co.uk>.
I didn't know that trick.
Could you point me this documentation?
Anyway, don't you think that is something wrong in discarding all documents
without any warning? It's returning 200 return code without any other
content on the SOLRJ response to updates and don't log anything on the
server side...
Regards,
Daniel
On 28/11/07 15:40, "Erik Hatcher" <er...@ehatchersolutions.com> wrote:
>
> On Nov 28, 2007, at 8:41 AM, Daniel Alheiros wrote:
>> I experienced a very unpleasant problem recently, when my search
>> indexing
>> adaptor was changed to add some new fields. The problem is my
>> schema didn't
>> follow those changes (new fields added), and after that SOLR was
>> silently
>> ignoring all documents I sent.
>
> Is your schema perhaps configured to ignore undefined fields?
>
> Erik
>
http://www.bbc.co.uk/
This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.
Re: SOLR 1.2 - Updates sent containing fields that are not on the Schema fail silently
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Nov 28, 2007, at 8:41 AM, Daniel Alheiros wrote:
> I experienced a very unpleasant problem recently, when my search
> indexing
> adaptor was changed to add some new fields. The problem is my
> schema didn't
> follow those changes (new fields added), and after that SOLR was
> silently
> ignoring all documents I sent.
Is your schema perhaps configured to ignore undefined fields?
Erik