You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Dotan Cohen <do...@gmail.com> on 2012/11/01 01:42:10 UTC
After adding field to schema, the field is not being returned in results.
I had stopped Solr 4.0, added a new stored and indexed field to
schema.xml, and then restarted Solr. I see that my application is
adding documents and I can query for and retrieve documents just fine.
However, the new field is not being returned in the XML. I can see
that if I try to sort on the field I get no error, whereas if I try to
sort on a nonexistent filed I do get an error, so it seems that Solr
does recognize the new field as actually existing. However even for
new documents I cannot get the field added to my XML output from
queries to Solr, even if the field is explicitly requested with 'fl'.
What must I do to have Solr return this new field in the XML output?
Thanks.
--
Dotan Cohen
http://gibberish.co.il
http://what-is-what.com
Re: After adding field to schema, the field is not being returned
in results.
Posted by Carl Roberts <ca...@gmail.com>.
I too am running into what appears to be the same thing.
Everything works and data is imported but I cannot see the new field in
the result.
Re: After adding field to schema, the field is not being returned in results.
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
And - just to get stupid options out of the way - you don't have any
parameters defined on the handlers that may list the fields to return?
Regards,
Alex.
Personal blog: http://blog.outerthoughts.com/
LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
- Time is the quality of nature that keeps events from happening all at
once. Lately, it doesn't seem to be working. (Anonymous - via GTD book)
On Wed, Oct 31, 2012 at 8:42 PM, Dotan Cohen <do...@gmail.com> wrote:
> I had stopped Solr 4.0, added a new stored and indexed field to
> schema.xml, and then restarted Solr. I see that my application is
> adding documents and I can query for and retrieve documents just fine.
> However, the new field is not being returned in the XML. I can see
> that if I try to sort on the field I get no error, whereas if I try to
> sort on a nonexistent filed I do get an error, so it seems that Solr
> does recognize the new field as actually existing. However even for
> new documents I cannot get the field added to my XML output from
> queries to Solr, even if the field is explicitly requested with 'fl'.
>
> What must I do to have Solr return this new field in the XML output?
>
> Thanks.
>
> --
> Dotan Cohen
>
> http://gibberish.co.il
> http://what-is-what.com
>
Re: After adding field to schema, the field is not being returned in results.
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
If you can index stuff into a new schema for test, try defining one
with dynamicField name=* stored=true indexed=true type=string. Your
schema may have one like this commented out and/or set to false.
This would show you exactly what you are indexing and solve whether
you have any spelling or format issues.
In fact, I usually start my schema with just that one catch-all field
(well, plus id and _version_) and reindexing over and over with
incremental fields definitions.
If that gets you nothing then you are either not sending it or doing
some magic UpdateRequestProcessor to blow it away.
Regards,
Alex.
----
Sign up for my Solr resources newsletter at http://www.solr-start.com/
On 4 November 2012 at 13:15, Dotan Cohen <do...@gmail.com> wrote:
> On Sat, Nov 3, 2012 at 4:23 AM, Lance Norskog <go...@gmail.com> wrote:
>> If any value is in a bogus format, the entire document batch in that HTTP request fails. That is the right timestamp format.
>> The index may be corrupted somehow. Can you try removing all of the fields in data/ and trying again?
>>
>
> Thanks. Seeing how all the documents are being added, either there is
> a valid format in the created_iso8601 field or it is empty. I've
> pretty much ruled out empty in code, but still nothing in the index.
>
> I'll play around some more and update the list. At least I am learning.
>
>
> --
> Dotan Cohen
>
> http://gibberish.co.il
> http://what-is-what.com
Re: After adding field to schema, the field is not being returned in results.
Posted by Dotan Cohen <do...@gmail.com>.
On Sat, Nov 3, 2012 at 4:23 AM, Lance Norskog <go...@gmail.com> wrote:
> If any value is in a bogus format, the entire document batch in that HTTP request fails. That is the right timestamp format.
> The index may be corrupted somehow. Can you try removing all of the fields in data/ and trying again?
>
Thanks. Seeing how all the documents are being added, either there is
a valid format in the created_iso8601 field or it is empty. I've
pretty much ruled out empty in code, but still nothing in the index.
I'll play around some more and update the list. At least I am learning.
--
Dotan Cohen
http://gibberish.co.il
http://what-is-what.com
Re: After adding field to schema, the field is not being returned in results.
Posted by Dotan Cohen <do...@gmail.com>.
On Fri, Nov 2, 2012 at 4:32 PM, Erick Erickson <er...@gmail.com> wrote:
> Well, I'm at my wits end. I tried your field definitions (using the
> exampledocs XML) and they work just fine. As far as if you mess up the date
> on the way in, you should be seeing stack traces in your log files.
>
Please don't go to wit's end on this! I'm a bit frustrated too, but I
really don't want to bring the frustration to the mailing list!
> The only way I see not getting the "Sorry, no Term Info available :("
> message is if you don't have any values in the field. So, my guess is that
> you're not getting the format right and the docs aren't getting indexed,
> but that's just a guess. You can freely sort even if there are no values at
> all in a particular field. This can be indicated if you sort asc and desc
> and the order doesn't change. It just means the field is defined in the
> schema, not necessarily that there are any values in it.
>
> So, I claim you have no date values in your index. The fact that you can
> sort is just an artifact of sortMissingFirst/Last doing something sensible.
>
> Next question, are you absolutely sure that your indexing program and your
> searching program are pointing at the same server?
>
> So what I'd do next is
> 1> create a simple XML doc that conforms to your schema and use the
> post.jar tool to send it to your server. Watch the output log for any date
> format exceptions.
> 2> Use the admin UI to insure that you can see terms in docs added this way.
> 3> from there back up and see what step in the indexing process isn't
> working (assuming that's the problem). Solr logs help here.
>
> Note I'm completely PHP-ignorant, I have no clue whether the formatting
> you're doing is OK or not. You might try logging the value somewhere in
> your php so you an post that and/or include it in your sample XML file...
>
Yes, it seems that there is a contradiction. On one hand, by appending
the value of the created_iso8601 field to another field, I can ensure
that the value is legal and does exist! On the other hand, it seems
that there is no such value being stored in the index, but new
documents are being added that ostensibly should have that value.
I'll try adding a document with post.jar and see what happens. I'll
update the thread.
Thanks!
--
Dotan Cohen
http://gibberish.co.il
http://what-is-what.com
Re: After adding field to schema, the field is not being returned
in results.
Posted by Lance Norskog <go...@gmail.com>.
If any value is in a bogus format, the entire document batch in that HTTP request fails. That is the right timestamp format.
The index may be corrupted somehow. Can you try removing all of the fields in data/ and trying again?
----- Original Message -----
| From: "Erick Erickson" <er...@gmail.com>
| To: solr-user@lucene.apache.org
| Sent: Friday, November 2, 2012 7:32:40 AM
| Subject: Re: After adding field to schema, the field is not being returned in results.
|
| Well, I'm at my wits end. I tried your field definitions (using the
| exampledocs XML) and they work just fine. As far as if you mess up
| the date
| on the way in, you should be seeing stack traces in your log files.
|
| The only way I see not getting the "Sorry, no Term Info available :("
| message is if you don't have any values in the field. So, my guess is
| that
| you're not getting the format right and the docs aren't getting
| indexed,
| but that's just a guess. You can freely sort even if there are no
| values at
| all in a particular field. This can be indicated if you sort asc and
| desc
| and the order doesn't change. It just means the field is defined in
| the
| schema, not necessarily that there are any values in it.
|
| So, I claim you have no date values in your index. The fact that you
| can
| sort is just an artifact of sortMissingFirst/Last doing something
| sensible.
|
| Next question, are you absolutely sure that your indexing program and
| your
| searching program are pointing at the same server?
|
| So what I'd do next is
| 1> create a simple XML doc that conforms to your schema and use the
| post.jar tool to send it to your server. Watch the output log for any
| date
| format exceptions.
| 2> Use the admin UI to insure that you can see terms in docs added
| this way.
| 3> from there back up and see what step in the indexing process isn't
| working (assuming that's the problem). Solr logs help here.
|
| Note I'm completely PHP-ignorant, I have no clue whether the
| formatting
| you're doing is OK or not. You might try logging the value somewhere
| in
| your php so you an post that and/or include it in your sample XML
| file...
|
| Best
| Erick
|
|
| On Fri, Nov 2, 2012 at 10:02 AM, Dotan Cohen <do...@gmail.com>
| wrote:
|
| > On Thu, Nov 1, 2012 at 9:28 PM, Lance Norskog <go...@gmail.com>
| > wrote:
| > > Have you uploaded data with that field populated? Solr is not
| > > like a
| > relational database. It does not automatically populate a new field
| > when
| > you add it to the schema. If you sort on a field, a document with
| > no data
| > in that field comes first or last (I don't know which).
| > >
| >
| > Thank you. In fact, I am being careful to try to pull up records
| > after
| > the date in which the application was updated to populate the
| > field.
| >
| >
| > --
| > Dotan Cohen
| >
| > http://gibberish.co.il
| > http://what-is-what.com
| >
|
Re: After adding field to schema, the field is not being returned in results.
Posted by Erick Erickson <er...@gmail.com>.
Well, I'm at my wits end. I tried your field definitions (using the
exampledocs XML) and they work just fine. As far as if you mess up the date
on the way in, you should be seeing stack traces in your log files.
The only way I see not getting the "Sorry, no Term Info available :("
message is if you don't have any values in the field. So, my guess is that
you're not getting the format right and the docs aren't getting indexed,
but that's just a guess. You can freely sort even if there are no values at
all in a particular field. This can be indicated if you sort asc and desc
and the order doesn't change. It just means the field is defined in the
schema, not necessarily that there are any values in it.
So, I claim you have no date values in your index. The fact that you can
sort is just an artifact of sortMissingFirst/Last doing something sensible.
Next question, are you absolutely sure that your indexing program and your
searching program are pointing at the same server?
So what I'd do next is
1> create a simple XML doc that conforms to your schema and use the
post.jar tool to send it to your server. Watch the output log for any date
format exceptions.
2> Use the admin UI to insure that you can see terms in docs added this way.
3> from there back up and see what step in the indexing process isn't
working (assuming that's the problem). Solr logs help here.
Note I'm completely PHP-ignorant, I have no clue whether the formatting
you're doing is OK or not. You might try logging the value somewhere in
your php so you an post that and/or include it in your sample XML file...
Best
Erick
On Fri, Nov 2, 2012 at 10:02 AM, Dotan Cohen <do...@gmail.com> wrote:
> On Thu, Nov 1, 2012 at 9:28 PM, Lance Norskog <go...@gmail.com> wrote:
> > Have you uploaded data with that field populated? Solr is not like a
> relational database. It does not automatically populate a new field when
> you add it to the schema. If you sort on a field, a document with no data
> in that field comes first or last (I don't know which).
> >
>
> Thank you. In fact, I am being careful to try to pull up records after
> the date in which the application was updated to populate the field.
>
>
> --
> Dotan Cohen
>
> http://gibberish.co.il
> http://what-is-what.com
>
Re: After adding field to schema, the field is not being returned in results.
Posted by Dotan Cohen <do...@gmail.com>.
On Thu, Nov 1, 2012 at 9:28 PM, Lance Norskog <go...@gmail.com> wrote:
> Have you uploaded data with that field populated? Solr is not like a relational database. It does not automatically populate a new field when you add it to the schema. If you sort on a field, a document with no data in that field comes first or last (I don't know which).
>
Thank you. In fact, I am being careful to try to pull up records after
the date in which the application was updated to populate the field.
--
Dotan Cohen
http://gibberish.co.il
http://what-is-what.com
Re: After adding field to schema, the field is not being returned in results.
Posted by Dotan Cohen <do...@gmail.com>.
On Thu, Nov 1, 2012 at 9:09 PM, Erick Erickson <er...@gmail.com> wrote:
> What happens if you sort ascending rather than descending? Depending on
> what (if anything) you've done with sortMissingFirst/Last on that field,
> it's possible that you're just seeing the results of the sort and docs with
> your new field are somewhere down the list. If you've done nothing, you
> should be seeing the docs with the new field at the top of the list with
> the query you posted, so this is grasping at straws a bit.
>
Thanks. Sorting ASC or DESC still does not show the field, even in
documents for which the field should exist based on the time that it
was created. However, I am starting to wonder that perhaps my
application is creating the wrong field values and perhaps that is why
the field don't exist. This is the field in question:
<fieldType name="tdate" class="solr.TrieDateField" omitNorms="true"
precisionStep="6" positionIncrementGap="0"/>
<field name="created_iso8601" type="tdate" stored="true"
multiValued="false" indexed="true"/>
My application is writing dates in this format (ISO 8601):
2012-11-02T13:48:10Z
Here is the PHP code:
date("Y-m-d\TH:i:s\Z")
I am setting the server timezone, as PHP >= 5.1 requires:
date_default_timezone_set('UTC');
> The solr admin page, try going to collection>>schema browser and choose the
> field in question from your drop-down. see if it looks like it is stored
> and indexed, and see what some of the values are. This is getting the vals
> from the indexed terms, but at least it should say "stored" in the schema
> and index sections. If it doesn't, then you somehow have a mismatch between
> your schema and what's actually in your index. This really shouldn't be the
> case since it's a brand-new field....
>
Sorry, no Term Info available :(
Alright, so it is an issue with the data that I'm feeding it. Would
Solr include the fields with good data and reject the fields with bad
data, but update the Document anyway? I can confirm that the variable
used to populate the field in question is not empty.
> Two other things I'd try.
> 1> If you have the ID of the document you're _sure_ has a date field in it
> try your query just on that, with fl=*. This would avoid any possible
> sortMissingFirst/Last issues.
>
Yes, I've done that, with and without fl.
> 2> Another way to restrict this would be to add an "fq" clause to the query
> so docs without the field would not be displayed, something like
> fq=[NOW-1YEAR TO NOW] assuming your dates are in the last year.
>
> But I guess we're down to needing to see the schema definition etc. if that
> doesn't work.
>
> Best
> Erick
Thanks, Erick. It does look like the issue is that the field remains
empty. Perhaps I'm writing ISO 8601 wrong, I'll get to looking at that
now. I'm surprised that Solr accepts the documents with bad data in
some of the fields, I will look into that too as well.
Have a peaceful Saturday.
--
Dotan Cohen
http://gibberish.co.il
http://what-is-what.com
Re: After adding field to schema, the field is not being returned in results.
Posted by Erick Erickson <er...@gmail.com>.
What happens if you sort ascending rather than descending? Depending on
what (if anything) you've done with sortMissingFirst/Last on that field,
it's possible that you're just seeing the results of the sort and docs with
your new field are somewhere down the list. If you've done nothing, you
should be seeing the docs with the new field at the top of the list with
the query you posted, so this is grasping at straws a bit.
The solr admin page, try going to collection>>schema browser and choose the
field in question from your drop-down. see if it looks like it is stored
and indexed, and see what some of the values are. This is getting the vals
from the indexed terms, but at least it should say "stored" in the schema
and index sections. If it doesn't, then you somehow have a mismatch between
your schema and what's actually in your index. This really shouldn't be the
case since it's a brand-new field....
Two other things I'd try.
1> If you have the ID of the document you're _sure_ has a date field in it
try your query just on that, with fl=*. This would avoid any possible
sortMissingFirst/Last issues.
2> Another way to restrict this would be to add an "fq" clause to the query
so docs without the field would not be displayed, something like
fq=[NOW-1YEAR TO NOW] assuming your dates are in the last year.
But I guess we're down to needing to see the schema definition etc. if that
doesn't work.
Best
Erick
On Thu, Nov 1, 2012 at 11:55 AM, Dotan Cohen <do...@gmail.com> wrote:
> On Thu, Nov 1, 2012 at 3:00 PM, Erick Erickson <er...@gmail.com>
> wrote:
> > I'd try several things....
> >
> > 1> just because you an sort has nothing to do with whether the field is
> > returned. Sorting uses the indexed data, returning it is the stored data.
> > So it's a bit of a red herring when you can sort on a field but not see
> it,
> > although it is a good test that your schema knows about the field.
> >
>
> Right, the only thing that I was testing for is that the field is
> recognised by Solr. If I had a typo in the field name in the query or
> in the schema, Solr would have complained.
>
>
> > 2> Try fl=* just for yucks.
> >
>
> yuck, yuck, but didn't work!
>
>
> > 3> Check your schema.xml for typos. stroed="true" for instance?
>
> No, tried that. Good thinking, though.
>
>
> > 4> Why are you restricting your returns to 1 and only one document? Are
> you
> > absolutely sure that that document has the new field? Solr happily sorts
> > documents that do not have a value for a field, that's the purpose of
> > sortMissingFirst/Last.
> >
>
> All the newest documents have the field, and I'm sorting by time
> descending. In fact, I did test with more rows, but for the mailing
> list I wanted the output to be concise.
>
> Thanks.
>
> --
> Dotan Cohen
>
> http://gibberish.co.il
> http://what-is-what.com
>
Re: After adding field to schema, the field is not being returned in results.
Posted by Dotan Cohen <do...@gmail.com>.
On Thu, Nov 1, 2012 at 3:00 PM, Erick Erickson <er...@gmail.com> wrote:
> I'd try several things....
>
> 1> just because you an sort has nothing to do with whether the field is
> returned. Sorting uses the indexed data, returning it is the stored data.
> So it's a bit of a red herring when you can sort on a field but not see it,
> although it is a good test that your schema knows about the field.
>
Right, the only thing that I was testing for is that the field is
recognised by Solr. If I had a typo in the field name in the query or
in the schema, Solr would have complained.
> 2> Try fl=* just for yucks.
>
yuck, yuck, but didn't work!
> 3> Check your schema.xml for typos. stroed="true" for instance?
No, tried that. Good thinking, though.
> 4> Why are you restricting your returns to 1 and only one document? Are you
> absolutely sure that that document has the new field? Solr happily sorts
> documents that do not have a value for a field, that's the purpose of
> sortMissingFirst/Last.
>
All the newest documents have the field, and I'm sorting by time
descending. In fact, I did test with more rows, but for the mailing
list I wanted the output to be concise.
Thanks.
--
Dotan Cohen
http://gibberish.co.il
http://what-is-what.com
Re: After adding field to schema, the field is not being returned in results.
Posted by Erick Erickson <er...@gmail.com>.
I'd try several things....
1> just because you an sort has nothing to do with whether the field is
returned. Sorting uses the indexed data, returning it is the stored data.
So it's a bit of a red herring when you can sort on a field but not see it,
although it is a good test that your schema knows about the field.
2> Try fl=* just for yucks.
3> Check your schema.xml for typos. stroed="true" for instance?
4> Why are you restricting your returns to 1 and only one document? Are you
absolutely sure that that document has the new field? Solr happily sorts
documents that do not have a value for a field, that's the purpose of
sortMissingFirst/Last.
FWIW
Erick
On Wed, Oct 31, 2012 at 9:54 PM, Dotan Cohen <do...@gmail.com> wrote:
> On Thu, Nov 1, 2012 at 2:52 AM, Otis Gospodnetic
> <ot...@gmail.com> wrote:
> > Hi,
> >
> > That should work just fine. It;s either a bug or you are doing something
> > you didn't mention. Maybe you can provide a small, self-enclosed unit
> test
> > and stick it in JIRA?
> >
>
> I would assume that it's me doing something wrong! How does this look:
>
>
> /solr/select?q=*&rows=1&sort=created_iso8601%20desc&fl=created_iso8601,created
>
> <response>
> <lst name="responseHeader">
> <int name="status">0</int>
> <int name="QTime">1</int>
> <lst name="params">
> <str name="q">*:*</str>
> <str name="rows">1</str>
> <str name="fl">created_iso8601,created</str>
> </lst>
> </lst>
> <result name="response" numFound="1037937" start="0">
> <doc>
> <int name="created">1350854389</int>
> </doc>
> </result>
> </response>
>
> Surely the sort parameter would throw an error if the
> created_iso8601field did not exist. That field is indexed and stored,
> with no parameters defined on handlers that may list the fields to
> return as Alexandre had mentioned.
>
>
> --
> Dotan Cohen
>
> http://gibberish.co.il
> http://what-is-what.com
>
Re: After adding field to schema, the field is not being returned
in results.
Posted by Lance Norskog <go...@gmail.com>.
Have you uploaded data with that field populated? Solr is not like a relational database. It does not automatically populate a new field when you add it to the schema. If you sort on a field, a document with no data in that field comes first or last (I don't know which).
----- Original Message -----
| From: "Dotan Cohen" <do...@gmail.com>
| To: solr-user@lucene.apache.org
| Sent: Wednesday, October 31, 2012 6:54:47 PM
| Subject: Re: After adding field to schema, the field is not being returned in results.
|
| On Thu, Nov 1, 2012 at 2:52 AM, Otis Gospodnetic
| <ot...@gmail.com> wrote:
| > Hi,
| >
| > That should work just fine. It;s either a bug or you are doing
| > something
| > you didn't mention. Maybe you can provide a small, self-enclosed
| > unit test
| > and stick it in JIRA?
| >
|
| I would assume that it's me doing something wrong! How does this
| look:
|
| /solr/select?q=*&rows=1&sort=created_iso8601%20desc&fl=created_iso8601,created
|
| <response>
| <lst name="responseHeader">
| <int name="status">0</int>
| <int name="QTime">1</int>
| <lst name="params">
| <str name="q">*:*</str>
| <str name="rows">1</str>
| <str name="fl">created_iso8601,created</str>
| </lst>
| </lst>
| <result name="response" numFound="1037937" start="0">
| <doc>
| <int name="created">1350854389</int>
| </doc>
| </result>
| </response>
|
| Surely the sort parameter would throw an error if the
| created_iso8601field did not exist. That field is indexed and stored,
| with no parameters defined on handlers that may list the fields to
| return as Alexandre had mentioned.
|
|
| --
| Dotan Cohen
|
| http://gibberish.co.il
| http://what-is-what.com
|
Re: After adding field to schema, the field is not being returned in results.
Posted by Dotan Cohen <do...@gmail.com>.
On Thu, Nov 1, 2012 at 2:52 AM, Otis Gospodnetic
<ot...@gmail.com> wrote:
> Hi,
>
> That should work just fine. It;s either a bug or you are doing something
> you didn't mention. Maybe you can provide a small, self-enclosed unit test
> and stick it in JIRA?
>
I would assume that it's me doing something wrong! How does this look:
/solr/select?q=*&rows=1&sort=created_iso8601%20desc&fl=created_iso8601,created
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">1</int>
<lst name="params">
<str name="q">*:*</str>
<str name="rows">1</str>
<str name="fl">created_iso8601,created</str>
</lst>
</lst>
<result name="response" numFound="1037937" start="0">
<doc>
<int name="created">1350854389</int>
</doc>
</result>
</response>
Surely the sort parameter would throw an error if the
created_iso8601field did not exist. That field is indexed and stored,
with no parameters defined on handlers that may list the fields to
return as Alexandre had mentioned.
--
Dotan Cohen
http://gibberish.co.il
http://what-is-what.com
Re: After adding field to schema, the field is not being returned in results.
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
That should work just fine. It;s either a bug or you are doing something
you didn't mention. Maybe you can provide a small, self-enclosed unit test
and stick it in JIRA?
Otis
--
Search Analytics - http://sematext.com/search-analytics/index.html
Performance Monitoring - http://sematext.com/spm/index.html
On Wed, Oct 31, 2012 at 8:42 PM, Dotan Cohen <do...@gmail.com> wrote:
> I had stopped Solr 4.0, added a new stored and indexed field to
> schema.xml, and then restarted Solr. I see that my application is
> adding documents and I can query for and retrieve documents just fine.
> However, the new field is not being returned in the XML. I can see
> that if I try to sort on the field I get no error, whereas if I try to
> sort on a nonexistent filed I do get an error, so it seems that Solr
> does recognize the new field as actually existing. However even for
> new documents I cannot get the field added to my XML output from
> queries to Solr, even if the field is explicitly requested with 'fl'.
>
> What must I do to have Solr return this new field in the XML output?
>
> Thanks.
>
> --
> Dotan Cohen
>
> http://gibberish.co.il
> http://what-is-what.com
>