DisMax queries referencing undefined fields

[Doug Daniels <dd...@rooftophq.com>]

Hi,

I noticed that the "field list" (fl) parameter ignores field names that 
it cannot locate, while the "query fields" (qf) parameter throws an 
exception when fields cannot be located.  Is there any way to override 
this behavior and have qf also ignore fields it cannot find?

This would be pretty helpful for us, as we're going to have a large 
number of dynamic, user-specific fields defined in our schema.  These 
fields will have canonical name formats (e.g. userid-comment), but they 
may not be defined for every document.  In fact, some fields may be 
defined for no documents, which I gather would be the ones that would 
throw exceptions.  It would be nice to provide solr a set of fields that 
could be searched and have it use the subset of those fields that exist.

Thanks in advance for any help,
Doug

Re: DisMax queries referencing undefined fields

[Chris Hostetter <ho...@fucit.org>]

: I noticed that the "field list" (fl) parameter ignores field names that it
: cannot locate, while the "query fields" (qf) parameter throws an exception
: when fields cannot be located.  Is there any way to override this behavior and
: have qf also ignore fields it cannot find?

Those parameters are radically different.  FL isn't evaluated untill after 
a query is executed and it's time to return documents ... just because the 
current range of documents being returned doesn't have a value doesn't 
mean there is a problem with the FL -- other documents in the same DocSet 
might have those values.  It's not that Solr ignores fields in the "fl" 
that it can't locate, as it is that Solr tests each field a document to be 
returned has, and only returns it if the field is in the FL. 

In theory, field names in the FL should be tested to see if a matching 
field or dynamic field exists that would match and generate a 
warning/error if it's not -- i would consider that an FL bug.

The semantics of QF follow directly from the semantics of the standard 
query parser: if you tell it to query against a field which does not exist 
for any document, then something is wrong with the request.  Unlike the 
FL case (which is lazy for not checking that the feild exists) dismax 
has to check each field because it needs to know how to analyze the input 
for every field in the QF -- if the field doesn't exist, it can't do that.

: This would be pretty helpful for us, as we're going to have a large number of
: dynamic, user-specific fields defined in our schema.  These fields will have
: canonical name formats (e.g. userid-comment), but they may not be defined for
: every document.  In fact, some fields may be defined for no documents, which I
: gather would be the ones that would throw exceptions.  It would be nice to
: provide solr a set of fields that could be searched and have it use the subset
: of those fields that exist.

i supose it would be possible to make an option for dismax to ignore any 
field it can't find, but that would be fairly kludgy and would introduce 
some really confusing edge cases (ie: what happens if non of the fields in 
QF can be found)

A better option would probably be to use something like this from the 
sample schema.xml...

   <!-- uncomment the following to ignore any fields that don't already 
match an existing 
        field name or dynamic field, rather than reporting them as an 
error. 
        alternately, change the type="ignored" to some other type e.g. 
"text" if you want 
        unknown fields indexed and/or stored by default --> 
   <!-- dynamicField name="*" type="ignored" / -->


...then any field name you want will work regardless of wether you are 
using dismax or hte standard request handler.


Hmmm: it might be better though if the "ignored" field type was a 
TextField with an Analyzer that produced no tokens ... then it would drop 
out of the query completely ... anyone want to submit a "NoOpAnalyzer" :)  


-Hoss

Re: DisMax queries referencing undefined fields

[Doug Daniels <dd...@rooftophq.com>]

Nevermind!  I figured out that the qf parameter handles any field names 
that match a dynamic field name pattern, regardless of whether a field 
is defined with that name.

-Doug

Doug Daniels wrote:
> Hi,
> 
> I noticed that the "field list" (fl) parameter ignores field names that 
> it cannot locate, while the "query fields" (qf) parameter throws an 
> exception when fields cannot be located.  Is there any way to override 
> this behavior and have qf also ignore fields it cannot find?
> 
> This would be pretty helpful for us, as we're going to have a large 
> number of dynamic, user-specific fields defined in our schema.  These 
> fields will have canonical name formats (e.g. userid-comment), but they 
> may not be defined for every document.  In fact, some fields may be 
> defined for no documents, which I gather would be the ones that would 
> throw exceptions.  It would be nice to provide solr a set of fields that 
> could be searched and have it use the subset of those fields that exist.
> 
> Thanks in advance for any help,
> Doug
>