Seeing fully expanded query, even if there's a search error

[Mark Bennett <mb...@ideaeng.com>]

Using nightly builds. (actually May 4th...)

When things go OK one of the options you can enable is to see the fully
expanded query in the search results.

But I have a custom schema, and clearly there's something in the dismax
logic that clashes with it.

This would normally be easy to fix, by looking at the expanded query in the
web UI and then making the appropriate changes, but the error seems to short
circuit getting back the results list, which would have shown the query.

What I'd really like is just a "testqp" utility, like Verity K2 used to
have:
* select the parser (or search component, etc)
* type in a "user" query
* see the fully expanded parse tree (without actually running the search)
* that old tool also ran from the command line, which was nice, though at
this point web or console would be OK

Though I suspect I could eventually extract this logic from the code base
myself, I'm not as familiar with this code base as y'all, and decided to
sacrifice my pride for expediency in this particular case.  :-)  I have made
some progress on extracting other bits of code.

I suppose there is some possibility, depending on how it's written, that
dismax CAN'T generate a fully expanded query if there's any problem with the
schema.  In that case I guess an (English) explanation of what dismax does
in terms of query generation would be the backup plan.  I've certainly
looked around at the doc for dismax.  There's a lot of "how to use it", what
the options are, etc.  But what I really wanted was to understand its query
generation logic, so that I could adapt it to my project.

Thanks in advance,
Mark

--
Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com

Re: Seeing fully expanded query, even if there's a search error

[Yonik Seeley <yo...@lucidimagination.com>]

The best we currently have is to add debugQuery=true to the request -
that will return what the actual Lucene query looks like (stemmed
forms, expanded fields, etc).

-Yonik
http://www.lucidimagination.com



On Mon, Jun 1, 2009 at 12:56 PM, Mark Bennett <mb...@ideaeng.com> wrote:
> Using nightly builds. (actually May 4th...)
>
> When things go OK one of the options you can enable is to see the fully
> expanded query in the search results.
>
> But I have a custom schema, and clearly there's something in the dismax
> logic that clashes with it.
>
> This would normally be easy to fix, by looking at the expanded query in the
> web UI and then making the appropriate changes, but the error seems to short
> circuit getting back the results list, which would have shown the query.
>
> What I'd really like is just a "testqp" utility, like Verity K2 used to
> have:
> * select the parser (or search component, etc)
> * type in a "user" query
> * see the fully expanded parse tree (without actually running the search)
> * that old tool also ran from the command line, which was nice, though at
> this point web or console would be OK
>
> Though I suspect I could eventually extract this logic from the code base
> myself, I'm not as familiar with this code base as y'all, and decided to
> sacrifice my pride for expediency in this particular case.  :-)  I have made
> some progress on extracting other bits of code.
>
> I suppose there is some possibility, depending on how it's written, that
> dismax CAN'T generate a fully expanded query if there's any problem with the
> schema.  In that case I guess an (English) explanation of what dismax does
> in terms of query generation would be the backup plan.  I've certainly
> looked around at the doc for dismax.  There's a lot of "how to use it", what
> the options are, etc.  But what I really wanted was to understand its query
> generation logic, so that I could adapt it to my project.
>
> Thanks in advance,
> Mark
>
> --
> Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
>

Re: Seeing fully expanded query, even if there's a search error

[Mark Bennett <mb...@ideaeng.com>]

Hi Hoss,

I understand this problem a bit better now.  The problem was that the
default solrconfig.xml file includes a dismax config with qf, pf and bf
options that refer to specific schema fields that I had removed.

I don't have the exact error, but essentially Solr replies back that you're
referencing fields that don't exist, instead of the parsed query.  I believe
the errors were in the log, when doing a query or when submitting a
document.

And this was odd since I wasn't explicitly referencing the alleged fields;
of course I had interited the references to them from the default dismax
config.

I don't have that setup right now so I can't give the exact errors.

--
Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513


On Mon, Jun 22, 2009 at 12:07 PM, Chris Hostetter
<ho...@fucit.org>wrote:

>
> : Date: Mon, 1 Jun 2009 09:56:35 -0700
> : From: Mark Bennett
> : Subject: Seeing fully expanded query, even if there's a search error
>
> : This would normally be easy to fix, by looking at the expanded query in
> the
> : web UI and then making the appropriate changes, but the error seems to
> short
> : circuit getting back the results list, which would have shown the query.
>
> when you say "error" what exactly do you mean?  are you getting back an
> exception? an HTTP error code? a stack trace in the logs?
>
> if you get an HTTP status code of "OK" you should be about to use
> debugQuery=true even if there are no results visible.
>
> if you get any other kind of error - then that's a bug in the dismax
> parser and if you give us all the details we can try to fix it.
>
> : I suppose there is some possibility, depending on how it's written, that
> : dismax CAN'T generate a fully expanded query if there's any problem with
> the
> : schema.  In that case I guess an (English) explanation of what dismax
> does
> : in terms of query generation would be the backup plan.  I've certainly
>
> it's certianly possible that a "broken" schema could break dismax ...
> although the only scenerio i can think of where that wouldn't depend on a
> "bug" in dismax as well is if your schema didn't match the data actually
> found in your index (ie: field types not matching what's actually their)
>
> : the options are, etc.  But what I really wanted was to understand its
> query
> : generation logic, so that I could adapt it to my project.
>
> this is a big fat gapping blank spot in the documentation ... largely
> because everytime i've tried to explain it in the past, the answer winds
> up being longer them the code.  pictures help, but my whiteboarding skills
> don't map well to photoshop skills.
>
> FWIW, i added some details to the bottom of the wiki...
>
> http://wiki.apache.org/solr/DisMaxRequestHandler#head-3f8e5bbbe05e8fb58e3a49a5888d0be32b3a09ce
>
>
>
> -Hoss
>
>

Re: Seeing fully expanded query, even if there's a search error

[Chris Hostetter <ho...@fucit.org>]

: Date: Mon, 1 Jun 2009 09:56:35 -0700
: From: Mark Bennett
: Subject: Seeing fully expanded query, even if there's a search error

: This would normally be easy to fix, by looking at the expanded query in the
: web UI and then making the appropriate changes, but the error seems to short
: circuit getting back the results list, which would have shown the query.

when you say "error" what exactly do you mean?  are you getting back an 
exception? an HTTP error code? a stack trace in the logs?

if you get an HTTP status code of "OK" you should be about to use 
debugQuery=true even if there are no results visible.

if you get any other kind of error - then that's a bug in the dismax 
parser and if you give us all the details we can try to fix it.

: I suppose there is some possibility, depending on how it's written, that
: dismax CAN'T generate a fully expanded query if there's any problem with the
: schema.  In that case I guess an (English) explanation of what dismax does
: in terms of query generation would be the backup plan.  I've certainly

it's certianly possible that a "broken" schema could break dismax ... 
although the only scenerio i can think of where that wouldn't depend on a 
"bug" in dismax as well is if your schema didn't match the data actually 
found in your index (ie: field types not matching what's actually their)

: the options are, etc.  But what I really wanted was to understand its query
: generation logic, so that I could adapt it to my project.

this is a big fat gapping blank spot in the documentation ... largely 
because everytime i've tried to explain it in the past, the answer winds 
up being longer them the code.  pictures help, but my whiteboarding skills 
don't map well to photoshop skills.

FWIW, i added some details to the bottom of the wiki...
http://wiki.apache.org/solr/DisMaxRequestHandler#head-3f8e5bbbe05e8fb58e3a49a5888d0be32b3a09ce



-Hoss