You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Dennis Gearon <ge...@sbcglobal.net> on 2010/10/10 16:55:42 UTC
Records from DIH not easily queried for
With a brand new setup, per the demo/tutorial, with fields slightly changed in the config and data, posting XML records results in a simple qiery being able to find records.
But records imported via a plain jane DIH request can only be found using 'q=*:*' queries.
There's no filtering, tokenizing, blah blah. It's the factory settings. The installation is as new at this as we are :-)
Anyone have any ideas why we can't query for DIH handled records? Do they have some magic juju done to them that XML Posts don't, or visa versa?
Dennis Gearon
Signature Warning
----------------
It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
EARTH has a Right To Life,
otherwise we all die.
Re: Records from DIH not easily queried for
Posted by Dennis Gearon <ge...@sbcglobal.net>.
hmmmm,
I am talking about the query for the DIH (Data Import Handler?) that queries the database to find records/documents to index. We originally were trying to import aout 20 fields, but have whittled down the schema file (my bad calling in config) to only try a few fields.
I myself have been STUDYING this whole Solr/Lucene product, while actually DOING the backend and interface to the front end. HE is doing the search engine and I'm trying to apply my limited knowledge and ask questions here.
I should have a pow wow with him so we present a united front to ask questions. I'll start a new thread when we do that. Thanks for the help, Otis. :-)
And for Solr/Lucene.
Dennis Gearon
Signature Warning
----------------
It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
EARTH has a Right To Life,
otherwise we all die.
--- On Sun, 10/10/10, Otis Gospodnetic <ot...@yahoo.com> wrote:
> From: Otis Gospodnetic <ot...@yahoo.com>
> Subject: Re: Records from DIH not easily queried for
> To: solr-user@lucene.apache.org
> Date: Sunday, October 10, 2010, 8:47 PM
> Dennis,
>
> The unique field does not have to be named "id".
> Fields are specified in schema.xml. There is no
> config.xml, although there is
> solrconfig.xml.
> If a query is returning fields that you do not have defined
> in schema.xml, it's
> time to stop working on weekends! ;) Seriously,
> though, that cannot be
> happening. Maybe you can show the relevant part of
> your schema file, your
> response, and your request?
>
>
> Otis----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Lucene ecosystem search :: http://search-lucene.com/
>
>
>
> ----- Original Message ----
> > From: Dennis Gearon <ge...@sbcglobal.net>
> > To: solr-user@lucene.apache.org
> > Sent: Sun, October 10, 2010 8:47:29 PM
> > Subject: Re: Records from DIH not easily queried for
> >
> > Thanks for answering on a Sunday :-)
> >
> > What happens if the query returns extra fields
> that are not specified in the
> >config.xml?
> >
> > And does the unque field have to be named 'id'?
> >
> > Dennis Gearon
> >
> > Signature Warning
> > ----------------
> > It is always a good idea to learn from your own
> mistakes. It is usually a
> >better idea to learn from others’ mistakes, so you
> do not have to make them
> >yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
> >
> > EARTH has a Right To Life,
> > otherwise we all die.
> >
> >
> > --- On Sun, 10/10/10, Erick Erickson <er...@gmail.com>
> wrote:
> >
> > > From: Erick Erickson <er...@gmail.com>
> > > Subject: Re: Records from DIH not easily
> queried for
> > > To: solr-user@lucene.apache.org
> > > Date: Sunday, October 10, 2010, 8:11 AM
> > > The phrase that jumps out is "with
> > > fields slightly modified". I'm
> > > guessing that your modifications are off by
> a little.
> > > Here's what
> > > I'd check first:
> > > 1> check the case. Sometimes the DB <->
> field link
> > > is case
> > > sensitive.
> > > 2> Look in your index via the admin page
> and look at
> > > your actual
> > > fields as reported there. Are they really
> what you expect?
> > > 3> Try your query with
> &debugQuery=on. Is what you
> > > get back
> > > what you expect?
> > > 4> Sometimes your browser cache will fool you,
> try the
> > > force-refresh
> > > combination on your browser.
> > >
> > > There's no magic here, nothing special or
> different about
> > > DIH
> > > imported data than any other sort. So it's
> almost
> > > certainly
> > > some innocent-seeming change that's not,
> typo, incorrect
> > > assumption, etc.
> > >
> > > If none of that works, you need to post your
> schema changes
> > > and
> > > your query results (with
> &debugQuery=on). Particularly,
> > > post
> > > the fieldType definitions as well as your
> field
> > > definitions...
> > >
> > > Best
> > > Erick
> > >
> > > On Sun, Oct 10, 2010 at 10:55 AM, Dennis Gearon
> <ge...@sbcglobal.net>wrote:
> > >
> > > > With a brand new setup, per the
> demo/tutorial, with
> > > fields slightly changed
> > > > in the config and data, posting XML
> records results in
> > > a simple qiery being
> > > > able to find records.
> > > >
> > > >
> > > > But records imported via a plain jane
> DIH request can
> > > only be found using
> > > > 'q=*:*' queries.
> > > >
> > > > There's no filtering, tokenizing, blah
> blah. It's the
> > > factory settings. The
> > > > installation is as new at this as we
> are :-)
> > > >
> > > > Anyone have any ideas why we can't
> query for DIH
> > > handled records? Do they
> > > > have some magic juju done to them that
> XML Posts
> > > don't, or visa versa?
> > > >
> > > > Dennis Gearon
> > > >
> > > > Signature Warning
> > > > ----------------
> > > > It is always a good idea to learn from
> your own
> > > mistakes. It is usually a
> > > > better idea to learn from others’
> mistakes, so you
> > > do not have to make them
> > > > yourself. from '
> > > > http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
> > > >
> > > > EARTH has a Right To Life,
> > > > otherwise we all die.
> > > >
> > >
> >
>
Re: Records from DIH not easily queried for
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Dennis,
The unique field does not have to be named "id".
Fields are specified in schema.xml. There is no config.xml, although there is
solrconfig.xml.
If a query is returning fields that you do not have defined in schema.xml, it's
time to stop working on weekends! ;) Seriously, though, that cannot be
happening. Maybe you can show the relevant part of your schema file, your
response, and your request?
Otis----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
----- Original Message ----
> From: Dennis Gearon <ge...@sbcglobal.net>
> To: solr-user@lucene.apache.org
> Sent: Sun, October 10, 2010 8:47:29 PM
> Subject: Re: Records from DIH not easily queried for
>
> Thanks for answering on a Sunday :-)
>
> What happens if the query returns extra fields that are not specified in the
>config.xml?
>
> And does the unque field have to be named 'id'?
>
> Dennis Gearon
>
> Signature Warning
> ----------------
> It is always a good idea to learn from your own mistakes. It is usually a
>better idea to learn from others’ mistakes, so you do not have to make them
>yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
>
> EARTH has a Right To Life,
> otherwise we all die.
>
>
> --- On Sun, 10/10/10, Erick Erickson <er...@gmail.com> wrote:
>
> > From: Erick Erickson <er...@gmail.com>
> > Subject: Re: Records from DIH not easily queried for
> > To: solr-user@lucene.apache.org
> > Date: Sunday, October 10, 2010, 8:11 AM
> > The phrase that jumps out is "with
> > fields slightly modified". I'm
> > guessing that your modifications are off by a little.
> > Here's what
> > I'd check first:
> > 1> check the case. Sometimes the DB <-> field link
> > is case
> > sensitive.
> > 2> Look in your index via the admin page and look at
> > your actual
> > fields as reported there. Are they really what you expect?
> > 3> Try your query with &debugQuery=on. Is what you
> > get back
> > what you expect?
> > 4> Sometimes your browser cache will fool you, try the
> > force-refresh
> > combination on your browser.
> >
> > There's no magic here, nothing special or different about
> > DIH
> > imported data than any other sort. So it's almost
> > certainly
> > some innocent-seeming change that's not, typo, incorrect
> > assumption, etc.
> >
> > If none of that works, you need to post your schema changes
> > and
> > your query results (with &debugQuery=on). Particularly,
> > post
> > the fieldType definitions as well as your field
> > definitions...
> >
> > Best
> > Erick
> >
> > On Sun, Oct 10, 2010 at 10:55 AM, Dennis Gearon
<ge...@sbcglobal.net>wrote:
> >
> > > With a brand new setup, per the demo/tutorial, with
> > fields slightly changed
> > > in the config and data, posting XML records results in
> > a simple qiery being
> > > able to find records.
> > >
> > >
> > > But records imported via a plain jane DIH request can
> > only be found using
> > > 'q=*:*' queries.
> > >
> > > There's no filtering, tokenizing, blah blah. It's the
> > factory settings. The
> > > installation is as new at this as we are :-)
> > >
> > > Anyone have any ideas why we can't query for DIH
> > handled records? Do they
> > > have some magic juju done to them that XML Posts
> > don't, or visa versa?
> > >
> > > Dennis Gearon
> > >
> > > Signature Warning
> > > ----------------
> > > It is always a good idea to learn from your own
> > mistakes. It is usually a
> > > better idea to learn from others’ mistakes, so you
> > do not have to make them
> > > yourself. from '
> > > http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
> > >
> > > EARTH has a Right To Life,
> > > otherwise we all die.
> > >
> >
>
Re: Records from DIH not easily queried for
Posted by Dennis Gearon <ge...@sbcglobal.net>.
Thanks for answering on a Sunday :-)
What happens if the query returns extra fields that are not specified in the config.xml?
And does the unque field have to be named 'id'?
Dennis Gearon
Signature Warning
----------------
It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
EARTH has a Right To Life,
otherwise we all die.
--- On Sun, 10/10/10, Erick Erickson <er...@gmail.com> wrote:
> From: Erick Erickson <er...@gmail.com>
> Subject: Re: Records from DIH not easily queried for
> To: solr-user@lucene.apache.org
> Date: Sunday, October 10, 2010, 8:11 AM
> The phrase that jumps out is "with
> fields slightly modified". I'm
> guessing that your modifications are off by a little.
> Here's what
> I'd check first:
> 1> check the case. Sometimes the DB <-> field link
> is case
> sensitive.
> 2> Look in your index via the admin page and look at
> your actual
> fields as reported there. Are they really what you expect?
> 3> Try your query with &debugQuery=on. Is what you
> get back
> what you expect?
> 4> Sometimes your browser cache will fool you, try the
> force-refresh
> combination on your browser.
>
> There's no magic here, nothing special or different about
> DIH
> imported data than any other sort. So it's almost
> certainly
> some innocent-seeming change that's not, typo, incorrect
> assumption, etc.
>
> If none of that works, you need to post your schema changes
> and
> your query results (with &debugQuery=on). Particularly,
> post
> the fieldType definitions as well as your field
> definitions...
>
> Best
> Erick
>
> On Sun, Oct 10, 2010 at 10:55 AM, Dennis Gearon <ge...@sbcglobal.net>wrote:
>
> > With a brand new setup, per the demo/tutorial, with
> fields slightly changed
> > in the config and data, posting XML records results in
> a simple qiery being
> > able to find records.
> >
> >
> > But records imported via a plain jane DIH request can
> only be found using
> > 'q=*:*' queries.
> >
> > There's no filtering, tokenizing, blah blah. It's the
> factory settings. The
> > installation is as new at this as we are :-)
> >
> > Anyone have any ideas why we can't query for DIH
> handled records? Do they
> > have some magic juju done to them that XML Posts
> don't, or visa versa?
> >
> > Dennis Gearon
> >
> > Signature Warning
> > ----------------
> > It is always a good idea to learn from your own
> mistakes. It is usually a
> > better idea to learn from others’ mistakes, so you
> do not have to make them
> > yourself. from '
> > http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
> >
> > EARTH has a Right To Life,
> > otherwise we all die.
> >
>
Re: Records from DIH not easily queried for
Posted by Dennis Gearon <ge...@sbcglobal.net>.
Well, found the problem, us of course.
We were using string instead of text for the field type in the schema config file. So it wasn't tokenizing words or doing other 'search by word' enabling preprocessing before storing the document in the index. We could have only found whole sentences.
Now it works! But, now the long road to tuning it to find what we WANT it to find . . . begins.
That and getting what we want out of geospatial. We're just starting on that.
Dennis Gearon
Signature Warning
----------------
It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
EARTH has a Right To Life,
otherwise we all die.
--- On Sun, 10/10/10, Erick Erickson <er...@gmail.com> wrote:
> From: Erick Erickson <er...@gmail.com>
> Subject: Re: Records from DIH not easily queried for
> To: solr-user@lucene.apache.org
> Date: Sunday, October 10, 2010, 8:11 AM
> The phrase that jumps out is "with
> fields slightly modified". I'm
> guessing that your modifications are off by a little.
> Here's what
> I'd check first:
> 1> check the case. Sometimes the DB <-> field link
> is case
> sensitive.
> 2> Look in your index via the admin page and look at
> your actual
> fields as reported there. Are they really what you expect?
> 3> Try your query with &debugQuery=on. Is what you
> get back
> what you expect?
> 4> Sometimes your browser cache will fool you, try the
> force-refresh
> combination on your browser.
>
> There's no magic here, nothing special or different about
> DIH
> imported data than any other sort. So it's almost
> certainly
> some innocent-seeming change that's not, typo, incorrect
> assumption, etc.
>
> If none of that works, you need to post your schema changes
> and
> your query results (with &debugQuery=on). Particularly,
> post
> the fieldType definitions as well as your field
> definitions...
>
> Best
> Erick
>
> On Sun, Oct 10, 2010 at 10:55 AM, Dennis Gearon <ge...@sbcglobal.net>wrote:
>
> > With a brand new setup, per the demo/tutorial, with
> fields slightly changed
> > in the config and data, posting XML records results in
> a simple qiery being
> > able to find records.
> >
> >
> > But records imported via a plain jane DIH request can
> only be found using
> > 'q=*:*' queries.
> >
> > There's no filtering, tokenizing, blah blah. It's the
> factory settings. The
> > installation is as new at this as we are :-)
> >
> > Anyone have any ideas why we can't query for DIH
> handled records? Do they
> > have some magic juju done to them that XML Posts
> don't, or visa versa?
> >
> > Dennis Gearon
> >
> > Signature Warning
> > ----------------
> > It is always a good idea to learn from your own
> mistakes. It is usually a
> > better idea to learn from others’ mistakes, so you
> do not have to make them
> > yourself. from '
> > http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
> >
> > EARTH has a Right To Life,
> > otherwise we all die.
> >
>
Re: Records from DIH not easily queried for
Posted by Erick Erickson <er...@gmail.com>.
The phrase that jumps out is "with fields slightly modified". I'm
guessing that your modifications are off by a little. Here's what
I'd check first:
1> check the case. Sometimes the DB <-> field link is case
sensitive.
2> Look in your index via the admin page and look at your actual
fields as reported there. Are they really what you expect?
3> Try your query with &debugQuery=on. Is what you get back
what you expect?
4> Sometimes your browser cache will fool you, try the force-refresh
combination on your browser.
There's no magic here, nothing special or different about DIH
imported data than any other sort. So it's almost certainly
some innocent-seeming change that's not, typo, incorrect
assumption, etc.
If none of that works, you need to post your schema changes and
your query results (with &debugQuery=on). Particularly, post
the fieldType definitions as well as your field definitions...
Best
Erick
On Sun, Oct 10, 2010 at 10:55 AM, Dennis Gearon <ge...@sbcglobal.net>wrote:
> With a brand new setup, per the demo/tutorial, with fields slightly changed
> in the config and data, posting XML records results in a simple qiery being
> able to find records.
>
>
> But records imported via a plain jane DIH request can only be found using
> 'q=*:*' queries.
>
> There's no filtering, tokenizing, blah blah. It's the factory settings. The
> installation is as new at this as we are :-)
>
> Anyone have any ideas why we can't query for DIH handled records? Do they
> have some magic juju done to them that XML Posts don't, or visa versa?
>
> Dennis Gearon
>
> Signature Warning
> ----------------
> It is always a good idea to learn from your own mistakes. It is usually a
> better idea to learn from others’ mistakes, so you do not have to make them
> yourself. from '
> http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
>
> EARTH has a Right To Life,
> otherwise we all die.
>