You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by jlist9 <jl...@gmail.com> on 2010/05/30 04:55:48 UTC
Luke browser does not show non-String Solr fields?
I tried the stand-alone Luke tool (not Luke request handler) to browse
a solr index and find a few strange things:
1. Queries like "id:123" which work fine in /solr/admin web interface
returns nothing in Luke. "*:*" returns everything fine in Luke.
2. When Luke displays records with query "*:*", it shows the string
values fine but the numeric fields and date fields shows blank. It shows
DocID OK, though.
Anyone else has tried Luke on a solr index?
Re: Luke browser does not show non-String Solr fields?
Posted by jlist9 <jl...@gmail.com>.
Thank you Chris. I'm clear now. I'll give Luke's latest version a try
when it's out.
On Wed, Jun 2, 2010 at 9:47 AM, Chris Hostetter
<ho...@fucit.org> wrote:
>
> : I see. It's still a little confusing to me but I'm fine as long as
> : this is the expected behavior. I also tried the "example" index
> : with data that come with the solr distribution and observe the
> : same behavior - only String fields are displayed. So Lucene is
> : sharing _some_ types with Solr but not all. It's still a bit puzzling
> : to me that Lucene is not able to understand the simple types
> : such as long. But I'm OK as long as there is a reason. Thanks
> : for the explanations!
>
> The key is that there are *no* types in Lucene ... older
> versions of Lucene only supported "Strin" and clinets that wanted to index
> other types had to encode those types in some way as needed. More
> recently lucene has started moving away from even dealing with Strings,
> and towards just indexing/searching raw byte[] ... all concepts of "field
> types" in Solr are specific to Solr
>
> (the caveat being that Lucene has, over the years, added utilities to help
> people make smart choices about how to encode some data types -- and in
> the case of the Trie numeric fields SOlr uses those utilites. But that
> data isn't stored anywhere in the index files themselves, so Luke has no
> way of knowing that it should attempt to "decode" the binary data of a
> field using the Trie utilities. That said: aparently Andrzej is working
> on making it possible to tell Luke "oh BTW, i indexed this field using
> this solr fieldType" ... i think he said it was on the Luke trunk)
>
>
> -Hoss
Re: Luke browser does not show non-String Solr fields?
Posted by Chris Hostetter <ho...@fucit.org>.
: I see. It's still a little confusing to me but I'm fine as long as
: this is the expected behavior. I also tried the "example" index
: with data that come with the solr distribution and observe the
: same behavior - only String fields are displayed. So Lucene is
: sharing _some_ types with Solr but not all. It's still a bit puzzling
: to me that Lucene is not able to understand the simple types
: such as long. But I'm OK as long as there is a reason. Thanks
: for the explanations!
The key is that there are *no* types in Lucene ... older
versions of Lucene only supported "Strin" and clinets that wanted to index
other types had to encode those types in some way as needed. More
recently lucene has started moving away from even dealing with Strings,
and towards just indexing/searching raw byte[] ... all concepts of "field
types" in Solr are specific to Solr
(the caveat being that Lucene has, over the years, added utilities to help
people make smart choices about how to encode some data types -- and in
the case of the Trie numeric fields SOlr uses those utilites. But that
data isn't stored anywhere in the index files themselves, so Luke has no
way of knowing that it should attempt to "decode" the binary data of a
field using the Trie utilities. That said: aparently Andrzej is working
on making it possible to tell Luke "oh BTW, i indexed this field using
this solr fieldType" ... i think he said it was on the Luke trunk)
-Hoss
Re: Luke browser does not show non-String Solr fields?
Posted by jlist9 <jl...@gmail.com>.
I see. It's still a little confusing to me but I'm fine as long as
this is the expected behavior. I also tried the "example" index
with data that come with the solr distribution and observe the
same behavior - only String fields are displayed. So Lucene is
sharing _some_ types with Solr but not all. It's still a bit puzzling
to me that Lucene is not able to understand the simple types
such as long. But I'm OK as long as there is a reason. Thanks
for the explanations!
On Tue, Jun 1, 2010 at 10:38 AM, Chris Hostetter
<ho...@fucit.org> wrote:
>
> : So it seems like Luke does not understand Solr's long type. This
> : is not a native Lucene type?
>
> No, Lucene has concept of "types" ... there are utilities to help encode
> some data in special ways (particularly numbers) but the underlying lucene
> index doesn't keep track of when/how you do ths -- so Luke has no way of
> knowing what "type" the field is.
>
> Schema information is specific to Solr.
>
>
> -Hoss
>
>
Re: Luke browser does not show non-String Solr fields?
Posted by Chris Hostetter <ho...@fucit.org>.
: So it seems like Luke does not understand Solr's long type. This
: is not a native Lucene type?
No, Lucene has concept of "types" ... there are utilities to help encode
some data in special ways (particularly numbers) but the underlying lucene
index doesn't keep track of when/how you do ths -- so Luke has no way of
knowing what "type" the field is.
Schema information is specific to Solr.
-Hoss
Re: Luke browser does not show non-String Solr fields?
Posted by jlist9 <jl...@gmail.com>.
The id field has type "long" in schema.xml. In Luke, they are shown
as "hex dump". When viewing a doc (returned by *:*), I pick the ID field
and press the "Show" button, Luke pops up a dialog that allows me
to change the "Show Content As" value. When I choose "Number",
I get an error message:
"Some values could not be properly represented in this format. They
are marked in grey and presented as a hex dump."
So it seems like Luke does not understand Solr's long type. This
is not a native Lucene type?
On Mon, May 31, 2010 at 9:52 AM, Chris Hostetter
<ho...@fucit.org> wrote:
>
> : 1. Queries like "id:123" which work fine in /solr/admin web interface but
> : returns nothing in Luke. Query "*:*" returns all records fine in Luke. I
> : expect Luke returns the same result as /solr/admin since it's essentially
> : a Lucene query?
>
> you haven't told us what fieldtype you are using for the "id" field -- but
> i'm going to go out on a limb and guess it's TrieIntFieldType (or possibly
> a SortedIntFieldType) ... those field types encode their values in such a
> way that they sort lexigraphicaly and produce faster range queries -- if
> Luke doesn't kow about that special encoding, it can search on them (or
> even display the terms properly)
>
> Luke has a "view terms" feature right? ... look at the raw terms in your
> "id" ifeld and i bet you'll see they look nothing like numbers -- and
> that's why you can search on them as numbers in Luke
>
> (when you serach on them in Solr, SOlr knows about your schema, and knows
> about your field types, and can do the proper encoding/decoding)
>
>
>
> -Hoss
>
>
Re: Luke browser does not show non-String Solr fields?
Posted by Chris Hostetter <ho...@fucit.org>.
: 1. Queries like "id:123" which work fine in /solr/admin web interface but
: returns nothing in Luke. Query "*:*" returns all records fine in Luke. I
: expect Luke returns the same result as /solr/admin since it's essentially
: a Lucene query?
you haven't told us what fieldtype you are using for the "id" field -- but
i'm going to go out on a limb and guess it's TrieIntFieldType (or possibly
a SortedIntFieldType) ... those field types encode their values in such a
way that they sort lexigraphicaly and produce faster range queries -- if
Luke doesn't kow about that special encoding, it can search on them (or
even display the terms properly)
Luke has a "view terms" feature right? ... look at the raw terms in your
"id" ifeld and i bet you'll see they look nothing like numbers -- and
that's why you can search on them as numbers in Luke
(when you serach on them in Solr, SOlr knows about your schema, and knows
about your field types, and can do the proper encoding/decoding)
-Hoss
Re: Luke browser does not show non-String Solr fields?
Posted by Trey Grainger <so...@gmail.com>.
I submitted a patch a few months back for a Solr Document Inspector which
allows one to see the indexed values for any document in a Solr index (
https://issues.apache.org/jira/browse/SOLR-1837). This is more or less a
port of Luke's DocumentReconstructor into Solr, but the tool additionally
has access to all the solr schema/field type information for display
purposes (i.e. Trie Fields are human-readable).
This won't help you search for values in an index or inspect anything at a
macro level (i.e. term counts across the index), but there are other tools
in Solr for that. Given a UniqueID, however, you can view all the indexed
values for each field in that particular document. You can always do a
search within Solr for the values you are looking for and then use this tool
to view the indexed values for any documents which match.
This may or may not help you (I'm can't tell what problem you are trying to
solve), but I thought it would be worth mentioning as one tool in your
toolbox.
-Trey
>
>
>
>
Re: Luke browser does not show non-String Solr fields?
Posted by Ahmet Arslan <io...@yahoo.com>.
> Thanks for the suggestion. I tried
> 0.9.9.1 but saw the same problem.
> I didn't see 0.9.9 on their download page.
http://www.getopt.org/luke/ has 0.9.9 version. But that may not be the issue.
I suspect that trie based fields causing this. Because they index each value at various levels of precision.
Do you have problems with other than trie based (tint, tdate, etc) types?
Re: Luke browser does not show non-String Solr fields?
Posted by jlist9 <jl...@gmail.com>.
Thanks for the suggestion. I tried 0.9.9.1 but saw the same problem.
I didn't see 0.9.9 on their download page.
On Mon, May 31, 2010 at 2:39 AM, Ahmet Arslan <io...@yahoo.com> wrote:
>
>> Solr 1.4
>>
>> > You haven't identified the version of Luke you're
>> using.
>>
>> Luke 1.0.1 (2010-04-01)
>>
>
> I think with solr you need to use Release 0.9.9.1 or 0.9.9
> Because solr 1.4.0 uses lucene 2.9.1
>
>
>
>
Re: Luke browser does not show non-String Solr fields?
Posted by Ahmet Arslan <io...@yahoo.com>.
> Solr 1.4
>
> > You haven't identified the version of Luke you're
> using.
>
> Luke 1.0.1 (2010-04-01)
>
I think with solr you need to use Release 0.9.9.1 or 0.9.9
Because solr 1.4.0 uses lucene 2.9.1
Re: Luke browser does not show non-String Solr fields?
Posted by jlist9 <jl...@gmail.com>.
Sorry. Let me add more info. (I assumed that anyone who tried it would see
the problem right away but that might not be the case.)
> You haven't shown the SOLR field type definitions.
Values of all non-String types in my index are not being shown.
In my case, this includes long, tint and date types.
> You haven't provided, say, the output from SOLR if you add &debugQuery=on.
Solr side of things seem to work fine. Is this still needed?
> You haven't shown the terms from either SOLR admin or Luke that they
> actually see in the index.
I'm not sure if I understand this question. Basically, if I do a query "id:694"
in /solr/admin, I get the result. But the same query doesn't return anything
in Luke.
> You haven't identified the version of SOLR that the index was created with
Solr 1.4
> You haven't identified the version of Luke you're using.
Luke 1.0.1 (2010-04-01)
> Both SOLR and Luke use Lucene under the covers, so the problem is almost
> certainly your expectations...
This may well be the case. Not knowing the details, I was still surprised
since I expected the same search behavior for simple queries between
solr and luke.
> Imagine a co-worker from another department has provided you the information
> you've provided us. What could you say?
I might give it a try and see the same thing :-D
Thanks
> On Sun, May 30, 2010 at 2:19 PM, jlist9 <jl...@gmail.com> wrote:
>
>> Oh, here's a modified/improved version of what I described in my first
>> email:
>>
>> 1. Queries like "id:123" which work fine in /solr/admin web interface but
>> returns nothing in Luke. Query "*:*" returns all records fine in Luke. I
>> expect Luke returns the same result as /solr/admin since it's essentially
>> a Lucene query?
>>
>> 2. When Luke displays records with query "*:*", it shows the string
>> values fine but the numeric fields and date fields show blank. It shows
>> DocID OK, though. I expect Luke to be able to show non-string values, too.
>>
>> On Sun, May 30, 2010 at 10:57 AM, Erick Erickson
>> <er...@gmail.com> wrote:
>> > Then you have to provide a lot more detail about what you did
>> > and what you're seeing and what you think you should see. You
>> > might review this page:
>> > http://wiki.apache.org/solr/UsingMailingLists
>> >
>> > Best
>> > Erick
>> >
>> > On Sun, May 30, 2010 at 1:41 PM, jlist9 <jl...@gmail.com> wrote:
>> >
>> >> I find in the Plugins tab that the default is PersianAnalyzer. I
>> switched
>> >> to StandardAnalyzer and tried a few different Lucene Compatibility
>> values
>> >> but it didn't help :-(
>> >>
>> >> On Sun, May 30, 2010 at 4:40 AM, Erick Erickson <
>> erickerickson@gmail.com>
>> >> wrote:
>> >> > The Solr admin page as access to (and uses) the field
>> >> > definitions you've put in the config file. Luke has no
>> >> > knowledge of this configuration, you have to choose
>> >> > your analyzer from the drop down and select the one
>> >> > closest to what's in your config file for SOLR. Are you
>> >> > perhaps using an analyzer in Luke that doesn't
>> >> > play nice with the definitions in SOLR?
>> >> >
>> >> > HTH
>> >> > Erick
>> >> >
>> >> >
>> >> > On Sat, May 29, 2010 at 10:55 PM, jlist9 <jl...@gmail.com> wrote:
>> >> >
>> >> >> I tried the stand-alone Luke tool (not Luke request handler) to
>> browse
>> >> >> a solr index and find a few strange things:
>> >> >>
>> >> >> 1. Queries like "id:123" which work fine in /solr/admin web interface
>> >> >> returns nothing in Luke. "*:*" returns everything fine in Luke.
>> >> >>
>> >> >> 2. When Luke displays records with query "*:*", it shows the string
>> >> >> values fine but the numeric fields and date fields shows blank. It
>> shows
>> >> >> DocID OK, though.
>> >> >>
>> >> >> Anyone else has tried Luke on a solr index?
>> >> >>
>> >> >
>> >>
>> >
>>
>
Re: Luke browser does not show non-String Solr fields?
Posted by Erick Erickson <er...@gmail.com>.
No, not nearly enough information.
You haven't shown the SOLR field type definitions.
You haven't provided, say, the output from SOLR if you add &debugQuery=on.
You haven't shown the terms from either SOLR admin or Luke that they
actually see in the index.
You haven't identified the version of SOLR that the index was created with
You haven't identified the version of Luke you're using.
Both SOLR and Luke use Lucene under the covers, so the problem is almost
certainly your expectations...
Imagine a co-worker from another department has provided you the information
you've provided us. What could you say?
Best
Erick
On Sun, May 30, 2010 at 2:19 PM, jlist9 <jl...@gmail.com> wrote:
> Oh, here's a modified/improved version of what I described in my first
> email:
>
> 1. Queries like "id:123" which work fine in /solr/admin web interface but
> returns nothing in Luke. Query "*:*" returns all records fine in Luke. I
> expect Luke returns the same result as /solr/admin since it's essentially
> a Lucene query?
>
> 2. When Luke displays records with query "*:*", it shows the string
> values fine but the numeric fields and date fields show blank. It shows
> DocID OK, though. I expect Luke to be able to show non-string values, too.
>
> On Sun, May 30, 2010 at 10:57 AM, Erick Erickson
> <er...@gmail.com> wrote:
> > Then you have to provide a lot more detail about what you did
> > and what you're seeing and what you think you should see. You
> > might review this page:
> > http://wiki.apache.org/solr/UsingMailingLists
> >
> > Best
> > Erick
> >
> > On Sun, May 30, 2010 at 1:41 PM, jlist9 <jl...@gmail.com> wrote:
> >
> >> I find in the Plugins tab that the default is PersianAnalyzer. I
> switched
> >> to StandardAnalyzer and tried a few different Lucene Compatibility
> values
> >> but it didn't help :-(
> >>
> >> On Sun, May 30, 2010 at 4:40 AM, Erick Erickson <
> erickerickson@gmail.com>
> >> wrote:
> >> > The Solr admin page as access to (and uses) the field
> >> > definitions you've put in the config file. Luke has no
> >> > knowledge of this configuration, you have to choose
> >> > your analyzer from the drop down and select the one
> >> > closest to what's in your config file for SOLR. Are you
> >> > perhaps using an analyzer in Luke that doesn't
> >> > play nice with the definitions in SOLR?
> >> >
> >> > HTH
> >> > Erick
> >> >
> >> >
> >> > On Sat, May 29, 2010 at 10:55 PM, jlist9 <jl...@gmail.com> wrote:
> >> >
> >> >> I tried the stand-alone Luke tool (not Luke request handler) to
> browse
> >> >> a solr index and find a few strange things:
> >> >>
> >> >> 1. Queries like "id:123" which work fine in /solr/admin web interface
> >> >> returns nothing in Luke. "*:*" returns everything fine in Luke.
> >> >>
> >> >> 2. When Luke displays records with query "*:*", it shows the string
> >> >> values fine but the numeric fields and date fields shows blank. It
> shows
> >> >> DocID OK, though.
> >> >>
> >> >> Anyone else has tried Luke on a solr index?
> >> >>
> >> >
> >>
> >
>
Re: Luke browser does not show non-String Solr fields?
Posted by jlist9 <jl...@gmail.com>.
Oh, here's a modified/improved version of what I described in my first email:
1. Queries like "id:123" which work fine in /solr/admin web interface but
returns nothing in Luke. Query "*:*" returns all records fine in Luke. I
expect Luke returns the same result as /solr/admin since it's essentially
a Lucene query?
2. When Luke displays records with query "*:*", it shows the string
values fine but the numeric fields and date fields show blank. It shows
DocID OK, though. I expect Luke to be able to show non-string values, too.
On Sun, May 30, 2010 at 10:57 AM, Erick Erickson
<er...@gmail.com> wrote:
> Then you have to provide a lot more detail about what you did
> and what you're seeing and what you think you should see. You
> might review this page:
> http://wiki.apache.org/solr/UsingMailingLists
>
> Best
> Erick
>
> On Sun, May 30, 2010 at 1:41 PM, jlist9 <jl...@gmail.com> wrote:
>
>> I find in the Plugins tab that the default is PersianAnalyzer. I switched
>> to StandardAnalyzer and tried a few different Lucene Compatibility values
>> but it didn't help :-(
>>
>> On Sun, May 30, 2010 at 4:40 AM, Erick Erickson <er...@gmail.com>
>> wrote:
>> > The Solr admin page as access to (and uses) the field
>> > definitions you've put in the config file. Luke has no
>> > knowledge of this configuration, you have to choose
>> > your analyzer from the drop down and select the one
>> > closest to what's in your config file for SOLR. Are you
>> > perhaps using an analyzer in Luke that doesn't
>> > play nice with the definitions in SOLR?
>> >
>> > HTH
>> > Erick
>> >
>> >
>> > On Sat, May 29, 2010 at 10:55 PM, jlist9 <jl...@gmail.com> wrote:
>> >
>> >> I tried the stand-alone Luke tool (not Luke request handler) to browse
>> >> a solr index and find a few strange things:
>> >>
>> >> 1. Queries like "id:123" which work fine in /solr/admin web interface
>> >> returns nothing in Luke. "*:*" returns everything fine in Luke.
>> >>
>> >> 2. When Luke displays records with query "*:*", it shows the string
>> >> values fine but the numeric fields and date fields shows blank. It shows
>> >> DocID OK, though.
>> >>
>> >> Anyone else has tried Luke on a solr index?
>> >>
>> >
>>
>
Re: Luke browser does not show non-String Solr fields?
Posted by Erick Erickson <er...@gmail.com>.
Then you have to provide a lot more detail about what you did
and what you're seeing and what you think you should see. You
might review this page:
http://wiki.apache.org/solr/UsingMailingLists
Best
Erick
On Sun, May 30, 2010 at 1:41 PM, jlist9 <jl...@gmail.com> wrote:
> I find in the Plugins tab that the default is PersianAnalyzer. I switched
> to StandardAnalyzer and tried a few different Lucene Compatibility values
> but it didn't help :-(
>
> On Sun, May 30, 2010 at 4:40 AM, Erick Erickson <er...@gmail.com>
> wrote:
> > The Solr admin page as access to (and uses) the field
> > definitions you've put in the config file. Luke has no
> > knowledge of this configuration, you have to choose
> > your analyzer from the drop down and select the one
> > closest to what's in your config file for SOLR. Are you
> > perhaps using an analyzer in Luke that doesn't
> > play nice with the definitions in SOLR?
> >
> > HTH
> > Erick
> >
> >
> > On Sat, May 29, 2010 at 10:55 PM, jlist9 <jl...@gmail.com> wrote:
> >
> >> I tried the stand-alone Luke tool (not Luke request handler) to browse
> >> a solr index and find a few strange things:
> >>
> >> 1. Queries like "id:123" which work fine in /solr/admin web interface
> >> returns nothing in Luke. "*:*" returns everything fine in Luke.
> >>
> >> 2. When Luke displays records with query "*:*", it shows the string
> >> values fine but the numeric fields and date fields shows blank. It shows
> >> DocID OK, though.
> >>
> >> Anyone else has tried Luke on a solr index?
> >>
> >
>
Re: Luke browser does not show non-String Solr fields?
Posted by jlist9 <jl...@gmail.com>.
I find in the Plugins tab that the default is PersianAnalyzer. I switched
to StandardAnalyzer and tried a few different Lucene Compatibility values
but it didn't help :-(
On Sun, May 30, 2010 at 4:40 AM, Erick Erickson <er...@gmail.com> wrote:
> The Solr admin page as access to (and uses) the field
> definitions you've put in the config file. Luke has no
> knowledge of this configuration, you have to choose
> your analyzer from the drop down and select the one
> closest to what's in your config file for SOLR. Are you
> perhaps using an analyzer in Luke that doesn't
> play nice with the definitions in SOLR?
>
> HTH
> Erick
>
>
> On Sat, May 29, 2010 at 10:55 PM, jlist9 <jl...@gmail.com> wrote:
>
>> I tried the stand-alone Luke tool (not Luke request handler) to browse
>> a solr index and find a few strange things:
>>
>> 1. Queries like "id:123" which work fine in /solr/admin web interface
>> returns nothing in Luke. "*:*" returns everything fine in Luke.
>>
>> 2. When Luke displays records with query "*:*", it shows the string
>> values fine but the numeric fields and date fields shows blank. It shows
>> DocID OK, though.
>>
>> Anyone else has tried Luke on a solr index?
>>
>
Re: Luke browser does not show non-String Solr fields?
Posted by Erick Erickson <er...@gmail.com>.
The Solr admin page as access to (and uses) the field
definitions you've put in the config file. Luke has no
knowledge of this configuration, you have to choose
your analyzer from the drop down and select the one
closest to what's in your config file for SOLR. Are you
perhaps using an analyzer in Luke that doesn't
play nice with the definitions in SOLR?
HTH
Erick
On Sat, May 29, 2010 at 10:55 PM, jlist9 <jl...@gmail.com> wrote:
> I tried the stand-alone Luke tool (not Luke request handler) to browse
> a solr index and find a few strange things:
>
> 1. Queries like "id:123" which work fine in /solr/admin web interface
> returns nothing in Luke. "*:*" returns everything fine in Luke.
>
> 2. When Luke displays records with query "*:*", it shows the string
> values fine but the numeric fields and date fields shows blank. It shows
> DocID OK, though.
>
> Anyone else has tried Luke on a solr index?
>