Re: Luke browser does not show non-String Solr fields?

[jlist9 <jl...@gmail.com>]

The id field has type "long" in schema.xml. In Luke, they are shown
as "hex dump". When viewing a doc (returned by *:*), I pick the ID field
and press the "Show" button, Luke pops up a dialog that allows me
to change the "Show Content As" value. When I choose "Number",
I get an error message:

"Some values could not be properly represented in this format. They
are marked in grey and presented as a hex dump."

So it seems like Luke does not understand Solr's long type. This
is not a native Lucene type?

On Mon, May 31, 2010 at 9:52 AM, Chris Hostetter
<ho...@fucit.org> wrote:
>
> : 1. Queries like "id:123" which work fine in /solr/admin web interface but
> : returns nothing in Luke. Query "*:*" returns all records fine in Luke. I
> : expect Luke returns the same result as /solr/admin since it's essentially
> : a Lucene query?
>
> you haven't told us what fieldtype you are using for the "id" field -- but
> i'm going to go out on a limb and guess it's TrieIntFieldType (or possibly
> a SortedIntFieldType) ... those field types encode their values  in such a
> way that they sort lexigraphicaly and produce faster range queries -- if
> Luke doesn't kow about that special encoding, it can search on them (or
> even display the terms properly)
>
> Luke has a "view terms" feature right? ... look at the raw terms in your
> "id" ifeld and i bet you'll see they look nothing like numbers -- and
> that's why you can search on them as numbers in Luke
>
> (when you serach on them in Solr, SOlr knows about your schema, and knows
> about your field types, and can do the proper encoding/decoding)
>
>
>
> -Hoss
>
>

Re: Luke browser does not show non-String Solr fields?

[jlist9 <jl...@gmail.com>]

Thank you Chris. I'm clear now. I'll give Luke's latest version a try
when it's out.

On Wed, Jun 2, 2010 at 9:47 AM, Chris Hostetter
<ho...@fucit.org> wrote:
>
> : I see. It's still a little confusing to me but I'm fine as long as
> : this is the expected behavior. I also tried the "example" index
> : with data that come with the solr distribution and observe the
> : same behavior - only String fields are displayed. So Lucene is
> : sharing _some_ types with Solr but not all. It's still a bit puzzling
> : to me that Lucene is not able to understand the simple types
> : such as long. But I'm OK as long as there is a reason. Thanks
> : for the explanations!
>
> The key is that there are *no* types in Lucene ... older
> versions of Lucene only supported "Strin" and clinets that wanted to index
> other types had to encode those types in some way as needed.  More
> recently lucene has started moving away from even dealing with Strings,
> and towards just indexing/searching raw byte[] ... all concepts of "field
> types" in Solr are specific to Solr
>
> (the caveat being that Lucene has, over the years, added utilities to help
> people make smart choices about how to encode some data types -- and in
> the case of the Trie numeric fields SOlr uses those utilites.  But that
> data isn't stored anywhere in the index files themselves, so Luke has no
> way of knowing that it should attempt to "decode" the binary data of a
> field using the Trie utilities.  That said: aparently Andrzej is working
> on making it possible to tell Luke "oh BTW, i indexed this field using
> this solr fieldType" ... i think he said it was on the Luke trunk)
>
>
> -Hoss

Re: Luke browser does not show non-String Solr fields?

[Chris Hostetter <ho...@fucit.org>]

: I see. It's still a little confusing to me but I'm fine as long as
: this is the expected behavior. I also tried the "example" index
: with data that come with the solr distribution and observe the
: same behavior - only String fields are displayed. So Lucene is
: sharing _some_ types with Solr but not all. It's still a bit puzzling
: to me that Lucene is not able to understand the simple types
: such as long. But I'm OK as long as there is a reason. Thanks
: for the explanations!

The key is that there are *no* types in Lucene ... older 
versions of Lucene only supported "Strin" and clinets that wanted to index 
other types had to encode those types in some way as needed.  More 
recently lucene has started moving away from even dealing with Strings, 
and towards just indexing/searching raw byte[] ... all concepts of "field 
types" in Solr are specific to Solr 

(the caveat being that Lucene has, over the years, added utilities to help 
people make smart choices about how to encode some data types -- and in 
the case of the Trie numeric fields SOlr uses those utilites.  But that 
data isn't stored anywhere in the index files themselves, so Luke has no 
way of knowing that it should attempt to "decode" the binary data of a 
field using the Trie utilities.  That said: aparently Andrzej is working 
on making it possible to tell Luke "oh BTW, i indexed this field using 
this solr fieldType" ... i think he said it was on the Luke trunk)


-Hoss

Re: Luke browser does not show non-String Solr fields?

[jlist9 <jl...@gmail.com>]

I see. It's still a little confusing to me but I'm fine as long as
this is the expected behavior. I also tried the "example" index
with data that come with the solr distribution and observe the
same behavior - only String fields are displayed. So Lucene is
sharing _some_ types with Solr but not all. It's still a bit puzzling
to me that Lucene is not able to understand the simple types
such as long. But I'm OK as long as there is a reason. Thanks
for the explanations!

On Tue, Jun 1, 2010 at 10:38 AM, Chris Hostetter
<ho...@fucit.org> wrote:
>
> : So it seems like Luke does not understand Solr's long type. This
> : is not a native Lucene type?
>
> No,  Lucene has concept of "types" ... there are utilities to help encode
> some data in special ways (particularly numbers) but the underlying lucene
> index doesn't keep track of when/how you do ths -- so Luke has no way of
> knowing what "type" the field is.
>
> Schema information is specific to Solr.
>
>
> -Hoss
>
>

Re: Luke browser does not show non-String Solr fields?

[Chris Hostetter <ho...@fucit.org>]

: So it seems like Luke does not understand Solr's long type. This
: is not a native Lucene type?

No,  Lucene has concept of "types" ... there are utilities to help encode 
some data in special ways (particularly numbers) but the underlying lucene 
index doesn't keep track of when/how you do ths -- so Luke has no way of 
knowing what "type" the field is.

Schema information is specific to Solr.


-Hoss