You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by jimmy nguyen <ji...@gmail.com> on 2013/10/04 00:36:35 UTC

Adding attributes to Solr fields ?

Hi all,

is it possible to add attributes to our Solr fields ?

 I'm indexing GATE-annotated documents into solr. The annotations produced
by my GATE application usually have several features (for example,
Person.title, Person.name, Person.phoneNumber...).
Now each of my documents may contain more than one Person annotation, and
each person might have more than one phone number... Unfortunately I don't
know how to index all the features for one annotation in one field in solr.

So instead, I would like to add an attribute "id" (or "offset") to each of
the features I'm sending to Solr in order to be able to find out, for
example, which Person.name goes with which Person.phoneNumber.

So instead of:
<doc> <str name="id">1</str>
<arr name="person"> <str>Jane Doe</str> <str>John Doe</str>
<arr name="person_phoneNumber"> <str>0123456789</str> <str>1234567890</str>
<str>2345678901</str>
</doc>

I'd like to get something like this in Solr:
<doc>
<str name="id">1</str>
<arr name="person"> <str id="1">Jane Doe</str> <str id="2">John Doe</str>
<arr name="person_phoneNumber"> <str id="1">0123456789</str> <str id="1">
1234567890</str>
<str id="2">2345678901</str>
</doc>

This way it is easy to link the 2 first phone numbers to Jane Doe and the
last one to John Doe.

Any idea ?

Thanks !
Jim

Re: Adding attributes to Solr fields ?

Posted by jimmy nguyen <ji...@gmail.com>.
Hello,

and thank you for your answer Shawn.

I tried to simplify my problem but I realize I chose a bad example : I
don't process phone numbers, and I do process unstructured documents.

My GATE application might return several annotations for the same group of
words (because I'm using an ontology). So for example, I will have an
Animal annotation, which marks the words "cat", "catfish" and "eider" as
Animal(s), and (depending on the ontology used) the "cat" annotation will
have 2 features : Animal.class=mammal and Animal.class="cat", the "catfish"
will have 1 feature Animal.class=fish, and the more specific term "eider"
will have 2 features : Animal.class=bird, Animal.class=duck.

I don't want to consider 1 solr "document" for each animal, I really want 1
index for each actual document. I'd like to be able to query my solr index
for "bird" and get all the documents containing the terms "bird", or any
subclass or instance (like "duck" or "eider"). Since all the words "bird",
"duck" and "eider" appearing in my documents will be tagged as Animal and
there will be an annotation with Animal.class=bird, it is easy to get Solr
to return the right documents.

But since I get something like :

<result>
  <doc>
    <str name="id">hdfs://...</str>
    <arr name="animal">
      <str>cat</str>
      <str>cat</str>
      <str>catfish</str>
      <str>eider</str>
      <str>eider</str>
    </arr>
    <arr name="class">
      <str>mammal</str>
      <str>cat</str>
      <str>fish</str>
      <str>bird</str>
      <str>duck</str>
    </arr>
    <arr name="instance">
      <str>http://.../Animal#catfish</str>
      <str>http://.../Animal#eider</str>
      <str>http://.../Animal#eider</str>
    </arr>
  </doc>
  <doc>
       ...
  </doc>
  <doc>
       ...
  </doc>
</result>

... when I want to generate a snippet of the document and highlight the
terms whose appearance made solr return the document (like the first
document containing "eider" when the user is searching for a "bird"), I'd
like to highlight the term "eider" in the snippet, but I don't know how to
do that. Having a correspondance between my solr "animal" and "class"
fields (for example, an id attribute that would link them : <str
id="5">eider</str> and the same id for the class "bird") would make it
easier to highlight my term "eider".

What do you think ?

Thanks !
Jim

Re: Adding attributes to Solr fields ?

Posted by Shawn Heisey <so...@elyograg.org>.
On 10/3/2013 4:36 PM, jimmy nguyen wrote:
> I'd like to get something like this in Solr:
> <doc>
> <str name="id">1</str>
> <arr name="person"> <str id="1">Jane Doe</str> <str id="2">John Doe</str>
> <arr name="person_phoneNumber"> <str id="1">0123456789</str> <str id="1">
> 1234567890</str>
> <str id="2">2345678901</str>
> </doc>
>
> This way it is easy to link the 2 first phone numbers to Jane Doe and the
> last one to John Doe.

Attributes like that are not something that Solr will do out of the 
box.  With some custom code, you might be able to create something like 
that, but I'm not sure that you really need to do that.

When dealing with a search engine, and Solr in particular, you need to 
think in terms of a flat data model, without relational features of any 
kind.  Solr does have some limited join capability where it can use two 
indexes to return results, but if you're wanting to jump right in and 
use that capability, there's a good chance that you need to back up and 
think about a redesign with a flat data model.  Solr is not a relational 
database, it's a search engine.  It does have features that let it fill 
a NoSQL database role, but that's not what it's designed to do.

The first decision is what a single Solr document will represent.  Do 
you want phone numbers or people to be the basic unit in your index?  
Here's what I think you'd probably actually want for a typical 
document.  In this example, phoneNumber is a multi-valued field:

<doc>
<str name="id">1</str>
<str name="firstName">Jane</str>
<strname="lastName">Doe</str>
<str name="phoneNumber">0123456789</str>
<str name="phoneNumber">1234567890</str>
</doc>

In case you were thinking of using Solr as your primary data store, 
don't.  Unless you've got a flat data model, it wouldn't be very good 
for that role.  Also, given that most schema changes and some config 
changes require a reindex, you'll want to have your data available 
elsewhere so you can accomplish that.

Thanks,
Shawn