Facets on multiple values

[Shishir Jain <sh...@gmail.com>]

Hi,

Am using Solr facets for my data and have a field which has multiple values
in its field am using ";" to delimit those values. So after doing a solr
search it returns me a facet array but that contains ";" in the facet value.
I want facet to return each as separate values.

For eg. am using the keyword field for faceting which has value
"graduation;post graduation; diploma" so facet returns
me a "graduation;post graduation; diploma" as a single value. but what i
want is that it should return me three different values 'graduation', 'post
graduation', 'diploma'

Appreciate any help in this.

Thanks & Regards,
Shishir

Re: Facets on multiple values

[Gora Mohanty <go...@srijan.in>]

On Thu, 29 Jul 2010 20:39:57 +0530
Shishir Jain <sh...@gmail.com> wrote:

> Hi,
> 
> Am using Solr facets for my data and have a field which has
> multiple values in its field am using ";" to delimit those
> values. So after doing a solr search it returns me a facet array
> but that contains ";" in the facet value. I want facet to return
> each as separate values.
[...]

Not sure that I follow. Where are the ";"-separated values coming
from? If you are using ";" in your data as a separator for multiple
values, probably the best approach is to use a multi-valued Solr
field for indexing these, and an appropriate transformer to import
the values into a Solr index. This should be quite easy, and there
is good Solr documentation on this.

Regards,
Gora

Re: AW: Facets on multiple values

[Shawn Heisey <so...@elyograg.org>]

  On 7/29/2010 1:13 PM, Chris Hostetter wrote:
> : My initial approach was to grab the values (which are in another table) with a
> : DIH subentity and store them in a multivalued field, but that reduced index
> : speed to a crawl.  That's because instead of one query for the entire import,
> : it was making an individual subquery for every document returned by the main
> : query.  Switching to a left join, I couldn't see any performance difference,
> : and it's still one query.
>
> It's not clera to me how you are getting the values i nthe first place
> that getting them as a multivalued field slowed down thta much, but
> if the data is already semi-colon delimited, then the RegexTransformer can
> make a mutlivalued field out of it using splitBy.

Here's the original query before adding this new field:

SELECT *,FROM_UNIXTIME(post_date) AS pd FROM ncdat WHERE blahblahblah

This is the new query:

SELECT d.*,FROM_UNIXTIME(post_date) AS pd,GROUP_CONCAT(w.webtable 
SEPARATOR ';') AS search_group FROM ncdat d LEFT JOIN ncdat_wt w ON 
d.feature=w.featurecode WHERE blahblahblah

The abandoned initial approach kept the same main query and used its 
primary key on a second query to gather the search groups.  With 7.5 
million rows in the first query, you get 7.5 million individual queries 
against the second table, which when it's complete will only have a few 
thousand rows.  It went from taking about 5 hours to index (database is 
the bottleneck, not Solr) to about 12 hours.  Is there a way to make 
this approach faster?

Thanks,
Shawn

Re: AW: Facets on multiple values

[Chris Hostetter <ho...@fucit.org>]

: My initial approach was to grab the values (which are in another table) with a
: DIH subentity and store them in a multivalued field, but that reduced index
: speed to a crawl.  That's because instead of one query for the entire import,
: it was making an individual subquery for every document returned by the main
: query.  Switching to a left join, I couldn't see any performance difference,
: and it's still one query.

It's not clera to me how you are getting the values i nthe first place 
that getting them as a multivalued field slowed down thta much, but 
if the data is already semi-colon delimited, then the RegexTransformer can 
make a mutlivalued field out of it using splitBy.


-Hoss

Re: AW: Facets on multiple values

[Shawn Heisey <so...@elyograg.org>]

  On 7/29/2010 12:18 PM, Chris Hostetter wrote:
> it also depends on what you want to get *out* if this is a stored field
> ... using an analyzer like this will deal with letting you facet on the
> individual terms, but the stored vaue returned with each document will
> still be a single semi-colon seperated string.
>
> modifying your indexing code (either DIH, or whatever) will actually
> result in an array of distinct values being returned for the documents.

Very true.  In my case, the added field is not stored, just indexed.  I 
would only store it for debugging purposes, so it would be OK to keep 
the semicolons even then.

My initial approach was to grab the values (which are in another table) 
with a DIH subentity and store them in a multivalued field, but that 
reduced index speed to a crawl.  That's because instead of one query for 
the entire import, it was making an individual subquery for every 
document returned by the main query.  Switching to a left join, I 
couldn't see any performance difference, and it's still one query.

If I were to revamp my build system to import in chunks, I could 
probably use the DIH option that caches the queries, but I really don't 
want to do that.

Re: AW: Facets on multiple values

[Chris Hostetter <ho...@fucit.org>]

: <tokenizer class="solr.PatternTokenizerFactory" pattern="; *" />
	...
: Whether to use this idea or Bastian's depends on how the original data source
: is organized.

it also depends on what you want to get *out* if this is a stored field 
... using an analyzer like this will deal with letting you facet on the 
individual terms, but the stored vaue returned with each document will 
still be a single semi-colon seperated string.

modifying your indexing code (either DIH, or whatever) will actually 
result in an array of distinct values being returned for the documents.


-Hoss

Re: AW: Facets on multiple values

[Shawn Heisey <so...@elyograg.org>]

  I'm developing a new schema that includes something similar.  The DIH 
database select statement uses a left join to gather a set of values for 
each main record into a new field, separated by semicolons.  I put the 
result into a fieldType with the following analyzer chain, which breaks 
it up into tokens at the semicolons, then does some additional processing:

<analyzer>
<tokenizer class="solr.PatternTokenizerFactory" pattern="; *" />
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory" />
<filter class="solr.TrimFilterFactory" />
</analyzer>

Whether to use this idea or Bastian's depends on how the original data 
source is organized.


On 7/29/2010 9:20 AM, Bastian Spitzer wrote:
> just define the keyword field as multivalued and add the keywords separatly, not as single-valued-string.
>
> cheers.
>
> -----Ursprüngliche Nachricht-----
> Von: Shishir Jain [mailto:shishir.jain@gmail.com]
> Gesendet: Donnerstag, 29. Juli 2010 17:10
> An: solr-user@lucene.apache.org
> Betreff: Facets on multiple values
>
> Hi,
>
> Am using Solr facets for my data and have a field which has multiple values in its field am using ";" to delimit those values. So after doing a solr search it returns me a facet array but that contains ";" in the facet value.
> I want facet to return each as separate values.
>
> For eg. am using the keyword field for faceting which has value "graduation;post graduation; diploma" so facet returns me a "graduation;post graduation; diploma" as a single value. but what i want is that it should return me three different values 'graduation', 'post graduation', 'diploma'

AW: Facets on multiple values

[Bastian Spitzer <bs...@magix.net>]

just define the keyword field as multivalued and add the keywords separatly, not as single-valued-string.

cheers. 

-----Ursprüngliche Nachricht-----
Von: Shishir Jain [mailto:shishir.jain@gmail.com] 
Gesendet: Donnerstag, 29. Juli 2010 17:10
An: solr-user@lucene.apache.org
Betreff: Facets on multiple values

Hi,

Am using Solr facets for my data and have a field which has multiple values in its field am using ";" to delimit those values. So after doing a solr search it returns me a facet array but that contains ";" in the facet value.
I want facet to return each as separate values.

For eg. am using the keyword field for faceting which has value "graduation;post graduation; diploma" so facet returns me a "graduation;post graduation; diploma" as a single value. but what i want is that it should return me three different values 'graduation', 'post graduation', 'diploma'

Appreciate any help in this.

Thanks & Regards,
Shishir