You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Jorge Luis Betancourt González <jl...@uci.cu> on 2015/06/23 15:06:36 UTC

Re: [MASSMAIL]Split content of metatag to multi value field

I don't think there is any built-in way of doing this, the problem is essentially that for your particular use case you want the value of the meta tag to be an array instead of only one value, this means that you'll need to either change the parse-metatags plugin (or index-metadata) to accomplish the desired goal. 

I'm guessing that you want to store as a multivalued field in Solr just to get the stored value as an array instead of a comma separated list right? because for search purposes the StandardTokenizerFactory will emit a stream of separated tokens for search, and you will be able to search for any of the language codes specified in the ORIGIN meta. 

Regards,

----- Original Message -----
From: "Peter Kraume" <pe...@gmx.de>
To: user@nutch.apache.org
Sent: Monday, June 22, 2015 6:51:32 PM
Subject: [MASSMAIL]Split content of metatag to multi value field

Hi everybody,

I have a meta tag with a comma separated list of values. Is it possible to put that as multi value field to a Solr index?

The meta tag looks like this:
<meta name="ORIGIN" content="de,us,be,in,il,it" />

Currently the whole content of ORIGIN is saved in Solr as single value:
"countries_stringM": [
	"de,us,be,in,il,it"
],

My nutch-site.xml looks like this:
<property>
  <name>metatags.names</name>
  <value>origin</value>
</property>

<property>
  <name>index.parse.md</name>
  <value>metatag.origin</value>
</property>

In solrindex-mapping.xml I've added this mapping:
<field source="metatag.origin" dest="countries_stringM" />

I’m really thankful for any hints!

Cheers
Peter

Re: [MASSMAIL]Re: Split content of metatag to multi value field

Posted by Jorge Luis Betancourt González <jl...@uci.cu>.

----- Original Message -----
From: "Peter Kraume" <pe...@gmx.de>
To: user@nutch.apache.org
Sent: Tuesday, June 23, 2015 10:38:46 AM
Subject: [MASSMAIL]Re: Split content of metatag to multi value field

> Am 23.06.2015 um 15:06 schrieb Jorge Luis Betancourt González <jl...@uci.cu>:
> 
> I don't think there is any built-in way of doing this, the problem is essentially that for your particular use case you want the value of the meta tag to be an array instead of only one value, this means that you'll need to either change the parse-metatags plugin (or index-metadata) to accomplish the desired goal. 

Maybe it’s easier to change the HTML code of the website to have multiple lines of <meta name="ORIGIN" content="xx" /> with single country codes since the metatags plugin can handle this as array.

Well if you can to change the website :-) 

> 
> I'm guessing that you want to store as a multivalued field in Solr just to get the stored value as an array instead of a comma separated list right? because for search purposes the StandardTokenizerFactory will emit a stream of separated tokens for search, and you will be able to search for any of the language codes specified in the ORIGIN meta. 

If I get you right, you think I can query a comma separated string like it would be an array in Solr?

This depend on the tokenizer and analyzers you're using in your field, can you share the relevant schema.xml section? the field declaration and field type of the "countries_stringM" field should be enough. 

The general answer is that you can query this string for it parts (terms), if you're indexing this field in a solr.TextField. When you use the solr.TextField in your schema, the you'll need to define a tokenizer which only purpose is to transform the original value sent to the field into a stream/list of tokens. In your particular case, the StandardTokenizerFactory[1] will generate a stream of tokens, which basically means that it will split your original text "de,us,be,in,il,it" into a list of tokens [de, us, be, in, il, it] and this will be the terms that solr will use to match the document to a given query, so if your issue a query like: (countries_stringM:us) it will match your document.

For instance a very basic fieldType that you can use in your schema.xml file is:

<fieldType name="keywords" class="solr.TextField" sortMissingLast="true">
            <analyzer>
                <tokenizer class="solr.StandardTokenizerFactory"/>
                <filter class="solr.LowerCaseFilterFactory"/>
                <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
            </analyzer>
</fieldType

and then declare your field to use this fieldType:

<field name="countries_stringM" type="keywords" indexed="true" stored="true"/>

[1] https://cwiki.apache.org/confluence/display/solr/Tokenizers

Cheers
Peter

Re: Split content of metatag to multi value field

Posted by Peter Kraume <pe...@gmx.de>.


> Am 23.06.2015 um 15:06 schrieb Jorge Luis Betancourt González <jl...@uci.cu>:
> 
> I don't think there is any built-in way of doing this, the problem is essentially that for your particular use case you want the value of the meta tag to be an array instead of only one value, this means that you'll need to either change the parse-metatags plugin (or index-metadata) to accomplish the desired goal. 

Maybe it’s easier to change the HTML code of the website to have multiple lines of <meta name="ORIGIN" content="xx" /> with single country codes since the metatags plugin can handle this as array.

> 
> I'm guessing that you want to store as a multivalued field in Solr just to get the stored value as an array instead of a comma separated list right? because for search purposes the StandardTokenizerFactory will emit a stream of separated tokens for search, and you will be able to search for any of the language codes specified in the ORIGIN meta. 

If I get you right, you think I can query a comma separated string like it would be an array in Solr?

Cheers
Peter