You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Tanner Postert <ta...@gmail.com> on 2011/02/15 23:43:32 UTC
solr.HTMLStripCharFilterFactory not working
I have several fields defined and one of the field types includes a
solr.HTMLStripCharFilterFactory field in the analyzer but it doesn't appear
to be affecting the field as I would expect.
I have tried a simple:
<charFilter class="solr.HTMLStripCharFilterFactory">
followed by the tokenizer
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
or the combined factory
<tokenizer class="solr.HTMLStripWhitespaceTokenizerFactory" />
but neither seems to work.
Returned search results from the webtitle & webdescription as well as text
include the original HTML characters that the title & description fields
have.
The relevant schema:
<types>
<fieldType name="string" class="solr.StrField" sortMissingLast="true"
omitNorms="true"/>
<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.HTMLStripWhitespaceTokenizerFactory" />
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt" enablePositionIncrements="true"/>
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
generateNumberParts="1" catenateWords="1" catenateNumbers="1"
catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory" language="English"
protected="protwords.txt"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.HTMLStripWhitespaceTokenizerFactory" />
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt" enablePositionIncrements="true"/>
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
generateNumberParts="1" catenateWords="0" catenateNumbers="0"
catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory" language="English"
protected="protwords.txt"/>
</analyzer>
</fieldType>
<fieldType name="textSpell" class="solr.TextField"
positionIncrementGap="100" omitNorms="true">
<analyzer type="index">
<tokenizer class="solr.HTMLStripStandardTokenizerFactory" />
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StandardFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.HTMLStripStandardTokenizerFactory" />
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.StandardFilterFactory"/>
</analyzer>
</fieldType>
</types>
<fields>
<field name="title" type="string" index="true"
stored="true" multiValued="false" />
<field name="webtitle" type="text" index="true"
stored="true" multiValued="false" />
<copyField source="title" dest="webtitle" />
<field name="description" type="string" index="true"
stored="true" multiValued="false" compressed="true" />
<field name="webdescription" type="text" index="true"
stored="true" mutliValued="false" compressed="true" />
<copyField source="description" dest="webdescription" />
<field name="spell" type="textSpell" index="true"
stored="true" multiValued="true" />
<copyField source="title" dest="spell" />
<copyField source="description" dest="spell" />
<field name="text" type="text" index="true" stored="true"
multiValued="true" />
<copyField source="title" dest="text" />
<copyField source="description" dest="text" />
</fields>
Re: solr.HTMLStripCharFilterFactory not working
Posted by Tanner Postert <ta...@gmail.com>.
I updated my data importer.
I used to have:
<field column="webtitle" stripHTML="true" />
<field column="webdescription" stripHTML="true" />
which wasn't working. But I changed that to
<field column="webtitle" name="webtitle" stripHTML="true" />
<field column="webdescription" name="webdescription" stripHTML="true" />
and it is working fine.
On Tue, Feb 15, 2011 at 5:50 PM, Koji Sekiguchi <ko...@r.email.ne.jp> wrote:
> (11/02/16 8:03), Tanner Postert wrote:
>
>> I am using the data import handler and using the HTMLStripTransformer
>> doesn't seem to be working either.
>>
>> I've changed webtitle and webdescription to not by copied from title and
>> description in the schema.xml file then set them both to just but
>> duplicates
>> of title and description in the data importer query:
>>
>> <document name="items">
>> <entity dataSource="db" name="item" transformer="HTMLStripTransformer"
>> query="select
>> title as title,
>> title as webtitle,
>> description as description,
>> description as webdescription
>> FROM ...>
>> <field column="webtitle" stripHTML="true" />
>> <field column="webdescription" stripHTML="true" />
>> </entity>
>> </document>
>>
>>
> Just for input (I'm not sure that I could help you), I'm using
> HTMLStripTransformer
> with PlainTextEntityProcessor and it works fine with me:
>
> <dataConfig>
> <dataSource name="f" type="URLDataSource" encoding="UTF-8"
> baseUrl="http://lucene.apache.org/"/>
> <document>
> <entity name="solr" processor="PlainTextEntityProcessor"
> transformer="HTMLStripTransformer"
> dataSource="f" url="solr/">
> <field column="plainText" name="text" stripHTML="true"/>
> </entity>
> </document>
> </dataConfig>
>
> Koji
> --
> http://www.rondhuit.com/en/
>
Re: solr.HTMLStripCharFilterFactory not working
Posted by Koji Sekiguchi <ko...@r.email.ne.jp>.
(11/02/16 8:03), Tanner Postert wrote:
> I am using the data import handler and using the HTMLStripTransformer
> doesn't seem to be working either.
>
> I've changed webtitle and webdescription to not by copied from title and
> description in the schema.xml file then set them both to just but duplicates
> of title and description in the data importer query:
>
> <document name="items">
> <entity dataSource="db" name="item" transformer="HTMLStripTransformer"
> query="select
> title as title,
> title as webtitle,
> description as description,
> description as webdescription
> FROM ...>
> <field column="webtitle" stripHTML="true" />
> <field column="webdescription" stripHTML="true" />
> </entity>
> </document>
>
Just for input (I'm not sure that I could help you), I'm using HTMLStripTransformer
with PlainTextEntityProcessor and it works fine with me:
<dataConfig>
<dataSource name="f" type="URLDataSource" encoding="UTF-8"
baseUrl="http://lucene.apache.org/"/>
<document>
<entity name="solr" processor="PlainTextEntityProcessor" transformer="HTMLStripTransformer"
dataSource="f" url="solr/">
<field column="plainText" name="text" stripHTML="true"/>
</entity>
</document>
</dataConfig>
Koji
--
http://www.rondhuit.com/en/
Re: solr.HTMLStripCharFilterFactory not working
Posted by Tanner Postert <ta...@gmail.com>.
I am using the data import handler and using the HTMLStripTransformer
doesn't seem to be working either.
I've changed webtitle and webdescription to not by copied from title and
description in the schema.xml file then set them both to just but duplicates
of title and description in the data importer query:
<document name="items">
<entity dataSource="db" name="item" transformer="HTMLStripTransformer"
query="select
title as title,
title as webtitle,
description as description,
description as webdescription
FROM ... >
<field column="webtitle" stripHTML="true" />
<field column="webdescription" stripHTML="true" />
</entity>
</document>
On Tue, Feb 15, 2011 at 3:49 PM, Tanner Postert <ta...@gmail.com>wrote:
> nevermind, I think I found my answer here:
> http://www.mail-archive.com/solr-user@lucene.apache.org/msg34622.html
>
> <http://www.mail-archive.com/solr-user@lucene.apache.org/msg34622.html>I
> will add the HTML stripper to the data importer and see how that goes
>
>
> On Tue, Feb 15, 2011 at 3:43 PM, Tanner Postert <ta...@gmail.com>wrote:
>
>> I have several fields defined and one of the field types includes a
>> solr.HTMLStripCharFilterFactory field in the analyzer but it doesn't
>> appear to be affecting the field as I would expect.
>> I have tried a simple:
>>
>> <charFilter class="solr.HTMLStripCharFilterFactory">
>> followed by the tokenizer
>> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>>
>> or the combined factory
>>
>> <tokenizer class="solr.HTMLStripWhitespaceTokenizerFactory" />
>>
>> but neither seems to work.
>>
>> Returned search results from the webtitle & webdescription as well as text
>> include the original HTML characters that the title & description fields
>> have.
>>
>> The relevant schema:
>>
>> <types>
>> <fieldType name="string" class="solr.StrField" sortMissingLast="true"
>> omitNorms="true"/>
>>
>> <fieldType name="text" class="solr.TextField" positionIncrementGap="100">
>> <analyzer type="index">
>> <tokenizer class="solr.HTMLStripWhitespaceTokenizerFactory" />
>>
>> <filter class="solr.StopFilterFactory" ignoreCase="true"
>> words="stopwords.txt" enablePositionIncrements="true"/>
>>
>> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
>> generateNumberParts="1" catenateWords="1" catenateNumbers="1"
>> catenateAll="0" splitOnCaseChange="1"/>
>> <filter class="solr.LowerCaseFilterFactory"/>
>> <filter class="solr.SnowballPorterFilterFactory" language="English"
>> protected="protwords.txt"/>
>> </analyzer>
>> <analyzer type="query">
>>
>> <tokenizer class="solr.HTMLStripWhitespaceTokenizerFactory" />
>>
>> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
>> ignoreCase="true" expand="true"/>
>>
>> <filter class="solr.StopFilterFactory" ignoreCase="true"
>> words="stopwords.txt" enablePositionIncrements="true"/>
>>
>> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
>> generateNumberParts="1" catenateWords="0" catenateNumbers="0"
>> catenateAll="0" splitOnCaseChange="1"/>
>>
>> <filter class="solr.LowerCaseFilterFactory"/>
>> <filter class="solr.SnowballPorterFilterFactory" language="English"
>> protected="protwords.txt"/>
>> </analyzer>
>> </fieldType>
>>
>> <fieldType name="textSpell" class="solr.TextField"
>> positionIncrementGap="100" omitNorms="true">
>> <analyzer type="index">
>> <tokenizer class="solr.HTMLStripStandardTokenizerFactory" />
>> <filter class="solr.StopFilterFactory" ignoreCase="true"
>> words="stopwords.txt"/>
>> <filter class="solr.LowerCaseFilterFactory"/>
>> <filter class="solr.StandardFilterFactory"/>
>> </analyzer>
>> <analyzer type="query">
>> <tokenizer class="solr.HTMLStripStandardTokenizerFactory" />
>> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
>> ignoreCase="true" expand="true"/>
>> <filter class="solr.StopFilterFactory" ignoreCase="true"
>> words="stopwords.txt"/>
>> <filter class="solr.LowerCaseFilterFactory"/>
>> <filter class="solr.StandardFilterFactory"/>
>> </analyzer>
>> </fieldType>
>> </types>
>>
>> <fields>
>> <field name="title" type="string" index="true"
>> stored="true" multiValued="false" />
>> <field name="webtitle" type="text" index="true"
>> stored="true" multiValued="false" />
>> <copyField source="title" dest="webtitle" />
>>
>> <field name="description" type="string" index="true"
>> stored="true" multiValued="false" compressed="true" />
>> <field name="webdescription" type="text" index="true"
>> stored="true" mutliValued="false" compressed="true" />
>> <copyField source="description" dest="webdescription" />
>>
>> <field name="spell" type="textSpell" index="true"
>> stored="true" multiValued="true" />
>> <copyField source="title" dest="spell" />
>> <copyField source="description" dest="spell" />
>>
>> <field name="text" type="text" index="true" stored="true"
>> multiValued="true" />
>> <copyField source="title" dest="text" />
>> <copyField source="description" dest="text" />
>>
>> </fields>
>>
>>
>
Re: solr.HTMLStripCharFilterFactory not working
Posted by Tanner Postert <ta...@gmail.com>.
nevermind, I think I found my answer here:
http://www.mail-archive.com/solr-user@lucene.apache.org/msg34622.html
<http://www.mail-archive.com/solr-user@lucene.apache.org/msg34622.html>I
will add the HTML stripper to the data importer and see how that goes
On Tue, Feb 15, 2011 at 3:43 PM, Tanner Postert <ta...@gmail.com>wrote:
> I have several fields defined and one of the field types includes a
> solr.HTMLStripCharFilterFactory field in the analyzer but it doesn't
> appear to be affecting the field as I would expect.
> I have tried a simple:
>
> <charFilter class="solr.HTMLStripCharFilterFactory">
> followed by the tokenizer
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>
> or the combined factory
>
> <tokenizer class="solr.HTMLStripWhitespaceTokenizerFactory" />
>
> but neither seems to work.
>
> Returned search results from the webtitle & webdescription as well as text
> include the original HTML characters that the title & description fields
> have.
>
> The relevant schema:
>
> <types>
> <fieldType name="string" class="solr.StrField" sortMissingLast="true"
> omitNorms="true"/>
>
> <fieldType name="text" class="solr.TextField" positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.HTMLStripWhitespaceTokenizerFactory" />
>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt" enablePositionIncrements="true"/>
>
> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
> generateNumberParts="1" catenateWords="1" catenateNumbers="1"
> catenateAll="0" splitOnCaseChange="1"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.SnowballPorterFilterFactory" language="English"
> protected="protwords.txt"/>
> </analyzer>
> <analyzer type="query">
>
> <tokenizer class="solr.HTMLStripWhitespaceTokenizerFactory" />
>
> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
> ignoreCase="true" expand="true"/>
>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt" enablePositionIncrements="true"/>
>
> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
> generateNumberParts="1" catenateWords="0" catenateNumbers="0"
> catenateAll="0" splitOnCaseChange="1"/>
>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.SnowballPorterFilterFactory" language="English"
> protected="protwords.txt"/>
> </analyzer>
> </fieldType>
>
> <fieldType name="textSpell" class="solr.TextField"
> positionIncrementGap="100" omitNorms="true">
> <analyzer type="index">
> <tokenizer class="solr.HTMLStripStandardTokenizerFactory" />
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StandardFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.HTMLStripStandardTokenizerFactory" />
> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
> ignoreCase="true" expand="true"/>
> <filter class="solr.StopFilterFactory" ignoreCase="true"
> words="stopwords.txt"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.StandardFilterFactory"/>
> </analyzer>
> </fieldType>
> </types>
>
> <fields>
> <field name="title" type="string" index="true"
> stored="true" multiValued="false" />
> <field name="webtitle" type="text" index="true"
> stored="true" multiValued="false" />
> <copyField source="title" dest="webtitle" />
>
> <field name="description" type="string" index="true"
> stored="true" multiValued="false" compressed="true" />
> <field name="webdescription" type="text" index="true"
> stored="true" mutliValued="false" compressed="true" />
> <copyField source="description" dest="webdescription" />
>
> <field name="spell" type="textSpell" index="true"
> stored="true" multiValued="true" />
> <copyField source="title" dest="spell" />
> <copyField source="description" dest="spell" />
>
> <field name="text" type="text" index="true" stored="true"
> multiValued="true" />
> <copyField source="title" dest="text" />
> <copyField source="description" dest="text" />
>
> </fields>
>
>