You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Ranveer Kumar <ra...@gmail.com> on 2010/04/14 11:24:52 UTC
HTMLStripCharFilterFactory configuration problem
Hi all,
I am facing problem to configure HTMLStripCharFilterFactory.
following is the schema :
<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<charFilter class="solr.HTMLStripCharFilterFactory"/>
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.StopFilterFactory"
ignoreCase="true"
words="stopwords.txt"
enablePositionIncrements="true"
/>
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="1"
catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory" language="English"
protected="protwords.txt"/>
</analyzer>
<analyzer type="query">
<charFilter class="solr.HTMLStripCharFilterFactory"/><!--
escapedTags="<,>"/> -->
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<!-- <filter class="solr.LengthFilterFactory" min="2" max="50" /> -->
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory"
ignoreCase="true"
words="stopwords.txt"
enablePositionIncrements="true"
/>
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1" generateNumberParts="1" catenateWords="0"
catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.SnowballPorterFilterFactory" language="English"
protected="protwords.txt"/>
</analyzer>
</fieldType>
when I am checking with analysis.jsp it giving true result. But in my query
result still I am getting html tage..
I am using solrj client..
please help me
Re: HTMLStripCharFilterFactory configuration problem
Posted by Ahmet Arslan <io...@yahoo.com>.
> Actually I am using SolrJ client..
> Is there anyway to do same using solrj.
>
> thanks
If you are using Java, life is easier. You can use this static function before adding a field to SolrInputDocument.
static String stripHTMLX(String value) {
StringBuilder out = new StringBuilder();
StringReader strReader = new StringReader(value);
try {
HTMLStripCharFilter html = new HTMLStripCharFilter(CharReader.get(strReader.markSupported() ? strReader : new BufferedReader(strReader)));
char[] cbuf = new char[1024 * 10];
while (true) {
int count = html.read(cbuf);
if (count == -1)
break; // end of stream mark is -1
if (count > 0)
out.append(cbuf, 0, count);
}
html.close();
} catch (IOException e) {
e.printStackTrace();
return null;
// "Failed stripping HTML for column: " + column, e);
}
return out.toString();
}
Re: HTMLStripCharFilterFactory configuration problem
Posted by Ranveer Kumar <ra...@gmail.com>.
thanks..
Actually I am using SolrJ client..
Is there anyway to do same using solrj.
thanks
On Sat, Apr 17, 2010 at 8:06 PM, Ahmet Arslan <io...@yahoo.com> wrote:
>
>
> > Thanks for reply..
> > but how will I get the stored value instead of indexed
> > value..
> > where I need to configure to get stored instead of indexed
> > value.
> > please help...
> >
>
> You need to remove html tags before analysis (charfilter, tokenizer,
> tokenfilter) phase. For example if you are using DIH to index, you can use
> HTMLStripTransformer[1]. How are you indexing your data?
>
> [1]http://wiki.apache.org/solr/DataImportHandler#HTMLStripTransformer
>
>
>
>
Re: HTMLStripCharFilterFactory configuration problem
Posted by Ahmet Arslan <io...@yahoo.com>.
> Thanks for reply..
> but how will I get the stored value instead of indexed
> value..
> where I need to configure to get stored instead of indexed
> value.
> please help...
>
You need to remove html tags before analysis (charfilter, tokenizer, tokenfilter) phase. For example if you are using DIH to index, you can use HTMLStripTransformer[1]. How are you indexing your data?
[1]http://wiki.apache.org/solr/DataImportHandler#HTMLStripTransformer
Re: HTMLStripCharFilterFactory configuration problem
Posted by Ranveer Kumar <ra...@gmail.com>.
Hi Sven,
Thanks for reply..
but how will I get the stored value instead of indexed value..
where I need to configure to get stored instead of indexed value.
please help...
thanks
with regards
On Wed, Apr 14, 2010 at 3:16 PM, Sven Maurmann <sv...@kippdata.de>wrote:
> Hi,
>
> please note that you get the stored value of the field as a result and
> not the indexed one.
>
> Cheers,
> Sven
>
>
> --On Wednesday, April 14, 2010 02:54:52 PM +0530 Ranveer Kumar <
> ranveer.solr@gmail.com> wrote:
>
> Hi all,
>>
>> I am facing problem to configure HTMLStripCharFilterFactory.
>> following is the schema :
>> <fieldType name="text" class="solr.TextField"
>> positionIncrementGap="100"> <analyzer type="index">
>> <charFilter class="solr.HTMLStripCharFilterFactory"/>
>> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>>
>> <filter class="solr.StopFilterFactory"
>> ignoreCase="true"
>> words="stopwords.txt"
>> enablePositionIncrements="true"
>> />
>> <filter class="solr.WordDelimiterFilterFactory"
>> generateWordParts="1" generateNumberParts="1" catenateWords="1"
>> catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
>> <filter class="solr.LowerCaseFilterFactory"/>
>> <filter class="solr.SnowballPorterFilterFactory"
>> language="English" protected="protwords.txt"/>
>> </analyzer>
>> <analyzer type="query">
>> <charFilter class="solr.HTMLStripCharFilterFactory"/><!--
>> escapedTags="<,>"/> -->
>> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>>
>> <!-- <filter class="solr.LengthFilterFactory" min="2" max="50"
>> /> --> <filter class="solr.SynonymFilterFactory"
>> synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
>> <filter class="solr.StopFilterFactory"
>> ignoreCase="true"
>> words="stopwords.txt"
>> enablePositionIncrements="true"
>> />
>> <filter class="solr.WordDelimiterFilterFactory"
>> generateWordParts="1" generateNumberParts="1" catenateWords="0"
>> catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
>> <filter class="solr.LowerCaseFilterFactory"/>
>> <filter class="solr.SnowballPorterFilterFactory"
>> language="English" protected="protwords.txt"/>
>> </analyzer>
>> </fieldType>
>>
>> when I am checking with analysis.jsp it giving true result. But in
>> my query result still I am getting html tage..
>> I am using solrj client..
>>
>> please help me
>>
>
>
>
> --
> kippdata informationstechnologie GmbH
> Sven Maurmann Tel: 0228 98549 -12
> Bornheimer Str. 33a Fax: 0228 98549 -50
> D-53111 Bonn sven.maurmann@kippdata.de
>
> HRB 8018 Amtsgericht Bonn / USt.-IdNr. DE 196 457 417
> Geschäftsführer: Dr. Thomas Höfer, Rainer Jung, Sven Maurmann
>
Re: HTMLStripCharFilterFactory configuration problem
Posted by Sven Maurmann <sv...@kippdata.de>.
Hi,
please note that you get the stored value of the field as a result and
not the indexed one.
Cheers,
Sven
--On Wednesday, April 14, 2010 02:54:52 PM +0530 Ranveer Kumar
<ra...@gmail.com> wrote:
> Hi all,
>
> I am facing problem to configure HTMLStripCharFilterFactory.
> following is the schema :
> <fieldType name="text" class="solr.TextField"
> positionIncrementGap="100"> <analyzer type="index">
> <charFilter class="solr.HTMLStripCharFilterFactory"/>
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>
> <filter class="solr.StopFilterFactory"
> ignoreCase="true"
> words="stopwords.txt"
> enablePositionIncrements="true"
> />
> <filter class="solr.WordDelimiterFilterFactory"
> generateWordParts="1" generateNumberParts="1" catenateWords="1"
> catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.SnowballPorterFilterFactory"
> language="English" protected="protwords.txt"/>
> </analyzer>
> <analyzer type="query">
> <charFilter class="solr.HTMLStripCharFilterFactory"/><!--
> escapedTags="<,>"/> -->
> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>
> <!-- <filter class="solr.LengthFilterFactory" min="2" max="50"
> /> --> <filter class="solr.SynonymFilterFactory"
> synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
> <filter class="solr.StopFilterFactory"
> ignoreCase="true"
> words="stopwords.txt"
> enablePositionIncrements="true"
> />
> <filter class="solr.WordDelimiterFilterFactory"
> generateWordParts="1" generateNumberParts="1" catenateWords="0"
> catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.SnowballPorterFilterFactory"
> language="English" protected="protwords.txt"/>
> </analyzer>
> </fieldType>
>
> when I am checking with analysis.jsp it giving true result. But in
> my query result still I am getting html tage..
> I am using solrj client..
>
> please help me
--
kippdata informationstechnologie GmbH
Sven Maurmann Tel: 0228 98549 -12
Bornheimer Str. 33a Fax: 0228 98549 -50
D-53111 Bonn sven.maurmann@kippdata.de
HRB 8018 Amtsgericht Bonn / USt.-IdNr. DE 196 457 417
Geschäftsführer: Dr. Thomas Höfer, Rainer Jung, Sven Maurmann