You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by György Frivolt <gy...@gmail.com> on 2010/03/02 17:07:02 UTC
Indexing HTML document
Hi, How to index properly HTML documents? All the documents are HTML, some
containing charaters encodid like ží ... Is there a character
filter for filtering these codes? Is there a way to strip the HTML tags out?
Does solr weight the terms in the document based on where they appear?..
words in headers (H1, H2,..) would be supposed to describe the document more
then words in paragraphs.
Thanks for help,
Georg
Re: Indexing HTML document
Posted by György Frivolt <fi...@gmail.com>.
Thank you! That's even more I wanted to know. ;)
Georg
On Tue, Mar 2, 2010 at 10:05 PM, Walter Underwood <wu...@wunderwood.org>wrote:
> You are in luck, because Avi Rappoport has just written a tutorial about
> how to do this. It is available from Lucid Imagination:
>
>
> http://www.lucidimagination.com/solutions/whitepapers/Indexing-Text-and-HTML-Files-with-Solr
>
> I've just started reviewing it, but knowing Avi, I expect it to be very
> helpful.
>
> wunder
>
> On Mar 2, 2010, at 8:28 AM, Siddhant Goel wrote:
>
> > There is an HTML filter documented here, which might be of some help -
> >
> http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.HTMLStripCharFilterFactory
> >
> > Control characters can be eliminated using code like this -
> >
> http://bitbucket.org/cogtree/python-solr/src/tip/pythonsolr/pysolr.py#cl-449
> >
> > On Tue, Mar 2, 2010 at 9:37 PM, György Frivolt <gyorgy.frivolt@gmail.com
> >wrote:
> >
> >> Hi, How to index properly HTML documents? All the documents are HTML,
> some
> >> containing charaters encodid like ží ... Is there a character
> >> filter for filtering these codes? Is there a way to strip the HTML tags
> >> out?
> >> Does solr weight the terms in the document based on where they appear?..
> >> words in headers (H1, H2,..) would be supposed to describe the document
> >> more
> >> then words in paragraphs.
> >>
> >> Thanks for help,
> >>
> >> Georg
> >>
> >
> >
> >
> > --
> > - Siddhant
>
>
Re: Indexing HTML document
Posted by Walter Underwood <wu...@wunderwood.org>.
You are in luck, because Avi Rappoport has just written a tutorial about how to do this. It is available from Lucid Imagination:
http://www.lucidimagination.com/solutions/whitepapers/Indexing-Text-and-HTML-Files-with-Solr
I've just started reviewing it, but knowing Avi, I expect it to be very helpful.
wunder
On Mar 2, 2010, at 8:28 AM, Siddhant Goel wrote:
> There is an HTML filter documented here, which might be of some help -
> http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.HTMLStripCharFilterFactory
>
> Control characters can be eliminated using code like this -
> http://bitbucket.org/cogtree/python-solr/src/tip/pythonsolr/pysolr.py#cl-449
>
> On Tue, Mar 2, 2010 at 9:37 PM, György Frivolt <gy...@gmail.com>wrote:
>
>> Hi, How to index properly HTML documents? All the documents are HTML, some
>> containing charaters encodid like ží ... Is there a character
>> filter for filtering these codes? Is there a way to strip the HTML tags
>> out?
>> Does solr weight the terms in the document based on where they appear?..
>> words in headers (H1, H2,..) would be supposed to describe the document
>> more
>> then words in paragraphs.
>>
>> Thanks for help,
>>
>> Georg
>>
>
>
>
> --
> - Siddhant
Re: Indexing HTML document
Posted by Siddhant Goel <si...@gmail.com>.
There is an HTML filter documented here, which might be of some help -
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.HTMLStripCharFilterFactory
Control characters can be eliminated using code like this -
http://bitbucket.org/cogtree/python-solr/src/tip/pythonsolr/pysolr.py#cl-449
On Tue, Mar 2, 2010 at 9:37 PM, György Frivolt <gy...@gmail.com>wrote:
> Hi, How to index properly HTML documents? All the documents are HTML, some
> containing charaters encodid like ží ... Is there a character
> filter for filtering these codes? Is there a way to strip the HTML tags
> out?
> Does solr weight the terms in the document based on where they appear?..
> words in headers (H1, H2,..) would be supposed to describe the document
> more
> then words in paragraphs.
>
> Thanks for help,
>
> Georg
>
--
- Siddhant