Ignoring Whitespace

["Gargate, Siddharth" <sg...@ptc.com>]

After parsing HTML documents, tika adds whitespaces (newlines and tabs)
and this content gets stored as is in SOLR. If I fetch the teasers, the
teaser contains these additonal whitespaces. How do I ignore these
whitespaces? At tika, solr or explicitly remove with my code?
 
Thanks,
Siddharth

Re: Ignoring Whitespace

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Fri, Mar 6, 2009 at 10:27 AM, Gargate, Siddharth <sg...@ptc.com> wrote:
> After parsing HTML documents, tika adds whitespaces (newlines and tabs) and
> this content gets stored as is in SOLR. If I fetch the teasers, the teaser
> contains these additonal whitespaces.

Tika adds the whitespace to help indexing of content like
"<p>one</p><p>two</p>". A naive indexer would just look at the
character content of the HTML document and see the non-word "onetwo"
instead of the correct sequence "one", "two".

There's been some discussion of only adding the extra whitespace when
the source document doesn't already contain it. For example
"<p>one</p> <p>two</p>" could be left unmodified. But such a feature
is currently not implemented.

> How do I ignore these whitespaces? At tika, solr or explicitly remove with my code?

If you need more accurate representation of the original document,
then it's probably best to store the it as-is along with the indexed
version. I'm not sure if or how Solr handles this internally.

BR,

Jukka Zitting