You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-user@lucene.apache.org by Sandro Feuillet <sa...@zehnplus.ch> on 2012/04/02 14:57:35 UTC

Problems with indexing of huge textfiles (drupal/tika/solr)

Hi,

We have troubles indexing big text files with Solr.
We extract PDF files with Tika and try to index them with Solr.
But Solr doesn't index the entire text. As soon as a certain amount of text
is reached Solr stopps indexing the rest. We haven't found a setting or
parameter wich defines the amount of text to index per Node/Document. Wher
is this limit set or how can we increase it?
At the Moment the Limit is somwhere around 40k Characters or 69kb Text.

Best Regards,
Sandro

-- 
.......... .
Sandro Feuillet

zehnplus GmbH
Binzmühlestrasse 210
CH-8050 Zürich

Telefon:  +41 43 288 58 49
Mobil:    +41 76 422 30 22
E-Mail:   sandro.feuillet@zehnplus.ch
Internet: http://www.zehnplus.ch
.......... .

Re: Problems with indexing of huge textfiles (drupal/tika/solr)

Posted by Erick Erickson <er...@gmail.com>.

And probably 10,000 tokens (words). See maxFieldLength
in solrconfig.xml.

Best
Erick

On Mon, Apr 2, 2012 at 8:57 AM, Sandro Feuillet
<sa...@zehnplus.ch> wrote:
> Hi,
>
> We have troubles indexing big text files with Solr.
> We extract PDF files with Tika and try to index them with Solr.
> But Solr doesn't index the entire text. As soon as a certain amount of text
> is reached Solr stopps indexing the rest. We haven't found a setting or
> parameter wich defines the amount of text to index per Node/Document. Wher
> is this limit set or how can we increase it?
> At the Moment the Limit is somwhere around 40k Characters or 69kb Text.
>
> Best Regards,
> Sandro
>
> --
> .......... .
> Sandro Feuillet
>
> zehnplus GmbH
> Binzmühlestrasse 210
> CH-8050 Zürich
>
> Telefon:  +41 43 288 58 49
> Mobil:    +41 76 422 30 22
> E-Mail:   sandro.feuillet@zehnplus.ch
> Internet: http://www.zehnplus.ch
> .......... .