Optimized memory used

[msaunier <ms...@citya.com>]

Hello Karl,

 

With my ManifoldCF, I have 12 Go used sometimes. 

I would like to know if certain actions make it possible to optimize this
use of memory. Commits more frequent for example?

 

Thanks,

Maxence,

Re: Optimized memory used

[Karl Wright <da...@gmail.com>]

ManifoldCF's usage of memory is bounded per thread, but obviously scales
with the number of worker threads you have.  If you are using Tika, the
amount of memory that may be used varies a lot, however, because Tika's
streaming document memory behavior is quite variable, depending on the kind
of document it is extracting from, and we have no control over that.  So
you need to set memory parameters experimentally when using Tika.

If you need to separate out who is causing memory problems and how much is
needed, you can run Tika in "server" mode and use the Tika external
transformer which will allow ManifoldCF to work with it.  That's not as
flexible but at least you can get a better idea of where the memory is
going and who to complain to.

Karl



On Tue, Jul 24, 2018 at 5:43 AM msaunier <ms...@citya.com> wrote:

> Hello Karl,
>
>
>
> With my ManifoldCF, I have 12 Go used sometimes.
>
> I would like to know if certain actions make it possible to optimize this
> use of memory. Commits more frequent for example?
>
>
>
> Thanks,
>
> Maxence,
>