You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tika.apache.org by Cristian Zamfir <cr...@cyberhaven.com> on 2021/05/27 21:31:36 UTC
best practices for avoiding OOM for tika docker
Hi,
I am running some stress tests of the latest tika server docker (not
modified in any way, just pulled from the registry) and seeing that after a
few hours I see OOM in the logs. The container has a limit of 4GB set in
K8S. I am wondering if you have any best practices on how to avoid this.
For instance:
- should I set the JVM heap size to match the memory limit on the
container?
- is there a way to restart in case of OOM?
- should I use spawnChild?
Thanks!
Cristi
Re: best practices for avoiding OOM for tika docker
Posted by Cristian Zamfir <cr...@cyberhaven.com>.
On Thu, May 27, 2021 at 11:31 PM Cristian Zamfir <cr...@cyberhaven.com>
wrote:
> Hi,
>
> I am running some stress tests of the latest tika server docker (not
> modified in any way, just pulled from the registry) and seeing that after a
> few hours I see OOM in the logs. The container has a limit of 4GB set in
> K8S. I am wondering if you have any best practices on how to avoid this.
> For instance:
> - should I set the JVM heap size to match the memory limit on the
> container?
> - is there a way to restart in case of OOM?
> - should I use spawnChild?
>
A related question is about health checks - a simple way to implement
external health checks would be to call any API entrypoint such as /tika
with a couple of bytes of input and check the reply, but I thought I would
check for a dedicated API for this. Thanks!
Re: best practices for avoiding OOM for tika docker
Posted by Nick Burch <ap...@gagravarr.org>.
On Wed, 2 Jun 2021, Cristian Zamfir wrote:
> 1. Do you have a recommendation for a stress test that would allow me to
> easily test OOM behavior?
Depends what kind of OOM you're interested in. If you fire a lot of
memory-hungry documents at a single server at once, you can trigger an
OOM. Alternately, if you send Tika one broken or nearly-broken document,
you can trigger an OOM. We try to fix cases of the latter when we can, but
they still occur, once of those is the easiest if you find one
> 2. For implementing a health check that detects when Tika is stuck, I could
> periodically send a simple request and check that the reply is correct, do
> you recommend a better approach?
Would /version work for you? Server needs to be up to reply, but it's a
very lightweight endpoint
Nick
Fwd: best practices for avoiding OOM for tika docker
Posted by Tim Allison <ta...@apache.org>.
dropped cc...
>I noticed that Tika prints in the logs OOM (null), but seems to recover by itself even when not using -spawnChild. Is this the expected behavior?
When not in -spawnChild mode, Tika is catching OOM exceptions (when it
can), but it isn't "recovering"... the jvm may be in an inconsistent
state, and it is safest to restart the jvm. It would probably be good
practice when in -spawnChild mode to use -XX:+ExitOnOutOfMemoryError,
or on the tika commandline -JXX:+ExitOnOutOfMemoryError.
I highly encourage you to use -spawnChild mode, or the new pipes
modules in 2.x if those will work for you at some point...those are
still beta. OOMs are one thing, but infinite loops are another.
1. Do you have a recommendation for a stress test that would allow me
to easily test OOM behavior?
The MockParser is built for exactly this:
https://cwiki.apache.org/confluence/display/TIKA/MockParser
Let us know if you have any questions about it. The key elements for
you are <fakeload/>, <throw/> <oom/> and probably <system_exit/>.
That's for synthetic load testing. If you want files in the wild, we
have 2TB of files from the wild:
https://corpora.tika.apache.org/base/docs/
2. For implementing a health check that detects when Tika is stuck, I
could periodically send a simple request and check that the reply is
correct, do you recommend a better approach?
We have a rudimentary /status endpoint, which will give you number of
restarts, number of files processed, milliseconds since last parse.
You have to turn it on via the commandline: -status.
On Wed, Jun 2, 2021 at 6:50 AM Cristian Zamfir <cr...@cyberhaven.com> wrote:
>
> Hi!
>
> I noticed that Tika prints in the logs OOM (null), but seems to recover by itself even when not using -spawnChild. Is this the expected behavior? I am trying to figure out when logs containing "OOM" are critical and would require a container restart.
>
> I also wanted to bring up two of my questions below, I am looking forward to your feedback:
> 1. Do you have a recommendation for a stress test that would allow me to easily test OOM behavior?
> 2. For implementing a health check that detects when Tika is stuck, I could periodically send a simple request and check that the reply is correct, do you recommend a better approach?
>
> Thanks,
> Cristi
>
> On Sat, May 29, 2021 at 2:58 PM Cristian Zamfir <cr...@cyberhaven.com> wrote:
>>
>>
>> > On 28 May 2021, at 19:03, Tim Allison <ta...@apache.org> wrote:
>> >
>> > Tika 2.x should help with this in pipes and async. Your system should
>> > expect to go oom or crash at some point if you're processing enough
>> > files.
>>
>> I believe that this is what is happening in my case, it’s not due to a single file, it happens under high load when processing many files at once.
>>
>> >
>> > Right --spawnChild is not default in 1.x, but it will be in 2.x. And,
>> > yes, you should be using it. To set the Xmx in the forked process add
>> > -J, as in -JXmx2g would set the Xmx for the forked process.
>>
>>
>> Did both now and I think this provides good recovery from OOM.
>>
>>
>> >
>> > I don't have experience to recommend bumping Xmx to close to your
>> > container's max memory. In java programs that do a bunch of work off
>> > heap, this would be a bad idea because you need to leave resources for
>> > your system os, but I don't think we do much off heap.
>>
>> What’s your take on a configuration in which the container is capped at 4GB and the spawned child has a heap limit of 3GB? Sounds like a pretty safe margin to me.
>>
>> >
>> > Which file types are causing OOMs? The MP4Parser is notorious, and
>> > we're looking to swap it out in 2.x for a different parser.
>>
>> Good to hear. I don’t know how to identify the root cause because there are many files sent at once.
>> However, it would be great to learn if there is a quick way to trigger a high load and test resiliency to OOM, do you have a recommendation?
>>
>>
>> >
>> > Yep, TIKA-3353 is the monitoring that Nick was mentioning.
>>
>> I am actually more interested in health checks, to detect when the system is stuck without automatically restarting. A built-in health check would certainly be a nice feature.
>>
>> Besides OOM, one other possible cause is if /tmp gets full - for instance I see here https://github.com/tongwang/tika-server-docker/blob/master/bin/healthcheck that /tmp is cleaned up periodically and the health check fails if it is too full.
>>
>> Are there any other situations that could indicate that the container is stuck and needs a restart and if yes, is there a way to detect the condition?
>>
>> Thanks,
>> Cristi
>>
>> >
>> > On Fri, May 28, 2021 at 9:08 AM Cristian Zamfir <cr...@cyberhaven.com> wrote:
>> >>
>> >> Thanks for your answer Nick!
>> >>
>> >> I am running apache/tika:latest-full which is using 1.25. Looks like I need at least version 1.26 for https://www.google.com/url?q=https://issues.apache.org/jira/browse/TIKA-3353&source=gmail-imap&ust=1622826254000000&usg=AOvVaw1we1l0Sh-gWif4FqbZ2qek, but I am not sure if this is not overkill for implementing basic liveness health checks.
>> >>
>> >> It's clear that –spawnChild and ForkParser are two must-haves that AFAIU are not default in apache/tika:latest-full
>> >>
>> >> My guess is that I also need to set the jvm heap size close to the memory resource limit for the container, but that's not ideal because the heap size would be statically configured while the memory resource limits are dynamic. Or maybe this is not necessary if I use -spawnChild?
>> >>
>> >> I am looking forward to your answers, thanks a lot!
>> >>
>> >> Cristi
>> >>
>> >>
>> >> On Fri, May 28, 2021 at 2:55 PM Nick Burch <ap...@gagravarr.org> wrote:
>> >>>
>> >>> On Thu, 27 May 2021, Cristian Zamfir wrote:
>> >>>> I am running some stress tests of the latest tika server docker (not
>> >>>> modified in any way, just pulled from the registry) and seeing that after a
>> >>>> few hours I see OOM in the logs. The container has a limit of 4GB set in
>> >>>> K8S. I am wondering if you have any best practices on how to avoid this.
>> >>>
>> >>> Hopefully one of our Tika+Docker experts will be along in a minute to help
>> >>> advise!
>> >>>
>> >>> For now, the general advice is documented at:
>> >>> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/TIKA/The%2BRobustness%2Bof%2BApache%2BTika&source=gmail-imap&ust=1622826254000000&usg=AOvVaw0p_ynGwlHapvMiy24sF1FP
>> >>>
>> >>> Also, which version of Tika are you on? There have been some contributions
>> >>> recently around monitoring the server, which you might want to upgrade
>> >>> for, eg TIKA-3353
>> >>>
>> >>> Nick
>>
Re: best practices for avoiding OOM for tika docker
Posted by Cristian Zamfir <cr...@cyberhaven.com>.
Actually maybe this is related:
java -jar ./tika-server-standard-2.0.0-BETA.jar
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
Tika 1.x had all the logging dependencies. I did not find something in the
changes log about it.
On Thu, Jun 10, 2021 at 6:51 PM Cristian Zamfir <cr...@cyberhaven.com>
wrote:
> I see, thanks. Has TIKA_CHILD_JVM_OPTS=-JXmx been replaced by the
> configuration option forkedJvmArgs or do they still both work? Guessing
> that it is fully replaced.
>
> When I switched to a config file for the server I noticed that some of the
> options I can see in the github repo do not seem to work. For instance log
> and includeStack. Have there been changes to these options in master
> compared to when 2.0.0-BETA was released?
>
> java -jar ./tika-server-standard-2.0.0-BETA.jar --config
> ./tika-server-config.xml
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> org.apache.tika.exception.TikaConfigException: Couldn't find setter:
> setLog for object class org.apache.tika.server.core.TikaServerConfig
> at org.apache.tika.config.ConfigBase.tryToSet(ConfigBase.java:389)
> at org.apache.tika.config.ConfigBase.setParams(ConfigBase.java:222)
> at org.apache.tika.config.ConfigBase.setParams(ConfigBase.java:190)
> at org.apache.tika.config.ConfigBase.configure(ConfigBase.java:432)
> at
> org.apache.tika.server.core.TikaServerConfig.load(TikaServerConfig.java:179)
> at
> org.apache.tika.server.core.TikaServerConfig.load(TikaServerConfig.java:172)
> at
> org.apache.tika.server.core.TikaServerConfig.load(TikaServerConfig.java:128)
> at org.apache.tika.server.core.TikaServerCli.execute(TikaServerCli.java:83)
> at org.apache.tika.server.core.TikaServerCli.main(TikaServerCli.java:66)
>
>
> <properties>
> <server>
> <params>
> <log>info</log>
> <!-- <includeStack>false</includeStack> -->
> <forkedJvmArgs>
> <arg>-Xmx3g</arg>
> </forkedJvmArgs>
> <endpoints>
> <endpoint>status</endpoint>
> <endpoint>tika</endpoint>
> </endpoints>
> </params>
> </server>
> </properties>
>
> Thanks,
> Cristi
>
> On Thu, Jun 10, 2021 at 5:36 PM Tim Allison <ta...@apache.org> wrote:
>
>> I just updated the wiki. I haven't put in an anchor yet, but see:
>> https://cwiki.apache.org/confluence/display/TIKA/TikaServer and search
>> for 'status' at the bottom of the page.
>>
>> Please let us know if you have any questions.
>>
>> Best,
>>
>> Tim
>>
>>
>> On Thu, Jun 10, 2021 at 11:22 AM Cristian Zamfir <cr...@cyberhaven.com>
>> wrote:
>> >
>> > It appears that the -status option was dropped in 2.x - was it replaced
>> by something else?
>> >
>> > Thanks,
>> > Cristi
>> >
>> >
>> > On Wed, Jun 2, 2021 at 4:54 PM Tim Allison <ta...@apache.org> wrote:
>> >>
>> >> >I wanted to double check that -JXX:+ExitOnOutOfMemoryError should be
>> provided to the main process or to the child, can you please confirm?
>> >>
>> >> Yes
>> >>
>> >> On Wed, Jun 2, 2021 at 10:49 AM Cristian Zamfir <cr...@cyberhaven.com>
>> wrote:
>> >> >
>> >> >
>> >> >
>> >> > > On 2 Jun 2021, at 15:33, Cristian Zamfir <cr...@cyberhaven.com>
>> wrote:
>> >> > >
>> >> > >>
>> >> > >> On 2 Jun 2021, at 14:43, Tim Allison <ta...@apache.org> wrote:
>> >> > >>
>> >> > >>> I noticed that Tika prints in the logs OOM (null), but seems to
>> recover by itself even when not using -spawnChild. Is this the expected
>> behavior?
>> >> > >>
>> >> > >> When not in -spawnChild mode, Tika is catching OOM exceptions
>> (when it
>> >> > >> can), but it isn't "recovering"... the jvm may be in an
>> inconsistent
>> >> > >> state, and it is safest to restart the jvm. It would probably be
>> good
>> >> > >> practice when in -spawnChild mode to use
>> -XX:+ExitOnOutOfMemoryError,
>> >> > >> or on the tika commandline -JXX:+ExitOnOutOfMemoryError.
>> >> > >
>> >> > > Thanks for the clarification, makes sense. Already migrated to
>> using -spawnChild.
>> >> > > It would be great to make these default for the docker container,
>> I suspect most people using the docker image will use it in a similar way
>> and can run into OOM.
>> >> >
>> >> > I tested that these args work:
>> >> > -spawnChild TIKA_CHILD_JVM_OPTS=-JXmx3g -JXX:+ExitOnOutOfMemoryError
>> -status
>> >> > I wanted to double check that -JXX:+ExitOnOutOfMemoryError should be
>> provided to the main process or to the child, can you please confirm?
>> >> >
>> >> >
>> >> > >
>> >> > >
>> >> > >>
>> >> > >> I highly encourage you to use -spawnChild mode, or the new pipes
>> >> > >> modules in 2.x if those will work for you at some point...those
>> are
>> >> > >> still beta. OOMs are one thing, but infinite loops are another.
>> >> > >>
>> >> > >> 1. Do you have a recommendation for a stress test that would
>> allow me
>> >> > >> to easily test OOM behavior?
>> >> > >> The MockParser is built for exactly this:
>> >> > >>
>> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/TIKA/MockParser&source=gmail-imap&ust=1623242610000000&usg=AOvVaw0Ojll8dNTrN4dPDsFxe27I
>> >> > >>
>> >> > >> Let us know if you have any questions about it. The key elements
>> for
>> >> > >> you are <fakeload/>, <throw/> <oom/> and probably <system_exit/>.
>> >> > >> That's for synthetic load testing. If you want files in the
>> wild, we
>> >> > >> have 2TB of files from the wild:
>> >> > >>
>> https://www.google.com/url?q=https://corpora.tika.apache.org/base/docs/&source=gmail-imap&ust=1623242610000000&usg=AOvVaw1GmwQmaIG2r-uVROvKDdeJ
>> >> > >
>> >> > > Looks great. Looks like I will need to tweak the container for
>> testing this, but that’s likely fine.
>> >> >
>> >> > Actually I tested that the server restarts on OOM using ulimit and
>> then a for loop with curl, it is easy to reproduce.
>> >> >
>> >> > >
>> >> > >>
>> >> > >> 2. For implementing a health check that detects when Tika is
>> stuck, I
>> >> > >> could periodically send a simple request and check that the reply
>> is
>> >> > >> correct, do you recommend a better approach?
>> >> > >> We have a rudimentary /status endpoint, which will give you
>> number of
>> >> > >> restarts, number of files processed, milliseconds since last
>> parse.
>> >> > >> You have to turn it on via the commandline: -status.
>> >> > >
>> >> > > The status endpoint looks like a possible, I can look for
>> "status": “OPERATING”. Sending a single byte file looks like a decent check
>> as well.
>> >> > >
>> >> > >
>> >> > >
>> >> > >>
>> >> > >> On Wed, Jun 2, 2021 at 6:50 AM Cristian Zamfir <
>> cristi@cyberhaven.com> wrote:
>> >> > >>>
>> >> > >>> Hi!
>> >> > >>>
>> >> > >>> I noticed that Tika prints in the logs OOM (null), but seems to
>> recover by itself even when not using -spawnChild. Is this the expected
>> behavior? I am trying to figure out when logs containing "OOM" are critical
>> and would require a container restart.
>> >> > >>>
>> >> > >>> I also wanted to bring up two of my questions below, I am
>> looking forward to your feedback:
>> >> > >>> 1. Do you have a recommendation for a stress test that would
>> allow me to easily test OOM behavior?
>> >> > >>> 2. For implementing a health check that detects when Tika is
>> stuck, I could periodically send a simple request and check that the reply
>> is correct, do you recommend a better approach?
>> >> > >>>
>> >> > >>> Thanks,
>> >> > >>> Cristi
>> >> > >>>
>> >> > >>> On Sat, May 29, 2021 at 2:58 PM Cristian Zamfir <
>> cristi@cyberhaven.com> wrote:
>> >> > >>>>
>> >> > >>>>
>> >> > >>>>> On 28 May 2021, at 19:03, Tim Allison <ta...@apache.org>
>> wrote:
>> >> > >>>>>
>> >> > >>>>> Tika 2.x should help with this in pipes and async. Your
>> system should
>> >> > >>>>> expect to go oom or crash at some point if you're processing
>> enough
>> >> > >>>>> files.
>> >> > >>>>
>> >> > >>>> I believe that this is what is happening in my case, it’s not
>> due to a single file, it happens under high load when processing many files
>> at once.
>> >> > >>>>
>> >> > >>>>>
>> >> > >>>>> Right --spawnChild is not default in 1.x, but it will be in
>> 2.x. And,
>> >> > >>>>> yes, you should be using it. To set the Xmx in the forked
>> process add
>> >> > >>>>> -J, as in -JXmx2g would set the Xmx for the forked process.
>> >> > >>>>
>> >> > >>>>
>> >> > >>>> Did both now and I think this provides good recovery from OOM.
>> >> > >>>>
>> >> > >>>>
>> >> > >>>>>
>> >> > >>>>> I don't have experience to recommend bumping Xmx to close to
>> your
>> >> > >>>>> container's max memory. In java programs that do a bunch of
>> work off
>> >> > >>>>> heap, this would be a bad idea because you need to leave
>> resources for
>> >> > >>>>> your system os, but I don't think we do much off heap.
>> >> > >>>>
>> >> > >>>> What’s your take on a configuration in which the container is
>> capped at 4GB and the spawned child has a heap limit of 3GB? Sounds like a
>> pretty safe margin to me.
>> >> > >>>>
>> >> > >>>>>
>> >> > >>>>> Which file types are causing OOMs? The MP4Parser is
>> notorious, and
>> >> > >>>>> we're looking to swap it out in 2.x for a different parser.
>> >> > >>>>
>> >> > >>>> Good to hear. I don’t know how to identify the root cause
>> because there are many files sent at once.
>> >> > >>>> However, it would be great to learn if there is a quick way to
>> trigger a high load and test resiliency to OOM, do you have a
>> recommendation?
>> >> > >>>>
>> >> > >>>>
>> >> > >>>>>
>> >> > >>>>> Yep, TIKA-3353 is the monitoring that Nick was mentioning.
>> >> > >>>>
>> >> > >>>> I am actually more interested in health checks, to detect when
>> the system is stuck without automatically restarting. A built-in health
>> check would certainly be a nice feature.
>> >> > >>>>
>> >> > >>>> Besides OOM, one other possible cause is if /tmp gets full -
>> for instance I see here
>> https://www.google.com/url?q=https://github.com/tongwang/tika-server-docker/blob/master/bin/healthcheck&source=gmail-imap&ust=1623242610000000&usg=AOvVaw3ELoyR3KnlYeRkxqI-n_sp
>> that /tmp is cleaned up periodically and the health check fails if it is
>> too full.
>> >> > >>>>
>> >> > >>>> Are there any other situations that could indicate that the
>> container is stuck and needs a restart and if yes, is there a way to detect
>> the condition?
>> >> > >>>>
>> >> > >>>> Thanks,
>> >> > >>>> Cristi
>> >> > >>>>
>> >> > >>>>>
>> >> > >>>>> On Fri, May 28, 2021 at 9:08 AM Cristian Zamfir <
>> cristi@cyberhaven.com> wrote:
>> >> > >>>>>>
>> >> > >>>>>> Thanks for your answer Nick!
>> >> > >>>>>>
>> >> > >>>>>> I am running apache/tika:latest-full which is using 1.25.
>> Looks like I need at least version 1.26 for
>> https://www.google.com/url?q=https://www.google.com/url?q%3Dhttps://issues.apache.org/jira/browse/TIKA-3353%26source%3Dgmail-imap%26ust%3D1622826254000000%26usg%3DAOvVaw1we1l0Sh-gWif4FqbZ2qek&source=gmail-imap&ust=1623242610000000&usg=AOvVaw1_cxA4lC8qLoQsbu4sQfsP,
>> but I am not sure if this is not overkill for implementing basic liveness
>> health checks.
>> >> > >>>>>>
>> >> > >>>>>> It's clear that –spawnChild and ForkParser are two must-haves
>> that AFAIU are not default in apache/tika:latest-full
>> >> > >>>>>>
>> >> > >>>>>> My guess is that I also need to set the jvm heap size close
>> to the memory resource limit for the container, but that's not ideal
>> because the heap size would be statically configured while the memory
>> resource limits are dynamic. Or maybe this is not necessary if I use
>> -spawnChild?
>> >> > >>>>>>
>> >> > >>>>>> I am looking forward to your answers, thanks a lot!
>> >> > >>>>>>
>> >> > >>>>>> Cristi
>> >> > >>>>>>
>> >> > >>>>>>
>> >> > >>>>>> On Fri, May 28, 2021 at 2:55 PM Nick Burch <
>> apache@gagravarr.org> wrote:
>> >> > >>>>>>>
>> >> > >>>>>>> On Thu, 27 May 2021, Cristian Zamfir wrote:
>> >> > >>>>>>>> I am running some stress tests of the latest tika server
>> docker (not
>> >> > >>>>>>>> modified in any way, just pulled from the registry) and
>> seeing that after a
>> >> > >>>>>>>> few hours I see OOM in the logs. The container has a limit
>> of 4GB set in
>> >> > >>>>>>>> K8S. I am wondering if you have any best practices on how
>> to avoid this.
>> >> > >>>>>>>
>> >> > >>>>>>> Hopefully one of our Tika+Docker experts will be along in a
>> minute to help
>> >> > >>>>>>> advise!
>> >> > >>>>>>>
>> >> > >>>>>>> For now, the general advice is documented at:
>> >> > >>>>>>>
>> https://www.google.com/url?q=https://www.google.com/url?q%3Dhttps://cwiki.apache.org/confluence/display/TIKA/The%252BRobustness%252Bof%252BApache%252BTika%26source%3Dgmail-imap%26ust%3D1622826254000000%26usg%3DAOvVaw0p_ynGwlHapvMiy24sF1FP&source=gmail-imap&ust=1623242610000000&usg=AOvVaw2un_ETGBn01eVOW2jexxPL
>> >> > >>>>>>>
>> >> > >>>>>>> Also, which version of Tika are you on? There have been some
>> contributions
>> >> > >>>>>>> recently around monitoring the server, which you might want
>> to upgrade
>> >> > >>>>>>> for, eg TIKA-3353
>> >> > >>>>>>>
>> >> > >>>>>>> Nick
>> >> >
>>
>
Re: best practices for avoiding OOM for tika docker
Posted by Cristian Zamfir <cr...@cyberhaven.com>.
I see, thanks. Has TIKA_CHILD_JVM_OPTS=-JXmx been replaced by the
configuration option forkedJvmArgs or do they still both work? Guessing
that it is fully replaced.
When I switched to a config file for the server I noticed that some of the
options I can see in the github repo do not seem to work. For instance log
and includeStack. Have there been changes to these options in master
compared to when 2.0.0-BETA was released?
java -jar ./tika-server-standard-2.0.0-BETA.jar --config
./tika-server-config.xml
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
org.apache.tika.exception.TikaConfigException: Couldn't find setter: setLog
for object class org.apache.tika.server.core.TikaServerConfig
at org.apache.tika.config.ConfigBase.tryToSet(ConfigBase.java:389)
at org.apache.tika.config.ConfigBase.setParams(ConfigBase.java:222)
at org.apache.tika.config.ConfigBase.setParams(ConfigBase.java:190)
at org.apache.tika.config.ConfigBase.configure(ConfigBase.java:432)
at
org.apache.tika.server.core.TikaServerConfig.load(TikaServerConfig.java:179)
at
org.apache.tika.server.core.TikaServerConfig.load(TikaServerConfig.java:172)
at
org.apache.tika.server.core.TikaServerConfig.load(TikaServerConfig.java:128)
at org.apache.tika.server.core.TikaServerCli.execute(TikaServerCli.java:83)
at org.apache.tika.server.core.TikaServerCli.main(TikaServerCli.java:66)
<properties>
<server>
<params>
<log>info</log>
<!-- <includeStack>false</includeStack> -->
<forkedJvmArgs>
<arg>-Xmx3g</arg>
</forkedJvmArgs>
<endpoints>
<endpoint>status</endpoint>
<endpoint>tika</endpoint>
</endpoints>
</params>
</server>
</properties>
Thanks,
Cristi
On Thu, Jun 10, 2021 at 5:36 PM Tim Allison <ta...@apache.org> wrote:
> I just updated the wiki. I haven't put in an anchor yet, but see:
> https://cwiki.apache.org/confluence/display/TIKA/TikaServer and search
> for 'status' at the bottom of the page.
>
> Please let us know if you have any questions.
>
> Best,
>
> Tim
>
>
> On Thu, Jun 10, 2021 at 11:22 AM Cristian Zamfir <cr...@cyberhaven.com>
> wrote:
> >
> > It appears that the -status option was dropped in 2.x - was it replaced
> by something else?
> >
> > Thanks,
> > Cristi
> >
> >
> > On Wed, Jun 2, 2021 at 4:54 PM Tim Allison <ta...@apache.org> wrote:
> >>
> >> >I wanted to double check that -JXX:+ExitOnOutOfMemoryError should be
> provided to the main process or to the child, can you please confirm?
> >>
> >> Yes
> >>
> >> On Wed, Jun 2, 2021 at 10:49 AM Cristian Zamfir <cr...@cyberhaven.com>
> wrote:
> >> >
> >> >
> >> >
> >> > > On 2 Jun 2021, at 15:33, Cristian Zamfir <cr...@cyberhaven.com>
> wrote:
> >> > >
> >> > >>
> >> > >> On 2 Jun 2021, at 14:43, Tim Allison <ta...@apache.org> wrote:
> >> > >>
> >> > >>> I noticed that Tika prints in the logs OOM (null), but seems to
> recover by itself even when not using -spawnChild. Is this the expected
> behavior?
> >> > >>
> >> > >> When not in -spawnChild mode, Tika is catching OOM exceptions
> (when it
> >> > >> can), but it isn't "recovering"... the jvm may be in an
> inconsistent
> >> > >> state, and it is safest to restart the jvm. It would probably be
> good
> >> > >> practice when in -spawnChild mode to use
> -XX:+ExitOnOutOfMemoryError,
> >> > >> or on the tika commandline -JXX:+ExitOnOutOfMemoryError.
> >> > >
> >> > > Thanks for the clarification, makes sense. Already migrated to
> using -spawnChild.
> >> > > It would be great to make these default for the docker container, I
> suspect most people using the docker image will use it in a similar way and
> can run into OOM.
> >> >
> >> > I tested that these args work:
> >> > -spawnChild TIKA_CHILD_JVM_OPTS=-JXmx3g -JXX:+ExitOnOutOfMemoryError
> -status
> >> > I wanted to double check that -JXX:+ExitOnOutOfMemoryError should be
> provided to the main process or to the child, can you please confirm?
> >> >
> >> >
> >> > >
> >> > >
> >> > >>
> >> > >> I highly encourage you to use -spawnChild mode, or the new pipes
> >> > >> modules in 2.x if those will work for you at some point...those are
> >> > >> still beta. OOMs are one thing, but infinite loops are another.
> >> > >>
> >> > >> 1. Do you have a recommendation for a stress test that would allow
> me
> >> > >> to easily test OOM behavior?
> >> > >> The MockParser is built for exactly this:
> >> > >>
> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/TIKA/MockParser&source=gmail-imap&ust=1623242610000000&usg=AOvVaw0Ojll8dNTrN4dPDsFxe27I
> >> > >>
> >> > >> Let us know if you have any questions about it. The key elements
> for
> >> > >> you are <fakeload/>, <throw/> <oom/> and probably <system_exit/>.
> >> > >> That's for synthetic load testing. If you want files in the wild,
> we
> >> > >> have 2TB of files from the wild:
> >> > >>
> https://www.google.com/url?q=https://corpora.tika.apache.org/base/docs/&source=gmail-imap&ust=1623242610000000&usg=AOvVaw1GmwQmaIG2r-uVROvKDdeJ
> >> > >
> >> > > Looks great. Looks like I will need to tweak the container for
> testing this, but that’s likely fine.
> >> >
> >> > Actually I tested that the server restarts on OOM using ulimit and
> then a for loop with curl, it is easy to reproduce.
> >> >
> >> > >
> >> > >>
> >> > >> 2. For implementing a health check that detects when Tika is
> stuck, I
> >> > >> could periodically send a simple request and check that the reply
> is
> >> > >> correct, do you recommend a better approach?
> >> > >> We have a rudimentary /status endpoint, which will give you number
> of
> >> > >> restarts, number of files processed, milliseconds since last parse.
> >> > >> You have to turn it on via the commandline: -status.
> >> > >
> >> > > The status endpoint looks like a possible, I can look for "status":
> “OPERATING”. Sending a single byte file looks like a decent check as well.
> >> > >
> >> > >
> >> > >
> >> > >>
> >> > >> On Wed, Jun 2, 2021 at 6:50 AM Cristian Zamfir <
> cristi@cyberhaven.com> wrote:
> >> > >>>
> >> > >>> Hi!
> >> > >>>
> >> > >>> I noticed that Tika prints in the logs OOM (null), but seems to
> recover by itself even when not using -spawnChild. Is this the expected
> behavior? I am trying to figure out when logs containing "OOM" are critical
> and would require a container restart.
> >> > >>>
> >> > >>> I also wanted to bring up two of my questions below, I am looking
> forward to your feedback:
> >> > >>> 1. Do you have a recommendation for a stress test that would
> allow me to easily test OOM behavior?
> >> > >>> 2. For implementing a health check that detects when Tika is
> stuck, I could periodically send a simple request and check that the reply
> is correct, do you recommend a better approach?
> >> > >>>
> >> > >>> Thanks,
> >> > >>> Cristi
> >> > >>>
> >> > >>> On Sat, May 29, 2021 at 2:58 PM Cristian Zamfir <
> cristi@cyberhaven.com> wrote:
> >> > >>>>
> >> > >>>>
> >> > >>>>> On 28 May 2021, at 19:03, Tim Allison <ta...@apache.org>
> wrote:
> >> > >>>>>
> >> > >>>>> Tika 2.x should help with this in pipes and async. Your system
> should
> >> > >>>>> expect to go oom or crash at some point if you're processing
> enough
> >> > >>>>> files.
> >> > >>>>
> >> > >>>> I believe that this is what is happening in my case, it’s not
> due to a single file, it happens under high load when processing many files
> at once.
> >> > >>>>
> >> > >>>>>
> >> > >>>>> Right --spawnChild is not default in 1.x, but it will be in
> 2.x. And,
> >> > >>>>> yes, you should be using it. To set the Xmx in the forked
> process add
> >> > >>>>> -J, as in -JXmx2g would set the Xmx for the forked process.
> >> > >>>>
> >> > >>>>
> >> > >>>> Did both now and I think this provides good recovery from OOM.
> >> > >>>>
> >> > >>>>
> >> > >>>>>
> >> > >>>>> I don't have experience to recommend bumping Xmx to close to
> your
> >> > >>>>> container's max memory. In java programs that do a bunch of
> work off
> >> > >>>>> heap, this would be a bad idea because you need to leave
> resources for
> >> > >>>>> your system os, but I don't think we do much off heap.
> >> > >>>>
> >> > >>>> What’s your take on a configuration in which the container is
> capped at 4GB and the spawned child has a heap limit of 3GB? Sounds like a
> pretty safe margin to me.
> >> > >>>>
> >> > >>>>>
> >> > >>>>> Which file types are causing OOMs? The MP4Parser is notorious,
> and
> >> > >>>>> we're looking to swap it out in 2.x for a different parser.
> >> > >>>>
> >> > >>>> Good to hear. I don’t know how to identify the root cause
> because there are many files sent at once.
> >> > >>>> However, it would be great to learn if there is a quick way to
> trigger a high load and test resiliency to OOM, do you have a
> recommendation?
> >> > >>>>
> >> > >>>>
> >> > >>>>>
> >> > >>>>> Yep, TIKA-3353 is the monitoring that Nick was mentioning.
> >> > >>>>
> >> > >>>> I am actually more interested in health checks, to detect when
> the system is stuck without automatically restarting. A built-in health
> check would certainly be a nice feature.
> >> > >>>>
> >> > >>>> Besides OOM, one other possible cause is if /tmp gets full - for
> instance I see here
> https://www.google.com/url?q=https://github.com/tongwang/tika-server-docker/blob/master/bin/healthcheck&source=gmail-imap&ust=1623242610000000&usg=AOvVaw3ELoyR3KnlYeRkxqI-n_sp
> that /tmp is cleaned up periodically and the health check fails if it is
> too full.
> >> > >>>>
> >> > >>>> Are there any other situations that could indicate that the
> container is stuck and needs a restart and if yes, is there a way to detect
> the condition?
> >> > >>>>
> >> > >>>> Thanks,
> >> > >>>> Cristi
> >> > >>>>
> >> > >>>>>
> >> > >>>>> On Fri, May 28, 2021 at 9:08 AM Cristian Zamfir <
> cristi@cyberhaven.com> wrote:
> >> > >>>>>>
> >> > >>>>>> Thanks for your answer Nick!
> >> > >>>>>>
> >> > >>>>>> I am running apache/tika:latest-full which is using 1.25.
> Looks like I need at least version 1.26 for
> https://www.google.com/url?q=https://www.google.com/url?q%3Dhttps://issues.apache.org/jira/browse/TIKA-3353%26source%3Dgmail-imap%26ust%3D1622826254000000%26usg%3DAOvVaw1we1l0Sh-gWif4FqbZ2qek&source=gmail-imap&ust=1623242610000000&usg=AOvVaw1_cxA4lC8qLoQsbu4sQfsP,
> but I am not sure if this is not overkill for implementing basic liveness
> health checks.
> >> > >>>>>>
> >> > >>>>>> It's clear that –spawnChild and ForkParser are two must-haves
> that AFAIU are not default in apache/tika:latest-full
> >> > >>>>>>
> >> > >>>>>> My guess is that I also need to set the jvm heap size close to
> the memory resource limit for the container, but that's not ideal because
> the heap size would be statically configured while the memory resource
> limits are dynamic. Or maybe this is not necessary if I use -spawnChild?
> >> > >>>>>>
> >> > >>>>>> I am looking forward to your answers, thanks a lot!
> >> > >>>>>>
> >> > >>>>>> Cristi
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> On Fri, May 28, 2021 at 2:55 PM Nick Burch <
> apache@gagravarr.org> wrote:
> >> > >>>>>>>
> >> > >>>>>>> On Thu, 27 May 2021, Cristian Zamfir wrote:
> >> > >>>>>>>> I am running some stress tests of the latest tika server
> docker (not
> >> > >>>>>>>> modified in any way, just pulled from the registry) and
> seeing that after a
> >> > >>>>>>>> few hours I see OOM in the logs. The container has a limit
> of 4GB set in
> >> > >>>>>>>> K8S. I am wondering if you have any best practices on how to
> avoid this.
> >> > >>>>>>>
> >> > >>>>>>> Hopefully one of our Tika+Docker experts will be along in a
> minute to help
> >> > >>>>>>> advise!
> >> > >>>>>>>
> >> > >>>>>>> For now, the general advice is documented at:
> >> > >>>>>>>
> https://www.google.com/url?q=https://www.google.com/url?q%3Dhttps://cwiki.apache.org/confluence/display/TIKA/The%252BRobustness%252Bof%252BApache%252BTika%26source%3Dgmail-imap%26ust%3D1622826254000000%26usg%3DAOvVaw0p_ynGwlHapvMiy24sF1FP&source=gmail-imap&ust=1623242610000000&usg=AOvVaw2un_ETGBn01eVOW2jexxPL
> >> > >>>>>>>
> >> > >>>>>>> Also, which version of Tika are you on? There have been some
> contributions
> >> > >>>>>>> recently around monitoring the server, which you might want
> to upgrade
> >> > >>>>>>> for, eg TIKA-3353
> >> > >>>>>>>
> >> > >>>>>>> Nick
> >> >
>
Re: best practices for avoiding OOM for tika docker
Posted by Tim Allison <ta...@apache.org>.
I just updated the wiki. I haven't put in an anchor yet, but see:
https://cwiki.apache.org/confluence/display/TIKA/TikaServer and search
for 'status' at the bottom of the page.
Please let us know if you have any questions.
Best,
Tim
On Thu, Jun 10, 2021 at 11:22 AM Cristian Zamfir <cr...@cyberhaven.com> wrote:
>
> It appears that the -status option was dropped in 2.x - was it replaced by something else?
>
> Thanks,
> Cristi
>
>
> On Wed, Jun 2, 2021 at 4:54 PM Tim Allison <ta...@apache.org> wrote:
>>
>> >I wanted to double check that -JXX:+ExitOnOutOfMemoryError should be provided to the main process or to the child, can you please confirm?
>>
>> Yes
>>
>> On Wed, Jun 2, 2021 at 10:49 AM Cristian Zamfir <cr...@cyberhaven.com> wrote:
>> >
>> >
>> >
>> > > On 2 Jun 2021, at 15:33, Cristian Zamfir <cr...@cyberhaven.com> wrote:
>> > >
>> > >>
>> > >> On 2 Jun 2021, at 14:43, Tim Allison <ta...@apache.org> wrote:
>> > >>
>> > >>> I noticed that Tika prints in the logs OOM (null), but seems to recover by itself even when not using -spawnChild. Is this the expected behavior?
>> > >>
>> > >> When not in -spawnChild mode, Tika is catching OOM exceptions (when it
>> > >> can), but it isn't "recovering"... the jvm may be in an inconsistent
>> > >> state, and it is safest to restart the jvm. It would probably be good
>> > >> practice when in -spawnChild mode to use -XX:+ExitOnOutOfMemoryError,
>> > >> or on the tika commandline -JXX:+ExitOnOutOfMemoryError.
>> > >
>> > > Thanks for the clarification, makes sense. Already migrated to using -spawnChild.
>> > > It would be great to make these default for the docker container, I suspect most people using the docker image will use it in a similar way and can run into OOM.
>> >
>> > I tested that these args work:
>> > -spawnChild TIKA_CHILD_JVM_OPTS=-JXmx3g -JXX:+ExitOnOutOfMemoryError -status
>> > I wanted to double check that -JXX:+ExitOnOutOfMemoryError should be provided to the main process or to the child, can you please confirm?
>> >
>> >
>> > >
>> > >
>> > >>
>> > >> I highly encourage you to use -spawnChild mode, or the new pipes
>> > >> modules in 2.x if those will work for you at some point...those are
>> > >> still beta. OOMs are one thing, but infinite loops are another.
>> > >>
>> > >> 1. Do you have a recommendation for a stress test that would allow me
>> > >> to easily test OOM behavior?
>> > >> The MockParser is built for exactly this:
>> > >> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/TIKA/MockParser&source=gmail-imap&ust=1623242610000000&usg=AOvVaw0Ojll8dNTrN4dPDsFxe27I
>> > >>
>> > >> Let us know if you have any questions about it. The key elements for
>> > >> you are <fakeload/>, <throw/> <oom/> and probably <system_exit/>.
>> > >> That's for synthetic load testing. If you want files in the wild, we
>> > >> have 2TB of files from the wild:
>> > >> https://www.google.com/url?q=https://corpora.tika.apache.org/base/docs/&source=gmail-imap&ust=1623242610000000&usg=AOvVaw1GmwQmaIG2r-uVROvKDdeJ
>> > >
>> > > Looks great. Looks like I will need to tweak the container for testing this, but that’s likely fine.
>> >
>> > Actually I tested that the server restarts on OOM using ulimit and then a for loop with curl, it is easy to reproduce.
>> >
>> > >
>> > >>
>> > >> 2. For implementing a health check that detects when Tika is stuck, I
>> > >> could periodically send a simple request and check that the reply is
>> > >> correct, do you recommend a better approach?
>> > >> We have a rudimentary /status endpoint, which will give you number of
>> > >> restarts, number of files processed, milliseconds since last parse.
>> > >> You have to turn it on via the commandline: -status.
>> > >
>> > > The status endpoint looks like a possible, I can look for "status": “OPERATING”. Sending a single byte file looks like a decent check as well.
>> > >
>> > >
>> > >
>> > >>
>> > >> On Wed, Jun 2, 2021 at 6:50 AM Cristian Zamfir <cr...@cyberhaven.com> wrote:
>> > >>>
>> > >>> Hi!
>> > >>>
>> > >>> I noticed that Tika prints in the logs OOM (null), but seems to recover by itself even when not using -spawnChild. Is this the expected behavior? I am trying to figure out when logs containing "OOM" are critical and would require a container restart.
>> > >>>
>> > >>> I also wanted to bring up two of my questions below, I am looking forward to your feedback:
>> > >>> 1. Do you have a recommendation for a stress test that would allow me to easily test OOM behavior?
>> > >>> 2. For implementing a health check that detects when Tika is stuck, I could periodically send a simple request and check that the reply is correct, do you recommend a better approach?
>> > >>>
>> > >>> Thanks,
>> > >>> Cristi
>> > >>>
>> > >>> On Sat, May 29, 2021 at 2:58 PM Cristian Zamfir <cr...@cyberhaven.com> wrote:
>> > >>>>
>> > >>>>
>> > >>>>> On 28 May 2021, at 19:03, Tim Allison <ta...@apache.org> wrote:
>> > >>>>>
>> > >>>>> Tika 2.x should help with this in pipes and async. Your system should
>> > >>>>> expect to go oom or crash at some point if you're processing enough
>> > >>>>> files.
>> > >>>>
>> > >>>> I believe that this is what is happening in my case, it’s not due to a single file, it happens under high load when processing many files at once.
>> > >>>>
>> > >>>>>
>> > >>>>> Right --spawnChild is not default in 1.x, but it will be in 2.x. And,
>> > >>>>> yes, you should be using it. To set the Xmx in the forked process add
>> > >>>>> -J, as in -JXmx2g would set the Xmx for the forked process.
>> > >>>>
>> > >>>>
>> > >>>> Did both now and I think this provides good recovery from OOM.
>> > >>>>
>> > >>>>
>> > >>>>>
>> > >>>>> I don't have experience to recommend bumping Xmx to close to your
>> > >>>>> container's max memory. In java programs that do a bunch of work off
>> > >>>>> heap, this would be a bad idea because you need to leave resources for
>> > >>>>> your system os, but I don't think we do much off heap.
>> > >>>>
>> > >>>> What’s your take on a configuration in which the container is capped at 4GB and the spawned child has a heap limit of 3GB? Sounds like a pretty safe margin to me.
>> > >>>>
>> > >>>>>
>> > >>>>> Which file types are causing OOMs? The MP4Parser is notorious, and
>> > >>>>> we're looking to swap it out in 2.x for a different parser.
>> > >>>>
>> > >>>> Good to hear. I don’t know how to identify the root cause because there are many files sent at once.
>> > >>>> However, it would be great to learn if there is a quick way to trigger a high load and test resiliency to OOM, do you have a recommendation?
>> > >>>>
>> > >>>>
>> > >>>>>
>> > >>>>> Yep, TIKA-3353 is the monitoring that Nick was mentioning.
>> > >>>>
>> > >>>> I am actually more interested in health checks, to detect when the system is stuck without automatically restarting. A built-in health check would certainly be a nice feature.
>> > >>>>
>> > >>>> Besides OOM, one other possible cause is if /tmp gets full - for instance I see here https://www.google.com/url?q=https://github.com/tongwang/tika-server-docker/blob/master/bin/healthcheck&source=gmail-imap&ust=1623242610000000&usg=AOvVaw3ELoyR3KnlYeRkxqI-n_sp that /tmp is cleaned up periodically and the health check fails if it is too full.
>> > >>>>
>> > >>>> Are there any other situations that could indicate that the container is stuck and needs a restart and if yes, is there a way to detect the condition?
>> > >>>>
>> > >>>> Thanks,
>> > >>>> Cristi
>> > >>>>
>> > >>>>>
>> > >>>>> On Fri, May 28, 2021 at 9:08 AM Cristian Zamfir <cr...@cyberhaven.com> wrote:
>> > >>>>>>
>> > >>>>>> Thanks for your answer Nick!
>> > >>>>>>
>> > >>>>>> I am running apache/tika:latest-full which is using 1.25. Looks like I need at least version 1.26 for https://www.google.com/url?q=https://www.google.com/url?q%3Dhttps://issues.apache.org/jira/browse/TIKA-3353%26source%3Dgmail-imap%26ust%3D1622826254000000%26usg%3DAOvVaw1we1l0Sh-gWif4FqbZ2qek&source=gmail-imap&ust=1623242610000000&usg=AOvVaw1_cxA4lC8qLoQsbu4sQfsP, but I am not sure if this is not overkill for implementing basic liveness health checks.
>> > >>>>>>
>> > >>>>>> It's clear that –spawnChild and ForkParser are two must-haves that AFAIU are not default in apache/tika:latest-full
>> > >>>>>>
>> > >>>>>> My guess is that I also need to set the jvm heap size close to the memory resource limit for the container, but that's not ideal because the heap size would be statically configured while the memory resource limits are dynamic. Or maybe this is not necessary if I use -spawnChild?
>> > >>>>>>
>> > >>>>>> I am looking forward to your answers, thanks a lot!
>> > >>>>>>
>> > >>>>>> Cristi
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> On Fri, May 28, 2021 at 2:55 PM Nick Burch <ap...@gagravarr.org> wrote:
>> > >>>>>>>
>> > >>>>>>> On Thu, 27 May 2021, Cristian Zamfir wrote:
>> > >>>>>>>> I am running some stress tests of the latest tika server docker (not
>> > >>>>>>>> modified in any way, just pulled from the registry) and seeing that after a
>> > >>>>>>>> few hours I see OOM in the logs. The container has a limit of 4GB set in
>> > >>>>>>>> K8S. I am wondering if you have any best practices on how to avoid this.
>> > >>>>>>>
>> > >>>>>>> Hopefully one of our Tika+Docker experts will be along in a minute to help
>> > >>>>>>> advise!
>> > >>>>>>>
>> > >>>>>>> For now, the general advice is documented at:
>> > >>>>>>> https://www.google.com/url?q=https://www.google.com/url?q%3Dhttps://cwiki.apache.org/confluence/display/TIKA/The%252BRobustness%252Bof%252BApache%252BTika%26source%3Dgmail-imap%26ust%3D1622826254000000%26usg%3DAOvVaw0p_ynGwlHapvMiy24sF1FP&source=gmail-imap&ust=1623242610000000&usg=AOvVaw2un_ETGBn01eVOW2jexxPL
>> > >>>>>>>
>> > >>>>>>> Also, which version of Tika are you on? There have been some contributions
>> > >>>>>>> recently around monitoring the server, which you might want to upgrade
>> > >>>>>>> for, eg TIKA-3353
>> > >>>>>>>
>> > >>>>>>> Nick
>> >
Re: best practices for avoiding OOM for tika docker
Posted by Cristian Zamfir <cr...@cyberhaven.com>.
It appears that the -status option was dropped in 2.x - was it replaced by
something else?
Thanks,
Cristi
On Wed, Jun 2, 2021 at 4:54 PM Tim Allison <ta...@apache.org> wrote:
> >I wanted to double check that -JXX:+ExitOnOutOfMemoryError should be
> provided to the main process or to the child, can you please confirm?
>
> Yes
>
> On Wed, Jun 2, 2021 at 10:49 AM Cristian Zamfir <cr...@cyberhaven.com>
> wrote:
> >
> >
> >
> > > On 2 Jun 2021, at 15:33, Cristian Zamfir <cr...@cyberhaven.com>
> wrote:
> > >
> > >>
> > >> On 2 Jun 2021, at 14:43, Tim Allison <ta...@apache.org> wrote:
> > >>
> > >>> I noticed that Tika prints in the logs OOM (null), but seems to
> recover by itself even when not using -spawnChild. Is this the expected
> behavior?
> > >>
> > >> When not in -spawnChild mode, Tika is catching OOM exceptions (when it
> > >> can), but it isn't "recovering"... the jvm may be in an inconsistent
> > >> state, and it is safest to restart the jvm. It would probably be good
> > >> practice when in -spawnChild mode to use -XX:+ExitOnOutOfMemoryError,
> > >> or on the tika commandline -JXX:+ExitOnOutOfMemoryError.
> > >
> > > Thanks for the clarification, makes sense. Already migrated to using
> -spawnChild.
> > > It would be great to make these default for the docker container, I
> suspect most people using the docker image will use it in a similar way and
> can run into OOM.
> >
> > I tested that these args work:
> > -spawnChild TIKA_CHILD_JVM_OPTS=-JXmx3g -JXX:+ExitOnOutOfMemoryError
> -status
> > I wanted to double check that -JXX:+ExitOnOutOfMemoryError should be
> provided to the main process or to the child, can you please confirm?
> >
> >
> > >
> > >
> > >>
> > >> I highly encourage you to use -spawnChild mode, or the new pipes
> > >> modules in 2.x if those will work for you at some point...those are
> > >> still beta. OOMs are one thing, but infinite loops are another.
> > >>
> > >> 1. Do you have a recommendation for a stress test that would allow me
> > >> to easily test OOM behavior?
> > >> The MockParser is built for exactly this:
> > >>
> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/TIKA/MockParser&source=gmail-imap&ust=1623242610000000&usg=AOvVaw0Ojll8dNTrN4dPDsFxe27I
> > >>
> > >> Let us know if you have any questions about it. The key elements for
> > >> you are <fakeload/>, <throw/> <oom/> and probably <system_exit/>.
> > >> That's for synthetic load testing. If you want files in the wild, we
> > >> have 2TB of files from the wild:
> > >>
> https://www.google.com/url?q=https://corpora.tika.apache.org/base/docs/&source=gmail-imap&ust=1623242610000000&usg=AOvVaw1GmwQmaIG2r-uVROvKDdeJ
> > >
> > > Looks great. Looks like I will need to tweak the container for testing
> this, but that’s likely fine.
> >
> > Actually I tested that the server restarts on OOM using ulimit and then
> a for loop with curl, it is easy to reproduce.
> >
> > >
> > >>
> > >> 2. For implementing a health check that detects when Tika is stuck, I
> > >> could periodically send a simple request and check that the reply is
> > >> correct, do you recommend a better approach?
> > >> We have a rudimentary /status endpoint, which will give you number of
> > >> restarts, number of files processed, milliseconds since last parse.
> > >> You have to turn it on via the commandline: -status.
> > >
> > > The status endpoint looks like a possible, I can look for "status":
> “OPERATING”. Sending a single byte file looks like a decent check as well.
> > >
> > >
> > >
> > >>
> > >> On Wed, Jun 2, 2021 at 6:50 AM Cristian Zamfir <cr...@cyberhaven.com>
> wrote:
> > >>>
> > >>> Hi!
> > >>>
> > >>> I noticed that Tika prints in the logs OOM (null), but seems to
> recover by itself even when not using -spawnChild. Is this the expected
> behavior? I am trying to figure out when logs containing "OOM" are critical
> and would require a container restart.
> > >>>
> > >>> I also wanted to bring up two of my questions below, I am looking
> forward to your feedback:
> > >>> 1. Do you have a recommendation for a stress test that would allow
> me to easily test OOM behavior?
> > >>> 2. For implementing a health check that detects when Tika is stuck,
> I could periodically send a simple request and check that the reply is
> correct, do you recommend a better approach?
> > >>>
> > >>> Thanks,
> > >>> Cristi
> > >>>
> > >>> On Sat, May 29, 2021 at 2:58 PM Cristian Zamfir <
> cristi@cyberhaven.com> wrote:
> > >>>>
> > >>>>
> > >>>>> On 28 May 2021, at 19:03, Tim Allison <ta...@apache.org> wrote:
> > >>>>>
> > >>>>> Tika 2.x should help with this in pipes and async. Your system
> should
> > >>>>> expect to go oom or crash at some point if you're processing enough
> > >>>>> files.
> > >>>>
> > >>>> I believe that this is what is happening in my case, it’s not due
> to a single file, it happens under high load when processing many files at
> once.
> > >>>>
> > >>>>>
> > >>>>> Right --spawnChild is not default in 1.x, but it will be in 2.x.
> And,
> > >>>>> yes, you should be using it. To set the Xmx in the forked process
> add
> > >>>>> -J, as in -JXmx2g would set the Xmx for the forked process.
> > >>>>
> > >>>>
> > >>>> Did both now and I think this provides good recovery from OOM.
> > >>>>
> > >>>>
> > >>>>>
> > >>>>> I don't have experience to recommend bumping Xmx to close to your
> > >>>>> container's max memory. In java programs that do a bunch of work
> off
> > >>>>> heap, this would be a bad idea because you need to leave resources
> for
> > >>>>> your system os, but I don't think we do much off heap.
> > >>>>
> > >>>> What’s your take on a configuration in which the container is
> capped at 4GB and the spawned child has a heap limit of 3GB? Sounds like a
> pretty safe margin to me.
> > >>>>
> > >>>>>
> > >>>>> Which file types are causing OOMs? The MP4Parser is notorious, and
> > >>>>> we're looking to swap it out in 2.x for a different parser.
> > >>>>
> > >>>> Good to hear. I don’t know how to identify the root cause because
> there are many files sent at once.
> > >>>> However, it would be great to learn if there is a quick way to
> trigger a high load and test resiliency to OOM, do you have a
> recommendation?
> > >>>>
> > >>>>
> > >>>>>
> > >>>>> Yep, TIKA-3353 is the monitoring that Nick was mentioning.
> > >>>>
> > >>>> I am actually more interested in health checks, to detect when the
> system is stuck without automatically restarting. A built-in health check
> would certainly be a nice feature.
> > >>>>
> > >>>> Besides OOM, one other possible cause is if /tmp gets full - for
> instance I see here
> https://www.google.com/url?q=https://github.com/tongwang/tika-server-docker/blob/master/bin/healthcheck&source=gmail-imap&ust=1623242610000000&usg=AOvVaw3ELoyR3KnlYeRkxqI-n_sp
> that /tmp is cleaned up periodically and the health check fails if it is
> too full.
> > >>>>
> > >>>> Are there any other situations that could indicate that the
> container is stuck and needs a restart and if yes, is there a way to detect
> the condition?
> > >>>>
> > >>>> Thanks,
> > >>>> Cristi
> > >>>>
> > >>>>>
> > >>>>> On Fri, May 28, 2021 at 9:08 AM Cristian Zamfir <
> cristi@cyberhaven.com> wrote:
> > >>>>>>
> > >>>>>> Thanks for your answer Nick!
> > >>>>>>
> > >>>>>> I am running apache/tika:latest-full which is using 1.25. Looks
> like I need at least version 1.26 for
> https://www.google.com/url?q=https://www.google.com/url?q%3Dhttps://issues.apache.org/jira/browse/TIKA-3353%26source%3Dgmail-imap%26ust%3D1622826254000000%26usg%3DAOvVaw1we1l0Sh-gWif4FqbZ2qek&source=gmail-imap&ust=1623242610000000&usg=AOvVaw1_cxA4lC8qLoQsbu4sQfsP,
> but I am not sure if this is not overkill for implementing basic liveness
> health checks.
> > >>>>>>
> > >>>>>> It's clear that –spawnChild and ForkParser are two must-haves
> that AFAIU are not default in apache/tika:latest-full
> > >>>>>>
> > >>>>>> My guess is that I also need to set the jvm heap size close to
> the memory resource limit for the container, but that's not ideal because
> the heap size would be statically configured while the memory resource
> limits are dynamic. Or maybe this is not necessary if I use -spawnChild?
> > >>>>>>
> > >>>>>> I am looking forward to your answers, thanks a lot!
> > >>>>>>
> > >>>>>> Cristi
> > >>>>>>
> > >>>>>>
> > >>>>>> On Fri, May 28, 2021 at 2:55 PM Nick Burch <ap...@gagravarr.org>
> wrote:
> > >>>>>>>
> > >>>>>>> On Thu, 27 May 2021, Cristian Zamfir wrote:
> > >>>>>>>> I am running some stress tests of the latest tika server docker
> (not
> > >>>>>>>> modified in any way, just pulled from the registry) and seeing
> that after a
> > >>>>>>>> few hours I see OOM in the logs. The container has a limit of
> 4GB set in
> > >>>>>>>> K8S. I am wondering if you have any best practices on how to
> avoid this.
> > >>>>>>>
> > >>>>>>> Hopefully one of our Tika+Docker experts will be along in a
> minute to help
> > >>>>>>> advise!
> > >>>>>>>
> > >>>>>>> For now, the general advice is documented at:
> > >>>>>>>
> https://www.google.com/url?q=https://www.google.com/url?q%3Dhttps://cwiki.apache.org/confluence/display/TIKA/The%252BRobustness%252Bof%252BApache%252BTika%26source%3Dgmail-imap%26ust%3D1622826254000000%26usg%3DAOvVaw0p_ynGwlHapvMiy24sF1FP&source=gmail-imap&ust=1623242610000000&usg=AOvVaw2un_ETGBn01eVOW2jexxPL
> > >>>>>>>
> > >>>>>>> Also, which version of Tika are you on? There have been some
> contributions
> > >>>>>>> recently around monitoring the server, which you might want to
> upgrade
> > >>>>>>> for, eg TIKA-3353
> > >>>>>>>
> > >>>>>>> Nick
> >
>
Re: best practices for avoiding OOM for tika docker
Posted by Cristian Zamfir <cr...@cyberhaven.com>.
Hi!
I noticed that Tika prints in the logs OOM (null), but seems to recover by
itself even when not using -spawnChild. Is this the expected behavior? I am
trying to figure out when logs containing "OOM" are critical and would
require a container restart.
I also wanted to bring up two of my questions below, I am looking forward
to your feedback:
1. Do you have a recommendation for a stress test that would allow me to
easily test OOM behavior?
2. For implementing a health check that detects when Tika is stuck, I could
periodically send a simple request and check that the reply is correct, do
you recommend a better approach?
Thanks,
Cristi
On Sat, May 29, 2021 at 2:58 PM Cristian Zamfir <cr...@cyberhaven.com>
wrote:
>
> > On 28 May 2021, at 19:03, Tim Allison <ta...@apache.org> wrote:
> >
> > Tika 2.x should help with this in pipes and async. Your system should
> > expect to go oom or crash at some point if you're processing enough
> > files.
>
> I believe that this is what is happening in my case, it’s not due to a
> single file, it happens under high load when processing many files at once.
>
> >
> > Right --spawnChild is not default in 1.x, but it will be in 2.x. And,
> > yes, you should be using it. To set the Xmx in the forked process add
> > -J, as in -JXmx2g would set the Xmx for the forked process.
>
>
> Did both now and I think this provides good recovery from OOM.
>
>
> >
> > I don't have experience to recommend bumping Xmx to close to your
> > container's max memory. In java programs that do a bunch of work off
> > heap, this would be a bad idea because you need to leave resources for
> > your system os, but I don't think we do much off heap.
>
> What’s your take on a configuration in which the container is capped at
> 4GB and the spawned child has a heap limit of 3GB? Sounds like a pretty
> safe margin to me.
>
> >
> > Which file types are causing OOMs? The MP4Parser is notorious, and
> > we're looking to swap it out in 2.x for a different parser.
>
> Good to hear. I don’t know how to identify the root cause because there
> are many files sent at once.
> However, it would be great to learn if there is a quick way to trigger a
> high load and test resiliency to OOM, do you have a recommendation?
>
>
> >
> > Yep, TIKA-3353 is the monitoring that Nick was mentioning.
>
> I am actually more interested in health checks, to detect when the system
> is stuck without automatically restarting. A built-in health check would
> certainly be a nice feature.
>
> Besides OOM, one other possible cause is if /tmp gets full - for instance
> I see here
> https://github.com/tongwang/tika-server-docker/blob/master/bin/healthcheck
> that /tmp is cleaned up periodically and the health check fails if it is
> too full.
>
> Are there any other situations that could indicate that the container is
> stuck and needs a restart and if yes, is there a way to detect the
> condition?
>
> Thanks,
> Cristi
>
> >
> > On Fri, May 28, 2021 at 9:08 AM Cristian Zamfir <cr...@cyberhaven.com>
> wrote:
> >>
> >> Thanks for your answer Nick!
> >>
> >> I am running apache/tika:latest-full which is using 1.25. Looks like I
> need at least version 1.26 for
> https://www.google.com/url?q=https://issues.apache.org/jira/browse/TIKA-3353&source=gmail-imap&ust=1622826254000000&usg=AOvVaw1we1l0Sh-gWif4FqbZ2qek,
> but I am not sure if this is not overkill for implementing basic liveness
> health checks.
> >>
> >> It's clear that –spawnChild and ForkParser are two must-haves that
> AFAIU are not default in apache/tika:latest-full
> >>
> >> My guess is that I also need to set the jvm heap size close to the
> memory resource limit for the container, but that's not ideal because the
> heap size would be statically configured while the memory resource limits
> are dynamic. Or maybe this is not necessary if I use -spawnChild?
> >>
> >> I am looking forward to your answers, thanks a lot!
> >>
> >> Cristi
> >>
> >>
> >> On Fri, May 28, 2021 at 2:55 PM Nick Burch <ap...@gagravarr.org>
> wrote:
> >>>
> >>> On Thu, 27 May 2021, Cristian Zamfir wrote:
> >>>> I am running some stress tests of the latest tika server docker (not
> >>>> modified in any way, just pulled from the registry) and seeing that
> after a
> >>>> few hours I see OOM in the logs. The container has a limit of 4GB set
> in
> >>>> K8S. I am wondering if you have any best practices on how to avoid
> this.
> >>>
> >>> Hopefully one of our Tika+Docker experts will be along in a minute to
> help
> >>> advise!
> >>>
> >>> For now, the general advice is documented at:
> >>>
> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/TIKA/The%2BRobustness%2Bof%2BApache%2BTika&source=gmail-imap&ust=1622826254000000&usg=AOvVaw0p_ynGwlHapvMiy24sF1FP
> >>>
> >>> Also, which version of Tika are you on? There have been some
> contributions
> >>> recently around monitoring the server, which you might want to upgrade
> >>> for, eg TIKA-3353
> >>>
> >>> Nick
>
>
Re: best practices for avoiding OOM for tika docker
Posted by Cristian Zamfir <cr...@cyberhaven.com>.
> On 28 May 2021, at 19:03, Tim Allison <ta...@apache.org> wrote:
>
> Tika 2.x should help with this in pipes and async. Your system should
> expect to go oom or crash at some point if you're processing enough
> files.
I believe that this is what is happening in my case, it’s not due to a single file, it happens under high load when processing many files at once.
>
> Right --spawnChild is not default in 1.x, but it will be in 2.x. And,
> yes, you should be using it. To set the Xmx in the forked process add
> -J, as in -JXmx2g would set the Xmx for the forked process.
Did both now and I think this provides good recovery from OOM.
>
> I don't have experience to recommend bumping Xmx to close to your
> container's max memory. In java programs that do a bunch of work off
> heap, this would be a bad idea because you need to leave resources for
> your system os, but I don't think we do much off heap.
What’s your take on a configuration in which the container is capped at 4GB and the spawned child has a heap limit of 3GB? Sounds like a pretty safe margin to me.
>
> Which file types are causing OOMs? The MP4Parser is notorious, and
> we're looking to swap it out in 2.x for a different parser.
Good to hear. I don’t know how to identify the root cause because there are many files sent at once.
However, it would be great to learn if there is a quick way to trigger a high load and test resiliency to OOM, do you have a recommendation?
>
> Yep, TIKA-3353 is the monitoring that Nick was mentioning.
I am actually more interested in health checks, to detect when the system is stuck without automatically restarting. A built-in health check would certainly be a nice feature.
Besides OOM, one other possible cause is if /tmp gets full - for instance I see here https://github.com/tongwang/tika-server-docker/blob/master/bin/healthcheck that /tmp is cleaned up periodically and the health check fails if it is too full.
Are there any other situations that could indicate that the container is stuck and needs a restart and if yes, is there a way to detect the condition?
Thanks,
Cristi
>
> On Fri, May 28, 2021 at 9:08 AM Cristian Zamfir <cr...@cyberhaven.com> wrote:
>>
>> Thanks for your answer Nick!
>>
>> I am running apache/tika:latest-full which is using 1.25. Looks like I need at least version 1.26 for https://www.google.com/url?q=https://issues.apache.org/jira/browse/TIKA-3353&source=gmail-imap&ust=1622826254000000&usg=AOvVaw1we1l0Sh-gWif4FqbZ2qek, but I am not sure if this is not overkill for implementing basic liveness health checks.
>>
>> It's clear that –spawnChild and ForkParser are two must-haves that AFAIU are not default in apache/tika:latest-full
>>
>> My guess is that I also need to set the jvm heap size close to the memory resource limit for the container, but that's not ideal because the heap size would be statically configured while the memory resource limits are dynamic. Or maybe this is not necessary if I use -spawnChild?
>>
>> I am looking forward to your answers, thanks a lot!
>>
>> Cristi
>>
>>
>> On Fri, May 28, 2021 at 2:55 PM Nick Burch <ap...@gagravarr.org> wrote:
>>>
>>> On Thu, 27 May 2021, Cristian Zamfir wrote:
>>>> I am running some stress tests of the latest tika server docker (not
>>>> modified in any way, just pulled from the registry) and seeing that after a
>>>> few hours I see OOM in the logs. The container has a limit of 4GB set in
>>>> K8S. I am wondering if you have any best practices on how to avoid this.
>>>
>>> Hopefully one of our Tika+Docker experts will be along in a minute to help
>>> advise!
>>>
>>> For now, the general advice is documented at:
>>> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/TIKA/The%2BRobustness%2Bof%2BApache%2BTika&source=gmail-imap&ust=1622826254000000&usg=AOvVaw0p_ynGwlHapvMiy24sF1FP
>>>
>>> Also, which version of Tika are you on? There have been some contributions
>>> recently around monitoring the server, which you might want to upgrade
>>> for, eg TIKA-3353
>>>
>>> Nick
Re: best practices for avoiding OOM for tika docker
Posted by Tim Allison <ta...@apache.org>.
Tika 2.x should help with this in pipes and async. Your system should
expect to go oom or crash at some point if you're processing enough
files.
Right --spawnChild is not default in 1.x, but it will be in 2.x. And,
yes, you should be using it. To set the Xmx in the forked process add
-J, as in -JXmx2g would set the Xmx for the forked process.
I don't have experience to recommend bumping Xmx to close to your
container's max memory. In java programs that do a bunch of work off
heap, this would be a bad idea because you need to leave resources for
your system os, but I don't think we do much off heap.
Which file types are causing OOMs? The MP4Parser is notorious, and
we're looking to swap it out in 2.x for a different parser.
Yep, TIKA-3353 is the monitoring that Nick was mentioning.
On Fri, May 28, 2021 at 9:08 AM Cristian Zamfir <cr...@cyberhaven.com> wrote:
>
> Thanks for your answer Nick!
>
> I am running apache/tika:latest-full which is using 1.25. Looks like I need at least version 1.26 for https://issues.apache.org/jira/browse/TIKA-3353, but I am not sure if this is not overkill for implementing basic liveness health checks.
>
> It's clear that –spawnChild and ForkParser are two must-haves that AFAIU are not default in apache/tika:latest-full
>
> My guess is that I also need to set the jvm heap size close to the memory resource limit for the container, but that's not ideal because the heap size would be statically configured while the memory resource limits are dynamic. Or maybe this is not necessary if I use -spawnChild?
>
> I am looking forward to your answers, thanks a lot!
>
> Cristi
>
>
> On Fri, May 28, 2021 at 2:55 PM Nick Burch <ap...@gagravarr.org> wrote:
>>
>> On Thu, 27 May 2021, Cristian Zamfir wrote:
>> > I am running some stress tests of the latest tika server docker (not
>> > modified in any way, just pulled from the registry) and seeing that after a
>> > few hours I see OOM in the logs. The container has a limit of 4GB set in
>> > K8S. I am wondering if you have any best practices on how to avoid this.
>>
>> Hopefully one of our Tika+Docker experts will be along in a minute to help
>> advise!
>>
>> For now, the general advice is documented at:
>> https://cwiki.apache.org/confluence/display/TIKA/The+Robustness+of+Apache+Tika
>>
>> Also, which version of Tika are you on? There have been some contributions
>> recently around monitoring the server, which you might want to upgrade
>> for, eg TIKA-3353
>>
>> Nick
Re: best practices for avoiding OOM for tika docker
Posted by Cristian Zamfir <cr...@cyberhaven.com>.
Thanks for your answer Nick!
I am running apache/tika:latest-full which is using 1.25. Looks like I need
at least version 1.26 for https://issues.apache.org/jira/browse/TIKA-3353,
but I am not sure if this is not overkill for implementing basic liveness
health checks.
It's clear that –spawnChild and ForkParser are two must-haves that AFAIU
are not default in apache/tika:latest-full
My guess is that I also need to set the jvm heap size close to the memory
resource limit for the container, but that's not ideal because the heap
size would be statically configured while the memory resource limits are
dynamic. Or maybe this is not necessary if I use -spawnChild?
I am looking forward to your answers, thanks a lot!
Cristi
On Fri, May 28, 2021 at 2:55 PM Nick Burch <ap...@gagravarr.org> wrote:
> On Thu, 27 May 2021, Cristian Zamfir wrote:
> > I am running some stress tests of the latest tika server docker (not
> > modified in any way, just pulled from the registry) and seeing that
> after a
> > few hours I see OOM in the logs. The container has a limit of 4GB set in
> > K8S. I am wondering if you have any best practices on how to avoid this.
>
> Hopefully one of our Tika+Docker experts will be along in a minute to help
> advise!
>
> For now, the general advice is documented at:
>
> https://cwiki.apache.org/confluence/display/TIKA/The+Robustness+of+Apache+Tika
>
> Also, which version of Tika are you on? There have been some contributions
> recently around monitoring the server, which you might want to upgrade
> for, eg TIKA-3353
>
> Nick
>
Re: best practices for avoiding OOM for tika docker
Posted by Tim Allison <ta...@apache.org>.
>Is it reasonable to assume that detect-only is much safer?
Safer, yes. Safe, no.
At one point, a user reported a feature of the Compressor/Package
detector that read a 5 (?) byte file and allocated 2GB (?):
https://issues.apache.org/jira/browse/TIKA-2330
If you're only doing detection with the Mime detector (e.g., you do
not have tika-parsers on your path), you're probably quite safe.
However, if you have tika-parsers on your path, some parsers will be
run to determine subtypes of zip, for example, and other container
file formats.
On Fri, May 28, 2021 at 10:12 AM John Ulric <uj...@gmail.com> wrote:
>
> > For now, the general advice is documented at: https://cwiki.apache.org/confluence/display/TIKA/The+Robustness+of+Apache+Tika
>
> General question here: How would you compare the robustness in that respect between calling the full parsers vs. detect-only? Is it reasonable to assume that detect-only is much safer?
>
> John
>
Re: best practices for avoiding OOM for tika docker
Posted by John Ulric <uj...@gmail.com>.
> For now, the general advice is documented at:
https://cwiki.apache.org/confluence/display/TIKA/The+Robustness+of+Apache+Tika
<https://cwiki.apache.org/confluence/display/TIKA/The+Robustness+of+Apache+Tika>
General question here: How would you compare the robustness in that respect
between calling the full parsers vs. detect-only? Is it reasonable to
assume that detect-only is much safer?
John
Re: best practices for avoiding OOM for tika docker
Posted by Nick Burch <ap...@gagravarr.org>.
On Thu, 27 May 2021, Cristian Zamfir wrote:
> I am running some stress tests of the latest tika server docker (not
> modified in any way, just pulled from the registry) and seeing that after a
> few hours I see OOM in the logs. The container has a limit of 4GB set in
> K8S. I am wondering if you have any best practices on how to avoid this.
Hopefully one of our Tika+Docker experts will be along in a minute to help
advise!
For now, the general advice is documented at:
https://cwiki.apache.org/confluence/display/TIKA/The+Robustness+of+Apache+Tika
Also, which version of Tika are you on? There have been some contributions
recently around monitoring the server, which you might want to upgrade
for, eg TIKA-3353
Nick