You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tika.apache.org by pgnd <pg...@dev-mail.net> on 2024/01/29 19:20:12 UTC
tika-server 3x build; non-fatal ERROR "org.apache.tika.pipes.PipesServer oom ... java.lang.OutOfMemoryError: oom" messages ?
i'm building tika src (currently, branch == 'main') on Fedora's COPR online builder, to pkg as rpm.
it's a straightforward build of -server, with
mvn clean install -am -pl :tika-server-standard
my build 'succeeds' -- and installable rpm is generated -- but the build logs show lots of OOMs,
iiuc, the OOM should be resolvable by tweaking the memory opts ?
atm, i've set
++ export MAVEN_OPTS='-Xmx2048m -Xms1024m -XX:MaxMetaspaceSize=512m -Djava.awt.headless=true'
mvn clean install -am -pl :tika-server-standard
with that setting, i'm still seeing lots of these OOMs, seen only in 'org.apache.tika.pipes.PipesServer',
...
ERROR [main] 19:05:34,227 org.apache.tika.pipes.PipesServer oom: myId-39
java.lang.OutOfMemoryError: oom message
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
at org.apache.tika.parser.mock.MockParser.throwIt(MockParser.java:427) ~[test-classes/:?]
at org.apache.tika.parser.mock.MockParser.throwIt(MockParser.java:373) ~[test-classes/:?]
at org.apache.tika.parser.mock.MockParser.executeAction(MockParser.java:154) ~[test-classes/:?]
at org.apache.tika.parser.mock.MockParser.parse(MockParser.java:133) ~[test-classes/:?]
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:298) ~[classes/:?]
at org.apache.tika.parser.ParserDecorator.parse(ParserDecorator.java:152) ~[classes/:?]
at org.apache.tika.parser.DigestingParser.parse(DigestingParser.java:57) ~[classes/:?]
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:298) ~[classes/:?]
at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:203) ~[classes/:?]
at org.apache.tika.parser.RecursiveParserWrapper.parse(RecursiveParserWrapper.java:164) ~[classes/:?]
at org.apache.tika.pipes.PipesServer.parseRecursive(PipesServer.java:567) ~[classes/:?]
at org.apache.tika.pipes.PipesServer.parse(PipesServer.java:495) ~[classes/:?]
at org.apache.tika.pipes.PipesServer.parseIt(PipesServer.java:442) ~[classes/:?]
at org.apache.tika.pipes.PipesServer.actuallyParse(PipesServer.java:351) [classes/:?]
at org.apache.tika.pipes.PipesServer.parseOne(PipesServer.java:322) [classes/:?]
at org.apache.tika.pipes.PipesServer.processRequests(PipesServer.java:243) [classes/:?]
at org.apache.tika.pipes.PipesServer.main(PipesServer.java:178) [classes/:?]
ERROR [pool-12-thread-4] 19:05:34,339 org.apache.tika.pipes.PipesClient pipesClientId=2: execution exception
java.util.concurrent.ExecutionException: java.io.IOException: problem reading response from server: -1
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205) ~[?:?]
at org.apache.tika.pipes.PipesClient.actuallyProcess(PipesClient.java:201) [classes/:?]
at org.apache.tika.pipes.PipesClient.process(PipesClient.java:147) [classes/:?]
at org.apache.tika.pipes.async.AsyncProcessor$FetchEmitWorker.call(AsyncProcessor.java:295) [classes/:?]
at org.apache.tika.pipes.async.AsyncProcessor$FetchEmitWorker.call(AsyncProcessor.java:261) [classes/:?]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: java.io.IOException: problem reading response from server: -1
at org.apache.tika.pipes.PipesClient.readResults(PipesClient.java:294) ~[classes/:?]
at org.apache.tika.pipes.PipesClient.lambda$actuallyProcess$0(PipesClient.java:177) ~[classes/:?]
... 4 more
Caused by: java.lang.IllegalArgumentException: byte must be > 0
at org.apache.tika.pipes.PipesServer$STATUS.lookup(PipesServer.java:115) ~[classes/:?]
at org.apache.tika.pipes.PipesClient.readResults(PipesClient.java:288) ~[classes/:?]
at org.apache.tika.pipes.PipesClient.lambda$actuallyProcess$0(PipesClient.java:177) ~[classes/:?]
... 4 more
...
, although they appear non-fatal.
here's a last full build log,
https://download.copr.fedorainfracloud.org/results/pgfed/tika-pgnd/fedora-39-x86_64/06971734-tika-server/builder-live.log.gz
that, despite the OOMs, results in an installable rpm.
or, is this Pipes-specific, and either a config issue or outright bug?
Re: tika-server 3x build; non-fatal ERROR "org.apache.tika.pipes.PipesServer oom ... java.lang.OutOfMemoryError: oom" messages ?
Posted by Tim Allison <ta...@apache.org>.
Sorry. Y. If you want to open a pr to turn off the noise, please do!
tika-server does have a /pipes and /async handler which use the pipes
module, so, y, that’s on the build path.
‘clean’ — it actually builds, y, sorry.
Let us know if you have any other questions or if you find anything that
needs to be fixed before the 3.0.0 release!
Thank you!
On Mon, Jan 29, 2024 at 6:55 PM pgnd <pg...@dev-mail.net> wrote:
> > We have a number of tests of the tika-pipes modules that intentionally
> trigger an oom (with a limited -Xmx) to make sure that tika-pipes correctly
> restarts the forked process. This creates _a lot_ of noise.
>
> ah, intentional noise. i spun wheels for a bit trying to eliminate it.
>
> are those tests supposed to run/trigger even in the "just build
> -server-standard" case, i.e.,
>
> mvn clean install -am -pl :tika-server-standard
>
> > If you're getting a clean build, I'd say you're good.
>
> not 100% clear what you mean by 'clean' here.
>
> no FATAL errors? yup.
> launches and (seems to) exec? yup.
> devoid of _any_ ERRORS and generally quiet on warnings? not nearly. yet,
> i suppose!
>
> thx o/
>
>
Re: tika-server 3x build; non-fatal ERROR "org.apache.tika.pipes.PipesServer oom ... java.lang.OutOfMemoryError: oom" messages ?
Posted by pgnd <pg...@dev-mail.net>.
> We have a number of tests of the tika-pipes modules that intentionally trigger an oom (with a limited -Xmx) to make sure that tika-pipes correctly restarts the forked process. This creates _a lot_ of noise.
ah, intentional noise. i spun wheels for a bit trying to eliminate it.
are those tests supposed to run/trigger even in the "just build -server-standard" case, i.e.,
mvn clean install -am -pl :tika-server-standard
> If you're getting a clean build, I'd say you're good.
not 100% clear what you mean by 'clean' here.
no FATAL errors? yup.
launches and (seems to) exec? yup.
devoid of _any_ ERRORS and generally quiet on warnings? not nearly. yet, i suppose!
thx o/
Re: tika-server 3x build; non-fatal ERROR "org.apache.tika.pipes.PipesServer oom ... java.lang.OutOfMemoryError: oom" messages ?
Posted by Tim Allison <ta...@apache.org>.
We have a number of tests of the tika-pipes modules that intentionally
trigger an oom (with a limited -Xmx) to make sure that tika-pipes correctly
restarts the forked process. This creates _a lot_ of noise. If you're
getting a clean build, I'd say you're good.
On Mon, Jan 29, 2024 at 2:20 PM pgnd <pg...@dev-mail.net> wrote:
> i'm building tika src (currently, branch == 'main') on Fedora's COPR
> online builder, to pkg as rpm.
>
> it's a straightforward build of -server, with
>
> mvn clean install -am -pl :tika-server-standard
>
> my build 'succeeds' -- and installable rpm is generated -- but the build
> logs show lots of OOMs,
>
> iiuc, the OOM should be resolvable by tweaking the memory opts ?
>
> atm, i've set
>
> ++ export MAVEN_OPTS='-Xmx2048m -Xms1024m -XX:MaxMetaspaceSize=512m
> -Djava.awt.headless=true'
> mvn clean install -am -pl :tika-server-standard
>
> with that setting, i'm still seeing lots of these OOMs, seen only in
> 'org.apache.tika.pipes.PipesServer',
>
> ...
> ERROR [main] 19:05:34,227 org.apache.tika.pipes.PipesServer oom:
> myId-39
> java.lang.OutOfMemoryError: oom message
> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[?:?]
> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
> ~[?:?]
> at
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> ~[?:?]
> at
> java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
> ~[?:?]
> at
> java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
> ~[?:?]
> at
> org.apache.tika.parser.mock.MockParser.throwIt(MockParser.java:427)
> ~[test-classes/:?]
> at
> org.apache.tika.parser.mock.MockParser.throwIt(MockParser.java:373)
> ~[test-classes/:?]
> at
> org.apache.tika.parser.mock.MockParser.executeAction(MockParser.java:154)
> ~[test-classes/:?]
> at
> org.apache.tika.parser.mock.MockParser.parse(MockParser.java:133)
> ~[test-classes/:?]
> at
> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:298)
> ~[classes/:?]
> at
> org.apache.tika.parser.ParserDecorator.parse(ParserDecorator.java:152)
> ~[classes/:?]
> at
> org.apache.tika.parser.DigestingParser.parse(DigestingParser.java:57)
> ~[classes/:?]
> at
> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:298)
> ~[classes/:?]
> at
> org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:203)
> ~[classes/:?]
> at
> org.apache.tika.parser.RecursiveParserWrapper.parse(RecursiveParserWrapper.java:164)
> ~[classes/:?]
> at
> org.apache.tika.pipes.PipesServer.parseRecursive(PipesServer.java:567)
> ~[classes/:?]
> at
> org.apache.tika.pipes.PipesServer.parse(PipesServer.java:495) ~[classes/:?]
> at
> org.apache.tika.pipes.PipesServer.parseIt(PipesServer.java:442)
> ~[classes/:?]
> at
> org.apache.tika.pipes.PipesServer.actuallyParse(PipesServer.java:351)
> [classes/:?]
> at
> org.apache.tika.pipes.PipesServer.parseOne(PipesServer.java:322)
> [classes/:?]
> at
> org.apache.tika.pipes.PipesServer.processRequests(PipesServer.java:243)
> [classes/:?]
> at
> org.apache.tika.pipes.PipesServer.main(PipesServer.java:178) [classes/:?]
> ERROR [pool-12-thread-4] 19:05:34,339
> org.apache.tika.pipes.PipesClient pipesClientId=2: execution exception
> java.util.concurrent.ExecutionException: java.io.IOException:
> problem reading response from server: -1
> at
> java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
> at
> java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205) ~[?:?]
> at
> org.apache.tika.pipes.PipesClient.actuallyProcess(PipesClient.java:201)
> [classes/:?]
> at
> org.apache.tika.pipes.PipesClient.process(PipesClient.java:147) [classes/:?]
> at
> org.apache.tika.pipes.async.AsyncProcessor$FetchEmitWorker.call(AsyncProcessor.java:295)
> [classes/:?]
> at
> org.apache.tika.pipes.async.AsyncProcessor$FetchEmitWorker.call(AsyncProcessor.java:261)
> [classes/:?]
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
> [?:?]
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> [?:?]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> [?:?]
> at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
> Caused by: java.io.IOException: problem reading response from
> server: -1
> at
> org.apache.tika.pipes.PipesClient.readResults(PipesClient.java:294)
> ~[classes/:?]
> at
> org.apache.tika.pipes.PipesClient.lambda$actuallyProcess$0(PipesClient.java:177)
> ~[classes/:?]
> ... 4 more
> Caused by: java.lang.IllegalArgumentException: byte must be > 0
> at
> org.apache.tika.pipes.PipesServer$STATUS.lookup(PipesServer.java:115)
> ~[classes/:?]
> at
> org.apache.tika.pipes.PipesClient.readResults(PipesClient.java:288)
> ~[classes/:?]
> at
> org.apache.tika.pipes.PipesClient.lambda$actuallyProcess$0(PipesClient.java:177)
> ~[classes/:?]
> ... 4 more
>
> ...
>
> , although they appear non-fatal.
>
> here's a last full build log,
>
>
> https://download.copr.fedorainfracloud.org/results/pgfed/tika-pgnd/fedora-39-x86_64/06971734-tika-server/builder-live.log.gz
>
> that, despite the OOMs, results in an installable rpm.
>
> or, is this Pipes-specific, and either a config issue or outright bug?
>
>
>
>
>