You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by Priya Arora <pr...@smartshore.nl> on 2019/09/11 05:43:14 UTC
Re: Unexpected HTTP result code: -1: null
Hi all,
Manifold CF crashes.
Raising this issue again, after enhancing memory and otherwise, still
facing the same issue. TRied every possible solution If anybody can help ,
please suggest me some solution.
*Problem:-* Manifoldcf crashes while crawling data from an intranet URL
hinting out below error in logs:-
agents process ran out of memory - shutting down
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.HashMap.resize(HashMap.java:704)
at java.util.HashMap.putVal(HashMap.java:629)
at java.util.HashMap.put(HashMap.java:612)
at
org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndEx
ecute(CacheManager.java:135)
at
org.apache.manifoldcf.agents.transformationconnection.TransformationC
onnectionManager.loadMultipleInternal(TransformationConnectionManager.java:265)
at
org.apache.manifoldcf.agents.transformationconnection.TransformationC
onnectionManager.loadMultiple(TransformationConnectionManager.java:246)
agents process could not start - shutting down
agents process ran out of memory - shutting down
java.lang.OutOfMemoryError: GC overhead limit exceeded
at
org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:11
93)
at
org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:15
77)
at
org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:36
7)
at
org.apache.manifoldcf.core.database.Database.execute(Database.java:87
3)
at
org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(D
atabase.java:696)
agents process ran out of memory - shutting down
java.lang.OutOfMemoryError: GC overhead limit exceeded
[Thread-491] INFO org.eclipse.jetty.server.ServerConnector - Stopped
ServerConne
ctor@3a4621bd{HTTP/1.1}{0.0.0.0:8345}
agents process ran out of memory - shutting down
java.lang.OutOfMemoryError: GC overhead limit exceeded
[Thread-491] INFO org.eclipse.jetty.server.handler.ContextHandler - Stopped
o.e.
j.w.WebAppContext@6a57ae10{/mcf-api-service,file:/tmp/jetty-0.0.0.0-8345-mcf-api
-service.war-_mcf-api-service-any-2010273384908625074.dir/webapp/,UNAVAILABLE}{/
usr/share/manifoldcf/example/./../web/war/mcf-api-service.war}
[Thread-491] INFO org.eclipse.jetty.server.handler.ContextHandler - Stopped
o.e.
j.w.WebAppContext@51c693d{/mcf-authority-service,file:/tmp/jetty-0.0.0.0-8345-mc
f-authority-service.war-_mcf-authority-service-any-1431262612693374489.dir/webap
p/,UNAVAILABLE}{/usr/share/manifoldcf/example/./../web/war/mcf-authority-service
.war}
agents process ran out of memory - shutting down
java.lang.OutOfMemoryError: Java heap space
agents process ran out of memory - shutting down
java.lang.OutOfMemoryError: Java heap space
agents process ran out of memory - shutting down
java.lang.OutOfMemoryError: Java heap space
Exception in thread "PostgreSQL-JDBC-SharedTimer-1"
java.lang.OutOfMemoryError:
Java heap space
All configuration are:-
I have -Xms1024m ,-Xmx1024m memory allocated in
*start-options.env.unix, start-options.env.win file.*
Also Configuration:-
1) *For Crawler server *- 16 GB RAM and 8-Core Intel(R) Xeon(R) CPU
E5-2660v3 @ 2.60GHz and
2) *For Elasticsearch server* - 48GB and 16-Core Intel(R) Xeon(R) CPU
E5-2660 v3 @ 2.60GHz and i am using postgres as database.
3) Database as PostgreSQL
4) No of worker threads:-20.
5) manifold Version :-2.13
Can anybody please suggest some solution to get rid of this error and
crash, as whenever this istutaion comes, manifoldCF UI results me this:_
[image: image.png]
Any suggestion would be highly appreciated.
Thanks
Priya
On Fri, Aug 16, 2019 at 6:41 PM Priya Arora <pr...@smartshore.nl> wrote:
> Sure I will try to do that.. and will get back to you
>
> Sent from my iPhone
>
> > On 16-Aug-2019, at 6:23 PM, Michael Cizmar <mi...@michaelcizmar.com>
> wrote:
> >
> > Priya - Was this right?
> >
> > 48GB and 1-Core Intel(R) Xeon(R) CPU
> >
> > While not directly related to an out of memory issue, you should have
> more
> > cores allocated to ES.
> >
> >> On Fri, Aug 16, 2019 at 1:09 AM Priya Arora <pr...@smartshore.nl>
> wrote:
> >>
> >> *Existing Threads/connections configuration is :-*
> >>
> >> How many worker threads do you have? - 15 worker threads has been
> >> allocated(in properties.xml file).
> >> And the Tika Extractor connections -10 connections are defined.
> >>
> >> Is this suggested to reduce the number more.
> >> If not, what else can be a solution
> >>
> >> Thanks
> >> Priya
> >>
> >>
> >>
> >>> On Wed, Aug 14, 2019 at 5:32 PM Karl Wright <da...@gmail.com>
> wrote:
> >>>
> >>> How many worker threads do you have?
> >>> Even if each worker thread is constrained in memory, and they should
> be,
> >>> you can easily cause things to run out of memory by giving too many
> >> worker
> >>> threads. Another way to keep Tika's usage constrained would be to
> reduce
> >>> the number of Tika Extractor connections, because that effectively
> limits
> >>> the number of extractions that can be going on at the same time.
> >>>
> >>> Karl
> >>>
> >>>
> >>>> On Wed, Aug 14, 2019 at 7:23 AM Priya Arora <pr...@smartshore.nl>
> wrote:
> >>>>
> >>>> Yes , I am using Tika Extractor. And the version used for manifold is
> >>> 2.13.
> >>>> Also I am using postgres as database.
> >>>>
> >>>> I have 4 types of jobs
> >>>> One is accessing/re crawling data from a public site. Other three are
> >>>> accessing intranet site.
> >>>> Out of which two are giving me correct output-without any error and
> >> third
> >>>> one which is having data more than the other two , and giving me this
> >>>> error.
> >>>>
> >>>> Is there any possibility with site accessibility issue. Can you please
> >>>> suggest some solution
> >>>> Thanks and regards
> >>>> Priya
> >>>>
> >>>> On Wed, Aug 14, 2019 at 3:11 PM Karl Wright <da...@gmail.com>
> >> wrote:
> >>>>
> >>>>> I will need to know more. Do you have the tika extractor in your
> >>>>> pipeline? If so, what version of ManifoldCF are you using? Tika has
> >>> had
> >>>>> bugs related to memory consumption in the past; the out of memory
> >>>> exception
> >>>>> may be coming from it and therefore a stack trace is critical to
> >> have.
> >>>>>
> >>>>> Alternatively, you can upgrade to the latest version of MCF (2.13)
> >> and
> >>>> that
> >>>>> has a newer version of Tika without those problem. But you may need
> >> to
> >>>> get
> >>>>> the agents process more memory.
> >>>>>
> >>>>> Another possible cause is that you're using hsqldb in production.
> >>> HSQLDB
> >>>>> keeps all of its tables in memory. If you have a large crawl, you do
> >>> not
> >>>>> want to use HSQLDB.
> >>>>>
> >>>>> Thanks,
> >>>>> Karl
> >>>>>
> >>>>>
> >>>>> On Wed, Aug 14, 2019 at 3:41 AM Priya Arora <pr...@smartshore.nl>
> >>> wrote:
> >>>>>
> >>>>>> Hi Karl,
> >>>>>>
> >>>>>> Manifold CF logs hints out me an error like :
> >>>>>> agents process ran out of memory - shutting down
> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >>>>>>
> >>>>>> Also I have -Xms1024m ,-Xmx1024m memory allocated in
> >>>>>> start-options.env.unix, start-options.env.win file.
> >>>>>> Also Configuration:-
> >>>>>> 1) For Crawler server - 16 GB RAM and 8-Core Intel(R) Xeon(R) CPU
> >>>> E5-2660
> >>>>>> v3 @ 2.60GHz and
> >>>>>>
> >>>>>> 2) For Elasticsearch server - 48GB and 1-Core Intel(R) Xeon(R) CPU
> >>>>> E5-2660
> >>>>>> v3 @ 2.60GHz and i am using postgres as database.
> >>>>>>
> >>>>>> Can you please help me out, what to do in this case.
> >>>>>>
> >>>>>> Thanks
> >>>>>> Priya
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Aug 14, 2019 at 12:33 PM Karl Wright <da...@gmail.com>
> >>>> wrote:
> >>>>>>
> >>>>>>> The error occurs, I believe, as the result of basic connection
> >>>>> problems,
> >>>>>>> e.g. the connection is getting rejected. You can find more
> >>>> information
> >>>>>> in
> >>>>>>> the simple history, and in the manifoldcf log.
> >>>>>>>
> >>>>>>> I would like to know the underlying cause, since the connector
> >>> should
> >>>>> be
> >>>>>>> resilient against errors of this kind.
> >>>>>>>
> >>>>>>> Karl
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Aug 14, 2019, 1:46 AM Priya Arora <pr...@smartshore.nl>
> >>>> wrote:
> >>>>>>>
> >>>>>>>> Hi Karl,
> >>>>>>>>
> >>>>>>>> I have an web Repository connector(Seeds:- an intranet Site).,
> >>> and
> >>>>> job
> >>>>>> i
> >>>>>>>> son Production server.
> >>>>>>>>
> >>>>>>>> When i ran job on PROD, the job stops itself 2 times with and
> >>>>>>> error:Error:
> >>>>>>>> Unexpected HTTP result code: -1: null.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Can you please provide me an idea, in which it happens so?
> >>>>>>>>
> >>>>>>>> Thanks and regards
> >>>>>>>> Priya Arora
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
Re: Unexpected HTTP result code: -1: null
Posted by Karl Wright <da...@gmail.com>.
"I have -Xms1024m ,-Xmx1024m memory allocated in
*start-options.env.unix, start-options.env.win file.*
Also Configuration:-
1) *For Crawler server *- 16 GB RAM and 8-Core Intel(R) Xeon(R) CPU
E5-2660v3 @ 2.60GHz and"
What is "crawler server"?
You said "single process example", right? That means there is one MCF
process. That process is running out of memory. You are giving it one
gigabyte. It is not enough. Add more memory to the -Xmx setting in
start-options.
Karl
On Wed, Sep 11, 2019 at 1:43 AM Priya Arora <pr...@smartshore.nl> wrote:
> Hi all,
>
> Manifold CF crashes.
> Raising this issue again, after enhancing memory and otherwise, still
> facing the same issue. TRied every possible solution If anybody can help ,
> please suggest me some solution.
> *Problem:-* Manifoldcf crashes while crawling data from an intranet URL
> hinting out below error in logs:-
>
> agents process ran out of memory - shutting down
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.HashMap.resize(HashMap.java:704)
> at java.util.HashMap.putVal(HashMap.java:629)
> at java.util.HashMap.put(HashMap.java:612)
> at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndEx
>
> ecute(CacheManager.java:135)
> at
> org.apache.manifoldcf.agents.transformationconnection.TransformationC
>
>
> onnectionManager.loadMultipleInternal(TransformationConnectionManager.java:265)
> at
> org.apache.manifoldcf.agents.transformationconnection.TransformationC
>
> onnectionManager.loadMultiple(TransformationConnectionManager.java:246)
> agents process could not start - shutting down
> agents process ran out of memory - shutting down
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at
> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:11
>
> 93)
> at
> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:15
>
> 77)
> at
> org.postgresql.jdbc.PgConnection.createStatement(PgConnection.java:36
>
> 7)
> at
> org.apache.manifoldcf.core.database.Database.execute(Database.java:87
>
> 3)
> at
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(D
>
> atabase.java:696)
> agents process ran out of memory - shutting down
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> [Thread-491] INFO org.eclipse.jetty.server.ServerConnector - Stopped
> ServerConne
> ctor@3a4621bd{HTTP/1.1}{0.0.0.0:8345}
> agents process ran out of memory - shutting down
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> [Thread-491] INFO org.eclipse.jetty.server.handler.ContextHandler -
> Stopped o.e.
> j.w.WebAppContext@6a57ae10{/mcf-api-service,file:/tmp/jetty-0.0.0.0-8345-mcf-api
>
>
> -service.war-_mcf-api-service-any-2010273384908625074.dir/webapp/,UNAVAILABLE}{/
>
> usr/share/manifoldcf/example/./../web/war/mcf-api-service.war}
> [Thread-491] INFO org.eclipse.jetty.server.handler.ContextHandler -
> Stopped o.e.
> j.w.WebAppContext@51c693d{/mcf-authority-service,file:/tmp/jetty-0.0.0.0-8345-mc
>
>
> f-authority-service.war-_mcf-authority-service-any-1431262612693374489.dir/webap
>
>
> p/,UNAVAILABLE}{/usr/share/manifoldcf/example/./../web/war/mcf-authority-service
>
> .war}
> agents process ran out of memory - shutting down
> java.lang.OutOfMemoryError: Java heap space
> agents process ran out of memory - shutting down
> java.lang.OutOfMemoryError: Java heap space
> agents process ran out of memory - shutting down
> java.lang.OutOfMemoryError: Java heap space
> Exception in thread "PostgreSQL-JDBC-SharedTimer-1"
> java.lang.OutOfMemoryError:
>
> Java heap space
>
> All configuration are:-
> I have -Xms1024m ,-Xmx1024m memory allocated in
> *start-options.env.unix, start-options.env.win file.*
> Also Configuration:-
> 1) *For Crawler server *- 16 GB RAM and 8-Core Intel(R) Xeon(R) CPU
> E5-2660v3 @ 2.60GHz and
>
> 2) *For Elasticsearch server* - 48GB and 16-Core Intel(R) Xeon(R) CPU
> E5-2660 v3 @ 2.60GHz and i am using postgres as database.
>
> 3) Database as PostgreSQL
> 4) No of worker threads:-20.
> 5) manifold Version :-2.13
>
> Can anybody please suggest some solution to get rid of this error and
> crash, as whenever this istutaion comes, manifoldCF UI results me this:_
> [image: image.png]
>
> Any suggestion would be highly appreciated.
>
> Thanks
> Priya
>
> On Fri, Aug 16, 2019 at 6:41 PM Priya Arora <pr...@smartshore.nl> wrote:
>
>> Sure I will try to do that.. and will get back to you
>>
>> Sent from my iPhone
>>
>> > On 16-Aug-2019, at 6:23 PM, Michael Cizmar <mi...@michaelcizmar.com>
>> wrote:
>> >
>> > Priya - Was this right?
>> >
>> > 48GB and 1-Core Intel(R) Xeon(R) CPU
>> >
>> > While not directly related to an out of memory issue, you should have
>> more
>> > cores allocated to ES.
>> >
>> >> On Fri, Aug 16, 2019 at 1:09 AM Priya Arora <pr...@smartshore.nl>
>> wrote:
>> >>
>> >> *Existing Threads/connections configuration is :-*
>> >>
>> >> How many worker threads do you have? - 15 worker threads has been
>> >> allocated(in properties.xml file).
>> >> And the Tika Extractor connections -10 connections are defined.
>> >>
>> >> Is this suggested to reduce the number more.
>> >> If not, what else can be a solution
>> >>
>> >> Thanks
>> >> Priya
>> >>
>> >>
>> >>
>> >>> On Wed, Aug 14, 2019 at 5:32 PM Karl Wright <da...@gmail.com>
>> wrote:
>> >>>
>> >>> How many worker threads do you have?
>> >>> Even if each worker thread is constrained in memory, and they should
>> be,
>> >>> you can easily cause things to run out of memory by giving too many
>> >> worker
>> >>> threads. Another way to keep Tika's usage constrained would be to
>> reduce
>> >>> the number of Tika Extractor connections, because that effectively
>> limits
>> >>> the number of extractions that can be going on at the same time.
>> >>>
>> >>> Karl
>> >>>
>> >>>
>> >>>> On Wed, Aug 14, 2019 at 7:23 AM Priya Arora <pr...@smartshore.nl>
>> wrote:
>> >>>>
>> >>>> Yes , I am using Tika Extractor. And the version used for manifold is
>> >>> 2.13.
>> >>>> Also I am using postgres as database.
>> >>>>
>> >>>> I have 4 types of jobs
>> >>>> One is accessing/re crawling data from a public site. Other three are
>> >>>> accessing intranet site.
>> >>>> Out of which two are giving me correct output-without any error and
>> >> third
>> >>>> one which is having data more than the other two , and giving me
>> this
>> >>>> error.
>> >>>>
>> >>>> Is there any possibility with site accessibility issue. Can you
>> please
>> >>>> suggest some solution
>> >>>> Thanks and regards
>> >>>> Priya
>> >>>>
>> >>>> On Wed, Aug 14, 2019 at 3:11 PM Karl Wright <da...@gmail.com>
>> >> wrote:
>> >>>>
>> >>>>> I will need to know more. Do you have the tika extractor in your
>> >>>>> pipeline? If so, what version of ManifoldCF are you using? Tika
>> has
>> >>> had
>> >>>>> bugs related to memory consumption in the past; the out of memory
>> >>>> exception
>> >>>>> may be coming from it and therefore a stack trace is critical to
>> >> have.
>> >>>>>
>> >>>>> Alternatively, you can upgrade to the latest version of MCF (2.13)
>> >> and
>> >>>> that
>> >>>>> has a newer version of Tika without those problem. But you may need
>> >> to
>> >>>> get
>> >>>>> the agents process more memory.
>> >>>>>
>> >>>>> Another possible cause is that you're using hsqldb in production.
>> >>> HSQLDB
>> >>>>> keeps all of its tables in memory. If you have a large crawl, you
>> do
>> >>> not
>> >>>>> want to use HSQLDB.
>> >>>>>
>> >>>>> Thanks,
>> >>>>> Karl
>> >>>>>
>> >>>>>
>> >>>>> On Wed, Aug 14, 2019 at 3:41 AM Priya Arora <pr...@smartshore.nl>
>> >>> wrote:
>> >>>>>
>> >>>>>> Hi Karl,
>> >>>>>>
>> >>>>>> Manifold CF logs hints out me an error like :
>> >>>>>> agents process ran out of memory - shutting down
>> >>>>>> java.lang.OutOfMemoryError: Java heap space
>> >>>>>>
>> >>>>>> Also I have -Xms1024m ,-Xmx1024m memory allocated in
>> >>>>>> start-options.env.unix, start-options.env.win file.
>> >>>>>> Also Configuration:-
>> >>>>>> 1) For Crawler server - 16 GB RAM and 8-Core Intel(R) Xeon(R) CPU
>> >>>> E5-2660
>> >>>>>> v3 @ 2.60GHz and
>> >>>>>>
>> >>>>>> 2) For Elasticsearch server - 48GB and 1-Core Intel(R) Xeon(R) CPU
>> >>>>> E5-2660
>> >>>>>> v3 @ 2.60GHz and i am using postgres as database.
>> >>>>>>
>> >>>>>> Can you please help me out, what to do in this case.
>> >>>>>>
>> >>>>>> Thanks
>> >>>>>> Priya
>> >>>>>>
>> >>>>>>
>> >>>>>> On Wed, Aug 14, 2019 at 12:33 PM Karl Wright <da...@gmail.com>
>> >>>> wrote:
>> >>>>>>
>> >>>>>>> The error occurs, I believe, as the result of basic connection
>> >>>>> problems,
>> >>>>>>> e.g. the connection is getting rejected. You can find more
>> >>>> information
>> >>>>>> in
>> >>>>>>> the simple history, and in the manifoldcf log.
>> >>>>>>>
>> >>>>>>> I would like to know the underlying cause, since the connector
>> >>> should
>> >>>>> be
>> >>>>>>> resilient against errors of this kind.
>> >>>>>>>
>> >>>>>>> Karl
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Wed, Aug 14, 2019, 1:46 AM Priya Arora <pr...@smartshore.nl>
>> >>>> wrote:
>> >>>>>>>
>> >>>>>>>> Hi Karl,
>> >>>>>>>>
>> >>>>>>>> I have an web Repository connector(Seeds:- an intranet Site).,
>> >>> and
>> >>>>> job
>> >>>>>> i
>> >>>>>>>> son Production server.
>> >>>>>>>>
>> >>>>>>>> When i ran job on PROD, the job stops itself 2 times with and
>> >>>>>>> error:Error:
>> >>>>>>>> Unexpected HTTP result code: -1: null.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> Can you please provide me an idea, in which it happens so?
>> >>>>>>>>
>> >>>>>>>> Thanks and regards
>> >>>>>>>> Priya Arora
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>>
>