Xindice under moderate load

[Lachlan Donald <la...@ljd.cc>]

I have posted previously about my major problems with Xindice and
OutOfMemory errors with no resolution so I went off and did some research
into possible causes of the problem.

My application is a content management system written in PHP. Web pages are
built by retrieving xml from Xindice and transforming it to XHTML with
xslts. I know that having to fetch content every time a web page is loaded
is a bad idea performance wise, hence the future plans are to cache the
output and allow sites to be bulk published via ftp like blogger.com does
it.

So I have about 5 sites that are being driven from this system, all of these
sites generate pretty small amounts of traffic, I would say a maximum of
200-300 hits a day per site. My problem is that I now have daily (sometimes
bi-daily) Xindice crashes with OutOfMemory exceptions.

Initially I thought it might be a systems problem so I added some more
memory and tried a variety of different servlet containers, first Tomcat,
then Jetty, then I tried a variety of Sun JDK's (couldn't get JRockIt to
run). It crashed consistantly in all of them.

My theory is that something in the Xindice system is having problems with
concurrant access (e.g too many threads). The OutOfMemory exceptions occur
when Xindice has 40+ threads running and lots of concurrant requests. From
what I have read Java throws this error when it can't create any more
threads.

The other thing I have read a few places is that Xerces is quite prone to
crashing with the OutOfMemory error when it is trying to parse specific
malformed xml, but I am pretty sure all of my XML is well formed before it
even gets to Xerces.

Any suggestions of what I can do? I have been pulling my hairs out at a
rather rapid rate :)

Cheers,
Lachlan Donald

Re: Xindice under moderate load

[Lachlan Donald <la...@ljd.cc>]

Sorry I got caught up in trying to get the JRockIt vm working and I missed
that part of your message.

The only java memory settings I have are set with
JAVA_OPTS="-Xms300m -Xmx300m" in my /etc/profile. They were used by Tomcat,
but I am not sure if Jetty (which I now use) picks up those settings.

As for OS memory settings all I have done is increase max_thread_proc to
10000.

What other settings would you recommend?

Cheers,
Lachlan Donald


----- Original Message -----
From: "Gianugo Rabellino" <gi...@apache.org>
To: <xi...@xml.apache.org>
Sent: Friday, January 17, 2003 7:34 PM
Subject: Re: Xindice under moderate load


> Lachlan Donald wrote:
>
> > So I have about 5 sites that are being driven from this system, all of
these
> > sites generate pretty small amounts of traffic, I would say a maximum of
> > 200-300 hits a day per site. My problem is that I now have daily
(sometimes
> > bi-daily) Xindice crashes with OutOfMemory exceptions.
>
> Next week I will be able to test and try to reproduce your problem. But
> you still didn't reply to my last questions about your Java settings.
> Can you give me more details on that (exp. Java's own memory settings)?
>
> Thanks,
>
> --
> Gianugo Rabellino
>
>

Re: Xindice under moderate load

[Gianugo Rabellino <gi...@apache.org>]

Lachlan Donald wrote:

> So I have about 5 sites that are being driven from this system, all of these
> sites generate pretty small amounts of traffic, I would say a maximum of
> 200-300 hits a day per site. My problem is that I now have daily (sometimes
> bi-daily) Xindice crashes with OutOfMemory exceptions.

Next week I will be able to test and try to reproduce your problem. But 
you still didn't reply to my last questions about your Java settings. 
Can you give me more details on that (exp. Java's own memory settings)?

Thanks,

-- 
Gianugo Rabellino