You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@roller.apache.org by Matt Raible <mr...@gmail.com> on 2005/07/25 01:29:12 UTC
Roller hosed up on raibledesigns.com
Guys,
Does anyone know what might be causing the following problem:
<quote from="emailBelow">
On starting your JVM, mysql's 'show processlist' shows you quickly get 6
connections, and then when I tried to surf to raibledesigns.com it quickly
grew to about 15 connections, and then slowly crawled up to 21, and then
the catalina.out spewed out it's errors and the connections dropped back
to 10. It then started climbing up, sitting at 15-16 connections for a
while, then climbed back to 21, then catalina.out spewed out a bunch more
messages, and the connections dropped back to 8....
</quote>
My site seems to start fine if I remove Roller, otherwise, it hangs
and the above scenario happens. My suspicion is possibly the search
index getting created, but that doesn't explain why so many
connections are getting created.
Thanks,
Matt
---------- Forwarded message ----------
From: Keith Bjorndahl <ke...@kgbinternet.com>
Date: Jul 24, 2005 5:20 PM
Subject: Re: Site is really screwed up
To: matt@raibledesigns.com
Hi Matt
> So you haven't restored anything back to the way my home directory was
> on those specific dates?
No, I thought I'd let you decide which files you want to keep and which to
replace. You can drop the whole Tomcat directory in if you want to try
that though.
> This still seems to be a database
> connectivity issue. Have you upgraded anything (hardware or software)
> in the last couple of days?
Not even in the last couple of weeks...a ssh upgrade in June, and a kernel
upgrade on June 1.
MySQL is set at
set-variable = max_connections=500
set-variable = max_user_connections=50
as it has been for a year or so, and the max_connections reached since the
last mysql restart (when you first reported the problem) is 89.
On starting your JVM, mysql's 'show processlist' shows you quickly get 6
connections, and then when I tried to surf to raibledesigns.com it quickly
grew to about 15 connections, and then slowly crawled up to 21, and then
the catalina.out spewed out it's errors and the connections dropped back
to 10. It then started climbing up, sitting at 15-16 connections for a
while, then climbed back to 21, then catalina.out spewed out a bunch more
messages, and the connections dropped back to 8....
It's like it keeps trying to get connections until it hit's your own pool
limit, and then drops a bunch and starts over again...any connection leaks
anywhere maybe?
Keith
>
> ERROR 2005-07-24 16:10:58,828 | JDBCExceptionReporter:logExceptions |
> Cannot get
> a connection, pool exhausted
> ERROR 2005-07-24 16:10:58,830 | JDBCException:<init> | Cannot open
> connection
> org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool
> exhaus
> ted
>
> Matt
>
> On 7/24/05, Keith Bjorndahl <ke...@kgbinternet.com> wrote:
>> I've put the files in the 'restore' sub-directory of your home
>> directory.
>> I've put the Jul1 backup in one diretory, and the Jul17 and Jul22
>> incrementals in their own directories under 'restore'. The Jul17 is an
>> incremental on the Jul1, and the Jul22 is an incremental on Jul17, so
>> you
>> can see what changed, if anything, up until the 22nd.
>>
>> Keith
>>
>> > I've tried a number of different things and I can't seem to get Tomcat
>> > started and serving pages no matter what I do. One strange thing I
>> > noticed is I can't delete $CATALINA_HOME/work/Catalina. I've noticed
>> > similar problems to what I'm experiencing - and cleaning out the work
>> > directory often solves them. Can you use your root privileges to
>> > clear out the contents of this directory. If doing that doesn't fix
>> > anything - we should probably restore from backup. Friday afternoon
>> > should work - although it would suck if we had to do this b/c there's
>> > a few folks that might've updated the wiki since then. Can you save
>> > ~/public_html/wiki so I can diff it against Friday's copy?
>> >
>> > Thanks,
>> >
>> > Matt
>> >
>> > On 7/24/05, Keith Bjorndahl <ke...@kgbinternet.com> wrote:
>> >> Oh....let me know if you want to restore any backups or anything.
>> >>
>> >> Keith
>> >>
>> >> > It appears that my site is really screwed up - I get 404 for pages
>> on
>> >> > my blog - as if something is hosed on the installation of Roller.
>> >> > This makes me suspect that someone might've hacked my site. Any
>> >> > ideas?
>> >> >
>> >> > Thanks,
>> >> >
>> >> > Matt
>> >> >
>> >>
>> >>
>> >>
>> >
>>
>>
>>
>
Re: Roller hosed up on raibledesigns.com
Posted by "Matthew P. Schmidt" <ma...@javalobby.org>.
Yea, its in the code.
-Matt
Matt Raible wrote:
>Did you turn off comment indexing in the code - or in a properties
>file? I'm guessing it's in the code?
>
>Thanks,
>
>Matt
>
>On 7/24/05, Matthew P. Schmidt <ma...@javalobby.org> wrote:
>
>
>>On JRoller, we had to turn off the comment indexing because it took too
>>long to fetch all of those, you may want to take that part out of the
>>build document process and see what happens. In addition, we're seeing
>>Resin have to close a lot of dangling connections. I'm not completely
>>sure what they are yet, but it most appear to happen in the
>>NavigationBarTag. For some reason we see these errors when the
>>PersistenceFilter isn't in teh chain as well. I'm not completely sure
>>how in the world that would happen since its mapped on /* with all the
>>proper FORWARD and REQUEST calls. Any ideas?
>>
>>
>>-Matt
>>
>>Matt Raible wrote:
>>
>>
>>
>>>Guys,
>>>
>>>Does anyone know what might be causing the following problem:
>>>
>>><quote from="emailBelow">
>>>On starting your JVM, mysql's 'show processlist' shows you quickly get 6
>>>connections, and then when I tried to surf to raibledesigns.com it quickly
>>>grew to about 15 connections, and then slowly crawled up to 21, and then
>>>the catalina.out spewed out it's errors and the connections dropped back
>>>to 10. It then started climbing up, sitting at 15-16 connections for a
>>>while, then climbed back to 21, then catalina.out spewed out a bunch more
>>>messages, and the connections dropped back to 8....
>>></quote>
>>>
>>>My site seems to start fine if I remove Roller, otherwise, it hangs
>>>and the above scenario happens. My suspicion is possibly the search
>>>index getting created, but that doesn't explain why so many
>>>connections are getting created.
>>>
>>>Thanks,
>>>
>>>Matt
>>>
>>>
>>>---------- Forwarded message ----------
>>>From: Keith Bjorndahl <ke...@kgbinternet.com>
>>>Date: Jul 24, 2005 5:20 PM
>>>Subject: Re: Site is really screwed up
>>>To: matt@raibledesigns.com
>>>
>>>
>>>Hi Matt
>>>
>>>
>>>
>>>
>>>
>>>>So you haven't restored anything back to the way my home directory was
>>>>on those specific dates?
>>>>
>>>>
>>>>
>>>>
>>>No, I thought I'd let you decide which files you want to keep and which to
>>>replace. You can drop the whole Tomcat directory in if you want to try
>>>that though.
>>>
>>>
>>>
>>>
>>>
>>>>This still seems to be a database
>>>>connectivity issue. Have you upgraded anything (hardware or software)
>>>>in the last couple of days?
>>>>
>>>>
>>>>
>>>>
>>>Not even in the last couple of weeks...a ssh upgrade in June, and a kernel
>>>upgrade on June 1.
>>>
>>>MySQL is set at
>>>set-variable = max_connections=500
>>>set-variable = max_user_connections=50
>>>
>>>as it has been for a year or so, and the max_connections reached since the
>>>last mysql restart (when you first reported the problem) is 89.
>>>
>>>On starting your JVM, mysql's 'show processlist' shows you quickly get 6
>>>connections, and then when I tried to surf to raibledesigns.com it quickly
>>>grew to about 15 connections, and then slowly crawled up to 21, and then
>>>the catalina.out spewed out it's errors and the connections dropped back
>>>to 10. It then started climbing up, sitting at 15-16 connections for a
>>>while, then climbed back to 21, then catalina.out spewed out a bunch more
>>>messages, and the connections dropped back to 8....
>>>
>>>It's like it keeps trying to get connections until it hit's your own pool
>>>limit, and then drops a bunch and starts over again...any connection leaks
>>>anywhere maybe?
>>>
>>>Keith
>>>
>>>
>>>
>>>
>>>
>>>
>>>>ERROR 2005-07-24 16:10:58,828 | JDBCExceptionReporter:logExceptions |
>>>>Cannot get
>>>>a connection, pool exhausted
>>>>ERROR 2005-07-24 16:10:58,830 | JDBCException:<init> | Cannot open
>>>>connection
>>>>org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool
>>>>exhaus
>>>>ted
>>>>
>>>>Matt
>>>>
>>>>On 7/24/05, Keith Bjorndahl <ke...@kgbinternet.com> wrote:
>>>>
>>>>
>>>>
>>>>
>>>>>I've put the files in the 'restore' sub-directory of your home
>>>>>directory.
>>>>>I've put the Jul1 backup in one diretory, and the Jul17 and Jul22
>>>>>incrementals in their own directories under 'restore'. The Jul17 is an
>>>>>incremental on the Jul1, and the Jul22 is an incremental on Jul17, so
>>>>>you
>>>>>can see what changed, if anything, up until the 22nd.
>>>>>
>>>>>Keith
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>I've tried a number of different things and I can't seem to get Tomcat
>>>>>>started and serving pages no matter what I do. One strange thing I
>>>>>>noticed is I can't delete $CATALINA_HOME/work/Catalina. I've noticed
>>>>>>similar problems to what I'm experiencing - and cleaning out the work
>>>>>>directory often solves them. Can you use your root privileges to
>>>>>>clear out the contents of this directory. If doing that doesn't fix
>>>>>>anything - we should probably restore from backup. Friday afternoon
>>>>>>should work - although it would suck if we had to do this b/c there's
>>>>>>a few folks that might've updated the wiki since then. Can you save
>>>>>>~/public_html/wiki so I can diff it against Friday's copy?
>>>>>>
>>>>>>Thanks,
>>>>>>
>>>>>>Matt
>>>>>>
>>>>>>On 7/24/05, Keith Bjorndahl <ke...@kgbinternet.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>Oh....let me know if you want to restore any backups or anything.
>>>>>>>
>>>>>>>Keith
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>It appears that my site is really screwed up - I get 404 for pages
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>on
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>>>my blog - as if something is hosed on the installation of Roller.
>>>>>>>>This makes me suspect that someone might've hacked my site. Any
>>>>>>>>ideas?
>>>>>>>>
>>>>>>>>Thanks,
>>>>>>>>
>>>>>>>>Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
Re: Roller hosed up on raibledesigns.com
Posted by Matt Raible <mr...@gmail.com>.
It works!
I don't know if it was a problem with Roller or with JSPWiki - but the
good news is its fixed now. The database running out of connections
on startup doesn't seem to be happening today - maybe the rollover to
a new day did something.
Here's what I did to fix the problem:
JSPWiki: I started w/ a "wiki pages" directory with no files in it -
and after ensuring
everything was running correctly, I added the files back in. The only
thing that's really changed b/w yesterday and today was I upgraded the
wiki software - but it doesn't even make a database connection. I
also switched to use JDK 5 and this seems to be working fine.
Now that everything is working - I don't think I'll tinker with it for
awhile - possibly later tonight. I did turn off the search indexing
on Roller. I'll try turning that back on later this evening.
Matt
On 7/25/05, Dave Johnson <da...@rollerweblogger.org> wrote:
> There is no way to disable "all startup processing", but if you turn on
> hibernate SQL logging we should be able to see what it is doing with
> those first connections. - Dave
>
>
>
> On Jul 25, 2005, at 1:45 AM, Matt Raible wrote:
>
> > I upgraded to 1.2 and tried turning off searching, but still no luck.
> > Not only does the connection pool get exhausted shortly after starting
> > up, but it also pegs the CPU for my user.
> >
> > Of course, everything works fine when I run it locally.
> >
> > I've tried upgrading to JDK 5 and Tomcat 5.5.9 on my server (and only
> > running Roller), but that didn't work either.
> >
> > We've also tried moving my account and home directory (w/ everything
> > in it) to a different server, but the same CPU and database
> > connections happens there too. Is there anyway to turn off all
> > startup processing (and database connecting) that Roller does?
> >
> > Thanks,
> >
> > Matt
> >
> > On 7/24/05, Dave Johnson <da...@rollerweblogger.org> wrote:
> >>
> >> Ouch!
> >>
> >> As a quick test to verify if search indexing is the problem, you could
> >> try turning off search entirely. This only works on 1.2, by the way.
> >> Change search.enabled to 'false' in roller.properties like so:
> >>
> >> search.enabled=false
> >>
> >>
> >> - Dave
> >>
> >>
> >> On Jul 24, 2005, at 8:08 PM, Matt Raible wrote:
> >>
> >>> Did you turn off comment indexing in the code - or in a properties
> >>> file? I'm guessing it's in the code?
> >>>
> >>> Thanks,
> >>>
> >>> Matt
> >>>
> >>> On 7/24/05, Matthew P. Schmidt <ma...@javalobby.org> wrote:
> >>>> On JRoller, we had to turn off the comment indexing because it took
> >>>> too
> >>>> long to fetch all of those, you may want to take that part out of
> >>>> the
> >>>> build document process and see what happens. In addition, we're
> >>>> seeing
> >>>> Resin have to close a lot of dangling connections. I'm not
> >>>> completely
> >>>> sure what they are yet, but it most appear to happen in the
> >>>> NavigationBarTag. For some reason we see these errors when the
> >>>> PersistenceFilter isn't in teh chain as well. I'm not completely
> >>>> sure
> >>>> how in the world that would happen since its mapped on /* with all
> >>>> the
> >>>> proper FORWARD and REQUEST calls. Any ideas?
> >>>>
> >>>>
> >>>> -Matt
> >>>>
> >>>> Matt Raible wrote:
> >>>>
> >>>>> Guys,
> >>>>>
> >>>>> Does anyone know what might be causing the following problem:
> >>>>>
> >>>>> <quote from="emailBelow">
> >>>>> On starting your JVM, mysql's 'show processlist' shows you quickly
> >>>>> get 6
> >>>>> connections, and then when I tried to surf to raibledesigns.com it
> >>>>> quickly
> >>>>> grew to about 15 connections, and then slowly crawled up to 21, and
> >>>>> then
> >>>>> the catalina.out spewed out it's errors and the connections dropped
> >>>>> back
> >>>>> to 10. It then started climbing up, sitting at 15-16 connections
> >>>>> for a
> >>>>> while, then climbed back to 21, then catalina.out spewed out a
> >>>>> bunch
> >>>>> more
> >>>>> messages, and the connections dropped back to 8....
> >>>>> </quote>
> >>>>>
> >>>>> My site seems to start fine if I remove Roller, otherwise, it hangs
> >>>>> and the above scenario happens. My suspicion is possibly the
> >>>>> search
> >>>>> index getting created, but that doesn't explain why so many
> >>>>> connections are getting created.
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Matt
> >>>>>
> >>>>>
> >>>>> ---------- Forwarded message ----------
> >>>>> From: Keith Bjorndahl <ke...@kgbinternet.com>
> >>>>> Date: Jul 24, 2005 5:20 PM
> >>>>> Subject: Re: Site is really screwed up
> >>>>> To: matt@raibledesigns.com
> >>>>>
> >>>>>
> >>>>> Hi Matt
> >>>>>
> >>>>>
> >>>>>
> >>>>>> So you haven't restored anything back to the way my home directory
> >>>>>> was
> >>>>>> on those specific dates?
> >>>>>>
> >>>>>>
> >>>>>
> >>>>> No, I thought I'd let you decide which files you want to keep and
> >>>>> which to
> >>>>> replace. You can drop the whole Tomcat directory in if you want to
> >>>>> try
> >>>>> that though.
> >>>>>
> >>>>>
> >>>>>
> >>>>>> This still seems to be a database
> >>>>>> connectivity issue. Have you upgraded anything (hardware or
> >>>>>> software)
> >>>>>> in the last couple of days?
> >>>>>>
> >>>>>>
> >>>>>
> >>>>> Not even in the last couple of weeks...a ssh upgrade in June, and a
> >>>>> kernel
> >>>>> upgrade on June 1.
> >>>>>
> >>>>> MySQL is set at
> >>>>> set-variable = max_connections=500
> >>>>> set-variable = max_user_connections=50
> >>>>>
> >>>>> as it has been for a year or so, and the max_connections reached
> >>>>> since the
> >>>>> last mysql restart (when you first reported the problem) is 89.
> >>>>>
> >>>>> On starting your JVM, mysql's 'show processlist' shows you quickly
> >>>>> get 6
> >>>>> connections, and then when I tried to surf to raibledesigns.com it
> >>>>> quickly
> >>>>> grew to about 15 connections, and then slowly crawled up to 21, and
> >>>>> then
> >>>>> the catalina.out spewed out it's errors and the connections dropped
> >>>>> back
> >>>>> to 10. It then started climbing up, sitting at 15-16 connections
> >>>>> for a
> >>>>> while, then climbed back to 21, then catalina.out spewed out a
> >>>>> bunch
> >>>>> more
> >>>>> messages, and the connections dropped back to 8....
> >>>>>
> >>>>> It's like it keeps trying to get connections until it hit's your
> >>>>> own
> >>>>> pool
> >>>>> limit, and then drops a bunch and starts over again...any
> >>>>> connection
> >>>>> leaks
> >>>>> anywhere maybe?
> >>>>>
> >>>>> Keith
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> ERROR 2005-07-24 16:10:58,828 |
> >>>>>> JDBCExceptionReporter:logExceptions
> >>>>>> |
> >>>>>> Cannot get
> >>>>>> a connection, pool exhausted
> >>>>>> ERROR 2005-07-24 16:10:58,830 | JDBCException:<init> | Cannot open
> >>>>>> connection
> >>>>>> org.apache.commons.dbcp.SQLNestedException: Cannot get a
> >>>>>> connection, pool
> >>>>>> exhaus
> >>>>>> ted
> >>>>>>
> >>>>>> Matt
> >>>>>>
> >>>>>> On 7/24/05, Keith Bjorndahl <ke...@kgbinternet.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>> I've put the files in the 'restore' sub-directory of your home
> >>>>>>> directory.
> >>>>>>> I've put the Jul1 backup in one diretory, and the Jul17 and Jul22
> >>>>>>> incrementals in their own directories under 'restore'. The Jul17
> >>>>>>> is an
> >>>>>>> incremental on the Jul1, and the Jul22 is an incremental on
> >>>>>>> Jul17,
> >>>>>>> so
> >>>>>>> you
> >>>>>>> can see what changed, if anything, up until the 22nd.
> >>>>>>>
> >>>>>>> Keith
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> I've tried a number of different things and I can't seem to get
> >>>>>>>> Tomcat
> >>>>>>>> started and serving pages no matter what I do. One strange
> >>>>>>>> thing
> >>>>>>>> I
> >>>>>>>> noticed is I can't delete $CATALINA_HOME/work/Catalina. I've
> >>>>>>>> noticed
> >>>>>>>> similar problems to what I'm experiencing - and cleaning out the
> >>>>>>>> work
> >>>>>>>> directory often solves them. Can you use your root privileges
> >>>>>>>> to
> >>>>>>>> clear out the contents of this directory. If doing that doesn't
> >>>>>>>> fix
> >>>>>>>> anything - we should probably restore from backup. Friday
> >>>>>>>> afternoon
> >>>>>>>> should work - although it would suck if we had to do this b/c
> >>>>>>>> there's
> >>>>>>>> a few folks that might've updated the wiki since then. Can you
> >>>>>>>> save
> >>>>>>>> ~/public_html/wiki so I can diff it against Friday's copy?
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>> Matt
> >>>>>>>>
> >>>>>>>> On 7/24/05, Keith Bjorndahl <ke...@kgbinternet.com> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Oh....let me know if you want to restore any backups or
> >>>>>>>>> anything.
> >>>>>>>>>
> >>>>>>>>> Keith
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> It appears that my site is really screwed up - I get 404 for
> >>>>>>>>>> pages
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>> on
> >>>>>>>
> >>>>>>>
> >>>>>>>>>> my blog - as if something is hosed on the installation of
> >>>>>>>>>> Roller.
> >>>>>>>>>> This makes me suspect that someone might've hacked my site.
> >>>>>>>>>> Any
> >>>>>>>>>> ideas?
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>>
> >>>>>>>>>> Matt
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> >>
> >
>
>
Re: Roller hosed up on raibledesigns.com
Posted by Dave Johnson <da...@rollerweblogger.org>.
There is no way to disable "all startup processing", but if you turn on
hibernate SQL logging we should be able to see what it is doing with
those first connections. - Dave
On Jul 25, 2005, at 1:45 AM, Matt Raible wrote:
> I upgraded to 1.2 and tried turning off searching, but still no luck.
> Not only does the connection pool get exhausted shortly after starting
> up, but it also pegs the CPU for my user.
>
> Of course, everything works fine when I run it locally.
>
> I've tried upgrading to JDK 5 and Tomcat 5.5.9 on my server (and only
> running Roller), but that didn't work either.
>
> We've also tried moving my account and home directory (w/ everything
> in it) to a different server, but the same CPU and database
> connections happens there too. Is there anyway to turn off all
> startup processing (and database connecting) that Roller does?
>
> Thanks,
>
> Matt
>
> On 7/24/05, Dave Johnson <da...@rollerweblogger.org> wrote:
>>
>> Ouch!
>>
>> As a quick test to verify if search indexing is the problem, you could
>> try turning off search entirely. This only works on 1.2, by the way.
>> Change search.enabled to 'false' in roller.properties like so:
>>
>> search.enabled=false
>>
>>
>> - Dave
>>
>>
>> On Jul 24, 2005, at 8:08 PM, Matt Raible wrote:
>>
>>> Did you turn off comment indexing in the code - or in a properties
>>> file? I'm guessing it's in the code?
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>> On 7/24/05, Matthew P. Schmidt <ma...@javalobby.org> wrote:
>>>> On JRoller, we had to turn off the comment indexing because it took
>>>> too
>>>> long to fetch all of those, you may want to take that part out of
>>>> the
>>>> build document process and see what happens. In addition, we're
>>>> seeing
>>>> Resin have to close a lot of dangling connections. I'm not
>>>> completely
>>>> sure what they are yet, but it most appear to happen in the
>>>> NavigationBarTag. For some reason we see these errors when the
>>>> PersistenceFilter isn't in teh chain as well. I'm not completely
>>>> sure
>>>> how in the world that would happen since its mapped on /* with all
>>>> the
>>>> proper FORWARD and REQUEST calls. Any ideas?
>>>>
>>>>
>>>> -Matt
>>>>
>>>> Matt Raible wrote:
>>>>
>>>>> Guys,
>>>>>
>>>>> Does anyone know what might be causing the following problem:
>>>>>
>>>>> <quote from="emailBelow">
>>>>> On starting your JVM, mysql's 'show processlist' shows you quickly
>>>>> get 6
>>>>> connections, and then when I tried to surf to raibledesigns.com it
>>>>> quickly
>>>>> grew to about 15 connections, and then slowly crawled up to 21, and
>>>>> then
>>>>> the catalina.out spewed out it's errors and the connections dropped
>>>>> back
>>>>> to 10. It then started climbing up, sitting at 15-16 connections
>>>>> for a
>>>>> while, then climbed back to 21, then catalina.out spewed out a
>>>>> bunch
>>>>> more
>>>>> messages, and the connections dropped back to 8....
>>>>> </quote>
>>>>>
>>>>> My site seems to start fine if I remove Roller, otherwise, it hangs
>>>>> and the above scenario happens. My suspicion is possibly the
>>>>> search
>>>>> index getting created, but that doesn't explain why so many
>>>>> connections are getting created.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Matt
>>>>>
>>>>>
>>>>> ---------- Forwarded message ----------
>>>>> From: Keith Bjorndahl <ke...@kgbinternet.com>
>>>>> Date: Jul 24, 2005 5:20 PM
>>>>> Subject: Re: Site is really screwed up
>>>>> To: matt@raibledesigns.com
>>>>>
>>>>>
>>>>> Hi Matt
>>>>>
>>>>>
>>>>>
>>>>>> So you haven't restored anything back to the way my home directory
>>>>>> was
>>>>>> on those specific dates?
>>>>>>
>>>>>>
>>>>>
>>>>> No, I thought I'd let you decide which files you want to keep and
>>>>> which to
>>>>> replace. You can drop the whole Tomcat directory in if you want to
>>>>> try
>>>>> that though.
>>>>>
>>>>>
>>>>>
>>>>>> This still seems to be a database
>>>>>> connectivity issue. Have you upgraded anything (hardware or
>>>>>> software)
>>>>>> in the last couple of days?
>>>>>>
>>>>>>
>>>>>
>>>>> Not even in the last couple of weeks...a ssh upgrade in June, and a
>>>>> kernel
>>>>> upgrade on June 1.
>>>>>
>>>>> MySQL is set at
>>>>> set-variable = max_connections=500
>>>>> set-variable = max_user_connections=50
>>>>>
>>>>> as it has been for a year or so, and the max_connections reached
>>>>> since the
>>>>> last mysql restart (when you first reported the problem) is 89.
>>>>>
>>>>> On starting your JVM, mysql's 'show processlist' shows you quickly
>>>>> get 6
>>>>> connections, and then when I tried to surf to raibledesigns.com it
>>>>> quickly
>>>>> grew to about 15 connections, and then slowly crawled up to 21, and
>>>>> then
>>>>> the catalina.out spewed out it's errors and the connections dropped
>>>>> back
>>>>> to 10. It then started climbing up, sitting at 15-16 connections
>>>>> for a
>>>>> while, then climbed back to 21, then catalina.out spewed out a
>>>>> bunch
>>>>> more
>>>>> messages, and the connections dropped back to 8....
>>>>>
>>>>> It's like it keeps trying to get connections until it hit's your
>>>>> own
>>>>> pool
>>>>> limit, and then drops a bunch and starts over again...any
>>>>> connection
>>>>> leaks
>>>>> anywhere maybe?
>>>>>
>>>>> Keith
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> ERROR 2005-07-24 16:10:58,828 |
>>>>>> JDBCExceptionReporter:logExceptions
>>>>>> |
>>>>>> Cannot get
>>>>>> a connection, pool exhausted
>>>>>> ERROR 2005-07-24 16:10:58,830 | JDBCException:<init> | Cannot open
>>>>>> connection
>>>>>> org.apache.commons.dbcp.SQLNestedException: Cannot get a
>>>>>> connection, pool
>>>>>> exhaus
>>>>>> ted
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>> On 7/24/05, Keith Bjorndahl <ke...@kgbinternet.com> wrote:
>>>>>>
>>>>>>
>>>>>>> I've put the files in the 'restore' sub-directory of your home
>>>>>>> directory.
>>>>>>> I've put the Jul1 backup in one diretory, and the Jul17 and Jul22
>>>>>>> incrementals in their own directories under 'restore'. The Jul17
>>>>>>> is an
>>>>>>> incremental on the Jul1, and the Jul22 is an incremental on
>>>>>>> Jul17,
>>>>>>> so
>>>>>>> you
>>>>>>> can see what changed, if anything, up until the 22nd.
>>>>>>>
>>>>>>> Keith
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> I've tried a number of different things and I can't seem to get
>>>>>>>> Tomcat
>>>>>>>> started and serving pages no matter what I do. One strange
>>>>>>>> thing
>>>>>>>> I
>>>>>>>> noticed is I can't delete $CATALINA_HOME/work/Catalina. I've
>>>>>>>> noticed
>>>>>>>> similar problems to what I'm experiencing - and cleaning out the
>>>>>>>> work
>>>>>>>> directory often solves them. Can you use your root privileges
>>>>>>>> to
>>>>>>>> clear out the contents of this directory. If doing that doesn't
>>>>>>>> fix
>>>>>>>> anything - we should probably restore from backup. Friday
>>>>>>>> afternoon
>>>>>>>> should work - although it would suck if we had to do this b/c
>>>>>>>> there's
>>>>>>>> a few folks that might've updated the wiki since then. Can you
>>>>>>>> save
>>>>>>>> ~/public_html/wiki so I can diff it against Friday's copy?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Matt
>>>>>>>>
>>>>>>>> On 7/24/05, Keith Bjorndahl <ke...@kgbinternet.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> Oh....let me know if you want to restore any backups or
>>>>>>>>> anything.
>>>>>>>>>
>>>>>>>>> Keith
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> It appears that my site is really screwed up - I get 404 for
>>>>>>>>>> pages
>>>>>>>>>>
>>>>>>>>>>
>>>>>>> on
>>>>>>>
>>>>>>>
>>>>>>>>>> my blog - as if something is hosed on the installation of
>>>>>>>>>> Roller.
>>>>>>>>>> This makes me suspect that someone might've hacked my site.
>>>>>>>>>> Any
>>>>>>>>>> ideas?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Matt
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>>
>>
>>
>
Re: Roller hosed up on raibledesigns.com
Posted by Matt Raible <mr...@gmail.com>.
I upgraded to 1.2 and tried turning off searching, but still no luck.
Not only does the connection pool get exhausted shortly after starting
up, but it also pegs the CPU for my user.
Of course, everything works fine when I run it locally.
I've tried upgrading to JDK 5 and Tomcat 5.5.9 on my server (and only
running Roller), but that didn't work either.
We've also tried moving my account and home directory (w/ everything
in it) to a different server, but the same CPU and database
connections happens there too. Is there anyway to turn off all
startup processing (and database connecting) that Roller does?
Thanks,
Matt
On 7/24/05, Dave Johnson <da...@rollerweblogger.org> wrote:
>
> Ouch!
>
> As a quick test to verify if search indexing is the problem, you could
> try turning off search entirely. This only works on 1.2, by the way.
> Change search.enabled to 'false' in roller.properties like so:
>
> search.enabled=false
>
>
> - Dave
>
>
> On Jul 24, 2005, at 8:08 PM, Matt Raible wrote:
>
> > Did you turn off comment indexing in the code - or in a properties
> > file? I'm guessing it's in the code?
> >
> > Thanks,
> >
> > Matt
> >
> > On 7/24/05, Matthew P. Schmidt <ma...@javalobby.org> wrote:
> >> On JRoller, we had to turn off the comment indexing because it took
> >> too
> >> long to fetch all of those, you may want to take that part out of the
> >> build document process and see what happens. In addition, we're
> >> seeing
> >> Resin have to close a lot of dangling connections. I'm not completely
> >> sure what they are yet, but it most appear to happen in the
> >> NavigationBarTag. For some reason we see these errors when the
> >> PersistenceFilter isn't in teh chain as well. I'm not completely sure
> >> how in the world that would happen since its mapped on /* with all the
> >> proper FORWARD and REQUEST calls. Any ideas?
> >>
> >>
> >> -Matt
> >>
> >> Matt Raible wrote:
> >>
> >>> Guys,
> >>>
> >>> Does anyone know what might be causing the following problem:
> >>>
> >>> <quote from="emailBelow">
> >>> On starting your JVM, mysql's 'show processlist' shows you quickly
> >>> get 6
> >>> connections, and then when I tried to surf to raibledesigns.com it
> >>> quickly
> >>> grew to about 15 connections, and then slowly crawled up to 21, and
> >>> then
> >>> the catalina.out spewed out it's errors and the connections dropped
> >>> back
> >>> to 10. It then started climbing up, sitting at 15-16 connections
> >>> for a
> >>> while, then climbed back to 21, then catalina.out spewed out a bunch
> >>> more
> >>> messages, and the connections dropped back to 8....
> >>> </quote>
> >>>
> >>> My site seems to start fine if I remove Roller, otherwise, it hangs
> >>> and the above scenario happens. My suspicion is possibly the search
> >>> index getting created, but that doesn't explain why so many
> >>> connections are getting created.
> >>>
> >>> Thanks,
> >>>
> >>> Matt
> >>>
> >>>
> >>> ---------- Forwarded message ----------
> >>> From: Keith Bjorndahl <ke...@kgbinternet.com>
> >>> Date: Jul 24, 2005 5:20 PM
> >>> Subject: Re: Site is really screwed up
> >>> To: matt@raibledesigns.com
> >>>
> >>>
> >>> Hi Matt
> >>>
> >>>
> >>>
> >>>> So you haven't restored anything back to the way my home directory
> >>>> was
> >>>> on those specific dates?
> >>>>
> >>>>
> >>>
> >>> No, I thought I'd let you decide which files you want to keep and
> >>> which to
> >>> replace. You can drop the whole Tomcat directory in if you want to
> >>> try
> >>> that though.
> >>>
> >>>
> >>>
> >>>> This still seems to be a database
> >>>> connectivity issue. Have you upgraded anything (hardware or
> >>>> software)
> >>>> in the last couple of days?
> >>>>
> >>>>
> >>>
> >>> Not even in the last couple of weeks...a ssh upgrade in June, and a
> >>> kernel
> >>> upgrade on June 1.
> >>>
> >>> MySQL is set at
> >>> set-variable = max_connections=500
> >>> set-variable = max_user_connections=50
> >>>
> >>> as it has been for a year or so, and the max_connections reached
> >>> since the
> >>> last mysql restart (when you first reported the problem) is 89.
> >>>
> >>> On starting your JVM, mysql's 'show processlist' shows you quickly
> >>> get 6
> >>> connections, and then when I tried to surf to raibledesigns.com it
> >>> quickly
> >>> grew to about 15 connections, and then slowly crawled up to 21, and
> >>> then
> >>> the catalina.out spewed out it's errors and the connections dropped
> >>> back
> >>> to 10. It then started climbing up, sitting at 15-16 connections
> >>> for a
> >>> while, then climbed back to 21, then catalina.out spewed out a bunch
> >>> more
> >>> messages, and the connections dropped back to 8....
> >>>
> >>> It's like it keeps trying to get connections until it hit's your own
> >>> pool
> >>> limit, and then drops a bunch and starts over again...any connection
> >>> leaks
> >>> anywhere maybe?
> >>>
> >>> Keith
> >>>
> >>>
> >>>
> >>>
> >>>> ERROR 2005-07-24 16:10:58,828 | JDBCExceptionReporter:logExceptions
> >>>> |
> >>>> Cannot get
> >>>> a connection, pool exhausted
> >>>> ERROR 2005-07-24 16:10:58,830 | JDBCException:<init> | Cannot open
> >>>> connection
> >>>> org.apache.commons.dbcp.SQLNestedException: Cannot get a
> >>>> connection, pool
> >>>> exhaus
> >>>> ted
> >>>>
> >>>> Matt
> >>>>
> >>>> On 7/24/05, Keith Bjorndahl <ke...@kgbinternet.com> wrote:
> >>>>
> >>>>
> >>>>> I've put the files in the 'restore' sub-directory of your home
> >>>>> directory.
> >>>>> I've put the Jul1 backup in one diretory, and the Jul17 and Jul22
> >>>>> incrementals in their own directories under 'restore'. The Jul17
> >>>>> is an
> >>>>> incremental on the Jul1, and the Jul22 is an incremental on Jul17,
> >>>>> so
> >>>>> you
> >>>>> can see what changed, if anything, up until the 22nd.
> >>>>>
> >>>>> Keith
> >>>>>
> >>>>>
> >>>>>
> >>>>>> I've tried a number of different things and I can't seem to get
> >>>>>> Tomcat
> >>>>>> started and serving pages no matter what I do. One strange thing
> >>>>>> I
> >>>>>> noticed is I can't delete $CATALINA_HOME/work/Catalina. I've
> >>>>>> noticed
> >>>>>> similar problems to what I'm experiencing - and cleaning out the
> >>>>>> work
> >>>>>> directory often solves them. Can you use your root privileges to
> >>>>>> clear out the contents of this directory. If doing that doesn't
> >>>>>> fix
> >>>>>> anything - we should probably restore from backup. Friday
> >>>>>> afternoon
> >>>>>> should work - although it would suck if we had to do this b/c
> >>>>>> there's
> >>>>>> a few folks that might've updated the wiki since then. Can you
> >>>>>> save
> >>>>>> ~/public_html/wiki so I can diff it against Friday's copy?
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Matt
> >>>>>>
> >>>>>> On 7/24/05, Keith Bjorndahl <ke...@kgbinternet.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>> Oh....let me know if you want to restore any backups or anything.
> >>>>>>>
> >>>>>>> Keith
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> It appears that my site is really screwed up - I get 404 for
> >>>>>>>> pages
> >>>>>>>>
> >>>>>>>>
> >>>>> on
> >>>>>
> >>>>>
> >>>>>>>> my blog - as if something is hosed on the installation of
> >>>>>>>> Roller.
> >>>>>>>> This makes me suspect that someone might've hacked my site. Any
> >>>>>>>> ideas?
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>> Matt
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>
> >
>
>
Re: Roller hosed up on raibledesigns.com
Posted by Dave Johnson <da...@rollerweblogger.org>.
Ouch!
As a quick test to verify if search indexing is the problem, you could
try turning off search entirely. This only works on 1.2, by the way.
Change search.enabled to 'false' in roller.properties like so:
search.enabled=false
- Dave
On Jul 24, 2005, at 8:08 PM, Matt Raible wrote:
> Did you turn off comment indexing in the code - or in a properties
> file? I'm guessing it's in the code?
>
> Thanks,
>
> Matt
>
> On 7/24/05, Matthew P. Schmidt <ma...@javalobby.org> wrote:
>> On JRoller, we had to turn off the comment indexing because it took
>> too
>> long to fetch all of those, you may want to take that part out of the
>> build document process and see what happens. In addition, we're
>> seeing
>> Resin have to close a lot of dangling connections. I'm not completely
>> sure what they are yet, but it most appear to happen in the
>> NavigationBarTag. For some reason we see these errors when the
>> PersistenceFilter isn't in teh chain as well. I'm not completely sure
>> how in the world that would happen since its mapped on /* with all the
>> proper FORWARD and REQUEST calls. Any ideas?
>>
>>
>> -Matt
>>
>> Matt Raible wrote:
>>
>>> Guys,
>>>
>>> Does anyone know what might be causing the following problem:
>>>
>>> <quote from="emailBelow">
>>> On starting your JVM, mysql's 'show processlist' shows you quickly
>>> get 6
>>> connections, and then when I tried to surf to raibledesigns.com it
>>> quickly
>>> grew to about 15 connections, and then slowly crawled up to 21, and
>>> then
>>> the catalina.out spewed out it's errors and the connections dropped
>>> back
>>> to 10. It then started climbing up, sitting at 15-16 connections
>>> for a
>>> while, then climbed back to 21, then catalina.out spewed out a bunch
>>> more
>>> messages, and the connections dropped back to 8....
>>> </quote>
>>>
>>> My site seems to start fine if I remove Roller, otherwise, it hangs
>>> and the above scenario happens. My suspicion is possibly the search
>>> index getting created, but that doesn't explain why so many
>>> connections are getting created.
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Keith Bjorndahl <ke...@kgbinternet.com>
>>> Date: Jul 24, 2005 5:20 PM
>>> Subject: Re: Site is really screwed up
>>> To: matt@raibledesigns.com
>>>
>>>
>>> Hi Matt
>>>
>>>
>>>
>>>> So you haven't restored anything back to the way my home directory
>>>> was
>>>> on those specific dates?
>>>>
>>>>
>>>
>>> No, I thought I'd let you decide which files you want to keep and
>>> which to
>>> replace. You can drop the whole Tomcat directory in if you want to
>>> try
>>> that though.
>>>
>>>
>>>
>>>> This still seems to be a database
>>>> connectivity issue. Have you upgraded anything (hardware or
>>>> software)
>>>> in the last couple of days?
>>>>
>>>>
>>>
>>> Not even in the last couple of weeks...a ssh upgrade in June, and a
>>> kernel
>>> upgrade on June 1.
>>>
>>> MySQL is set at
>>> set-variable = max_connections=500
>>> set-variable = max_user_connections=50
>>>
>>> as it has been for a year or so, and the max_connections reached
>>> since the
>>> last mysql restart (when you first reported the problem) is 89.
>>>
>>> On starting your JVM, mysql's 'show processlist' shows you quickly
>>> get 6
>>> connections, and then when I tried to surf to raibledesigns.com it
>>> quickly
>>> grew to about 15 connections, and then slowly crawled up to 21, and
>>> then
>>> the catalina.out spewed out it's errors and the connections dropped
>>> back
>>> to 10. It then started climbing up, sitting at 15-16 connections
>>> for a
>>> while, then climbed back to 21, then catalina.out spewed out a bunch
>>> more
>>> messages, and the connections dropped back to 8....
>>>
>>> It's like it keeps trying to get connections until it hit's your own
>>> pool
>>> limit, and then drops a bunch and starts over again...any connection
>>> leaks
>>> anywhere maybe?
>>>
>>> Keith
>>>
>>>
>>>
>>>
>>>> ERROR 2005-07-24 16:10:58,828 | JDBCExceptionReporter:logExceptions
>>>> |
>>>> Cannot get
>>>> a connection, pool exhausted
>>>> ERROR 2005-07-24 16:10:58,830 | JDBCException:<init> | Cannot open
>>>> connection
>>>> org.apache.commons.dbcp.SQLNestedException: Cannot get a
>>>> connection, pool
>>>> exhaus
>>>> ted
>>>>
>>>> Matt
>>>>
>>>> On 7/24/05, Keith Bjorndahl <ke...@kgbinternet.com> wrote:
>>>>
>>>>
>>>>> I've put the files in the 'restore' sub-directory of your home
>>>>> directory.
>>>>> I've put the Jul1 backup in one diretory, and the Jul17 and Jul22
>>>>> incrementals in their own directories under 'restore'. The Jul17
>>>>> is an
>>>>> incremental on the Jul1, and the Jul22 is an incremental on Jul17,
>>>>> so
>>>>> you
>>>>> can see what changed, if anything, up until the 22nd.
>>>>>
>>>>> Keith
>>>>>
>>>>>
>>>>>
>>>>>> I've tried a number of different things and I can't seem to get
>>>>>> Tomcat
>>>>>> started and serving pages no matter what I do. One strange thing
>>>>>> I
>>>>>> noticed is I can't delete $CATALINA_HOME/work/Catalina. I've
>>>>>> noticed
>>>>>> similar problems to what I'm experiencing - and cleaning out the
>>>>>> work
>>>>>> directory often solves them. Can you use your root privileges to
>>>>>> clear out the contents of this directory. If doing that doesn't
>>>>>> fix
>>>>>> anything - we should probably restore from backup. Friday
>>>>>> afternoon
>>>>>> should work - although it would suck if we had to do this b/c
>>>>>> there's
>>>>>> a few folks that might've updated the wiki since then. Can you
>>>>>> save
>>>>>> ~/public_html/wiki so I can diff it against Friday's copy?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>> On 7/24/05, Keith Bjorndahl <ke...@kgbinternet.com> wrote:
>>>>>>
>>>>>>
>>>>>>> Oh....let me know if you want to restore any backups or anything.
>>>>>>>
>>>>>>> Keith
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> It appears that my site is really screwed up - I get 404 for
>>>>>>>> pages
>>>>>>>>
>>>>>>>>
>>>>> on
>>>>>
>>>>>
>>>>>>>> my blog - as if something is hosed on the installation of
>>>>>>>> Roller.
>>>>>>>> This makes me suspect that someone might've hacked my site. Any
>>>>>>>> ideas?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>
>
Re: Roller hosed up on raibledesigns.com
Posted by Matt Raible <mr...@gmail.com>.
Did you turn off comment indexing in the code - or in a properties
file? I'm guessing it's in the code?
Thanks,
Matt
On 7/24/05, Matthew P. Schmidt <ma...@javalobby.org> wrote:
> On JRoller, we had to turn off the comment indexing because it took too
> long to fetch all of those, you may want to take that part out of the
> build document process and see what happens. In addition, we're seeing
> Resin have to close a lot of dangling connections. I'm not completely
> sure what they are yet, but it most appear to happen in the
> NavigationBarTag. For some reason we see these errors when the
> PersistenceFilter isn't in teh chain as well. I'm not completely sure
> how in the world that would happen since its mapped on /* with all the
> proper FORWARD and REQUEST calls. Any ideas?
>
>
> -Matt
>
> Matt Raible wrote:
>
> >Guys,
> >
> >Does anyone know what might be causing the following problem:
> >
> ><quote from="emailBelow">
> >On starting your JVM, mysql's 'show processlist' shows you quickly get 6
> >connections, and then when I tried to surf to raibledesigns.com it quickly
> >grew to about 15 connections, and then slowly crawled up to 21, and then
> >the catalina.out spewed out it's errors and the connections dropped back
> >to 10. It then started climbing up, sitting at 15-16 connections for a
> >while, then climbed back to 21, then catalina.out spewed out a bunch more
> >messages, and the connections dropped back to 8....
> ></quote>
> >
> >My site seems to start fine if I remove Roller, otherwise, it hangs
> >and the above scenario happens. My suspicion is possibly the search
> >index getting created, but that doesn't explain why so many
> >connections are getting created.
> >
> >Thanks,
> >
> >Matt
> >
> >
> >---------- Forwarded message ----------
> >From: Keith Bjorndahl <ke...@kgbinternet.com>
> >Date: Jul 24, 2005 5:20 PM
> >Subject: Re: Site is really screwed up
> >To: matt@raibledesigns.com
> >
> >
> >Hi Matt
> >
> >
> >
> >>So you haven't restored anything back to the way my home directory was
> >>on those specific dates?
> >>
> >>
> >
> >No, I thought I'd let you decide which files you want to keep and which to
> >replace. You can drop the whole Tomcat directory in if you want to try
> >that though.
> >
> >
> >
> >>This still seems to be a database
> >>connectivity issue. Have you upgraded anything (hardware or software)
> >>in the last couple of days?
> >>
> >>
> >
> >Not even in the last couple of weeks...a ssh upgrade in June, and a kernel
> >upgrade on June 1.
> >
> >MySQL is set at
> >set-variable = max_connections=500
> >set-variable = max_user_connections=50
> >
> >as it has been for a year or so, and the max_connections reached since the
> >last mysql restart (when you first reported the problem) is 89.
> >
> >On starting your JVM, mysql's 'show processlist' shows you quickly get 6
> >connections, and then when I tried to surf to raibledesigns.com it quickly
> >grew to about 15 connections, and then slowly crawled up to 21, and then
> >the catalina.out spewed out it's errors and the connections dropped back
> >to 10. It then started climbing up, sitting at 15-16 connections for a
> >while, then climbed back to 21, then catalina.out spewed out a bunch more
> >messages, and the connections dropped back to 8....
> >
> >It's like it keeps trying to get connections until it hit's your own pool
> >limit, and then drops a bunch and starts over again...any connection leaks
> >anywhere maybe?
> >
> >Keith
> >
> >
> >
> >
> >>ERROR 2005-07-24 16:10:58,828 | JDBCExceptionReporter:logExceptions |
> >>Cannot get
> >> a connection, pool exhausted
> >>ERROR 2005-07-24 16:10:58,830 | JDBCException:<init> | Cannot open
> >>connection
> >>org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool
> >>exhaus
> >>ted
> >>
> >>Matt
> >>
> >>On 7/24/05, Keith Bjorndahl <ke...@kgbinternet.com> wrote:
> >>
> >>
> >>>I've put the files in the 'restore' sub-directory of your home
> >>>directory.
> >>>I've put the Jul1 backup in one diretory, and the Jul17 and Jul22
> >>>incrementals in their own directories under 'restore'. The Jul17 is an
> >>>incremental on the Jul1, and the Jul22 is an incremental on Jul17, so
> >>>you
> >>>can see what changed, if anything, up until the 22nd.
> >>>
> >>>Keith
> >>>
> >>>
> >>>
> >>>>I've tried a number of different things and I can't seem to get Tomcat
> >>>>started and serving pages no matter what I do. One strange thing I
> >>>>noticed is I can't delete $CATALINA_HOME/work/Catalina. I've noticed
> >>>>similar problems to what I'm experiencing - and cleaning out the work
> >>>>directory often solves them. Can you use your root privileges to
> >>>>clear out the contents of this directory. If doing that doesn't fix
> >>>>anything - we should probably restore from backup. Friday afternoon
> >>>>should work - although it would suck if we had to do this b/c there's
> >>>>a few folks that might've updated the wiki since then. Can you save
> >>>>~/public_html/wiki so I can diff it against Friday's copy?
> >>>>
> >>>>Thanks,
> >>>>
> >>>>Matt
> >>>>
> >>>>On 7/24/05, Keith Bjorndahl <ke...@kgbinternet.com> wrote:
> >>>>
> >>>>
> >>>>>Oh....let me know if you want to restore any backups or anything.
> >>>>>
> >>>>>Keith
> >>>>>
> >>>>>
> >>>>>
> >>>>>>It appears that my site is really screwed up - I get 404 for pages
> >>>>>>
> >>>>>>
> >>>on
> >>>
> >>>
> >>>>>>my blog - as if something is hosed on the installation of Roller.
> >>>>>>This makes me suspect that someone might've hacked my site. Any
> >>>>>>ideas?
> >>>>>>
> >>>>>>Thanks,
> >>>>>>
> >>>>>>Matt
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>>
>
Re: Roller hosed up on raibledesigns.com
Posted by "Matthew P. Schmidt" <ma...@javalobby.org>.
On JRoller, we had to turn off the comment indexing because it took too
long to fetch all of those, you may want to take that part out of the
build document process and see what happens. In addition, we're seeing
Resin have to close a lot of dangling connections. I'm not completely
sure what they are yet, but it most appear to happen in the
NavigationBarTag. For some reason we see these errors when the
PersistenceFilter isn't in teh chain as well. I'm not completely sure
how in the world that would happen since its mapped on /* with all the
proper FORWARD and REQUEST calls. Any ideas?
-Matt
Matt Raible wrote:
>Guys,
>
>Does anyone know what might be causing the following problem:
>
><quote from="emailBelow">
>On starting your JVM, mysql's 'show processlist' shows you quickly get 6
>connections, and then when I tried to surf to raibledesigns.com it quickly
>grew to about 15 connections, and then slowly crawled up to 21, and then
>the catalina.out spewed out it's errors and the connections dropped back
>to 10. It then started climbing up, sitting at 15-16 connections for a
>while, then climbed back to 21, then catalina.out spewed out a bunch more
>messages, and the connections dropped back to 8....
></quote>
>
>My site seems to start fine if I remove Roller, otherwise, it hangs
>and the above scenario happens. My suspicion is possibly the search
>index getting created, but that doesn't explain why so many
>connections are getting created.
>
>Thanks,
>
>Matt
>
>
>---------- Forwarded message ----------
>From: Keith Bjorndahl <ke...@kgbinternet.com>
>Date: Jul 24, 2005 5:20 PM
>Subject: Re: Site is really screwed up
>To: matt@raibledesigns.com
>
>
>Hi Matt
>
>
>
>>So you haven't restored anything back to the way my home directory was
>>on those specific dates?
>>
>>
>
>No, I thought I'd let you decide which files you want to keep and which to
>replace. You can drop the whole Tomcat directory in if you want to try
>that though.
>
>
>
>>This still seems to be a database
>>connectivity issue. Have you upgraded anything (hardware or software)
>>in the last couple of days?
>>
>>
>
>Not even in the last couple of weeks...a ssh upgrade in June, and a kernel
>upgrade on June 1.
>
>MySQL is set at
>set-variable = max_connections=500
>set-variable = max_user_connections=50
>
>as it has been for a year or so, and the max_connections reached since the
>last mysql restart (when you first reported the problem) is 89.
>
>On starting your JVM, mysql's 'show processlist' shows you quickly get 6
>connections, and then when I tried to surf to raibledesigns.com it quickly
>grew to about 15 connections, and then slowly crawled up to 21, and then
>the catalina.out spewed out it's errors and the connections dropped back
>to 10. It then started climbing up, sitting at 15-16 connections for a
>while, then climbed back to 21, then catalina.out spewed out a bunch more
>messages, and the connections dropped back to 8....
>
>It's like it keeps trying to get connections until it hit's your own pool
>limit, and then drops a bunch and starts over again...any connection leaks
>anywhere maybe?
>
>Keith
>
>
>
>
>>ERROR 2005-07-24 16:10:58,828 | JDBCExceptionReporter:logExceptions |
>>Cannot get
>> a connection, pool exhausted
>>ERROR 2005-07-24 16:10:58,830 | JDBCException:<init> | Cannot open
>>connection
>>org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool
>>exhaus
>>ted
>>
>>Matt
>>
>>On 7/24/05, Keith Bjorndahl <ke...@kgbinternet.com> wrote:
>>
>>
>>>I've put the files in the 'restore' sub-directory of your home
>>>directory.
>>>I've put the Jul1 backup in one diretory, and the Jul17 and Jul22
>>>incrementals in their own directories under 'restore'. The Jul17 is an
>>>incremental on the Jul1, and the Jul22 is an incremental on Jul17, so
>>>you
>>>can see what changed, if anything, up until the 22nd.
>>>
>>>Keith
>>>
>>>
>>>
>>>>I've tried a number of different things and I can't seem to get Tomcat
>>>>started and serving pages no matter what I do. One strange thing I
>>>>noticed is I can't delete $CATALINA_HOME/work/Catalina. I've noticed
>>>>similar problems to what I'm experiencing - and cleaning out the work
>>>>directory often solves them. Can you use your root privileges to
>>>>clear out the contents of this directory. If doing that doesn't fix
>>>>anything - we should probably restore from backup. Friday afternoon
>>>>should work - although it would suck if we had to do this b/c there's
>>>>a few folks that might've updated the wiki since then. Can you save
>>>>~/public_html/wiki so I can diff it against Friday's copy?
>>>>
>>>>Thanks,
>>>>
>>>>Matt
>>>>
>>>>On 7/24/05, Keith Bjorndahl <ke...@kgbinternet.com> wrote:
>>>>
>>>>
>>>>>Oh....let me know if you want to restore any backups or anything.
>>>>>
>>>>>Keith
>>>>>
>>>>>
>>>>>
>>>>>>It appears that my site is really screwed up - I get 404 for pages
>>>>>>
>>>>>>
>>>on
>>>
>>>
>>>>>>my blog - as if something is hosed on the installation of Roller.
>>>>>>This makes me suspect that someone might've hacked my site. Any
>>>>>>ideas?
>>>>>>
>>>>>>Thanks,
>>>>>>
>>>>>>Matt
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>>>