You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by va...@gmail.com on 2013/08/19 17:36:41 UTC
Re: Subversion 1.8 httpd.exe taking 100% CPU
> We just upgraded subversion from 1.7 to 1.8 and noticed that the process
httpd.exe takes 100% and maxes the box and we have to keep killing the
httpd.exe, are you aware of this problem?
>
>
> * What's your environment (svn client / server / Apache HTTP Server
> version)?
>
> We using TortoiseSVN 1.8 / CollabNet Edge / Apache HTTP Server version
> 2.4.4
>
> * What exactly do you do when the httpd.exe starts to consume 100% CPU
> time?
>
> We don’t know exactly what causes it because nothing is written to any
> logs as far we can see, however once we kill the httpd.exe then it’s find
> for another couple of hours.
>
>
>
Was there ever a solution/progress on this? I am having the exact same
issue, with the same setup.
Also usful to know we're using LDAP authentication against our AD domain.
Subversion edge is running on Windows Server 2008 R2 x64.
Re: Subversion 1.8 httpd.exe taking 100% CPU
Posted by va...@gmail.com.
On Tuesday, August 20, 2013 12:15:32 PM UTC+2, Ivan Zhakov wrote:
>
> On Tue, Aug 20, 2013 at 2:05 PM, <valentij...@gmail.com <javascript:>>
> wrote:
> > On Tuesday, August 20, 2013 10:47:22 AM UTC+2, Ivan Zhakov wrote:
> >>>
> >>> On Tue, Aug 20, 2013 at 12:20 PM, <va...@gmail.com> wrote:
> >>> > On Monday, August 19, 2013 5:41:55 PM UTC+2, Dinesh Hirani wrote:
> >>> >>
> >>> >> I did not find a solution however I wrote an monitor application
> that
> >>> >> checks if the httpd.exe process hits 100%, if so I KILL the process
> >>> >> and
> >>> >> CollabNet then restarts another instance.
> >>>
> > Last night the problem occurred again, this time I had debug logging
> enabled
> > for apache, and left it running for a while.
> >
> > Error log shows last succesfull requests at 19:51. Some hours later all
> > apache threads are busy, as reported in the error log:
> >
> > Mon Aug 19 19:51:49.378688 2013] [authz_core:debug] [pid 568:tid 932]
> > mod_authz_core.c(799): [client 172.17.1.15:2283] AH01626: authorization
> > result of <RequireAny>: denied (no authenticated user yet)
> > [Mon Aug 19 19:51:51.660084 2013] [authz_core:debug] [pid 568:tid 932]
> > mod_authz_core.c(799): [client 172.17.1.15:2283] AH01626: authorization
> > result of Require valid-user : denied (no authenticated user yet)
> > [Mon Aug 19 19:51:51.660084 2013] [authz_core:debug] [pid 568:tid 932]
> > mod_authz_core.c(799): [client 172.17.1.15:2283] AH01626: authorization
> > result of <RequireAny>: denied (no authenticated user yet)
> > [Mon Aug 19 19:51:51.660084 2013] [authnz_ldap:debug] [pid 568:tid 932]
> > mod_authnz_ldap.c(500): [client 172.17.1.15:2283] AH01691: auth_ldap
> > authenticate: using URL
> > ldap://172.17.1.1:3268/DC=isaac,DC=local?sAMAccountName?sub
> > [Mon Aug 19 21:06:55.298940 2013] [mpm_winnt:error] [pid 568:tid 964]
> > AH00326: Server ran out of threads to serve requests. Consider raising
> the
> > ThreadsPerChild setting
> As I replied you privately I cannot investigate provided dumps because
> you're using CollabNet distribution and I don't have access to PDB
> files required for crash dumps debugging.
>
> But from what I see process is stuck in
> libaprutil-1.dll!000000007489fc50(), given the last message in debug
> log is "auth_ldap authenticate: using URL" it most likely problem with
> communication with LDAP server (ldap protocol implemented in
> libapr-util). It could be misconfiguration or bug in libapr-util.
>
> With subversion edge 3.x we used to have the same problem, until I changed
the ldap settings to connect to AD port 3268 instead of 389.
With 3268 the problem was solved. But in subversion edge 4.x the problem is
back, probably because of the upgrade to apache 2.4
We have had other problems with apache 2.4 on windows.
Re: Subversion 1.8 httpd.exe taking 100% CPU
Posted by xn wang <rh...@gmail.com>.
LDAPSharedCacheSize 0
http://svn.haxx.se/users/archive-2014-05/0000.shtml
在 2014年4月23日星期三 UTC+8下午9:40:14，mark_w写道：
>
>
> On Wed, Dec 11, 2013 at 12:28 AM, Ben Reser <ben@> wrote:
>
> > I came to the conclusion that there was an LDAP problem as well.
> >
> > Things that would be interesting to know is if the httpd version changed
> > along
> > with the Subversion version or if httpd was left alone and only
> Subversion
> > was
> > updated.
> >
>
> We have experienced the 100% CPU consumption for some time now as well. We
> are using mpm_winnit and LDAP. I did manage to get a stack trace of the
> thread (085:a1c) running in a hard loop consuming 100% of 1 CPU on a 4 CPU
> machine. Here is our version info and the trace:
>
> [Thu Apr 17 10:27:28.528141 2014] [mpm_winnt:notice] [pid 3844:tid 436]
> AH00455: Apache/2.4.7 (Win64) SVN/1.8.8 OpenSSL/1.0.1f configured --
> resuming normal operations
> [Thu Apr 17 10:27:28.528141 2014] [mpm_winnt:notice] [pid 3844:tid 436]
> AH00456: Server built: Feb 14 2014 06:04:26
>
> 085:a1c
> libaprutil_1!apr_reslist_cleanup_order_set+0xc2
> libaprutil_1!apr_rmm_calloc+0x84
> mod_ldap+0x61d9
> mod_ldap+0x6907
> mod_ldap+0x39f2
> mod_authnz_ldap+0x1ae3
> mod_auth_basic+0x17f7
> libhttpd!ap_run_check_user_id+0x35
> libhttpd!ap_process_request_internal+0x3c4
> libhttpd!ap_die+0x4bf
> libhttpd!ap_die+0x577
> libhttpd!ap_psignature+0x1865
> libhttpd!ap_run_process_connection+0x35
> libhttpd!ap_regkey_value_remove+0x1603
> kernel32!BaseThreadInitThunk+0xd
> ntdll!RtlUserThreadStart+0x1d
>
> 4 other threads had similar stack traces and appeared to be looping as
> well
> though those threads appeared to be running between 30% to 50%. In
> aggregate
> all of the threads of the httpd child process consumed 99% to 100% of our
> 4
> CPU server. My conjecture is that there is some thread safety issue
> (please
> excuse my lack of proper thread terminology) that is causing the looping
> when multiple threads enter the same LDAP code shown in the traces. Once
> that happens the thread that passes connections into the worker threads
> can't do and so that thread eventually uses up all the worker threads but
> is
> unable to actually pass the work off to the workers. Most of the worker
> threads stack traces show them blocked on "ntdll!ZwRemoveIoCompletion+0xa"
> which I think is the normal wait state. Here's a two of the other four
> suspect stack traces which are all similar.
>
> 087:1264
> libaprutil_1!apr_reslist_cleanup_order_set+0xc2
> libaprutil_1!apr_rmm_calloc+0x84
> mod_ldap+0x624b
> mod_ldap+0x5fc6
> mod_ldap+0x69c1
> mod_ldap+0x39f2
> mod_authnz_ldap+0x1ae3
> mod_auth_basic+0x17f7
> libhttpd!ap_run_check_user_id+0x35
> libhttpd!ap_process_request_internal+0x3c4
> libhttpd!ap_die+0x4bf
> libhttpd!ap_die+0x577
> libhttpd!ap_psignature+0x1865
> libhttpd!ap_run_process_connection+0x35
> libhttpd!ap_regkey_value_remove+0x1603
> kernel32!BaseThreadInitThunk+0xd
> ntdll!RtlUserThreadStart+0x1d
>
> 090:7b0
> libaprutil_1!apr_reslist_cleanup_order_set+0xc6
> libaprutil_1!apr_rmm_calloc+0x84
> mod_ldap+0x61d9
> mod_ldap+0x6907
> mod_ldap+0x39f2
> mod_authnz_ldap+0x1ae3
> mod_auth_basic+0x17f7
> libhttpd!ap_run_check_user_id+0x35
> libhttpd!ap_process_request_internal+0x3c4
> libhttpd!ap_die+0x4bf
> libhttpd!ap_die+0x577
> libhttpd!ap_psignature+0x1865
> libhttpd!ap_run_process_connection+0x35
> libhttpd!ap_regkey_value_remove+0x1603
> kernel32!BaseThreadInitThunk+0xd
> ntdll!RtlUserThreadStart+0x1d
>
> We upgraded to the latest version of Subversion on Monday and the problem
> reoccurred early this morning though we were not able to get a process
> dump.
>
> [Wed Apr 23 04:15:17.065606 2014] [mpm_winnt:notice] [pid 1728:tid 436]
> AH00455: Apache/2.4.9 (Win64) SVN/1.8.8 OpenSSL/1.0.1g configured --
> resuming normal operations
> [Wed Apr 23 04:15:17.065606 2014] [mpm_winnt:notice] [pid 1728:tid 436]
> AH00456: Server built: Apr 8 2014 03:06:16
>
> Any advice or suggestions for alleviating this chronic issue would be
> appreciated. Let me know if anyone wants additional information or
> clarification.
>
> mark w
>
>
>
>
>
>
> --
> View this message in context:
> http://subversion.1072662.n5.nabble.com/Subversion-1-8-httpd-exe-taking-100-CPU-tp182657p188345.html
> Sent from the Subversion Users mailing list archive at Nabble.com.
>
Re: Subversion 1.8 httpd.exe taking 100% CPU
Posted by mark_w <ma...@thehartford.com>.
On Wed, Dec 11, 2013 at 12:28 AM, Ben Reser <ben@> wrote:
> I came to the conclusion that there was an LDAP problem as well.
>
> Things that would be interesting to know is if the httpd version changed
> along
> with the Subversion version or if httpd was left alone and only Subversion
> was
> updated.
>
We have experienced the 100% CPU consumption for some time now as well. We
are using mpm_winnit and LDAP. I did manage to get a stack trace of the
thread (085:a1c) running in a hard loop consuming 100% of 1 CPU on a 4 CPU
machine. Here is our version info and the trace:
[Thu Apr 17 10:27:28.528141 2014] [mpm_winnt:notice] [pid 3844:tid 436]
AH00455: Apache/2.4.7 (Win64) SVN/1.8.8 OpenSSL/1.0.1f configured --
resuming normal operations
[Thu Apr 17 10:27:28.528141 2014] [mpm_winnt:notice] [pid 3844:tid 436]
AH00456: Server built: Feb 14 2014 06:04:26
085:a1c
libaprutil_1!apr_reslist_cleanup_order_set+0xc2
libaprutil_1!apr_rmm_calloc+0x84
mod_ldap+0x61d9
mod_ldap+0x6907
mod_ldap+0x39f2
mod_authnz_ldap+0x1ae3
mod_auth_basic+0x17f7
libhttpd!ap_run_check_user_id+0x35
libhttpd!ap_process_request_internal+0x3c4
libhttpd!ap_die+0x4bf
libhttpd!ap_die+0x577
libhttpd!ap_psignature+0x1865
libhttpd!ap_run_process_connection+0x35
libhttpd!ap_regkey_value_remove+0x1603
kernel32!BaseThreadInitThunk+0xd
ntdll!RtlUserThreadStart+0x1d
4 other threads had similar stack traces and appeared to be looping as well
though those threads appeared to be running between 30% to 50%. In aggregate
all of the threads of the httpd child process consumed 99% to 100% of our 4
CPU server. My conjecture is that there is some thread safety issue (please
excuse my lack of proper thread terminology) that is causing the looping
when multiple threads enter the same LDAP code shown in the traces. Once
that happens the thread that passes connections into the worker threads
can't do and so that thread eventually uses up all the worker threads but is
unable to actually pass the work off to the workers. Most of the worker
threads stack traces show them blocked on "ntdll!ZwRemoveIoCompletion+0xa"
which I think is the normal wait state. Here's a two of the other four
suspect stack traces which are all similar.
087:1264
libaprutil_1!apr_reslist_cleanup_order_set+0xc2
libaprutil_1!apr_rmm_calloc+0x84
mod_ldap+0x624b
mod_ldap+0x5fc6
mod_ldap+0x69c1
mod_ldap+0x39f2
mod_authnz_ldap+0x1ae3
mod_auth_basic+0x17f7
libhttpd!ap_run_check_user_id+0x35
libhttpd!ap_process_request_internal+0x3c4
libhttpd!ap_die+0x4bf
libhttpd!ap_die+0x577
libhttpd!ap_psignature+0x1865
libhttpd!ap_run_process_connection+0x35
libhttpd!ap_regkey_value_remove+0x1603
kernel32!BaseThreadInitThunk+0xd
ntdll!RtlUserThreadStart+0x1d
090:7b0
libaprutil_1!apr_reslist_cleanup_order_set+0xc6
libaprutil_1!apr_rmm_calloc+0x84
mod_ldap+0x61d9
mod_ldap+0x6907
mod_ldap+0x39f2
mod_authnz_ldap+0x1ae3
mod_auth_basic+0x17f7
libhttpd!ap_run_check_user_id+0x35
libhttpd!ap_process_request_internal+0x3c4
libhttpd!ap_die+0x4bf
libhttpd!ap_die+0x577
libhttpd!ap_psignature+0x1865
libhttpd!ap_run_process_connection+0x35
libhttpd!ap_regkey_value_remove+0x1603
kernel32!BaseThreadInitThunk+0xd
ntdll!RtlUserThreadStart+0x1d
We upgraded to the latest version of Subversion on Monday and the problem
reoccurred early this morning though we were not able to get a process dump.
[Wed Apr 23 04:15:17.065606 2014] [mpm_winnt:notice] [pid 1728:tid 436]
AH00455: Apache/2.4.9 (Win64) SVN/1.8.8 OpenSSL/1.0.1g configured --
resuming normal operations
[Wed Apr 23 04:15:17.065606 2014] [mpm_winnt:notice] [pid 1728:tid 436]
AH00456: Server built: Apr 8 2014 03:06:16
Any advice or suggestions for alleviating this chronic issue would be
appreciated. Let me know if anyone wants additional information or
clarification.
mark w
--
View this message in context: http://subversion.1072662.n5.nabble.com/Subversion-1-8-httpd-exe-taking-100-CPU-tp182657p188345.html
Sent from the Subversion Users mailing list archive at Nabble.com.
Re: Subversion 1.8 httpd.exe taking 100% CPU
Posted by Mark Phippard <ma...@gmail.com>.
On Wed, Dec 11, 2013 at 12:28 AM, Ben Reser <be...@reser.org> wrote:
> On 8/20/13 3:15 AM, Ivan Zhakov wrote:
> > But from what I see process is stuck in
> > libaprutil-1.dll!000000007489fc50(), given the last message in debug
> > log is "auth_ldap authenticate: using URL" it most likely problem with
> > communication with LDAP server (ldap protocol implemented in
> > libapr-util). It could be misconfiguration or bug in libapr-util.
>
> Note there was another similar thread on this here:
>
> https://mail-archives.apache.org/mod_mbox/subversion-users/201311.mbox/%3CCEB51A88.2B46A%25davek%40gamehouse.com%3E
>
> I came to the conclusion that there was an LDAP problem as well.
>
> Things that would be interesting to know is if the httpd version changed
> along
> with the Subversion version or if httpd was left alone and only Subversion
> was
> updated.
>
I would be happy to answer any questions on the included version of Apache
components in SVN Edge. That said, you can also just check the Apache
error log from the SVN Edge web UI and the line when the server is started
gives a breakdown of the versions. The current SVN Edge release is 4.0.4
and it includes SVN 1.8.5 and httpd 2.4.7:
https://ctf.open.collab.net/sf/wiki/do/viewPage/projects.svnedge/wiki/OpenSourceComponents
Based on Valentijn's request from this summer, we have also started
capturing the PDB files from the build process and I can supply them to
anyone that needs them. Just email me. Valentijn changed his server to
Linux before we got a chance to provide them to him.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
Re: Subversion 1.8 httpd.exe taking 100% CPU
Posted by Ben Reser <be...@reser.org>.
On 8/20/13 3:15 AM, Ivan Zhakov wrote:
> But from what I see process is stuck in
> libaprutil-1.dll!000000007489fc50(), given the last message in debug
> log is "auth_ldap authenticate: using URL" it most likely problem with
> communication with LDAP server (ldap protocol implemented in
> libapr-util). It could be misconfiguration or bug in libapr-util.
Note there was another similar thread on this here:
https://mail-archives.apache.org/mod_mbox/subversion-users/201311.mbox/%3CCEB51A88.2B46A%25davek%40gamehouse.com%3E
I came to the conclusion that there was an LDAP problem as well.
Things that would be interesting to know is if the httpd version changed along
with the Subversion version or if httpd was left alone and only Subversion was
updated.
Both threads seemed to be using CollabNet's Subversion Edge (basing that on the
other thread saying they had version 4.0.2-3732.117 which seems to corollate
with CollabNet's Subversion Edge version numbers, they're up to 4.0.4 now
though). I believe that httpd is included as part of Subversion Edge so I'd
guess that httpd version changed as well.
Re: Subversion 1.8 httpd.exe taking 100% CPU
Posted by Ivan Zhakov <iv...@visualsvn.com>.
On Tue, Aug 20, 2013 at 2:05 PM, <va...@gmail.com> wrote:
> On Tuesday, August 20, 2013 10:47:22 AM UTC+2, Ivan Zhakov wrote:
>>>
>>> On Tue, Aug 20, 2013 at 12:20 PM, <va...@gmail.com> wrote:
>>> > On Monday, August 19, 2013 5:41:55 PM UTC+2, Dinesh Hirani wrote:
>>> >>
>>> >> I did not find a solution however I wrote an monitor application that
>>> >> checks if the httpd.exe process hits 100%, if so I KILL the process
>>> >> and
>>> >> CollabNet then restarts another instance.
>>>
> Last night the problem occurred again, this time I had debug logging enabled
> for apache, and left it running for a while.
>
> Error log shows last succesfull requests at 19:51. Some hours later all
> apache threads are busy, as reported in the error log:
>
> Mon Aug 19 19:51:49.378688 2013] [authz_core:debug] [pid 568:tid 932]
> mod_authz_core.c(799): [client 172.17.1.15:2283] AH01626: authorization
> result of <RequireAny>: denied (no authenticated user yet)
> [Mon Aug 19 19:51:51.660084 2013] [authz_core:debug] [pid 568:tid 932]
> mod_authz_core.c(799): [client 172.17.1.15:2283] AH01626: authorization
> result of Require valid-user : denied (no authenticated user yet)
> [Mon Aug 19 19:51:51.660084 2013] [authz_core:debug] [pid 568:tid 932]
> mod_authz_core.c(799): [client 172.17.1.15:2283] AH01626: authorization
> result of <RequireAny>: denied (no authenticated user yet)
> [Mon Aug 19 19:51:51.660084 2013] [authnz_ldap:debug] [pid 568:tid 932]
> mod_authnz_ldap.c(500): [client 172.17.1.15:2283] AH01691: auth_ldap
> authenticate: using URL
> ldap://172.17.1.1:3268/DC=isaac,DC=local?sAMAccountName?sub
> [Mon Aug 19 21:06:55.298940 2013] [mpm_winnt:error] [pid 568:tid 964]
> AH00326: Server ran out of threads to serve requests. Consider raising the
> ThreadsPerChild setting
As I replied you privately I cannot investigate provided dumps because
you're using CollabNet distribution and I don't have access to PDB
files required for crash dumps debugging.
But from what I see process is stuck in
libaprutil-1.dll!000000007489fc50(), given the last message in debug
log is "auth_ldap authenticate: using URL" it most likely problem with
communication with LDAP server (ldap protocol implemented in
libapr-util). It could be misconfiguration or bug in libapr-util.
--
Ivan Zhakov
Re: Subversion 1.8 httpd.exe taking 100% CPU
Posted by va...@gmail.com.
On Tuesday, August 20, 2013 10:47:22 AM UTC+2, Ivan Zhakov wrote:
> On Tue, Aug 20, 2013 at 12:20 PM, <va...@gmail.com> wrote:
>> > On Monday, August 19, 2013 5:41:55 PM UTC+2, Dinesh Hirani wrote:
>> >>
>> >> I did not find a solution however I wrote an monitor application that
>> >> checks if the httpd.exe process hits 100%, if so I KILL the process
>> and
>> >> CollabNet then restarts another instance.
>>
>> Last night the problem occurred again, this time I had debug logging
enabled for apache, and left it running for a while.
Error log shows last succesfull requests at 19:51. Some hours later all
apache threads are busy, as reported in the error log:
Mon Aug 19 19:51:49.378688 2013] [authz_core:debug] [pid 568:tid 932]
mod_authz_core.c(799): [client 172.17.1.15:2283] AH01626: authorization
result of <RequireAny>: denied (no authenticated user yet)
[Mon Aug 19 19:51:51.660084 2013] [authz_core:debug] [pid 568:tid 932]
mod_authz_core.c(799): [client 172.17.1.15:2283] AH01626: authorization
result of Require valid-user : denied (no authenticated user yet)
[Mon Aug 19 19:51:51.660084 2013] [authz_core:debug] [pid 568:tid 932]
mod_authz_core.c(799): [client 172.17.1.15:2283] AH01626: authorization
result of <RequireAny>: denied (no authenticated user yet)
[Mon Aug 19 19:51:51.660084 2013] [authnz_ldap:debug] [pid 568:tid 932]
mod_authnz_ldap.c(500): [client 172.17.1.15:2283] AH01691: auth_ldap
authenticate: using URL
ldap://172.17.1.1:3268/DC=isaac,DC=local?sAMAccountName?sub
[Mon Aug 19 21:06:55.298940 2013] [mpm_winnt:error] [pid 568:tid 964]
AH00326: Server ran out of threads to serve requests. Consider raising the
ThreadsPerChild setting
[Mon Aug 19 21:06:56.299004 2013] [mpm_winnt:debug] [pid 568:tid 964]
child.c(187): AH00327: mpm_get_completion_context: Failed to get a free
context within 1 second
[Mon Aug 19 21:07:01.549340 2013] [mpm_winnt:debug] [pid 568:tid 964]
child.c(187): AH00327: mpm_get_completion_context: Failed to get a free
context within 1 second
[Mon Aug 19 21:07:06.830928 2013] [mpm_winnt:debug] [pid 568:tid 964]
child.c(187): AH00327: mpm_get_completion_context: Failed to get a free
context within 1 second
[Mon Aug 19 21:07:12.159394 2013] [mpm_winnt:debug] [pid 568:tid 964]
child.c(187): AH00327: mpm_get_completion_context: Failed to get a free
context within 1 second
[Mon Aug 19 21:07:17.440982 2013] [mpm_winnt:debug] [pid 568:tid 964]
child.c(187): AH00327: mpm_get_completion_context: Failed to get a free
context within 1 second
[Mon Aug 19 21:07:22.722570 2013] [mpm_winnt:debug] [pid 568:tid 964]
child.c(187): AH00327: mpm_get_completion_context: Failed to get a free
context within 1 second
[Mon Aug 19 21:07:28.129166 2013] [mpm_winnt:debug] [pid 568:tid 964]
child.c(187): AH00327: mpm_get_completion_context: Failed to get a free
context within 1 second
Re: Subversion 1.8 httpd.exe taking 100% CPU
Posted by va...@gmail.com.
On Tuesday, August 20, 2013 10:47:22 AM UTC+2, Ivan Zhakov wrote:
>
> On Tue, Aug 20, 2013 at 12:20 PM, <valentij...@gmail.com <javascript:>>
> wrote:
> > On Monday, August 19, 2013 5:41:55 PM UTC+2, Dinesh Hirani wrote:
> >>
> >> I did not find a solution however I wrote an monitor application that
> >> checks if the httpd.exe process hits 100%, if so I KILL the process and
> >> CollabNet then restarts another instance.
> >
> > Is it something you'd like to share?
> >
> > I used ProcDump to create a dump when the process went to 100%. Would it
> be
> > usefull to post it here?
> >
> Could you post just call stack here?
>
> I just sent you an email with the dump.Not sure if a process dump can
contain confidential data.
Re: Subversion 1.8 httpd.exe taking 100% CPU
Posted by Ivan Zhakov <iv...@visualsvn.com>.
On Tue, Aug 20, 2013 at 12:20 PM, <va...@gmail.com> wrote:
> On Monday, August 19, 2013 5:41:55 PM UTC+2, Dinesh Hirani wrote:
>>
>> I did not find a solution however I wrote an monitor application that
>> checks if the httpd.exe process hits 100%, if so I KILL the process and
>> CollabNet then restarts another instance.
>
> Is it something you'd like to share?
>
> I used ProcDump to create a dump when the process went to 100%. Would it be
> usefull to post it here?
>
Could you post just call stack here?
--
Ivan Zhakov
Re: Subversion 1.8 httpd.exe taking 100% CPU
Posted by Daniel Shahaf <da...@elego.de>.
valentijnscholten@gmail.com wrote on Tue, Aug 20, 2013 at 01:20:42 -0700:
> I used ProcDump to create a dump when the process went to 100%. Would it be
> usefull to post it here?
Please don't post large binary attachments to this list. Upload them
somewhere and (after ensuring they don't contain passwords, etc) post
a link to them.
RE: Subversion 1.8 httpd.exe taking 100% CPU
Posted by Dinesh Hirani <Di...@decuragroup.com>.
I’ve included the my main class that does the work, you can wrap it around the windows service. Call the start method when the service starts and call the dispose method when the service stops.
The code below binds to the httpd.exe PID and monitor the CPU load, once it hits 99% or greater than I KILL that PID. CollabNet will restart it automatically.
public class Manager : IDisposable
{
readonly ILog _log;
#region Constructor / Deconstructor
/// <summary>
/// Initializes a new instance of the <see cref="Manager" /> class.
/// </summary>
public Manager()
{
_log = LogManager.GetLogger(typeof(Manager));
}
/// <summary>
/// Finalizes an instance of the <see cref="Manager" /> class.
/// </summary>
~Manager()
{
Dispose(false);
}
#endregion
/// <summary>
/// Starts this instance.
/// </summary>
public void Start()
{
try
{
// now monitor these
WorkingThread = new Thread(() =>
{
while (!IsExit)
{
var httpdProcesses = Process.GetProcessesByName("httpd");
foreach (var httpdProcess in httpdProcesses)
{
if (GetCpuPercentById(httpdProcess.Id) >= 99)
{
try
{
_log.InfoFormat("Killing PID {0}", httpdProcess.Id);
httpdProcess.Kill();
_log.InfoFormat("Killed PID {0}", httpdProcess.Id);
}
catch (Exception ex)
{
_log.Warn(ex.Message, ex);
}
}
}
Thread.Sleep(100);
}
});
WorkingThread.Start();
}
catch (Exception ex)
{
_log.WarnFormat("Failed to process because {0}", ex.Message);
}
}
private int GetCpuPercentById(int pid)
{
PerformanceCounter pc = new PerformanceCounter("Process", "% Processor Time",
GetPerformanceCounterProcessName(pid), true);
pc.NextValue();
Thread.Sleep(1000);
int cpuPercent = (int) pc.NextValue() / Environment.ProcessorCount;
return cpuPercent;
}
private string GetPerformanceCounterProcessName(int pid)
{
return GetPerformanceCounterProcessName(pid, System.Diagnostics.Process.GetProcessById(pid).ProcessName);
}
private string GetPerformanceCounterProcessName(int pid, string processName)
{
int nameIndex = 1;
string value = processName;
string counterName = processName + "#" + nameIndex;
PerformanceCounter pc = new PerformanceCounter("Process", "ID Process", counterName, true);
while (true)
{
try
{
if (pid == (int) pc.NextValue())
{
value = counterName;
break;
}
else
{
nameIndex++;
counterName = processName + "#" + nameIndex;
pc = new PerformanceCounter("Process", "ID Process", counterName, true);
}
}
catch (SystemException ex)
{
if (ex.Message == "Instance '" + counterName + "' does not exist in the specified Category.")
{
break;
}
else
{
throw;
}
}
}
return value;
}
public bool IsExit { get; set; }
private Thread WorkingThread;
#region IDisposable
private bool _disposed;
/// <summary>
/// Releases unmanaged and - optionally - managed resources.
/// </summary>
/// <param name="disposing"><c>true</c> to release both managed and unmanaged resources; <c>false</c> to release only unmanaged resources.</param>
protected virtual void Dispose(bool disposing)
{
if (!_disposed)
{
if (disposing)
{
// Dispose managed resources.
IsExit = true;
if (WorkingThread != null)
{
WorkingThread.Join();
WorkingThread = null;
}
}
// There are no unmanaged resources to release, but
// if we add them, they need to be released here.
}
_disposed = true;
}
/// <summary>
/// Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.
/// </summary>
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
#endregion
}
From: valentijnscholten@gmail.com [mailto:valentijnscholten@gmail.com]
Sent: 20 August 2013 09:21
To: subversion_users@googlegroups.com
Cc: valentijnscholten@gmail.com; Pavel Lyalyakin; users@subversion.apache.org; Dinesh Hirani
Subject: Re: Subversion 1.8 httpd.exe taking 100% CPU
On Monday, August 19, 2013 5:41:55 PM UTC+2, Dinesh Hirani wrote:
I did not find a solution however I wrote an monitor application that checks if the httpd.exe process hits 100%, if so I KILL the process and CollabNet then restarts another instance.
Is it something you'd like to share?
I used ProcDump to create a dump when the process went to 100%. Would it be usefull to post it here?
The information in this e-mail is confidential and may be legally privileged. It is intended solely for the addressee(s). Access by any other person to this e-mail is not authorized. If you are not the intended recipient, please delete this e-mail. Any disclosure of this e-mail or of the parties to it and any copying or distribution of it is prohibited (and may be unlawful). The information expressed herein may be changed at any time without notice or obligation to update. We do not represent that this message is virus-free, complete or accurate and it should not be relied upon as such. Electronic communications may be monitored for operational and business purposes to the extent permitted by applicable law. This email does not constitute legal or tax advice, and the information contained in this communication should not be regarded as such.
Decura IM LLP is authorised and regulated by the Financial Conduct Authority. Registered office address: 11-12 St James’s Square, London SW1Y 4LB. Registered in England and Wales: OC375344
Decura LLP is authorised and regulated by the Financial Conduct Authority. Registered office address: 11-12 St James’s Square, London SW1Y 4LB. Registered in England and Wales: OC377231
Re: Subversion 1.8 httpd.exe taking 100% CPU
Posted by va...@gmail.com.
On Monday, August 19, 2013 5:41:55 PM UTC+2, Dinesh Hirani wrote:
> I did not find a solution however I wrote an monitor application that
> checks if the httpd.exe process hits 100%, if so I KILL the process and
> CollabNet then restarts another instance.
>
Is it something you'd like to share?
I used ProcDump to create a dump when the process went to 100%. Would it be
usefull to post it here?
RE: Subversion 1.8 httpd.exe taking 100% CPU
Posted by Dinesh Hirani <Di...@decuragroup.com>.
I did not find a solution however I wrote an monitor application that checks if the httpd.exe process hits 100%, if so I KILL the process and CollabNet then restarts another instance.
From: valentijnscholten@gmail.com [mailto:valentijnscholten@gmail.com]
Sent: 19 August 2013 16:37
To: subversion_users@googlegroups.com
Cc: Pavel Lyalyakin; users@subversion.apache.org; Dinesh Hirani
Subject: Re: Subversion 1.8 httpd.exe taking 100% CPU
> We just upgraded subversion from 1.7 to 1.8 and noticed that the process httpd.exe takes 100% and maxes the box and we have to keep killing the httpd.exe, are you aware of this problem?
* What's your environment (svn client / server / Apache HTTP Server version)?
We using TortoiseSVN 1.8 / CollabNet Edge / Apache HTTP Server version 2.4.4
* What exactly do you do when the httpd.exe starts to consume 100% CPU time?
We don’t know exactly what causes it because nothing is written to any logs as far we can see, however once we kill the httpd.exe then it’s find for another couple of hours.
Was there ever a solution/progress on this? I am having the exact same issue, with the same setup.
Also usful to know we're using LDAP authentication against our AD domain.
Subversion edge is running on Windows Server 2008 R2 x64.
The information in this e-mail is confidential and may be legally privileged. It is intended solely for the addressee(s). Access by any other person to this e-mail is not authorized. If you are not the intended recipient, please delete this e-mail. Any disclosure of this e-mail or of the parties to it and any copying or distribution of it is prohibited (and may be unlawful). The information expressed herein may be changed at any time without notice or obligation to update. We do not represent that this message is virus-free, complete or accurate and it should not be relied upon as such. Electronic communications may be monitored for operational and business purposes to the extent permitted by applicable law. This email does not constitute legal or tax advice, and the information contained in this communication should not be regarded as such.
Decura IM LLP is authorised and regulated by the Financial Conduct Authority. Registered office address: 11-12 St James’s Square, London SW1Y 4LB. Registered in England and Wales: OC375344
Decura LLP is authorised and regulated by the Financial Conduct Authority. Registered office address: 11-12 St James’s Square, London SW1Y 4LB. Registered in England and Wales: OC377231