You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modules-dev@httpd.apache.org by "Croteau, Beau" <Be...@ca.com> on 2007/06/14 17:15:41 UTC
Apache Thread Creation/Destroy Problem
I'm running the following code inside of an apache module:
apr_thread_t *thd_arr;
apr_pool_create(&ai->mp, NULL);
apr_threadattr_t *thd_attr;
apr_threadattr_create(&thd_attr, (ai->mp));
apr_threadattr_detach_set(thd_attr,1);
apr_thread_create(&thd_arr, thd_attr, ExecuteAction, ai, ai->mp);
The module runs in the mod_gsoap module (it's running as a gsoap web
service). It's running on Apache 2.2.3 on Windows.
The problem is that we're seeing a thread handle leak. I can see in the
logs that a thread is created and should be exiting correctly, but when
I use Handle.exe to dump all handles I can still see that the thread ID
is still in the list, even after exiting.
I've added the above code into a simple test program and the program
seems to run as expected, the handle count doesn't go up, the threads
are created, and when the threads die the thread count goes down.
I've also used the standard windows ::CreateThread and _beginthread
calls with similar types of 'leaks'.
When I dump the handle list for the apache process I see the thread
handle for the thread that should've since been released.
Any help would be greatly appreciated. Thanks.
RE: Apache Thread Creation/Destroy Problem (mod_ssl problem)
Posted by "Croteau, Beau" <Be...@ca.com>.
I have tried the LibAPR thread methods as well and the same thing still
occurs.
Joe mentioned the 'resumable' possibility. Does anybody have any idea
if part of this is maintaining current 'state' of the threads being
executed?
--Beau
-----Original Message-----
From: Joe Lewis [mailto:joe@joe-lewis.com]
Sent: Monday, June 18, 2007 4:24 PM
To: modules-dev@httpd.apache.org
Subject: Re: Apache Thread Creation/Destroy Problem (mod_ssl problem)
Croteau, Beau wrote:
> Does anybody have any ideas. I don't see any 'hooks' in OpenSSL for
> thread creation or destruction, but maybe I'm missing something.
(Yes,
> I have used the APR thread functions to no avail as well).
>
> I'm really perplexed by the behavior.
>
I DO know that the SSL extension for apache does attempt to use global
mutexes - the reason is that as an SSL connection is terminated, it
needs to be "resumable" with a fast handshake (as per the SSL
handshakes), which could happen across clients. Now, I don't see it
creating, so that may not be an issue.
Something I noticed is that your sleep() function is for 83 minutes
(repetatively, no less). (You could be seeing issues with the threads
running for so long - mutexes work in conjunction with threads and
semaphores to keep things in sync). Additionally, you are using
_beginthread() [which are native Microsoft calls] when you could be
using apr_thread_create (I don't know how portable those commands are to
winders, but you might want to try and implement those). See
http://apr.apache.org/docs/apr/0.9/group__apr__thread__proc.html
for more details regarding the functions.
I'd try the aprlib functions to see if things improve - which I expect
they will - and if they do, what you are experiencing is a collision of
threads and shared memory routines between the windows platform and the
actual APRLIB utilized by mod_ssl.
Joe
--
Joseph Lewis <http://sharktooth.org/>
"Divide the fire, and you will sooner put it out." - Publius Syrus
Re: Apache Thread Creation/Destroy Problem (mod_ssl problem)
Posted by Joe Lewis <jo...@joe-lewis.com>.
Croteau, Beau wrote:
> Does anybody have any ideas. I don't see any 'hooks' in OpenSSL for
> thread creation or destruction, but maybe I'm missing something. (Yes,
> I have used the APR thread functions to no avail as well).
>
> I'm really perplexed by the behavior.
>
I DO know that the SSL extension for apache does attempt to use global
mutexes - the reason is that as an SSL connection is terminated, it
needs to be "resumable" with a fast handshake (as per the SSL
handshakes), which could happen across clients. Now, I don't see it
creating, so that may not be an issue.
Something I noticed is that your sleep() function is for 83 minutes
(repetatively, no less). (You could be seeing issues with the threads
running for so long - mutexes work in conjunction with threads and
semaphores to keep things in sync). Additionally, you are using
_beginthread() [which are native Microsoft calls] when you could be
using apr_thread_create (I don't know how portable those commands are to
winders, but you might want to try and implement those). See
http://apr.apache.org/docs/apr/0.9/group__apr__thread__proc.html
for more details regarding the functions.
I'd try the aprlib functions to see if things improve - which I expect
they will - and if they do, what you are experiencing is a collision of
threads and shared memory routines between the windows platform and the
actual APRLIB utilized by mod_ssl.
Joe
--
Joseph Lewis <http://sharktooth.org/>
"Divide the fire, and you will sooner put it out." - Publius Syrus
RE: Apache Thread Creation/Destroy Problem (mod_ssl problem)
Posted by "Croteau, Beau" <Be...@ca.com>.
I've taken mod_gsoap out of it and implemented my own, bare-bones
module.
The code for the module is this:
#include "stdafx.h"
#include "helloworld.h"
#include "stdafx.h"
#if !defined(__GNUC__) || __GNUC__ < 2 || \
(__GNUC__ == 2 && __GNUC_MINOR__ < 7) ||\
defined(NEXT)
#ifndef __attribute__
#define __attribute__(__x)
#endif
#endif
#include <stdio.h>
#include <assert.h>
#include "winsock2.h"
#include "apr_strings.h"
#include "apr_fnmatch.h"
#include "apr_strings.h"
#include "apr_lib.h"
#include "apr_pools.h"
#define APR_WANT_STRFUNC
#include "apr_want.h"
#include "httpd.h"
#include "http_request.h"
#include "http_config.h"
#include "http_core.h"
#include "http_log.h"
#include "http_main.h"
#include "http_protocol.h"
#include "http_request.h"
#include "util_script.h"
#include <httpd.h>
void testThread2(void* input)
{
printf("Sample module: Another thread being launching -
%d\n",GetCurrentThreadId());
return;
}
void testThread(void* input)
{
while(1)
{
printf("Sample module, launching thread.\n");
_beginthread(testThread2,0,NULL);
Sleep(5000);
}
return;
}
static int helloworld_handler(request_rec *r) {
/* First, some housekeeping. */
if (!r->handler || strcasecmp(r->handler, "helloworld") != 0) {
/* r->handler wasn't "helloworld", so it's none of our business */
return DECLINED;
}
if (r->method_number != M_GET) {
/* We only accept GET and HEAD requests.
* They are identical for the purposes of a content generator
* Returning an HTTP error code causes Apache to return an
* error page (ErrorDocument) to the client.
*/
return HTTP_METHOD_NOT_ALLOWED;
}
/* OK, we're happy with this request, so we'll return the response. */
ap_set_content_type(r, "text/html");
ap_rputs("<title>Hello World!</title> .... etc...starting threads...",
r);
_beginthread(testThread,0,NULL);
/* we return OK to indicate that we have successfully processed
* the request. No further processing is required.
*/
return OK;
}
static void helloworld_hooks(apr_pool_t *pool) {
/* hook helloworld_handler in to Apache */
ap_hook_handler(helloworld_handler, NULL, NULL, APR_HOOK_MIDDLE);
}
module AP_MODULE_DECLARE_DATA helloworld_module = {
STANDARD20_MODULE_STUFF,
NULL, /* per-directory config creator */
NULL, /* dir config merger */
NULL, /* server config creator */
NULL, /* server config merger */
NULL, /* command table */
helloworld_hooks, /*hooks */
};
Once I hit http://localhost:8080/helloworld in my browser the thread
handle leak starts.
This is evident by the output:
C:\Program Files\Apache Software Foundation\Apache2.2\bin>httpd.exe -X
Sample module, launching thread.
Sample module: Another thread being launching - 15156
Sample module, launching thread.
Sample module: Another thread being launching - 15712
Sample module, launching thread.
Sample module: Another thread being launching - 16300
Sample module, launching thread.
Sample module: Another thread being launching - 16372
Sample module, launching thread.
Sample module: Another thread being launching - 16912
Sample module, launching thread.
Sample module: Another thread being launching - 13972
Sample module, launching thread.
Sample module: Another thread being launching - 13980
Sample module, launching thread.
Sample module: Another thread being launching - 14528
Sample module, launching thread.
Sample module: Another thread being launching - 17488
Sample module, launching thread.
Sample module: Another thread being launching - 17508
C:\>handle -a -p 13112 | find "15156"
310: Thread httpd.exe(13112): 15156
If I comment out the mod_ssl module:
C:\Program Files\Apache Software Foundation\Apache2.2\bin>httpd.exe -X
Sample module, launching thread.
Sample module: Another thread being launching - 33200
Sample module, launching thread.
Sample module: Another thread being launching - 33832
Sample module, launching thread.
Sample module: Another thread being launching - 34348
Sample module, launching thread.
Sample module: Another thread being launching - 34352
Sample module, launching thread.
Sample module: Another thread being launching - 34900
Sample module, launching thread.
C:\>handle -a -p 32212 | find "33200"
It clearly is 'freed' correctly.
Does anybody have any ideas. I don't see any 'hooks' in OpenSSL for
thread creation or destruction, but maybe I'm missing something. (Yes,
I have used the APR thread functions to no avail as well).
I'm really perplexed by the behavior.
--Beau
-----Original Message-----
From: Croteau, Beau [mailto:Beau.Croteau@ca.com]
Sent: Thursday, June 14, 2007 11:59 AM
To: modules-dev@httpd.apache.org
Subject: RE: Apache Thread Creation/Destroy Problem
The thread is created inside of the gsoap_handler. Though these threads
aren't necessarily mapped to a one to one gsoap_handler call. We have a
web service 'startup' function that creates several background threads
to do the processing and from there don't necessary have to have the
gsoap_handler called again for the thread leak to start happening.
--Beau
-----Original Message-----
From: Joe Lewis [mailto:joe@joe-lewis.com]
Sent: Thursday, June 14, 2007 11:59 AM
To: modules-dev@httpd.apache.org
Subject: Re: Apache Thread Creation/Destroy Problem
Croteau, Beau wrote:
> I'm sorry, I don't understand the question.
>
> Mod_gsoap hooks in as such:
> static void
> gsoap_hooks(apr_pool_t * p)
> {
> // I think this is the call to make to register a handler for
method
> calls (GET PUT et. al.).
> // We will ask to be last so that the comment has a higher
tendency
> to
> // go at the end.
> ap_hook_handler(gsoap_handler, NULL, NULL, APR_HOOK_LAST);
>
> //Register a handler for post config processing
> ap_hook_post_config(gsoap_init, NULL, NULL, APR_HOOK_MIDDLE);
> }
>
[snip]
I think we need to know if you create the thread in the gsoap_init
function or the gsoap_handler function (not a copy of the register_hooks
function, but which hook that register_hooks uses to create the thread).
Joe
RE: Apache Thread Creation/Destroy Problem
Posted by "Croteau, Beau" <Be...@ca.com>.
I did some more investigating and found the following behavior.
If I load the mod_ssl apache module then the thread handle leak occurs. If I don't then the leak doesn't appear to occur.
The code I'm using to test:
void nothing(void*input)
{
printf("Nothing - %d\n",::GetCurrentThreadId());
return;
}
unsigned __stdcall test2(void *data)
{
testClass *a = (testClass*)data;
printf("test1 - %d\n",::GetCurrentThreadId());
//testDLL testdll;
while(1)
{
printf("Spawning a new thread.\n");
_beginthread(nothing,0,NULL);
Sleep(5000);
}
delete a;
return 0;
}
test2 is a thread called by the main soap call.
If I comment out the line in httpd.conf:
LoadModule ssl_module modules/mod_ssl.so
Then the thread handle leak doesn't appear to occur, but when I load the module the thread count increases by one each iteration. Once again, this is inside of the mod_gsoap module as well, but if the only thing I change is mod_ssl being loaded then the thread leak doesn't occur.
Does anybody have any thoughts on this? I'm a bit confused by how this could happen. Keep in mind I'm a bit out of my realm as I know next to nothing about the Apache OR SSL code bases.
Thanks.
-----Original Message-----
From: Joe Lewis [mailto:joe@joe-lewis.com]
Sent: Thu 6/14/2007 12:22 PM
To: modules-dev@httpd.apache.org
Subject: Re: Apache Thread Creation/Destroy Problem
Croteau, Beau wrote:
> The thread is created inside of the gsoap_handler. Though these threads
> aren't necessarily mapped to a one to one gsoap_handler call. We have a
> web service 'startup' function that creates several background threads
> to do the processing and from there don't necessary have to have the
> gsoap_handler called again for the thread leak to start happening.
>
> --Beau
>
Thanks! Someone who is more familiar with SOAP and threads should be
able to chime in with more info at this point. Again, thanks for
identifying the hook that this is happening in.
--
Joseph Lewis <http://sharktooth.org/>
"Divide the fire, and you will sooner put it out." - Publius Syrus
Re: Apache Thread Creation/Destroy Problem
Posted by Joe Lewis <jo...@joe-lewis.com>.
Croteau, Beau wrote:
> The thread is created inside of the gsoap_handler. Though these threads
> aren't necessarily mapped to a one to one gsoap_handler call. We have a
> web service 'startup' function that creates several background threads
> to do the processing and from there don't necessary have to have the
> gsoap_handler called again for the thread leak to start happening.
>
> --Beau
>
Thanks! Someone who is more familiar with SOAP and threads should be
able to chime in with more info at this point. Again, thanks for
identifying the hook that this is happening in.
--
Joseph Lewis <http://sharktooth.org/>
"Divide the fire, and you will sooner put it out." - Publius Syrus
RE: Apache Thread Creation/Destroy Problem
Posted by "Croteau, Beau" <Be...@ca.com>.
The thread is created inside of the gsoap_handler. Though these threads
aren't necessarily mapped to a one to one gsoap_handler call. We have a
web service 'startup' function that creates several background threads
to do the processing and from there don't necessary have to have the
gsoap_handler called again for the thread leak to start happening.
--Beau
-----Original Message-----
From: Joe Lewis [mailto:joe@joe-lewis.com]
Sent: Thursday, June 14, 2007 11:59 AM
To: modules-dev@httpd.apache.org
Subject: Re: Apache Thread Creation/Destroy Problem
Croteau, Beau wrote:
> I'm sorry, I don't understand the question.
>
> Mod_gsoap hooks in as such:
> static void
> gsoap_hooks(apr_pool_t * p)
> {
> // I think this is the call to make to register a handler for
method
> calls (GET PUT et. al.).
> // We will ask to be last so that the comment has a higher
tendency
> to
> // go at the end.
> ap_hook_handler(gsoap_handler, NULL, NULL, APR_HOOK_LAST);
>
> //Register a handler for post config processing
> ap_hook_post_config(gsoap_init, NULL, NULL, APR_HOOK_MIDDLE);
> }
>
[snip]
I think we need to know if you create the thread in the gsoap_init
function or the gsoap_handler function (not a copy of the register_hooks
function, but which hook that register_hooks uses to create the thread).
Joe
Re: Apache Thread Creation/Destroy Problem
Posted by Joe Lewis <jo...@joe-lewis.com>.
Croteau, Beau wrote:
> I'm sorry, I don't understand the question.
>
> Mod_gsoap hooks in as such:
> static void
> gsoap_hooks(apr_pool_t * p)
> {
> // I think this is the call to make to register a handler for method
> calls (GET PUT et. al.).
> // We will ask to be last so that the comment has a higher tendency
> to
> // go at the end.
> ap_hook_handler(gsoap_handler, NULL, NULL, APR_HOOK_LAST);
>
> //Register a handler for post config processing
> ap_hook_post_config(gsoap_init, NULL, NULL, APR_HOOK_MIDDLE);
> }
>
[snip]
I think we need to know if you create the thread in the gsoap_init
function or the gsoap_handler function (not a copy of the register_hooks
function, but which hook that register_hooks uses to create the thread).
Joe
RE: Apache Thread Creation/Destroy Problem
Posted by "Croteau, Beau" <Be...@ca.com>.
I'm sorry, I don't understand the question.
Mod_gsoap hooks in as such:
static void
gsoap_hooks(apr_pool_t * p)
{
// I think this is the call to make to register a handler for method
calls (GET PUT et. al.).
// We will ask to be last so that the comment has a higher tendency
to
// go at the end.
ap_hook_handler(gsoap_handler, NULL, NULL, APR_HOOK_LAST);
//Register a handler for post config processing
ap_hook_post_config(gsoap_init, NULL, NULL, APR_HOOK_MIDDLE);
}
static const command_rec gsoap_cmds[] = {
AP_INIT_TAKE1("SOAPLibrary", ///< directive name
(command_function_interface) cmd_SoapLibrary, ///<
config action routine
NULL, ///< argument to include in call
ACCESS_CONF, ///< where available
"SOAP shared Library that will be dynamically loaded.
- 1 argument (path)" ///< directive description
),
AP_INIT_TAKE1("SupportLibrary", ///< directive name
(command_function_interface) cmd_SupportLibrary, ///<
config action routine
NULL, ///< argument to include in call
ACCESS_CONF, ///< where available
"additional library that must be dynamically loaded -
1 argument (path)" ///< directive description
),
{NULL}
};
module AP_MODULE_DECLARE_DATA gsoap_module = {
STANDARD20_MODULE_STUFF,
gsoap_create_dir_config, /* per-directory config creator */
gsoap_merge_dir_config, /* dir config merger */
gsoap_create_server_config, /* server config creator */
gsoap_merge_server_config, /* server config merger */
gsoap_cmds, /* command table */
gsoap_hooks, /*hooks */
};
We register as a "SOAPLibrary" with several other "SupportLibrary"'s
used.
Inside of our SOAPLibrary we use CreateThread to create multiple threads
to do various tasks. Inside one of these threads we want to execute an
action that is detached, we don't care what happens inside the thread
and we want to be able to continue processing.
In my logs I see:
{8416,11748}- Successfully disconnected from performance database.
8416 is the parent apache process. 11748 is the thread ID for the
asynchronously executing thread. The message signifies that the thread
is going through the destructor code for one of the objects it
possesses, which means that it is exiting.
But when I run the following command:
handle -a -p 8416 | find "11748"
I get:
17E0: Thread httpd.exe(8416): 11748
This means that the thread handle isn't being released for some reason.
When we let it run over time it continues to grow the thread-handle
count.
-----Original Message-----
From: Nick Kew [mailto:nick@webthing.com]
Sent: Thursday, June 14, 2007 11:29 AM
To: modules-dev@httpd.apache.org
Subject: Re: Apache Thread Creation/Destroy Problem
On Thu, 14 Jun 2007 11:15:41 -0400
"Croteau, Beau" <Be...@ca.com> wrote:
> I'm running the following code inside of an apache module:
You keep asking the same thing, but we need more information
to say anything useful.
[14:29] <crobe03> Does anybody have any idea why thread handles would
be leaked inside of an
[14:29] <crobe03> +apache module using APR but not in a stand-alone
program using the same code?
[14:31] <niq> where are they created?
[14:31] <crobe03> we have a web service that utilizes mod_gsoap
[14:32] <niq> *where* are they created? Which of apache's hooks?
So for the third time, where are they created?
If you ignore me, I'll be happy to ignore you.
--
Nick Kew
Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/
Re: Apache Thread Creation/Destroy Problem
Posted by Nick Kew <ni...@webthing.com>.
On Thu, 14 Jun 2007 11:15:41 -0400
"Croteau, Beau" <Be...@ca.com> wrote:
> I'm running the following code inside of an apache module:
You keep asking the same thing, but we need more information
to say anything useful.
[14:29] <crobe03> Does anybody have any idea why thread handles would
be leaked inside of an
[14:29] <crobe03> +apache module using APR but not in a stand-alone
program using the same code?
[14:31] <niq> where are they created?
[14:31] <crobe03> we have a web service that utilizes mod_gsoap
[14:32] <niq> *where* are they created? Which of apache's hooks?
So for the third time, where are they created?
If you ignore me, I'll be happy to ignore you.
--
Nick Kew
Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/