You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Issac Goldstand <ma...@beamartyr.net> on 2007/02/20 14:14:16 UTC
[RFC] Adding UDP support to server
I'm working on a protocol module for Apache 2.2 and ran into the lack of
UDP support in httpd. I'd like to try and remedy the situation in a
manner best suited for merging to trunk + backporting where possible. I
know that people have asked about it in the past, and if we really want
to be "d", we need support beyond TCP/SOCK_STREAM sockets.
I tried to go through the socket-related routines in httpd (those of the
server; not of modules, clients, etc (mod_ssl, mod_proxy, mod_cgid,
etc)) and came up with a list of areas which needed attention:
listen.c - alloc_listener is the focal point of the socket creation in
the server. There are a few issues that I can see with patching this
function:
1) Currently it calls apr_socket_create with SOCK_STREAM, 0 (for type,
protocol). For adding UDP support only, we could make do with adding a
flag to the function to toggle between the existing values and
SOCK_DGRAM, APR_PROTO_UDP (which I'm personally against). To be more
portable, we'd pass both type and protocol as parameters (which I
personally prefer). Neither of these options makes much of a mess, as
AFAIK only ap_set_listener calls alloc_listener.
2) In the code to check if we can re-use an existing listener, compare
the socket-descriptor's type and protocol against the ones passed.
3) Additionally, I'd suggest making alloc_listener a public API function
(we've already added arguments, so no extra harm in renaming it to
ap_alloc_listener and adding it to ap_listen.h is there?) so that module
authors can create directives for protocols/types beyond SOCK_STREAM/TCP
and SOCK_DGRAM/UDP (see below). [This would be a minor bump, right? It
adds functionality without any breakage or backwards compatibility
issues, since as mentioned above only ap_set_listener currently calls
alloc_listener today]
listen.c - make_sock sets a bunch of socket options. I don't profess to
be expert enough in socket programming to know what's kosher or not for
passing to non-"SOCK_STREAM TCP" sockets. More importantly, make_sock
calls apr_socket_listen, which isn't correct for datagram sockets. To
get around this, perhaps wrap the call in "if (s->type == SOCK_STREAM) "
(or whatever should be there to know if this type of socket needs to be
listen()ed to). Finally, similar logic (to detect if the socket needs
to be accept()ed) should be applied to the accept_func for MPMs which
utilize it. IMHO, rewriting standard MPM's that use an accept_func to
check if it's NULL before running it seems the smartest way to do it
(would that be a bigger bump than the ap_alloc_listeners described
above?). Barring that, we could define an "empty" default listener
function which does nothing, and have non-listening sockets have that
assigned as their alloc_func instead of MPM_ACCEPT_FUNC
listen.c - ap_apply_accept_filter sets the accept filter for the
protocol (["http", "https", etc], not [TCP, UDP, etc]) As above, I'm
not sure how "kosher" this is, but TCP_DEFER_ACCEPT on UDP sounds wrong
:-) That could be solved by enclosing the filter in an "if (s->protocol
== APR_PROTO_TCP)" block
listen.c - ap_set_listener is the user's interface to adding a listener.
I can't think of a nice graceful way to add the stream-type and
protocol to the existing listen directive while remaining backwards
compatible and keeping some resemblance of simplicity to the directive,
so I propose adding a ListenUDP directive which does the same logic
except that it calls alloc_listener with SOCK_DGRAM, APR_PROTO_UDP
(since UDP has been asked about several times). For other combinations,
module authors can call ap_alloc_listener as described above
core.c - core_pre_connection disables the nagle algorithm by setting
APR_TCP_NODELAY. Can this be done safely to a non-TCP socket? If not,
just check "if (csd->protocl == APRC_PROTO_TCP)" as above
mpm_common.c - ap_sock_disable_nagle disables the nagle algorithm as
above.
mpm_common.c - dummy_connection currently makes assumptions about
SOCK_STREAM and TCP. However, it also only connects to the port in the
first listener's (in global ap_listener) port. I suppose changing it to
use ap_listeners->sd->type, ap_listeners->sd->protocol shouldn't hurt...
should it?
Connection I/O should also be fairly easy. The core input filter just
uses a socket bucket to read from; if apr_bucket_type_socket supports
UDP sockets, we should be good to go on that end. The core output
filter is also fairly simple - just check that we're a SOCK_STREAM
socket before attempting to sendfile.
I'm also assuming (possibly incorrectly) that polling and socket timeout
related code should all work as-is.
I *think* that's all that should be needed... I have a patch against
trunk demonstrating most of this (attached). I'd really appreciate it
if people who have a firmer understanding of the relevant bits of code
that I'm proposing changes to could chime in and let me know if and what
I'm doing wrong, or not taking into consideration.
Issac
Re: [RFC] Adding UDP support to server
Posted by Taisuke Yamada <ty...@list.rakugaki.org>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Last year, I posted a question on implementation strategy of
HTTP-over-UDP(multicast) module to modules-dev list. I was
able to implement fully functional "UDP echo" service, but
for HTTP/U module, it suffered severe limitation due to how
core_filter handles output.
- - http://www.mail-archive.com/modules-dev@httpd.apache.org/msg00129.html
Here were my findings at the time (with 2.2.4-dev):
- - For initial UDP socket setup, current httpd implementation
(listen.c) is OK, as I was able to bypass it by inserting
custom ap_listen_rec (with custom accept_func function) into
ap_listeners at open_log stage.
- - Another issue was ap_lingering_close called after request
processing, but that too, can be worked around by doing
ap_set_module_config(c->conn_config, &core_module, NULL);
at process_connection stage.
- - Probe error from dummy_connection was an issue, but was not
a blocker as I can suppress that by listening on same TCP
port (Type2 implementation in above URL).
- - There was a timeout issue which caused server to stop
responding when handling UDP socket. I worked around that
by setting custom timeout value at process_connection stage,
but this was only a kludge and requires deeper look for
complete solution.
- - core_output_filter (core_filters.c) was a major blocker.
This (to be precise, underlying apr_socket_sendv called)
expects output socket to be able to return data with simple
writev(2). That can't be done for un-connected socket.
To workaround last issue, I did apr_socket_connect to UDP socket,
but that destroys listening UDP socket. In order to handle this,
I tried several methods below:
- - Do a connect(2) on current socket, pass it to bucket handler,
and dup2(2) newly created listening socket into socket registered
to ap_listeners.
- -- This almost worked, but apache never responded to further
request even though it continued to listen on UDP port.
- - Disconnect "connected" UDP socket by connecting with AF_UNSPEC
- -- Same as above...forgot the detail, but it didn't work.
- - Pass newly created "connected" socket to bucket handler
- -- This was a stupid idea as there's no way to pass received
data to this new socket (only if there's accept(2) for UDP...).
All failed, and as a result this HTTP/U module handled only
first HTTP/UDP request properly, but no more.
Although I have no specific idea on implementation, I think the
patch posted can be made simpler by focusing on core_filter issue.
But maybe this is an APR issue rather than of httpd?
> I'm working on a protocol module for Apache 2.2 and ran into the lack of
> UDP support in httpd. I'd like to try and remedy the situation in a
> manner best suited for merging to trunk + backporting where possible. I
> know that people have asked about it in the past, and if we really want
> to be "d", we need support beyond TCP/SOCK_STREAM sockets.
>
> I tried to go through the socket-related routines in httpd (those of the
> server; not of modules, clients, etc (mod_ssl, mod_proxy, mod_cgid,
> etc)) and came up with a list of areas which needed attention:
>
> listen.c - alloc_listener is the focal point of the socket creation in
> the server. There are a few issues that I can see with patching this
> function:
> 1) Currently it calls apr_socket_create with SOCK_STREAM, 0 (for type,
> protocol). For adding UDP support only, we could make do with adding a
> flag to the function to toggle between the existing values and
> SOCK_DGRAM, APR_PROTO_UDP (which I'm personally against). To be more
> portable, we'd pass both type and protocol as parameters (which I
> personally prefer). Neither of these options makes much of a mess, as
> AFAIK only ap_set_listener calls alloc_listener.
> 2) In the code to check if we can re-use an existing listener, compare
> the socket-descriptor's type and protocol against the ones passed.
> 3) Additionally, I'd suggest making alloc_listener a public API function
> (we've already added arguments, so no extra harm in renaming it to
> ap_alloc_listener and adding it to ap_listen.h is there?) so that module
> authors can create directives for protocols/types beyond SOCK_STREAM/TCP
> and SOCK_DGRAM/UDP (see below). [This would be a minor bump, right? It
> adds functionality without any breakage or backwards compatibility
> issues, since as mentioned above only ap_set_listener currently calls
> alloc_listener today]
>
> listen.c - make_sock sets a bunch of socket options. I don't profess to
> be expert enough in socket programming to know what's kosher or not for
> passing to non-"SOCK_STREAM TCP" sockets. More importantly, make_sock
> calls apr_socket_listen, which isn't correct for datagram sockets. To
> get around this, perhaps wrap the call in "if (s->type == SOCK_STREAM) "
> (or whatever should be there to know if this type of socket needs to be
> listen()ed to). Finally, similar logic (to detect if the socket needs
> to be accept()ed) should be applied to the accept_func for MPMs which
> utilize it. IMHO, rewriting standard MPM's that use an accept_func to
> check if it's NULL before running it seems the smartest way to do it
> (would that be a bigger bump than the ap_alloc_listeners described
> above?). Barring that, we could define an "empty" default listener
> function which does nothing, and have non-listening sockets have that
> assigned as their alloc_func instead of MPM_ACCEPT_FUNC
>
> listen.c - ap_apply_accept_filter sets the accept filter for the
> protocol (["http", "https", etc], not [TCP, UDP, etc]) As above, I'm
> not sure how "kosher" this is, but TCP_DEFER_ACCEPT on UDP sounds wrong
> :-) That could be solved by enclosing the filter in an "if (s->protocol
> == APR_PROTO_TCP)" block
>
> listen.c - ap_set_listener is the user's interface to adding a listener.
> I can't think of a nice graceful way to add the stream-type and
> protocol to the existing listen directive while remaining backwards
> compatible and keeping some resemblance of simplicity to the directive,
> so I propose adding a ListenUDP directive which does the same logic
> except that it calls alloc_listener with SOCK_DGRAM, APR_PROTO_UDP
> (since UDP has been asked about several times). For other combinations,
> module authors can call ap_alloc_listener as described above
>
> core.c - core_pre_connection disables the nagle algorithm by setting
> APR_TCP_NODELAY. Can this be done safely to a non-TCP socket? If not,
> just check "if (csd->protocl == APRC_PROTO_TCP)" as above
>
> mpm_common.c - ap_sock_disable_nagle disables the nagle algorithm as
> above.
>
> mpm_common.c - dummy_connection currently makes assumptions about
> SOCK_STREAM and TCP. However, it also only connects to the port in the
> first listener's (in global ap_listener) port. I suppose changing it to
> use ap_listeners->sd->type, ap_listeners->sd->protocol shouldn't hurt...
> should it?
>
> Connection I/O should also be fairly easy. The core input filter just
> uses a socket bucket to read from; if apr_bucket_type_socket supports
> UDP sockets, we should be good to go on that end. The core output
> filter is also fairly simple - just check that we're a SOCK_STREAM
> socket before attempting to sendfile.
>
> I'm also assuming (possibly incorrectly) that polling and socket timeout
> related code should all work as-is.
>
> I *think* that's all that should be needed... I have a patch against
> trunk demonstrating most of this (attached). I'd really appreciate it
> if people who have a firmer understanding of the relevant bits of code
> that I'm proposing changes to could chime in and let me know if and what
> I'm doing wrong, or not taking into consideration.
- --
Taisuke Yamada <ty...@spam.rakugaki.org>, http://rakugaki.org/
2268 E9A2 D4F9 014E F11D 1DF7 DCA3 83BC 78E5 CD3A
Message to my public address may not be handled in a timely manner.
For a direct contact, please use my private address on my namecard.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFF3q6b3KODvHjlzToRAq6JAJ9HB3XWDBPb1t+d8HMOwNC8P3rlFwCghLUW
+UqYm2BkxGa9rK5aHoIHe7w=
=7TEe
-----END PGP SIGNATURE-----
Re: [RFC] Adding UDP support to server
Posted by Issac Goldstand <ma...@beamartyr.net>.
Jean-Frederic wrote:
> On Tue, 2007-02-20 at 15:14 +0200, Issac Goldstand wrote:
>> I'm working on a protocol module for Apache 2.2 and ran into the lack of
>> UDP support in httpd. I'd like to try and remedy the situation in a
>> manner best suited for merging to trunk + backporting where possible. I
>> know that people have asked about it in the past, and if we really want
>> to be "d", we need support beyond TCP/SOCK_STREAM sockets.
>
> Having UDP support doesn't sound a bad idea.
>
> You don't need to change all the worker stuff. Just set accept_funct =
> to a dummy fonction for UPD sockets in make_sock().
That's possible, but we'd need to muck with the MPMs anyway. Although I
neglected to include it in my previous post, not all MPMs use an
accept_func and those that don't also need to be modified to provide the
option of not calling apr_socket_accept [or alternatively, have the MPMs
silently ignore a EOPNOTSUPP error returned from the underlying accept()
call, if that's portable and deemed "safe"]
> You don't need to do anything in dummy_connection() when using a UPD
> socket.
In general, I think the entire accept_func is not currently implemented
so well; the idea is to provide a flexible way to control how a TCP
connection is accept()ed, yet each MPM that uses it pretty much
hardcodes its use anyway (with the single exception of per-child on
trunk, which changes it at run-time).
If we're looking to do the UDP thing right, I'd suggest changing the way
accept_func works to be either an APR_OPTIONAL_FN or, better yet, an
AP_HOOK, and for MPMs to exclusively call the socket's accept_func (in
either case, the default TCP accept_func would just wrap
apr_socket_accept - it could also handle non-TCP functions by not
calling apr_socket_accept if apr_socket_type_get != SOCK_STREAM).
Issac
Re: [RFC] Adding UDP support to server
Posted by Jean-Frederic <jf...@gmail.com>.
On Tue, 2007-02-20 at 15:14 +0200, Issac Goldstand wrote:
> I'm working on a protocol module for Apache 2.2 and ran into the lack of
> UDP support in httpd. I'd like to try and remedy the situation in a
> manner best suited for merging to trunk + backporting where possible. I
> know that people have asked about it in the past, and if we really want
> to be "d", we need support beyond TCP/SOCK_STREAM sockets.
Having UDP support doesn't sound a bad idea.
You don't need to change all the worker stuff. Just set accept_funct =
to a dummy fonction for UPD sockets in make_sock().
You don't need to do anything in dummy_connection() when using a UPD
socket.
Cheers
Jean-Frederic
Re: determining MPM type from module ?
Posted by Gregory Nicholls <gn...@bellsouth.net>.
Ah .. thanks ..
G.
Rainer Jung wrote:
> To find out, if the MPM uses multiple threads, you can use the query API
> documented in ap_mpm.h (excerpt for Apache 2.0):
>
> #define AP_MPMQ_MAX_DAEMON_USED 1 /* Max # of daemons used so far */
> #define AP_MPMQ_IS_THREADED 2 /* MPM can do threading */
> #define AP_MPMQ_IS_FORKED 3 /* MPM can do forking */
> #define AP_MPMQ_HARD_LIMIT_DAEMONS 4 /* The compiled max # daemons */
> #define AP_MPMQ_HARD_LIMIT_THREADS 5 /* The compiled max # threads */
> #define AP_MPMQ_MAX_THREADS 6 /* # of threads/child by config */
> #define AP_MPMQ_MIN_SPARE_DAEMONS 7 /* Min # of spare daemons */
> #define AP_MPMQ_MIN_SPARE_THREADS 8 /* Min # of spare threads */
> #define AP_MPMQ_MAX_SPARE_DAEMONS 9 /* Max # of spare daemons */
> #define AP_MPMQ_MAX_SPARE_THREADS 10 /* Max # of spare threads */
> #define AP_MPMQ_MAX_REQUESTS_DAEMON 11 /* Max # of requests per daemon */
> #define AP_MPMQ_MAX_DAEMONS 12 /* Max # of daemons by config */
> #define AP_MPMQ_MPM_STATE 13 /* starting, running, stopping */
>
> /**
> * Query a property of the current MPM.
> * @param query_code One of APM_MPMQ_*
> * @param result A location to place the result of the query
> * @return APR_SUCCESS or APR_ENOTIMPL
> * @deffunc int ap_mpm_query(int query_code, int *result)
> */
> AP_DECLARE(apr_status_t) ap_mpm_query(int query_code, int *result);
>
> Regards,
>
> Rainer
>
>
> Gregory Nicholls schrieb:
>
>> Hi,
>> Is there any decent way of determining from inside a module, what mpm
>> is being used ? I have to share some memory between requests, which
>> needs a shared mem segment if we're using processes/kids but I can get
>> away with regular mem if it thread-based or some other single-process
>> mpm. Don't want to waste a shared-mem segment if I don't have to.
>> Thanks,
>> Greg.
>>
>>
>>
>
>
>
Re: determining MPM type from module ?
Posted by Rainer Jung <ra...@kippdata.de>.
To find out, if the MPM uses multiple threads, you can use the query API
documented in ap_mpm.h (excerpt for Apache 2.0):
#define AP_MPMQ_MAX_DAEMON_USED 1 /* Max # of daemons used so far */
#define AP_MPMQ_IS_THREADED 2 /* MPM can do threading */
#define AP_MPMQ_IS_FORKED 3 /* MPM can do forking */
#define AP_MPMQ_HARD_LIMIT_DAEMONS 4 /* The compiled max # daemons */
#define AP_MPMQ_HARD_LIMIT_THREADS 5 /* The compiled max # threads */
#define AP_MPMQ_MAX_THREADS 6 /* # of threads/child by config */
#define AP_MPMQ_MIN_SPARE_DAEMONS 7 /* Min # of spare daemons */
#define AP_MPMQ_MIN_SPARE_THREADS 8 /* Min # of spare threads */
#define AP_MPMQ_MAX_SPARE_DAEMONS 9 /* Max # of spare daemons */
#define AP_MPMQ_MAX_SPARE_THREADS 10 /* Max # of spare threads */
#define AP_MPMQ_MAX_REQUESTS_DAEMON 11 /* Max # of requests per daemon */
#define AP_MPMQ_MAX_DAEMONS 12 /* Max # of daemons by config */
#define AP_MPMQ_MPM_STATE 13 /* starting, running, stopping */
/**
* Query a property of the current MPM.
* @param query_code One of APM_MPMQ_*
* @param result A location to place the result of the query
* @return APR_SUCCESS or APR_ENOTIMPL
* @deffunc int ap_mpm_query(int query_code, int *result)
*/
AP_DECLARE(apr_status_t) ap_mpm_query(int query_code, int *result);
Regards,
Rainer
Gregory Nicholls schrieb:
> Hi,
> Is there any decent way of determining from inside a module, what mpm
> is being used ? I have to share some memory between requests, which
> needs a shared mem segment if we're using processes/kids but I can get
> away with regular mem if it thread-based or some other single-process
> mpm. Don't want to waste a shared-mem segment if I don't have to.
> Thanks,
> Greg.
>
>
determining MPM type from module ?
Posted by Gregory Nicholls <gn...@bellsouth.net>.
Hi,
Is there any decent way of determining from inside a module, what
mpm is being used ? I have to share some memory between requests, which
needs a shared mem segment if we're using processes/kids but I can get
away with regular mem if it thread-based or some other single-process
mpm. Don't want to waste a shared-mem segment if I don't have to.
Thanks,
Greg.