Non-threadsafe functions used in threads? (1.3, 2.0)

[Manoj Kasichainula <ma...@raleigh.ibm.com>]

David Colasurdo, one of the local AS/400 guys, found a few
non-threadsafe functions in use in the core. For example, we use
ap_get_time in http_log.c, which uses ctime. ctime generally uses a
statically allocated buffer, meaning that threads can overwrite each
other's values. Here are the ones that I know or suspect are used
during threaded execution:

ctime
getgrgid
getgrnam (in ap_gname2id; not used in the Apache distribution but it's
          an API call)
gethostbyaddr
gethostbyname (API func: ap_get_virthost_addr)
getpwnam
getpwuid
gmtime
inet_ntoa
localtime
rand
readdir
strtok

Single Unix provides _r functions for many of these, but I don't know
yet if the pthreads spec requires it.

I'm guessing that most of these will have to be dealt with in APR. But
I'm guessing some of these (i.e. ctime) are potential bugs on Windows
in 1.3, too.

-- 
Manoj Kasichainula - manojk@raleigh.ibm.com
IBM, Apache Development

Re: Question. fopen() in ap_pfopen() background ?

[Dean Gaudet <dg...@arctic.org>]

greg's wrong, buffering is necessary.

as to having buffered support in the same API as non-buffered direct fd
access... that could be wrong.  i haven't studied APRs API though.  i tend
to like having both direct fd stuff, and having something like BUFF... but
it works both ways.  (layering and buffering is "fun".)

Dean

On Tue, 12 Oct 1999, Ryan Bloom wrote:

> 
> Okay, I hate to say this, but I will anyway.  I wrote it this way to be
> lazy.  Apache has always had buffering support in it.  If it isn't
> necessary, we never have to use the APR_BUFFERED flag in ap_open.  Cool.
> If we decide we don't want APR_BUFFERED, I will be more than happy to take
> it out.  I just didn't want to remove the possiblity of having BUFFERED
> output in case Apache really wanted it.  I also didn't have the time to
> figure out if Apache needed it.  The solution?  Provide the ability to
> have buffered I/O, and move on.
> 
> I am more than happy to remove buffered I/O support.  If somebody will
> just tell me that Apache doesn't want to support it.
> 
> Ryan
> 
> > I do understand that FILE* can avoid a kernel context switch by
> > buffering in libc, but is there really a big perf gain? Don't most
> > modern operating systems provide plenty of builtin buffering?
> > 
> > i.e. why write it ourselves when the OS will usually do fine?
> > 
> > Hrm... what I'm trying to say is: can anybody characterize the actual
> > different between an fd and a FILE* on a good OS?
> > 
> > Cheers,
> > -g
> > 
> > p.s. obviously, I've never run this perf test... if there really is a
> > big difference, then I'll shut up now :-)
> > 
> > --
> > Greg Stein, http://www.lyra.org/
> > 
> 
> _______________________________________________________________________
> Ryan Bloom		rbb@raleigh.ibm.com
> 4205 S Miami Blvd	
> RTP, NC 27709		It's a beautiful sight to see good dancers 
> 			doing simple steps.  It's a painful sight to
> 			see beginners doing complicated patterns.	
> 
> 

Re: Question. fopen() in ap_pfopen() background ?

[Ryan Bloom <rb...@raleigh.ibm.com>]

Okay, I hate to say this, but I will anyway.  I wrote it this way to be
lazy.  Apache has always had buffering support in it.  If it isn't
necessary, we never have to use the APR_BUFFERED flag in ap_open.  Cool.
If we decide we don't want APR_BUFFERED, I will be more than happy to take
it out.  I just didn't want to remove the possiblity of having BUFFERED
output in case Apache really wanted it.  I also didn't have the time to
figure out if Apache needed it.  The solution?  Provide the ability to
have buffered I/O, and move on.

I am more than happy to remove buffered I/O support.  If somebody will
just tell me that Apache doesn't want to support it.

Ryan

> I do understand that FILE* can avoid a kernel context switch by
> buffering in libc, but is there really a big perf gain? Don't most
> modern operating systems provide plenty of builtin buffering?
> 
> i.e. why write it ourselves when the OS will usually do fine?
> 
> Hrm... what I'm trying to say is: can anybody characterize the actual
> different between an fd and a FILE* on a good OS?
> 
> Cheers,
> -g
> 
> p.s. obviously, I've never run this perf test... if there really is a
> big difference, then I'll shut up now :-)
> 
> --
> Greg Stein, http://www.lyra.org/
> 

_______________________________________________________________________
Ryan Bloom		rbb@raleigh.ibm.com
4205 S Miami Blvd	
RTP, NC 27709		It's a beautiful sight to see good dancers 
			doing simple steps.  It's a painful sight to
			see beginners doing complicated patterns.	

Re: Question. fopen() in ap_pfopen() background ?

[Dean Gaudet <dg...@arctic.org>]

On Tue, 12 Oct 1999, Greg Stein wrote:

> I do understand that FILE* can avoid a kernel context switch by
> buffering in libc, but is there really a big perf gain?

yes.  try timing a program which does read(fd, &c, 1) against another
which does fgetc.

> Don't most
> modern operating systems provide plenty of builtin buffering?

it's irrelevant.  a system call is at least an order of magnitude more
expensive than a function call (which is more expensive than an inline
fgetc style macro).  system calls involve privilege level changes -- which
means verifying the integrity of all the data passed from userland.

Dean

Re: Question. fopen() in ap_pfopen() background ?

[Greg Stein <gs...@lyra.org>]

Ryan Bloom wrote:
> 
> > > This is not true.  APR uses both file descriptors as well as FILE *'s,
> > > depending on what the situation calls for.
> >
> > it would be nice if we could avoid ever using FILE *... a project for
> > someone.
> 
> It wouldn't be hard, we would just need to put our own buffering into APR.
> For the sake of getting APR up and working and stable, I decided to use
> what was available for the first version of APR.  I would think that in
> APR 1.1, we could probably get buffering into APR itself, and remove the
> need for FILE *'s completely.  I agree, it does make somethings ugly to
> have those FILE *'s in there.

I do understand that FILE* can avoid a kernel context switch by
buffering in libc, but is there really a big perf gain? Don't most
modern operating systems provide plenty of builtin buffering?

i.e. why write it ourselves when the OS will usually do fine?

Hrm... what I'm trying to say is: can anybody characterize the actual
different between an fd and a FILE* on a good OS?

Cheers,
-g

p.s. obviously, I've never run this perf test... if there really is a
big difference, then I'll shut up now :-)

--
Greg Stein, http://www.lyra.org/

Re: Question. fopen() in ap_pfopen() background ?

[Ryan Bloom <rb...@raleigh.ibm.com>]

> > This is not true.  APR uses both file descriptors as well as FILE *'s,
> > depending on what the situation calls for.
> 
> it would be nice if we could avoid ever using FILE *... a project for
> someone.

It wouldn't be hard, we would just need to put our own buffering into APR.
For the sake of getting APR up and working and stable, I decided to use
what was available for the first version of APR.  I would think that in
APR 1.1, we could probably get buffering into APR itself, and remove the
need for FILE *'s completely.  I agree, it does make somethings ugly to
have those FILE *'s in there.

Ryan

_______________________________________________________________________
Ryan Bloom		rbb@raleigh.ibm.com
4205 S Miami Blvd	
RTP, NC 27709		It's a beautiful sight to see good dancers 
			doing simple steps.  It's a painful sight to
			see beginners doing complicated patterns.	

Re: Question. fopen() in ap_pfopen() background ?

[Dean Gaudet <dg...@arctic.org>]

On Fri, 8 Oct 1999, Ryan Bloom wrote:

> 
> > Apache 2.0 will use file descriptors instead of (FILE *)'s, since APR does.
> 
> This is not true.  APR uses both file descriptors as well as FILE *'s,
> depending on what the situation calls for.

it would be nice if we could avoid ever using FILE *... a project for
someone.

Dean

Re: Question. fopen() in ap_pfopen() background ?

[Ryan Bloom <rb...@raleigh.ibm.com>]

> Apache 2.0 will use file descriptors instead of (FILE *)'s, since APR does.

This is not true.  APR uses both file descriptors as well as FILE *'s,
depending on what the situation calls for.

Ryan

_______________________________________________________________________
Ryan Bloom		rbb@raleigh.ibm.com
4205 S Miami Blvd	
RTP, NC 27709		It's a beautiful sight to see good dancers 
			doing simple steps.  It's a painful sight to
			see beginners doing complicated patterns.	

Re: Question. fopen() in ap_pfopen() background ?

[Manoj Kasichainula <ma...@io.com>]

Please compose a new message rather than replying to an existing one
if you are not really replying. Otherwise, the thread trees get messed
up. Thanks.

On Fri, Oct 08, 1999 at 12:38:19AM -0700, saeedt@ix.netcom.com wrote:
> 3) The Solaris implementation has a problem with any fopen() which
> opens a file above the 256 file descriptor. HPUX and AIX do not have
> this problem. This means that some fopen's will fail if there are
> many fd's open on the Solaris platform. Isn't open()/read()/write()
> a cleaner implementation for logging ?

Apache 2.0 will use file descriptors instead of (FILE *)'s, since APR does.

-- 
Manoj Kasichainula - manojk at io dot com - http://www.io.com/~manojk/

Re: Question. fopen() in ap_pfopen() background ?

[Dean Gaudet <dg...@arctic.org>]

On Fri, 8 Oct 1999 saeedt@ix.netcom.com wrote:

> 1) What is the problem with fopen() "-a" and on what platform?

it's probably on some ancient unix.

> 2) Why is fopen() used for opening the error log in 
>    src/main/http_log.c file at open_error_log() function?
>    The buffering performance boost that fopen() provides is very minimal
>    in the case of writing to error log.

for historical reasons (fprintf).

> 3) The Solaris implementation has a problem with any fopen() which
>    opens a file above the 256 file descriptor. HPUX and AIX do not have
>    this problem. This means that some fopen's will fail if there are
> many
>    fd's open on the Solaris platform. Isn't open()/read()/write() a 
>    cleaner implementation for logging ?

yes.

Dean

Question. fopen() in ap_pfopen() background ?

[sa...@ix.netcom.com]

Folks, I have a few questions from the developers on this list.
The following code is from src/main/alloc.c file at
ap_pfopen()function.
-----------------------------------------------------------------
    if (*mode == 'a') {
	/* Work around faulty implementations of fopen */
	baseFlag = (*(mode + 1) == '+') ? O_RDWR : O_WRONLY;
	desc = open(name, baseFlag | O_APPEND | O_CREAT,
		    modeFlags);
	if (desc >= 0) {
	    desc = ap_slack(desc, AP_SLACK_LOW);
	    fd = ap_fdopen(desc, mode);
	}
    }
    else {
	fd = fopen(name, mode);
    }
------------------------------------------------------------------
According to the comment some fopen() implementations have a 
problem with fopen() and "-a" option. 

Here are my questions:
1) What is the problem with fopen() "-a" and on what platform?

2) Why is fopen() used for opening the error log in 
   src/main/http_log.c file at open_error_log() function?
   The buffering performance boost that fopen() provides is very minimal
   in the case of writing to error log.

3) The Solaris implementation has a problem with any fopen() which
   opens a file above the 256 file descriptor. HPUX and AIX do not have
   this problem. This means that some fopen's will fail if there are
many
   fd's open on the Solaris platform. Isn't open()/read()/write() a 
   cleaner implementation for logging ?

    Thanks,
    ST.

Re: Non-threadsafe functions used in threads? (1.3, 2.0)

[Dean Gaudet <dg...@arctic.org>]

fwiw, in NSPR they just replaced all the time code with their own code...
which gets rid of the portability nightmare... but that lands you in a
localisation nightmare instead...

i'm like totally surprised that readdir() isn't threadsafe!

Dean

On Mon, 27 Sep 1999, Manoj Kasichainula wrote:

> David Colasurdo, one of the local AS/400 guys, found a few
> non-threadsafe functions in use in the core. For example, we use
> ap_get_time in http_log.c, which uses ctime. ctime generally uses a
> statically allocated buffer, meaning that threads can overwrite each
> other's values. Here are the ones that I know or suspect are used
> during threaded execution:
> 
> ctime
> getgrgid
> getgrnam (in ap_gname2id; not used in the Apache distribution but it's
>           an API call)
> gethostbyaddr
> gethostbyname (API func: ap_get_virthost_addr)
> getpwnam
> getpwuid
> gmtime
> inet_ntoa
> localtime
> rand
> readdir
> strtok
> 
> Single Unix provides _r functions for many of these, but I don't know
> yet if the pthreads spec requires it.
> 
> I'm guessing that most of these will have to be dealt with in APR. But
> I'm guessing some of these (i.e. ctime) are potential bugs on Windows
> in 1.3, too.
> 
> -- 
> Manoj Kasichainula - manojk@raleigh.ibm.com
> IBM, Apache Development
>