You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apr.apache.org by Tim Bray <Ti...@Sun.COM> on 2008/10/03 08:40:28 UTC
apr_file_* weirdness on Mac OS X
I'm developing a module which accepts a large number of POST requests,
receives the request bodies, and persists them into ordinary files in
the filesystem. On OS X, there are occasions where the file is
(rarely) transiently not there. The idiom looks like this
status = apr_file_open(&fp, tempname, APR_FOPEN_WRITE |
APR_FOPEN_CREATE,
PERMS, pool);
/* ... do lots of writing ... */
apr_file_flush(fp);
apr_file_close(fp);
status = apr_file_rename(tempname, filename, pool); /* filename
is in the same directory as tempname */
BOOM! strerror() says "no such file or directory"
Then if I go and look, what do you know, the tempfile actually is
there. If I process 500 requests in a big hurry, 100 each from 5
concurrent clients, this will happen maybe half a dozen times, in an
unpredictable order. There are ways to work around this, but still,
it's kinda disturbing. It's either, in decreasing order of
probability (a) me doing something stupid, (b) OS X filesystem
weirdness, (c) an APR bug. apr_file_write_full() doesn't seem to make
a difference.
Same code running in -X single-threaded mode, no problem. Same code
on Solaris, no problem. (Well, yes, there are other weird Solaris
problems with mutexes, but we'll get to those).
-T
Re: apr_file_* weirdness on Mac OS X
Posted by Gordon Henriksen <go...@me.com>.
On 2008-10-03, at 02:40, Tim Bray wrote:
> I'm developing a module which accepts a large number of POST
> requests, receives the request bodies, and persists them into
> ordinary files in the filesystem. On OS X, there are occasions
> where the file is (rarely) transiently not there. The idiom looks
> like this
>
> Same code running in -X single-threaded mode, no problem. Same code
> on Solaris, no problem. (Well, yes, there are other weird Solaris
> problems with mutexes, but we'll get to those).
Sounds like a race. I'd suggest you replace this:
> status = apr_file_open(&fp, tempname, APR_FOPEN_WRITE |
> APR_FOPEN_CREATE,
> PERMS, pool);
with
apr_status_t status;
do {
... generate random filename ...
status = apr_file_open(..., APR_FOPEN_WRITE|APR_FOPEN_CREATE|
APR_FOPEN_EXCL, ...);
} while (APR_IS_EEXIST(status));
in order to hedge against another thread operating on the same
filename concurrently. apr_file_mktemp will handle this detail
transparently for you.
— Gordon
Re: apr_file_* weirdness on Mac OS X
Posted by Bojan Smojver <bo...@rexursive.com>.
On Fri, 2008-10-03 at 08:50 -0700, Tim Bray wrote:
> Um, I know that. But apr_file_mktemp doesn't let you control where
> the temp file goes, right?
If you give it full pathname in the template, doesn't that allow you to
put it where you want it?
--
Bojan
Re: apr_file_* weirdness on Mac OS X
Posted by Ruediger Pluem <rp...@apache.org>.
On 10/03/2008 05:50 PM, Tim Bray wrote:
> Um, I know that. But apr_file_mktemp doesn't let you control where the
> temp file goes, right? And now that I think of it, the file can be
No, of course you can control where the file goes. Just fill the templ
parameter appropriately:
http://apr.apache.org/docs/apr/1.3/group__apr__file__io.html#gc34981f9ff41f85e4377a6dde185fe52
Regards
Rüdiger
Re: apr_file_* weirdness on Mac OS X
Posted by Tim Bray <Ti...@Sun.COM>.
On Oct 3, 2008, at 8:41 AM, Erik Huelsmann wrote:
>
>> Well, this isn't actually a temp file. Once it's written and
>> closed, I
>> want to move it into its final location as quickly as possible, so
>> I thought
>> it would be good to have it in the same directory where it's going
>> to end up
>> and avoid doing a copy. But if this kind of thing is going to
>> happen, you
>> probably have a point. -Tim
>>
> You don't need it in the same directory, you need it on the same
> volume to avoid the copy.
Um, I know that. But apr_file_mktemp doesn't let you control where
the temp file goes, right? And now that I think of it, the file can
be fairly large. Grmph. If it is OS X, this seems like an
*egregious* violation of unix filesystem semantics. -T
Re: apr_file_* weirdness on Mac OS X
Posted by Erik Huelsmann <eh...@gmail.com>.
On Fri, Oct 3, 2008 at 5:15 PM, Tim Bray <Ti...@sun.com> wrote:
> On Oct 3, 2008, at 6:22 AM, Bojan Smojver wrote:
>
>> On Fri, 2008-10-03 at 01:13 -0700, Tim Bray wrote:
>>
>>> i.e. a randomized name in the same directory.
>>
>> Have you tried using apr_file_mktemp()?
>
> Well, this isn't actually a temp file. Once it's written and closed, I
> want to move it into its final location as quickly as possible, so I thought
> it would be good to have it in the same directory where it's going to end up
> and avoid doing a copy. But if this kind of thing is going to happen, you
> probably have a point. -Tim
>
You don't need it in the same directory, you need it on the same
volume to avoid the copy.
Bye,
Erik.
Re: apr_file_* weirdness on Mac OS X
Posted by Tim Bray <Ti...@Sun.COM>.
On Oct 3, 2008, at 6:22 AM, Bojan Smojver wrote:
> On Fri, 2008-10-03 at 01:13 -0700, Tim Bray wrote:
>
>> i.e. a randomized name in the same directory.
>
> Have you tried using apr_file_mktemp()?
Well, this isn't actually a temp file. Once it's written and closed,
I want to move it into its final location as quickly as possible, so I
thought it would be good to have it in the same directory where it's
going to end up and avoid doing a copy. But if this kind of thing is
going to happen, you probably have a point. -Tim
Re: apr_file_* weirdness on Mac OS X
Posted by Bojan Smojver <bo...@rexursive.com>.
On Fri, 2008-10-03 at 01:13 -0700, Tim Bray wrote:
> i.e. a randomized name in the same directory.
Have you tried using apr_file_mktemp()?
--
Bojan
Re: apr_file_* weirdness on Mac OS X
Posted by Tim Bray <Ti...@Sun.COM>.
On Oct 2, 2008, at 11:58 PM, Bojan Smojver wrote:
> On Thu, 2008-10-02 at 23:40 -0700, Tim Bray wrote:
>
>> status = apr_file_open(&fp, tempname, APR_FOPEN_WRITE |
>> APR_FOPEN_CREATE, PERMS, pool);
>
> Where does the tempname come from?
static char * dir_of(apr_pool_t * pool, char * path) {
char * s;
s = apr_pstrdup(pool, path);
*(rindex(s, '/')) = 0;
return s;
}
/* make a temp filename for 'filename' */
static char * temp_filename(apr_pool_t * pool, char * filename) {
char * tempname = dir_of(pool, filename);
return apr_psprintf(pool, "%s/%s%ld-%ld", tempname,
TEMP_PREFIX, arandom(), arandom());
}
i.e. a randomized name in the same directory.
Here's an example: filename is /wf1/data/m-a/a/pub/e/entries/
collection.atom and tempname is /wf1/data/m-a/a/pub/e/entries/T.
493459176-1261197826
-Tim
Re: apr_file_* weirdness on Mac OS X
Posted by Bojan Smojver <bo...@rexursive.com>.
On Thu, 2008-10-02 at 23:40 -0700, Tim Bray wrote:
> status = apr_file_open(&fp, tempname, APR_FOPEN_WRITE |
> APR_FOPEN_CREATE, PERMS, pool);
Where does the tempname come from?
--
Bojan