You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Damien Katz <da...@gmail.com> on 2008/07/02 19:01:59 UTC
Custom file driver for OS X and Windows Re: CouchDB 0.9 and 1.0
So erlang still has some file driver troubles. The old issue of a 2gig
limit is long gone, however two other issues remain:
Erlangs built-in file api for disk sync/flush doesn't work on all
platforms. The main two that I know where it doesn't work are windows
(at least it didn't when a looked over a year ago) and OS X (perhaps
other BSDs?). On OS X, the fix seems to be as simple as passing the
F_FULLFSYNC flag to fcntl. Without this disk sync, there is no way
CouchDB can safely store data. This means we either need the Erlang
folks to fix their drivers, or create our own file driver for the
problem platforms. Blech.
Windows has the same fsync problem, plus it also doesn't pass the
flags to sallow the renaming of our own open files, needed during file
compaction. Again there can be fixed in core Erlang, or our own drivers.
On Jul 2, 2008, at 3:08 AM, Jan Lehnardt wrote:
> Hello everybody,
> this thread is meant to collect missing work items (features and
> bugs) for for our 1.0 release and a discussion about how to split
> them up between 0.9 and 1.0.
>
> Take it away: Damien.
>
> Cheers
> Jan
> --
Re: Custom file driver for OS X and Windows Re: CouchDB 0.9 and 1.0
Posted by Jan Lehnardt <ja...@apache.org>.
On Jul 2, 2008, at 19:01, Damien Katz wrote:
> So erlang still has some file driver troubles. The old issue of a
> 2gig limit is long gone, however two other issues remain:
>
> Erlangs built-in file api for disk sync/flush doesn't work on all
> platforms. The main two that I know where it doesn't work are
> windows (at least it didn't when a looked over a year ago) and OS X
> (perhaps other BSDs?). On OS X, the fix seems to be as simple as
> passing the F_FULLFSYNC flag to fcntl. Without this disk sync, there
> is no way CouchDB can safely store data. This means we either need
> the Erlang folks to fix their drivers, or create our own file driver
> for the problem platforms. Blech.
>
> Windows has the same fsync problem, plus it also doesn't pass the
> flags to sallow the renaming of our own open files, needed during
> file compaction. Again there can be fixed in core Erlang, or our own
> drivers.
I have forwarded this to the Erlang folks to get their insight on this
issue.
I feel the only sensible way to fix this is to fix the file driver.
--
On a related note: At the Erlang eXchange last week I talked to Klacke
Wikström (the author of (d)ets, mnesia, the IO system and a lot of more
parts of core Erlang) and he mentioned a product of his where they
bypass the Erlang file driver for speed reasons completely. I don't
think this is relevant for us now, but we might want to keep in mind
that want to do the same eventually.
Cheers
Jan
--
>
>
> On Jul 2, 2008, at 3:08 AM, Jan Lehnardt wrote:
>
>> Hello everybody,
>> this thread is meant to collect missing work items (features and
>> bugs) for for our 1.0 release and a discussion about how to split
>> them up between 0.9 and 1.0.
>>
>> Take it away: Damien.
>>
>> Cheers
>> Jan
>> --
>
>
Re: Custom file driver for OS X and Windows Re: CouchDB 0.9 and 1.0
Posted by Jan Lehnardt <ja...@apache.org>.
On Jul 2, 2008, at 19:01, Damien Katz wrote:
> So erlang still has some file driver troubles. The old issue of a
> 2gig limit is long gone, however two other issues remain:
>
> Erlangs built-in file api for disk sync/flush doesn't work on all
> platforms. The main two that I know where it doesn't work are
> windows (at least it didn't when a looked over a year ago) and OS X
> (perhaps other BSDs?). On OS X, the fix seems to be as simple as
> passing the F_FULLFSYNC flag to fcntl. Without this disk sync, there
> is no way CouchDB can safely store data. This means we either need
> the Erlang folks to fix their drivers, or create our own file driver
> for the problem platforms. Blech.
I'm not a C hacker and all but would this patch solve the issue on
Darwin / Mac OS X?
I worked solely off documentation:
http://developer.apple.com/documentation/Darwin/Reference/ManPages/man2/fsync.2.html#/
/apple_ref/doc/man/2/fsync
http://developer.apple.com/documentation/Darwin/Reference/ManPages/man2/fcntl.2.html
The diff is against OTP R12B-3:
--- erts/emulator/drivers/unix/unix_efile.c 2008-07-17
03:06:13.000000000 +0200
+++ erts/emulator/drivers/unix/unix_efile_new.c 2008-07-17
03:06:57.000000000 +0200
@@ -44,11 +44,20 @@
#endif
#endif /* _OSE_ */
+#if defined(__APPLE__) && defined(__MACH__) && !defined(__DARWIN__)
+#define DARWIN 1
+#endif
+
+#ifdef DARWIN
+#include <fcntl.h>
+#endif /* DARWIN */
+
#ifdef VXWORKS
#include <ioLib.h>
#include <dosFsLib.h>
#include <nfsLib.h>
#include <sys/stat.h>
+
/*
** Not nice to include usrLib.h as MANY normal variable names get
reported
** as shadowing globals, like 'i' for example.
@@ -818,7 +827,11 @@
undefined fsync
#endif /* VXWORKS */
#else
+#ifdef DARWIN
+ return check_error(fcntl(fd, F_FULLFSYNC), errInfo);
+#else /* all other unix systems */
return check_error(fsync(fd), errInfo);
+#endif /* DARWIN */
#endif /* NO_FSYNC */
}
If that looks okay, I'll forward it to the erlang-patches mailing list
for inclusion.
Cheers
Jan
--