You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Richard Wallace <rw...@thewallacepack.net> on 2005/04/09 06:01:18 UTC
OT: Psuedo Async File IO
Hey everyone,
I'm getting to a point with my smtp protocol provider that I need to
start thinking about file IO. Since there is no support for
non-blocking IO in Java yet I need something that fakes it. I was
wondering if anyone had any experience with any existing components that
do this. If there aren't any really good ones that anyone can recommend
I'll probably wind up writing something on my own.
I'm kind of thinking it should basically be a component that uses a
single thread to monitor a queue for IO requests and then uses something
like the IoHandler in MINA to let clients know when reads/writes have
started/stopped/failed.
Any input on existing components or what would be necessary to maybe
repurpose parts of MINA for this task would be great.
Thanks,
Rich
Re: OT: Psuedo Async File IO
Posted by Richard Wallace <rw...@thewallacepack.net>.
Trustin Lee wrote:
> Hi Richard,
>
>
>>I'm getting to a point with my smtp protocol provider that I need to
>>start thinking about file IO. Since there is no support for
>>non-blocking IO in Java yet I need something that fakes it. I was
>>wondering if anyone had any experience with any existing components that
>>do this. If there aren't any really good ones that anyone can recommend
>>I'll probably wind up writing something on my own.
>
>
> How about Coconut AIO or stuff from IBM developerWorks? or,, you
> could implement your own one using Doug Lea's concurrent package
> easily.
>
Never heard of Coconut before. But it looks like its not being actively
developed anymore. I'll take a closer look and maybe email them. The
only thing I found at IBM is this: http://alphaworks.ibm.com/tech/aio4j.
That what you meant? I saw that but wasn't sure of the licensing on
it. Using Doug Lea's concurrent package is definitely a good idea if
I do go the custom route.
>
>>I'm kind of thinking it should basically be a component that uses a
>>single thread to monitor a queue for IO requests and then uses something
>>like the IoHandler in MINA to let clients know when reads/writes have
>>started/stopped/failed.
>
>
> It would be nice if MINA can support file I/O, but I don't see any
> easy way to do it retaining current API. Of course we could read one
> file sequentially as requests and write the other file sequentially as
> responses. WDYT?
>
Actually, I wasn't thinking too much of actually changing MINA as maybe
taking some of the interfaces and modifying them to make them more
appropriate for file IO. I could probably use a good deal of the things
in the o.a.mina.io package and the filter subpackage. The filters could
be handy for adding things like compression and encryption. Then using
something like the ProtocolHandler but without the
session{Opened|Closed|Idle}() methods for notifying the caller that data
has been written or read. Sound reasonable?
Rich
Re: OT: Psuedo Async File IO
Posted by Trustin Lee <tr...@gmail.com>.
Hi Richard,
> I'm getting to a point with my smtp protocol provider that I need to
> start thinking about file IO. Since there is no support for
> non-blocking IO in Java yet I need something that fakes it. I was
> wondering if anyone had any experience with any existing components that
> do this. If there aren't any really good ones that anyone can recommend
> I'll probably wind up writing something on my own.
How about Coconut AIO or stuff from IBM developerWorks? or,, you
could implement your own one using Doug Lea's concurrent package
easily.
> I'm kind of thinking it should basically be a component that uses a
> single thread to monitor a queue for IO requests and then uses something
> like the IoHandler in MINA to let clients know when reads/writes have
> started/stopped/failed.
It would be nice if MINA can support file I/O, but I don't see any
easy way to do it retaining current API. Of course we could read one
file sequentially as requests and write the other file sequentially as
responses. WDYT?
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
Re: OT: Psuedo Async File IO
Posted by Berin Loritsch <bl...@d-haven.org>.
Richard Wallace wrote:
> It's a definitely possibility. I'll probably wind up still creating
> some kind of simple wrapper framework so there can be a
> pseudo-non-blocking implementation for platforms not supported by Matts
> Java NBIO package (since it uses JNI). Then you can easily switch your
> implementation over to the Java non-blocking file IO whenever they get
> around to adding it in Java 1.6 or 1.7 or 1.8 or whenever.
>
> Rich
FYI, Matt Welsh's NBIO is close to the same thing as Java NIO. There is
still no direct support for asynchronous File IO in that system.
The way the SEDA team worked around this issue is by using a
RandomAccessFile, and by having a dedicated queue that would process
messages placed into the queue. The stage processing the messages would
then perform the actions within its thread, and then send along any
messages as a result from the actions.
Essentially if you want async file IO in the current Java runtimes, you
have to farm your file access out to a separate thread. Keep in mind
that the separate thread doesn't have to be strictly set aside for file
IO, it can be used for any async command that runs in the background.
Re: OT: Psuedo Async File IO
Posted by Richard Wallace <rw...@thewallacepack.net>.
It's a definitely possibility. I'll probably wind up still creating
some kind of simple wrapper framework so there can be a
pseudo-non-blocking implementation for platforms not supported by Matts
Java NBIO package (since it uses JNI). Then you can easily switch your
implementation over to the Java non-blocking file IO whenever they get
around to adding it in Java 1.6 or 1.7 or 1.8 or whenever.
Rich
Emmanuel Lecharny wrote:
> Yep, you're right. I answer a little bit too fast, thinking that you
> needed something simpler. The point is that FileLocks just give you a
> lock to files shared with an external process, not with other threads in
> the same JVM.
>
> What about http://www.eecs.harvard.edu/~mdw/proj/java-nbio/ ?
>
> On Sat, 2005-04-09 at 12:55 -0700, Richard Wallace wrote:
>
>>The try lock method is for getting an exclusive lock on a file. It has
>>nothing to do with the actual reading or writing of the file, other than
>>disallowing other threads or processes from opening the file and reading
>>or writing it.
>>
>>Rich
>>
>>Emmanuel Lecharny wrote:
>>
>>>Hi,
>>>
>>>can't you use the FileChannel's tryLock() method?
>>>(http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/FileChannel.html#tryLock(long,%20long,%20boolean))
>>>
>>>
>>>
>>>On Fri, 2005-04-08 at 22:38 -0700, Richard Wallace wrote:
>>>
>>>
>>>>Java NIO doesn't support non-blocking file IO. Only non-blocking socket
>>>>IO. If you look you'll see that the FileChannel does not extend the
>>>>SelectableChannel and so cannot be set to non-blocking mode or
>>>>registered with a selector.
>>>>
>>>>Ben Walding wrote:
>>>>
>>>>
>>>>>Am I confused, or is Java NIO not what you're looking for?
>>>>>
>>>>>http://java.sun.com/j2se/1.4.2/docs/guide/nio/
>>>>>
>>>>>Mike Spille wrote some wrappers around NIO, but I'm not sure how active
>>>>>they are:
>>>>>
>>>>>http://sourceforge.net/projects/pyrasun/
>>>>>
>>>>>Richard Wallace wrote:
>>>>>
>>>>>
>>>>>
>>>>>>Hey everyone,
>>>>>>
>>>>>>I'm getting to a point with my smtp protocol provider that I need to
>>>>>>start thinking about file IO. Since there is no support for
>>>>>>non-blocking IO in Java yet I need something that fakes it. I was
>>>>>>wondering if anyone had any experience with any existing components
>>>>>>that do this. If there aren't any really good ones that anyone can
>>>>>>recommend I'll probably wind up writing something on my own.
>>>>>>
>>>>>>I'm kind of thinking it should basically be a component that uses a
>>>>>>single thread to monitor a queue for IO requests and then uses
>>>>>>something like the IoHandler in MINA to let clients know when
>>>>>>reads/writes have started/stopped/failed.
>>>>>>
>>>>>>Any input on existing components or what would be necessary to maybe
>>>>>>repurpose parts of MINA for this task would be great.
>>>>>>
>>>>>>Thanks,
>>>>>>Rich
>>>>>>
>>>>>
>>>
>
>
Re: OT: Psuedo Async File IO
Posted by Emmanuel Lecharny <el...@apache.org>.
Yep, you're right. I answer a little bit too fast, thinking that you
needed something simpler. The point is that FileLocks just give you a
lock to files shared with an external process, not with other threads in
the same JVM.
What about http://www.eecs.harvard.edu/~mdw/proj/java-nbio/ ?
On Sat, 2005-04-09 at 12:55 -0700, Richard Wallace wrote:
> The try lock method is for getting an exclusive lock on a file. It has
> nothing to do with the actual reading or writing of the file, other than
> disallowing other threads or processes from opening the file and reading
> or writing it.
>
> Rich
>
> Emmanuel Lecharny wrote:
> > Hi,
> >
> > can't you use the FileChannel's tryLock() method?
> > (http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/FileChannel.html#tryLock(long,%20long,%20boolean))
> >
> >
> >
> > On Fri, 2005-04-08 at 22:38 -0700, Richard Wallace wrote:
> >
> >>Java NIO doesn't support non-blocking file IO. Only non-blocking socket
> >>IO. If you look you'll see that the FileChannel does not extend the
> >>SelectableChannel and so cannot be set to non-blocking mode or
> >>registered with a selector.
> >>
> >>Ben Walding wrote:
> >>
> >>>Am I confused, or is Java NIO not what you're looking for?
> >>>
> >>>http://java.sun.com/j2se/1.4.2/docs/guide/nio/
> >>>
> >>>Mike Spille wrote some wrappers around NIO, but I'm not sure how active
> >>>they are:
> >>>
> >>>http://sourceforge.net/projects/pyrasun/
> >>>
> >>>Richard Wallace wrote:
> >>>
> >>>
> >>>>Hey everyone,
> >>>>
> >>>>I'm getting to a point with my smtp protocol provider that I need to
> >>>>start thinking about file IO. Since there is no support for
> >>>>non-blocking IO in Java yet I need something that fakes it. I was
> >>>>wondering if anyone had any experience with any existing components
> >>>>that do this. If there aren't any really good ones that anyone can
> >>>>recommend I'll probably wind up writing something on my own.
> >>>>
> >>>>I'm kind of thinking it should basically be a component that uses a
> >>>>single thread to monitor a queue for IO requests and then uses
> >>>>something like the IoHandler in MINA to let clients know when
> >>>>reads/writes have started/stopped/failed.
> >>>>
> >>>>Any input on existing components or what would be necessary to maybe
> >>>>repurpose parts of MINA for this task would be great.
> >>>>
> >>>>Thanks,
> >>>>Rich
> >>>>
> >>>
> >
> >
>
Re: OT: Psuedo Async File IO
Posted by Richard Wallace <rw...@thewallacepack.net>.
The try lock method is for getting an exclusive lock on a file. It has
nothing to do with the actual reading or writing of the file, other than
disallowing other threads or processes from opening the file and reading
or writing it.
Rich
Emmanuel Lecharny wrote:
> Hi,
>
> can't you use the FileChannel's tryLock() method?
> (http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/FileChannel.html#tryLock(long,%20long,%20boolean))
>
>
>
> On Fri, 2005-04-08 at 22:38 -0700, Richard Wallace wrote:
>
>>Java NIO doesn't support non-blocking file IO. Only non-blocking socket
>>IO. If you look you'll see that the FileChannel does not extend the
>>SelectableChannel and so cannot be set to non-blocking mode or
>>registered with a selector.
>>
>>Ben Walding wrote:
>>
>>>Am I confused, or is Java NIO not what you're looking for?
>>>
>>>http://java.sun.com/j2se/1.4.2/docs/guide/nio/
>>>
>>>Mike Spille wrote some wrappers around NIO, but I'm not sure how active
>>>they are:
>>>
>>>http://sourceforge.net/projects/pyrasun/
>>>
>>>Richard Wallace wrote:
>>>
>>>
>>>>Hey everyone,
>>>>
>>>>I'm getting to a point with my smtp protocol provider that I need to
>>>>start thinking about file IO. Since there is no support for
>>>>non-blocking IO in Java yet I need something that fakes it. I was
>>>>wondering if anyone had any experience with any existing components
>>>>that do this. If there aren't any really good ones that anyone can
>>>>recommend I'll probably wind up writing something on my own.
>>>>
>>>>I'm kind of thinking it should basically be a component that uses a
>>>>single thread to monitor a queue for IO requests and then uses
>>>>something like the IoHandler in MINA to let clients know when
>>>>reads/writes have started/stopped/failed.
>>>>
>>>>Any input on existing components or what would be necessary to maybe
>>>>repurpose parts of MINA for this task would be great.
>>>>
>>>>Thanks,
>>>>Rich
>>>>
>>>
>
>
Re: OT: Psuedo Async File IO
Posted by Emmanuel Lecharny <el...@apache.org>.
Hi,
can't you use the FileChannel's tryLock() method?
(http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/FileChannel.html#tryLock(long,%20long,%20boolean))
On Fri, 2005-04-08 at 22:38 -0700, Richard Wallace wrote:
> Java NIO doesn't support non-blocking file IO. Only non-blocking socket
> IO. If you look you'll see that the FileChannel does not extend the
> SelectableChannel and so cannot be set to non-blocking mode or
> registered with a selector.
>
> Ben Walding wrote:
> > Am I confused, or is Java NIO not what you're looking for?
> >
> > http://java.sun.com/j2se/1.4.2/docs/guide/nio/
> >
> > Mike Spille wrote some wrappers around NIO, but I'm not sure how active
> > they are:
> >
> > http://sourceforge.net/projects/pyrasun/
> >
> > Richard Wallace wrote:
> >
> >> Hey everyone,
> >>
> >> I'm getting to a point with my smtp protocol provider that I need to
> >> start thinking about file IO. Since there is no support for
> >> non-blocking IO in Java yet I need something that fakes it. I was
> >> wondering if anyone had any experience with any existing components
> >> that do this. If there aren't any really good ones that anyone can
> >> recommend I'll probably wind up writing something on my own.
> >>
> >> I'm kind of thinking it should basically be a component that uses a
> >> single thread to monitor a queue for IO requests and then uses
> >> something like the IoHandler in MINA to let clients know when
> >> reads/writes have started/stopped/failed.
> >>
> >> Any input on existing components or what would be necessary to maybe
> >> repurpose parts of MINA for this task would be great.
> >>
> >> Thanks,
> >> Rich
> >>
> >
>
Re: Minor Documentation Quibble
Posted by Emmanuel Lecharny <el...@apache.org>.
outch! We know it, the XML doc is up to date but the site has to be re-
deployed. There is already a JIRA issue about it.
The ApacheDS is not anymore in incubator, it has quit its nest, but the
first flight is difficult !
On Sat, 2005-04-09 at 16:34 +1000, Chris Betts wrote:
> Hi Folks,
>
> following Emmanuel's advice, I ditched the 0.8 downloads and
> checked out the head, and it all seems to build nicely (yay!).
>
> one thing though: the documentation at
> http://directory.apache.org/svn.html is out of date; it refers to the
> svn directory being
> https://svn.apache.org/repos/asf/incubator/directory - apparently this
> doesn't exist any more (as the project is all growed up and no longer
> in the incubator?) - do I guessed that I just remove the '/directory'
> in the URL, and that seemed to work.
>
> cheers,
>
> Chris
>
>
Minor Documentation Quibble
Posted by Chris Betts <ch...@pegacat.com>.
Hi Folks,
following Emmanuel's advice, I ditched the 0.8 downloads and
checked out the head, and it all seems to build nicely (yay!).
one thing though: the documentation at
http://directory.apache.org/svn.html is out of date; it refers to the
svn directory being
https://svn.apache.org/repos/asf/incubator/directory - apparently this
doesn't exist any more (as the project is all growed up and no longer
in the incubator?) - do I guessed that I just remove the '/directory'
in the URL, and that seemed to work.
cheers,
Chris
Re: OT: Psuedo Async File IO
Posted by Richard Wallace <rw...@thewallacepack.net>.
Java NIO doesn't support non-blocking file IO. Only non-blocking socket
IO. If you look you'll see that the FileChannel does not extend the
SelectableChannel and so cannot be set to non-blocking mode or
registered with a selector.
Ben Walding wrote:
> Am I confused, or is Java NIO not what you're looking for?
>
> http://java.sun.com/j2se/1.4.2/docs/guide/nio/
>
> Mike Spille wrote some wrappers around NIO, but I'm not sure how active
> they are:
>
> http://sourceforge.net/projects/pyrasun/
>
> Richard Wallace wrote:
>
>> Hey everyone,
>>
>> I'm getting to a point with my smtp protocol provider that I need to
>> start thinking about file IO. Since there is no support for
>> non-blocking IO in Java yet I need something that fakes it. I was
>> wondering if anyone had any experience with any existing components
>> that do this. If there aren't any really good ones that anyone can
>> recommend I'll probably wind up writing something on my own.
>>
>> I'm kind of thinking it should basically be a component that uses a
>> single thread to monitor a queue for IO requests and then uses
>> something like the IoHandler in MINA to let clients know when
>> reads/writes have started/stopped/failed.
>>
>> Any input on existing components or what would be necessary to maybe
>> repurpose parts of MINA for this task would be great.
>>
>> Thanks,
>> Rich
>>
>
Re: OT: Psuedo Async File IO
Posted by Ben Walding <be...@walding.com>.
Am I confused, or is Java NIO not what you're looking for?
http://java.sun.com/j2se/1.4.2/docs/guide/nio/
Mike Spille wrote some wrappers around NIO, but I'm not sure how active
they are:
http://sourceforge.net/projects/pyrasun/
Richard Wallace wrote:
> Hey everyone,
>
> I'm getting to a point with my smtp protocol provider that I need to
> start thinking about file IO. Since there is no support for
> non-blocking IO in Java yet I need something that fakes it. I was
> wondering if anyone had any experience with any existing components
> that do this. If there aren't any really good ones that anyone can
> recommend I'll probably wind up writing something on my own.
>
> I'm kind of thinking it should basically be a component that uses a
> single thread to monitor a queue for IO requests and then uses
> something like the IoHandler in MINA to let clients know when
> reads/writes have started/stopped/failed.
>
> Any input on existing components or what would be necessary to maybe
> repurpose parts of MINA for this task would be great.
>
> Thanks,
> Rich
>