You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@mina.apache.org by Benjamin WATINE - Logixys <b....@logixys.fr> on 2009/03/02 20:03:26 UTC
Re: File transfer
Thomas Harning Jr. a écrit :
> On Tue, Feb 24, 2009 at 6:07 PM, Emmanuel Lecharny <el...@apache.org> wrote:
>> Benjamin WATINE - Logixys wrote:
>>> Hi,
>>>
>>> I plan to use mina to transfer files between hosts. I will have to
>>> transfer a lot of files, some very big (2 GB). What would be mina
>>> performance on this task ?
>>> I'm going to try to make a ProtocolCodecFilter for that. May be this
>>> module exists already ?
>>> Any hint on how to deal with big files and IoBuffer ?
>> What about using the FtpServer project for that purpose ?
>>
> For very large files, I would suggest taking advantage of IoBuffers
> wrapping NIO buffers that are MemMapped views of the file. If you run
> into a file that cannot be memmapped and for the case where the file
> needs to be sent in chunks... I would setup a filter/codec that
> progressively sends out chunks as completion events are made.
>
> As an aside, does anyone know if Java provides for emulating MemMapped
> files when the underlying OS says no (ex: certain filesystems in Linux
> don't implement memmapping)
First, sorry for the late answer, I didn't find time to work on it before.
So you mean that just using IoBuffer (in direct mode I suppose) will do
the job ? Great news !
But I'm not familiar with java buffers. So I'm trying to send a simple
file to a server, and write it back to a file.
The client side (that send the file) seems to work fine, but I don't
know how to implement the server side (that receive the stream, and
write it to fs using IoBuffer.
My code on the client side :
FileInputStream fis = new FileInputStream(path);
FileChannel fc = fis.getChannel();
ByteBuffer fb = ByteBuffer.allocate( 1024 * 15 );
fc.read(fb);
fb.flip();
IoBuffer fiob = IoBuffer.wrap(fb);
session.write(fiob);
Is it a good way to do this ?
For the server, in the handler.messageReceived() function, I try to cast
the message to an IoBuffer, but it's probably not a good way. It returns
cast error from String (why ?) to IoBuffer.
How can I do this ? A ByteBuffer that I wrap in a IoBuffer ? A piece of
code showing this would be greatly appreciated :)
Thanks Emmanuel for the pointer to FtpServer, but it's too heavy for my
project. I need a very simple filetransfer implementation.
Regards,
Ben
Re: File transfer
Posted by Erix Yao <er...@gmail.com>.
I prefer more Javadoc to be added . A lot of methods are marked as "//TODO
add doc", which makes me hard to understand the purpose of the method or
class.
2009/3/3 Emmanuel Lecharny <el...@apache.org>
> Erix Yao wrote:
>
>> After reading the source code of Mina, I found a lot of features that I
>> didn't know before. I think somebody can write a documentation to uncover
>> the usage of Mina.
>>
>>
> Sure ! And also add tests, Javadoc and samples :)
>
> Everone is very welcome to participate !
>
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
>
>
>
Re: File transfer
Posted by Emmanuel Lecharny <el...@apache.org>.
Erix Yao wrote:
> After reading the source code of Mina, I found a lot of features that I
> didn't know before. I think somebody can write a documentation to uncover
> the usage of Mina.
>
Sure ! And also add tests, Javadoc and samples :)
Everone is very welcome to participate !
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org
Re: File transfer
Posted by Ashish <pa...@gmail.com>.
On Tue, Mar 3, 2009 at 12:03 PM, Erix Yao <er...@gmail.com> wrote:
> After reading the source code of Mina, I found a lot of features that I
> didn't know before. I think somebody can write a documentation to uncover
> the usage of Mina.
Any specific use cases that you think can be useful. I can take a note
of these and update the wiki.
Software Programming is an Art and you can use same thing in so many
different ways.
thanks
ashish
Re: File transfer
Posted by Erix Yao <er...@gmail.com>.
After reading the source code of Mina, I found a lot of features that I
didn't know before. I think somebody can write a documentation to uncover
the usage of Mina.
2009/3/3 Emmanuel Lecharny <el...@apache.org>
> Erix Yao wrote:
>
>> How about directly transfer thee file? In MINA, after call
>> session.write(file) , it will check whether the messsage you have written
>> is
>> a file and use fileChannel.transfterTo() to transfer the file.
>>
>>
> For the transmission of a File, you're right. I missed this option...
>
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
>
>
>
Re: File transfer
Posted by Emmanuel Lecharny <el...@apache.org>.
Erix Yao wrote:
> How about directly transfer thee file? In MINA, after call
> session.write(file) , it will check whether the messsage you have written is
> a file and use fileChannel.transfterTo() to transfer the file.
>
For the transmission of a File, you're right. I missed this option...
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org
Re: File transfer
Posted by Erix Yao <er...@gmail.com>.
How about directly transfer thee file? In MINA, after call
session.write(file) , it will check whether the messsage you have written is
a file and use fileChannel.transfterTo() to transfer the file.
2009/3/3 Emmanuel Lecharny <el...@apache.org>
> Thomas Harning wrote:
>
>> On Mar 2, 2009, at 2:33 PM, Emmanuel Lecharny wrote:
>>
>>> You will create huge buffers. If your files are 2Gb big, it will suck up
>>> all your memory. Just split the file in small chunks, it works exactly the
>>> same way.
>>>
>> Would it? I would think that the IO system would behave as a
>> memory-mapped file... paging data in as it needed it and out as it's done...
>>
> The internal buffers in MINA ate wrapped around ByteBuffer. This is what is
> transmitted ultimately.
>
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
>
>
>
Re: File transfer
Posted by Emmanuel Lecharny <el...@apache.org>.
Thomas Harning wrote:
> On Mar 2, 2009, at 2:33 PM, Emmanuel Lecharny wrote:
>> You will create huge buffers. If your files are 2Gb big, it will suck
>> up all your memory. Just split the file in small chunks, it works
>> exactly the same way.
> Would it? I would think that the IO system would behave as a
> memory-mapped file... paging data in as it needed it and out as it's
> done...
The internal buffers in MINA ate wrapped around ByteBuffer. This is what
is transmitted ultimately.
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org
Re: File transfer
Posted by Thomas Harning <ha...@gmail.com>.
On Mar 2, 2009, at 2:33 PM, Emmanuel Lecharny wrote:
> You will create huge buffers. If your files are 2Gb big, it will
> suck up all your memory. Just split the file in small chunks, it
> works exactly the same way.
Would it? I would think that the IO system would behave as a memory-
mapped file... paging data in as it needed it and out as it's done...
Mem-mapped file would take up address space... so on 32-bit machine,
you would have the problem of using up address space if you allocate
big 2GB buffers...
Example:
reading file into buffers
loop over file - read 64 MB buffers
send buffer
- have to wait for file to be read entirely for each chunk
- 64MB of actual memory will be in use for the phase, potentially
getting paged out to disk
reading file using mem-buffers
loop over file - mapping 64 MB buffers
send buffer
- address space for 64MB is used up, but is purely disk-backed
- if "paging" were to occur, nothing would write to disk since OS is
using file-backed memory already
..... my argument falls apart if Java doesn't use memory-mapped files
for its mappedbytebuffers...
Re: File transfer
Posted by Emmanuel Lecharny <el...@apache.org>.
> So you mean that just using IoBuffer (in direct mode I suppose) will
> do the job ? Great news !
> But I'm not familiar with java buffers.
Time to read Java NIO from Ron Hitchens ...
> So I'm trying to send a simple file to a server, and write it back to
> a file.
>
> The client side (that send the file) seems to work fine, but I don't
> know how to implement the server side (that receive the stream, and
> write it to fs using IoBuffer.
>
> My code on the client side :
>
> FileInputStream fis = new FileInputStream(path);
> FileChannel fc = fis.getChannel();
> ByteBuffer fb = ByteBuffer.allocate( 1024 * 15 );
> fc.read(fb);
> fb.flip();
> IoBuffer fiob = IoBuffer.wrap(fb);
> session.write(fiob);
>
> Is it a good way to do this ?
You will create huge buffers. If your files are 2Gb big, it will suck up
all your memory. Just split the file in small chunks, it works exactly
the same way.
>
> For the server, in the handler.messageReceived() function, I try to
> cast the message to an IoBuffer, but it's probably not a good way. It
> returns cast error from String (why ?) to IoBuffer.
You propably have a ProtocolCodec in the chain which does some kind of
transformation.
Be aware that the file will be received in chunks too, whatever you do.
In any case, *don't* store everything into a big IoBuffer before storing
the data in a file. Just open the file when you open the session, and
store the bytes as they arrive in the open file.
You will also have to define a minimal protocol : the first bytes you
will receive will probably be the file name, or something like that. The
ProtocolDecoder will have to handle this.
> How can I do this ? A ByteBuffer that I wrap in a IoBuffer ? A piece
> of code showing this would be greatly appreciated :)
IoBuffer alread wrap a ByteBuffer. You may have a loo, at the "Imagine"
sample : it's documented on the web site :
http://mina.apache.org/tutorial-on-protocolcodecfilter-for-mina-2x.html
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org
Re: File transfer
Posted by Thomas Harning <ha...@gmail.com>.
On Mar 2, 2009, at 2:03 PM, Benjamin WATINE - Logixys wrote:
> Thomas Harning Jr. a écrit :
>> For very large files, I would suggest taking advantage of IoBuffers
>> wrapping NIO buffers that are MemMapped views of the file. If you
>> run
>> into a file that cannot be memmapped and for the case where the file
>> needs to be sent in chunks... I would setup a filter/codec that
>> progressively sends out chunks as completion events are made.
>> As an aside, does anyone know if Java provides for emulating
>> MemMapped
>> files when the underlying OS says no (ex: certain filesystems in
>> Linux
>> don't implement memmapping)
>
> First, sorry for the late answer, I didn't find time to work on it
> before.
>
> So you mean that just using IoBuffer (in direct mode I suppose) will
> do the job ? Great news !
> But I'm not familiar with java buffers. So I'm trying to send a
> simple file to a server, and write it back to a file.
>
> The client side (that send the file) seems to work fine, but I don't
> know how to implement the server side (that receive the stream, and
> write it to fs using IoBuffer.
>
> My code on the client side :
>
> FileInputStream fis = new FileInputStream(path);
> FileChannel fc = fis.getChannel();
> ByteBuffer fb = ByteBuffer.allocate( 1024 * 15 );
> fc.read(fb);
> fb.flip();
> IoBuffer fiob = IoBuffer.wrap(fb);
> session.write(fiob);
>
> Is it a good way to do this ?
I was intending that you use the file in memory-mapped fashion and
directly send that out. That way you don't have to manually pipe the
data from file to buffer to Mina.
MappedByteBuffer is what you'd want to use. Example call would be:
MappedByteBuffer fb = fc.map(FileChannel.MapMode.READ_ONLY, 0,
length-of-file); /* If file longer than address space, map it
progressively */
Not 100% sure how we'd handle writing this out and managing file
freeing... my guess is that you'd have to add some sort of on-sent
event.
>
>
> For the server, in the handler.messageReceived() function, I try to
> cast the message to an IoBuffer, but it's probably not a good way.
> It returns cast error from String (why ?) to IoBuffer.
> How can I do this ? A ByteBuffer that I wrap in a IoBuffer ? A piece
> of code showing this would be greatly appreciated :)
Haven't written either of these code items yet... its on the roadmap
for a product I'm working on... but way down the line.
>
>
> Thanks Emmanuel for the pointer to FtpServer, but it's too heavy for
> my project. I need a very simple filetransfer implementation.