You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Lou Parisi <Lo...@photomask.com> on 2009/05/05 00:47:23 UTC
Large File Transfer
I have made some sample applications using ActiveMQ BlobMessage and
everything is working. I have a few questions regarding the best
implementation.
1. Since these are large files is there any built in capability to resume
the file transfer from point of failure if a failure occurs while receiving
the file at the destination?
I have some questions from our infrastructure team asking why the file must
be sent by http or ftp and why not send by direct tcp socket connection like
some peer-to-peer connection between sender and consumer.
1. Is it possible to do something like this using the BlobMessage framework
2. Are there drawbacks to such an implementation that directed you to using
the Jetty HTTP server for transfer?
Thanks for any feedback or comments.
--
View this message in context: http://www.nabble.com/Large-File-Transfer-tp23378044p23378044.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: Large File Transfer
Posted by Sachindra Nath <Sa...@efi.com>.
Hi Lou,
My two cents on this. The activemq team possibly chose http/ftp over raw tcp sockets to get around
1. firewall/proxy setups that may be possible between an ActiveMQ broker and the file server. Using pre-defined ports can allow you to control the traffic.
2. A higher level protocol like http/ftp would ensure that you could use the broker and the file server on discrete systems (different architectures, different languages without caring for the byte ordering issues that are prevalent in raw tcp connections).
Regards
Sachindra
-----Original Message-----
From: Lou Parisi [mailto:Lou.Parisi@photomask.com]
Sent: Tuesday, May 05, 2009 4:17 AM
To: users@activemq.apache.org
Subject: Large File Transfer
I have made some sample applications using ActiveMQ BlobMessage and
everything is working. I have a few questions regarding the best
implementation.
1. Since these are large files is there any built in capability to resume
the file transfer from point of failure if a failure occurs while receiving
the file at the destination?
I have some questions from our infrastructure team asking why the file must
be sent by http or ftp and why not send by direct tcp socket connection like
some peer-to-peer connection between sender and consumer.
1. Is it possible to do something like this using the BlobMessage framework
2. Are there drawbacks to such an implementation that directed you to using
the Jetty HTTP server for transfer?
Thanks for any feedback or comments.
--
View this message in context: http://www.nabble.com/Large-File-Transfer-tp23378044p23378044.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Large File Transfer
Posted by Dejan Bosanac <de...@nighttale.net>.
Hi,
I was referring to blob messages as natural way to exchange large files
http://activemq.apache.org/blob-messages.html
In that scenario you don't send message payloads to the broker but to the
external HTTP or FTP server (those two are currently implemented).
You can find the implementation of the current stuff here
http://fisheye6.atlassian.com/browse/activemq/trunk/activemq-core/src/main/java/org/apache/activemq/blob
and you can try adding needed functionality (or create a new one).
<http://activemq.apache.org/blob-messages.html>Cheers
--
Dejan Bosanac - http://twitter.com/dejanb
Open Source Integration - http://fusesource.com/
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Thu, Jun 10, 2010 at 4:07 PM, kevinle92618 <ke...@yahoo.com>wrote:
>
> Hi Dejan,
>
> Currently we have a hard requirement to support large file transfer resume.
> We're very keen on using ActiveMQ, along with Camel, thus we must implement
> this "resume" feature. I'm in the investigation stage but I was thinking
> of
> implement this the follow way:
>
> Following Lou's suggestion, the producer will break down the large file to
> be sent into chunks and send them to the broker (ActiveMQ) one at a time
> until the entire file is sent. In the event of a file transfer
> interruption, the producer will detect this event and when connection is
> restore, it will only send the remainder to the broker (with a resume flag
> or something like that). The broker will then continue to receive the rest
> of the file knowing that it will have to concatenate the files together
> before putting it on the queue for consumption.
>
> I'm not sure if this is the best solution or not. If you have a better
> suggestion please feel free to let me know. My personal email address is
> kevinle92618@yahoo.com
>
>
>
> Dejan Bosanac wrote:
> >
> > Hi,
> >
> > just to confirm that ActiveMQ comes with HTTP and FTP strategies for
> > upload
> > and download, that doesn't support resume. This can be implemented as a
> > separate strategy. All contributions are welcomed.
> >
> > Cheers
> > --
> > Dejan Bosanac - http://twitter.com/dejanb
> >
> > Open Source Integration - http://fusesource.com/
> > ActiveMQ in Action - http://www.manning.com/snyder/
> > Blog - http://www.nighttale.net
> >
> >
> > On Thu, Jun 10, 2010 at 12:28 AM, kevinle92618
> > <ke...@yahoo.com>wrote:
> >
> >>
> >> Thanks Lou. I was afraid of this. If I understand you correctly,
> >> ActiveMQ
> >> does NOT support resume transfer. However, you were able to get around
> >> that
> >> problem by implementing the snippet codes you mentioned in your previous
> >> post right? Do you mind posting the codes for the TransferStatus class
> >> as
> >> well. Many thanks.
> >>
> >>
> >> Lou Parisi wrote:
> >> >
> >> >
> >> > kevinle92618 wrote:
> >> >>
> >> >> I don't have answers to your questions. I'm interested in knowing if
> >> >> ActiveMQ has the "resume" transfer capability for large file. Please
> >> >> reply if you have the answer. Thanks.
> >> >>
> >> >
> >> > It has been a while since I worked on this but I did get the "resume"
> >> > working. As I recall, I kept track of the number of bytes transferred
> >> and
> >> > if there was a failure I just had to resume and read past that many
> >> bytes
> >> > and restart transfer. Here is a snippet from some sample code I
> >> created
> >> > for testing.
> >> >
> >> >
> >> > private static void startTransfer() throws Exception {
> >> > RandomAccessFile outFile = null;
> >> > InputStream in = null;
> >> > byte[] buf = new byte[bufSize];
> >> > int len = 0;
> >> > URLConnection conn = null;
> >> >
> >> > // len == -1 when end of InputStream is read
> >> > while (len != -1) {
> >> > try {
> >> > conn = url.openConnection();
> >> > conn.setConnectTimeout(connectionTimeout);
> >> > in = getInputStream(conn);
> >> > in.skip(transferStatus.getBytesDownloaded());
> >> >
> >> > int count = 0;
> >> > System.out.println("downloading");
> >> > transferStatus.setStatus(TransferStatus.DOWNLOADING);
> >> > outFile = new RandomAccessFile("build/outfile", "rw");
> >> > outFile.seek(transferStatus.getBytesDownloaded());
> >> > while ((len = in.read(buf)) > 0) {
> >> > outFile.write(buf, 0, len);
> >> > transferStatus.incrementBytesDownloaded(len);
> >> > updateElapsedTime();
> >> > }
> >> > } catch (Exception e) {
> >> > System.out.println("Exception transfering data: " +
> >> > e.getMessage());
> >> > } finally {
> >> > if ( in != null ) {
> >> > in.close();
> >> > }
> >> > if ( outFile != null ) {
> >> > outFile.close();
> >> > }
> >> > conn = null;
> >> > }
> >> > }
> >> > System.out.println("transfer complete");
> >> > transferStatus.setStatus(TransferStatus.COMPLETE);
> >> > }
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >> http://old.nabble.com/Large-File-Transfer-tp23378044p28836565.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > -----
> > Dejan Bosanac
> >
> > Open Source Integration - http://fusesource.com/
> > ActiveMQ in Action - http://www.manning.com/snyder/
> > Blog - http://www.nighttale.net
> >
>
> --
> View this message in context:
> http://old.nabble.com/Large-File-Transfer-tp23378044p28843604.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
Re: Large File Transfer
Posted by kevinle92618 <ke...@yahoo.com>.
Hi Dejan,
Currently we have a hard requirement to support large file transfer resume.
We're very keen on using ActiveMQ, along with Camel, thus we must implement
this "resume" feature. I'm in the investigation stage but I was thinking of
implement this the follow way:
Following Lou's suggestion, the producer will break down the large file to
be sent into chunks and send them to the broker (ActiveMQ) one at a time
until the entire file is sent. In the event of a file transfer
interruption, the producer will detect this event and when connection is
restore, it will only send the remainder to the broker (with a resume flag
or something like that). The broker will then continue to receive the rest
of the file knowing that it will have to concatenate the files together
before putting it on the queue for consumption.
I'm not sure if this is the best solution or not. If you have a better
suggestion please feel free to let me know. My personal email address is
kevinle92618@yahoo.com
Dejan Bosanac wrote:
>
> Hi,
>
> just to confirm that ActiveMQ comes with HTTP and FTP strategies for
> upload
> and download, that doesn't support resume. This can be implemented as a
> separate strategy. All contributions are welcomed.
>
> Cheers
> --
> Dejan Bosanac - http://twitter.com/dejanb
>
> Open Source Integration - http://fusesource.com/
> ActiveMQ in Action - http://www.manning.com/snyder/
> Blog - http://www.nighttale.net
>
>
> On Thu, Jun 10, 2010 at 12:28 AM, kevinle92618
> <ke...@yahoo.com>wrote:
>
>>
>> Thanks Lou. I was afraid of this. If I understand you correctly,
>> ActiveMQ
>> does NOT support resume transfer. However, you were able to get around
>> that
>> problem by implementing the snippet codes you mentioned in your previous
>> post right? Do you mind posting the codes for the TransferStatus class
>> as
>> well. Many thanks.
>>
>>
>> Lou Parisi wrote:
>> >
>> >
>> > kevinle92618 wrote:
>> >>
>> >> I don't have answers to your questions. I'm interested in knowing if
>> >> ActiveMQ has the "resume" transfer capability for large file. Please
>> >> reply if you have the answer. Thanks.
>> >>
>> >
>> > It has been a while since I worked on this but I did get the "resume"
>> > working. As I recall, I kept track of the number of bytes transferred
>> and
>> > if there was a failure I just had to resume and read past that many
>> bytes
>> > and restart transfer. Here is a snippet from some sample code I
>> created
>> > for testing.
>> >
>> >
>> > private static void startTransfer() throws Exception {
>> > RandomAccessFile outFile = null;
>> > InputStream in = null;
>> > byte[] buf = new byte[bufSize];
>> > int len = 0;
>> > URLConnection conn = null;
>> >
>> > // len == -1 when end of InputStream is read
>> > while (len != -1) {
>> > try {
>> > conn = url.openConnection();
>> > conn.setConnectTimeout(connectionTimeout);
>> > in = getInputStream(conn);
>> > in.skip(transferStatus.getBytesDownloaded());
>> >
>> > int count = 0;
>> > System.out.println("downloading");
>> > transferStatus.setStatus(TransferStatus.DOWNLOADING);
>> > outFile = new RandomAccessFile("build/outfile", "rw");
>> > outFile.seek(transferStatus.getBytesDownloaded());
>> > while ((len = in.read(buf)) > 0) {
>> > outFile.write(buf, 0, len);
>> > transferStatus.incrementBytesDownloaded(len);
>> > updateElapsedTime();
>> > }
>> > } catch (Exception e) {
>> > System.out.println("Exception transfering data: " +
>> > e.getMessage());
>> > } finally {
>> > if ( in != null ) {
>> > in.close();
>> > }
>> > if ( outFile != null ) {
>> > outFile.close();
>> > }
>> > conn = null;
>> > }
>> > }
>> > System.out.println("transfer complete");
>> > transferStatus.setStatus(TransferStatus.COMPLETE);
>> > }
>> >
>> >
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Large-File-Transfer-tp23378044p28836565.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> -----
> Dejan Bosanac
>
> Open Source Integration - http://fusesource.com/
> ActiveMQ in Action - http://www.manning.com/snyder/
> Blog - http://www.nighttale.net
>
--
View this message in context: http://old.nabble.com/Large-File-Transfer-tp23378044p28843604.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Large File Transfer
Posted by Dejan Bosanac <de...@nighttale.net>.
Hi,
just to confirm that ActiveMQ comes with HTTP and FTP strategies for upload
and download, that doesn't support resume. This can be implemented as a
separate strategy. All contributions are welcomed.
Cheers
--
Dejan Bosanac - http://twitter.com/dejanb
Open Source Integration - http://fusesource.com/
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Thu, Jun 10, 2010 at 12:28 AM, kevinle92618 <ke...@yahoo.com>wrote:
>
> Thanks Lou. I was afraid of this. If I understand you correctly, ActiveMQ
> does NOT support resume transfer. However, you were able to get around
> that
> problem by implementing the snippet codes you mentioned in your previous
> post right? Do you mind posting the codes for the TransferStatus class as
> well. Many thanks.
>
>
> Lou Parisi wrote:
> >
> >
> > kevinle92618 wrote:
> >>
> >> I don't have answers to your questions. I'm interested in knowing if
> >> ActiveMQ has the "resume" transfer capability for large file. Please
> >> reply if you have the answer. Thanks.
> >>
> >
> > It has been a while since I worked on this but I did get the "resume"
> > working. As I recall, I kept track of the number of bytes transferred
> and
> > if there was a failure I just had to resume and read past that many bytes
> > and restart transfer. Here is a snippet from some sample code I created
> > for testing.
> >
> >
> > private static void startTransfer() throws Exception {
> > RandomAccessFile outFile = null;
> > InputStream in = null;
> > byte[] buf = new byte[bufSize];
> > int len = 0;
> > URLConnection conn = null;
> >
> > // len == -1 when end of InputStream is read
> > while (len != -1) {
> > try {
> > conn = url.openConnection();
> > conn.setConnectTimeout(connectionTimeout);
> > in = getInputStream(conn);
> > in.skip(transferStatus.getBytesDownloaded());
> >
> > int count = 0;
> > System.out.println("downloading");
> > transferStatus.setStatus(TransferStatus.DOWNLOADING);
> > outFile = new RandomAccessFile("build/outfile", "rw");
> > outFile.seek(transferStatus.getBytesDownloaded());
> > while ((len = in.read(buf)) > 0) {
> > outFile.write(buf, 0, len);
> > transferStatus.incrementBytesDownloaded(len);
> > updateElapsedTime();
> > }
> > } catch (Exception e) {
> > System.out.println("Exception transfering data: " +
> > e.getMessage());
> > } finally {
> > if ( in != null ) {
> > in.close();
> > }
> > if ( outFile != null ) {
> > outFile.close();
> > }
> > conn = null;
> > }
> > }
> > System.out.println("transfer complete");
> > transferStatus.setStatus(TransferStatus.COMPLETE);
> > }
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/Large-File-Transfer-tp23378044p28836565.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
Re: Large File Transfer
Posted by Lou Parisi <Lo...@photomask.com>.
kevinle92618 wrote:
>
> Thanks Lou. I was afraid of this. If I understand you correctly,
> ActiveMQ does NOT support resume transfer. However, you were able to get
> around that problem by implementing the snippet codes you mentioned in
> your previous post right? Do you mind posting the codes for the
> TransferStatus class as well. Many thanks.
>
What I posted is not production code but only a sample that I used to
confirm the resume functionality could be implemented. In fact, I did
implement this in ServiceMix with apache Camel and I can confirm it works.
I cannot say if it is the best way.
The TransferStatus class is only a bean I implemented to hold the transfer
state. I implemented it as a JMX bean and can then view the status of
transfers using JMX. What you need to have in the bean will be dependent on
the state you need to maintain for your scenario. At a minimum you will
need to keep track of the bytes transferred so you can do the resume.
--
View this message in context: http://old.nabble.com/Large-File-Transfer-tp23378044p28844082.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Large File Transfer
Posted by kevinle92618 <ke...@yahoo.com>.
Thanks Lou. I was afraid of this. If I understand you correctly, ActiveMQ
does NOT support resume transfer. However, you were able to get around that
problem by implementing the snippet codes you mentioned in your previous
post right? Do you mind posting the codes for the TransferStatus class as
well. Many thanks.
Lou Parisi wrote:
>
>
> kevinle92618 wrote:
>>
>> I don't have answers to your questions. I'm interested in knowing if
>> ActiveMQ has the "resume" transfer capability for large file. Please
>> reply if you have the answer. Thanks.
>>
>
> It has been a while since I worked on this but I did get the "resume"
> working. As I recall, I kept track of the number of bytes transferred and
> if there was a failure I just had to resume and read past that many bytes
> and restart transfer. Here is a snippet from some sample code I created
> for testing.
>
>
> private static void startTransfer() throws Exception {
> RandomAccessFile outFile = null;
> InputStream in = null;
> byte[] buf = new byte[bufSize];
> int len = 0;
> URLConnection conn = null;
>
> // len == -1 when end of InputStream is read
> while (len != -1) {
> try {
> conn = url.openConnection();
> conn.setConnectTimeout(connectionTimeout);
> in = getInputStream(conn);
> in.skip(transferStatus.getBytesDownloaded());
>
> int count = 0;
> System.out.println("downloading");
> transferStatus.setStatus(TransferStatus.DOWNLOADING);
> outFile = new RandomAccessFile("build/outfile", "rw");
> outFile.seek(transferStatus.getBytesDownloaded());
> while ((len = in.read(buf)) > 0) {
> outFile.write(buf, 0, len);
> transferStatus.incrementBytesDownloaded(len);
> updateElapsedTime();
> }
> } catch (Exception e) {
> System.out.println("Exception transfering data: " +
> e.getMessage());
> } finally {
> if ( in != null ) {
> in.close();
> }
> if ( outFile != null ) {
> outFile.close();
> }
> conn = null;
> }
> }
> System.out.println("transfer complete");
> transferStatus.setStatus(TransferStatus.COMPLETE);
> }
>
>
--
View this message in context: http://old.nabble.com/Large-File-Transfer-tp23378044p28836565.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Large File Transfer
Posted by Lou Parisi <Lo...@photomask.com>.
kevinle92618 wrote:
>
> I don't have answers to your questions. I'm interested in knowing if
> ActiveMQ has the "resume" transfer capability for large file. Please
> reply if you have the answer. Thanks.
>
It has been a while since I worked on this but I did get the "resume"
working. As I recall, I kept track of the number of bytes transferred and
if there was a failure I just had to resume and read past that many bytes
and restart transfer. Here is a snippet from some sample code I created for
testing.
private static void startTransfer() throws Exception {
RandomAccessFile outFile = null;
InputStream in = null;
byte[] buf = new byte[bufSize];
int len = 0;
URLConnection conn = null;
// len == -1 when end of InputStream is read
while (len != -1) {
try {
conn = url.openConnection();
conn.setConnectTimeout(connectionTimeout);
in = getInputStream(conn);
in.skip(transferStatus.getBytesDownloaded());
int count = 0;
System.out.println("downloading");
transferStatus.setStatus(TransferStatus.DOWNLOADING);
outFile = new RandomAccessFile("build/outfile", "rw");
outFile.seek(transferStatus.getBytesDownloaded());
while ((len = in.read(buf)) > 0) {
outFile.write(buf, 0, len);
transferStatus.incrementBytesDownloaded(len);
updateElapsedTime();
}
} catch (Exception e) {
System.out.println("Exception transfering data: " +
e.getMessage());
} finally {
if ( in != null ) {
in.close();
}
if ( outFile != null ) {
outFile.close();
}
conn = null;
}
}
System.out.println("transfer complete");
transferStatus.setStatus(TransferStatus.COMPLETE);
}
--
View this message in context: http://old.nabble.com/Large-File-Transfer-tp23378044p28832355.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Large File Transfer
Posted by kevinle92618 <ke...@yahoo.com>.
I don't have answers to your questions. I'm interested in knowing if
ActiveMQ has the "resume" transfer capability for large file. Please reply
if you have the answer. Thanks.
Lou Parisi wrote:
>
> I have made some sample applications using ActiveMQ BlobMessage and
> everything is working. I have a few questions regarding the best
> implementation.
>
> 1. Since these are large files is there any built in capability to resume
> the file transfer from point of failure if a failure occurs while
> receiving the file at the destination?
>
> I have some questions from our infrastructure team asking why the file
> must be sent by http or ftp and why not send by direct tcp socket
> connection like some peer-to-peer connection between sender and consumer.
> 1. Is it possible to do something like this using the BlobMessage
> framework
> 2. Are there drawbacks to such an implementation that directed you to
> using the Jetty HTTP server for transfer?
>
> Thanks for any feedback or comments.
>
--
View this message in context: http://old.nabble.com/Large-File-Transfer-tp23378044p28831914.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.