You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by ANEESHCH <an...@gmail.com> on 2008/09/20 15:43:25 UTC
Max file size which can be transmitted using HttPClient
Hi,
I have a requirement for sending a huge stream, created by uploading a 2GB
file, using fileupload to a remote file share using Http. When I'm tried
with java.net.HttpURLConnection, it was failing by throwing OutOfmemory
error.
1) Can I make use of HttpClient in this scenario?
2)What's the max file size which can be send using HttpClient?
3)Is it request streaming which I have to use for this purpose.
My application is running on, Websphere Application Server ND 5.1. and the
current max heap size of eah JVM is 768M.
Any help is appreciated..
Thanks in advance
-A
--
View this message in context: http://www.nabble.com/Max-file-size-which-can-be-transmitted-using-HttPClient-tp19585600p19585600.html
Sent from the HttpClient-User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Max file size which can be transmitted using HttPClient
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sat, 2008-09-20 at 06:43 -0700, ANEESHCH wrote:
> Hi,
> I have a requirement for sending a huge stream, created by uploading a 2GB
> file, using fileupload to a remote file share using Http. When I'm tried
> with java.net.HttpURLConnection, it was failing by throwing OutOfmemory
> error.
> 1) Can I make use of HttpClient in this scenario?
> 2)What's the max file size which can be send using HttpClient?
> 3)Is it request streaming which I have to use for this purpose.
>
> My application is running on, Websphere Application Server ND 5.1. and the
> current max heap size of eah JVM is 768M.
>
>
> Any help is appreciated..
>
> Thanks in advance
>
HttpClient can stream out entities up to 2^63-1 (Long#MAX_VALUE) when
using content-length delimited content or unlimited when using chunk
encoding.
See the example:
http://svn.apache.org/repos/asf/httpcomponents/oac.hc3x/trunk/src/examples/UnbufferedPost.java
Hope this helps
Oleg
> -A
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Max file size which can be transmitted using HttPClient
Posted by ANEESHCH <an...@gmail.com>.
Hi,
When I try the url, which I pass to the constructor of PostMethod from a
browser I'm not getting 404. Also the same url was working with
MultipartRequestEntity. When I was in the intranet, I was able to upload
files upto 63Mb, with this method. But when I tried, a 240MB I started
getting Sockettimeout exception, as well websphere reported some threads are
hung. The same issue happened when I tried with a 1K file through VPN.
As a note, even though the code snippet which I posted, was hard coding the
file, actually the file getting uploaded using struts 1.1 file upload, and
the input stream from the upload form is passsed instead of the file.
You thoughts please..
-A
olegk wrote:
>
> On Sun, 2008-09-21 at 14:11 -0700, ANEESHCH wrote:
>> Hi,
>> I tried the following options so far.
>> 1) Based on the unbuffered post, used
>> httppost.setRequestEntity(new InputStreamRequestEntity(
>> new FileInputStream(file), file.length()));
>>
>> It gave me the error, org.apache.commons.httpclient.ProtocolException:
>> Unbuffered entity enclosing request can not be repeated. in the system
>> error
>> file.
>>
>> 2) Then I tried using FileRequestEntity, to make sure that content can be
>> recreated, as well as for marking the entity as repeatable
>> No error was reported any where, but the file was not uploaded to the
>> remote
>> server.
>> Httpstatus code 404 was returned by the excecutemethod();
>>
>
> Status code 404 means the server has not found anything matching the
> Request-URI.
>
> This problem has nothing to do with HttpClient
>
> Oleg
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/Max-file-size-which-can-be-transmitted-using-HttPClient-tp19585600p19598823.html
Sent from the HttpClient-User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Max file size which can be transmitted using HttPClient
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sun, 2008-09-21 at 14:11 -0700, ANEESHCH wrote:
> Hi,
> I tried the following options so far.
> 1) Based on the unbuffered post, used
> httppost.setRequestEntity(new InputStreamRequestEntity(
> new FileInputStream(file), file.length()));
>
> It gave me the error, org.apache.commons.httpclient.ProtocolException:
> Unbuffered entity enclosing request can not be repeated. in the system error
> file.
>
> 2) Then I tried using FileRequestEntity, to make sure that content can be
> recreated, as well as for marking the entity as repeatable
> No error was reported any where, but the file was not uploaded to the remote
> server.
> Httpstatus code 404 was returned by the excecutemethod();
>
Status code 404 means the server has not found anything matching the
Request-URI.
This problem has nothing to do with HttpClient
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Max file size which can be transmitted using HttPClient
Posted by ANEESHCH <an...@gmail.com>.
Hi,
I tried the following options so far.
1) Based on the unbuffered post, used
httppost.setRequestEntity(new InputStreamRequestEntity(
new FileInputStream(file), file.length()));
It gave me the error, org.apache.commons.httpclient.ProtocolException:
Unbuffered entity enclosing request can not be repeated. in the system error
file.
2) Then I tried using FileRequestEntity, to make sure that content can be
recreated, as well as for marking the entity as repeatable
No error was reported any where, but the file was not uploaded to the remote
server.
Httpstatus code 404 was returned by the excecutemethod();
But initially, when I didn't set any value for the contenttype, it gave me
an IO Exception, read error, then I set the content type as "tex/plain;
UTF-8", the IO Error went away.
Please share your thoughts on this, some how I want to make this work.
<code snippet>
public void write(InputStream inStream, String url, int port, String user,
String password, String host) {
PostMethod httppost = null;
try {
System.out.println("Inside File Streamer: url=" + url + "; port="
+ port + "; user=" + user + ";password=" + password);
httppost = new PostMethod(url);
File file = new File("F:\\FileServers.txt");
httppost.setRequestEntity(new FileRequestEntity(file));
HttpClient client = new HttpClient();
client.getHttpConnectionManager().getParams().setConnectionTimeout(
5000);
client.getState().setCredentials(
new AuthScope(host, port,
AuthScope.ANY_REALM),
new UsernamePasswordCredentials(user, password));
httppost.setDoAuthentication(true);
int status = client.executeMethod(httppost);
if (status == HttpStatus.SC_OK) {
System.out.println("HttpStatus:"
+ httppost.getResponseBodyAsString());
} else {
System.out.println("Unexpected failure: " +
httppost.getStatusLine().toString());
}
} finally {
if (httppost != null) {
httppost.releaseConnection();
}
}
}
</code snippet>
ANEESHCH wrote:
>
> Hi,
> I have a requirement for sending a huge stream, created by uploading a 2GB
> file, using fileupload to a remote file share using Http. When I'm tried
> with java.net.HttpURLConnection, it was failing by throwing OutOfmemory
> error.
> 1) Can I make use of HttpClient in this scenario?
> 2)What's the max file size which can be send using HttpClient?
> 3)Is it request streaming which I have to use for this purpose.
>
> My application is running on, Websphere Application Server ND 5.1. and the
> current max heap size of eah JVM is 768M.
>
>
> Any help is appreciated..
>
> Thanks in advance
>
> -A
>
--
View this message in context: http://www.nabble.com/Max-file-size-which-can-be-transmitted-using-HttPClient-tp19585600p19598508.html
Sent from the HttpClient-User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Max file size which can be transmitted using HttPClient
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sun, 2008-09-21 at 09:37 -0700, ANEESHCH wrote:
> Hi,
> When I tried using UnbufferedPost, I got the following error:
>
...
> ................................
> Then tried with a custom entity class, and set the flag isRepeatable to
> true, I strated getting the following errror...
Marking an entity as repeatable, when it is not, does not really help,
does it? Make sure the entity is capable of recreating its content
InputStream in order for it to be repeatable. See
http://hc.apache.org/httpclient-3.x/xref/org/apache/commons/httpclient/methods/FileRequestEntity.html
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Max file size which can be transmitted using HttPClient
Posted by ANEESHCH <an...@gmail.com>.
Hi,
When I tried using UnbufferedPost, I got the following error:
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr R
org.apache.commons.httpclient.ProtocolException: Unbuffered entity enclosing
request can not be repeated.
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr R at
org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:487)
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr R at
org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114)
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr R at
org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr R at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr R at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr R at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr R at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr R at
com.ey.weblib.webdav.docstore.FileStreamer.write(FileStreamer.java:67)
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr R at
com.ey.weblib.webdav.docstore.RemoteDocFileManager.writeStream(RemoteDocFileManager.java:349)
[9/20/08 11:27:14:206 EDT] 2a4e4dea SystemErr R at
com.ey.weblib.webdav.docstore.DocFileCoordinator.uploadAFile(DocFileCoordinator.java:394)
................................
Then tried with a custom entity class, and set the flag isRepeatable to
true, I strated getting the following errror...
[9/21/08 0:12:36:648 EDT] 2a4e4dea SystemErr R java.io.IOException: Read
error
[9/21/08 0:12:36:648 EDT] 2a4e4dea SystemErr R at
java.io.FileInputStream.readBytes(Native Method)
[9/21/08 0:12:36:648 EDT] 2a4e4dea SystemErr R at
java.io.FileInputStream.read(FileInputStream.java:203)
[9/21/08 0:12:36:648 EDT] 2a4e4dea SystemErr R at
com.ey.weblib.webdav.docstore.InputStreamCustomEntity.writeRequest(InputStreamCustomEntity.java:50)
[9/21/08 0:12:36:648 EDT] 2a4e4dea SystemErr R at
org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:499)
[9/21/08 0:12:36:648 EDT] 2a4e4dea SystemErr R at
org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114)
[9/21/08 0:12:36:648 EDT] 2a4e4dea SystemErr R at
org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
[9/21/08 0:12:36:648 EDT] 2a4e4dea SystemErr R at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
[9/21/08 0:12:36:648 EDT] 2a4e4dea SystemErr R at
..................................
=========================================================
Please give me your opinion
-A
ANEESHCH wrote:
>
> Hi,
> I have a requirement for sending a huge stream, created by uploading a 2GB
> file, using fileupload to a remote file share using Http. When I'm tried
> with java.net.HttpURLConnection, it was failing by throwing OutOfmemory
> error.
> 1) Can I make use of HttpClient in this scenario?
> 2)What's the max file size which can be send using HttpClient?
> 3)Is it request streaming which I have to use for this purpose.
>
> My application is running on, Websphere Application Server ND 5.1. and the
> current max heap size of eah JVM is 768M.
>
>
> Any help is appreciated..
>
> Thanks in advance
>
> -A
>
--
View this message in context: http://www.nabble.com/Max-file-size-which-can-be-transmitted-using-HttPClient-tp19585600p19595998.html
Sent from the HttpClient-User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org