You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Håkon Sagehaug <Ha...@bccs.uib.no> on 2009/11/06 15:08:46 UTC
making a client for downloading file through http
Hi all,
I'm making a web service that should be bale to download files from ftp,
http etc. And now I-m making the http part and wondering the best way of
doing that using the httpcomponents libraries. The files that can be
downloaded can be small or large(100->Mb). The scenario is that the client
gives the service a ftp or http url and the service downloads it to the
services local file system.
For now I've used the http core package for downloading files, without
connection re-use but will add that. So my question is what is the most
efficient way of doing this using the libraries. I've looked a little at nio
component. What I want i guess is that the client calling the service should
not wait until the download is complete before an response is given, but
just be notified that the download has been started. I guess one option is
to do that in a thread, another is to use httpnio
Any tips or suggestions how to solve this?
cheers, Håkon
--
Håkon Sagehaug, Scientific Programmer
Parallab, Bergen Center for Computational Science (BCCS)
UNIFOB AS (University of Bergen Research Company)
Hakon.Sagehaug@bccs.uib.no, phone +47 55584125
Re: making a client for downloading file through http
Posted by sebb <se...@gmail.com>.
You might also want to look at Commons VFS:
http://commons.apache.org/vfs/
On 06/11/2009, Håkon Sagehaug <Ha...@bccs.uib.no> wrote:
> Hi
>
> can one add a retry handler to the http core way of doing it also?
>
> cheers, Håkon
>
>
> 2009/11/6 Oleg Kalnichevski <ol...@apache.org>
>
> > Håkon Sagehaug wrote:
> >
> >> Hi Oleg
> >>
> >> see inline
> >>
> >> 2009/11/6 Oleg Kalnichevski <ol...@apache.org>
> >>
> >> Håkon Sagehaug wrote:
> >>>
> >>> Hi Oleg
> >>>>
> >>>> Thanks for the tips, I think I'll go for the classic way. I followed the
> >>>> [1]
> >>>> instroduction and I guess thats a good starting point.
> >>>>
> >>>>
> >>>> Hi Håkon
> >>>
> >>> Make sure you have also read this tutorial:
> >>>
> >>> http://hc.apache.org/httpcomponents-core/tutorial/html/index.html
> >>>
> >>
> >> Yes, I've actually looked at this also
> >>
> >>
> >>>
> >>> I get sometimes this error
> >>>
> >>>> org.apache.http.NoHttpResponseException: The target server failed to
> >>>> respond
> >>>>
> >>>> Waht can i do to avoid this, is it a timeout isssue. My code is mroe or
> >>>> less
> >>>> that same setup as shown in [1].
> >>>>
> >>>>
> >>>> HttpCore throws this exception if the target server fails to return
> >>> _any_
> >>> response of what so ever and prematurely terminates the connection. This
> >>> can
> >>> happen, for instance, when the target server is being under heavy load
> >>> and
> >>> is unable to allocate a worker for a new connection, or due to unexpected
> >>> error in the request processing code. However, there can be no 100%
> >>> certainty about that. Usually the server logs is the only definitive
> >>> source
> >>> of information about that kind of problems.
> >>>
> >>> The strange thing is that if I used HTTPClient for the job of
> >> downloading I
> >> don't seem to have the problem. So just wondering if there where some
> >> parameters one have to set in the http-core that is set in the http-client
> >> component. But I can check the server logs also
> >>
> >>
> > There is no special parameter. HttpClient simply employs a retry handler to
> > re-try such requests automatically:
> >
> >
> > http://hc.apache.org/httpcomponents-client/tutorial/html/fundamentals.html#d4e280
> >
> > Oleg
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
>
>
>
> --
> Håkon Sagehaug, Scientific Programmer
> Parallab, Bergen Center for Computational Science (BCCS)
> UNIFOB AS (University of Bergen Research Company)
> Hakon.Sagehaug@bccs.uib.no, phone +47 55584125
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: making a client for downloading file through http
Posted by Oleg Kalnichevski <ol...@apache.org>.
Håkon Sagehaug wrote:
> Hi
>
> Soemthing like
>
> HttpRequestRetryHandler myRetryHandler = new HttpRequestRetryHandler() {
>
> public boolean retryRequest(IOException exception,
> int executionCount, HttpContext context) {
> if (executionCount >= 5) {
> // Do not retry if over max retry count
> return false;
> }
> if (exception instanceof NoHttpResponseException) {
> // Retry if the server dropped connection on us
> return true;
> }
> if (exception instanceof SSLHandshakeException) {
> // Do not retry on SSL handshake exception
> return false;
> }
> HttpRequest request = (HttpRequest) context
> .getAttribute(ExecutionContext.HTTP_REQUEST);
> boolean idempotent = !(request instanceof
> HttpEntityEnclosingRequest);
> if (idempotent) {
> // Retry if the request is considered idempotent
> return true;
> }
> return false;
> }
>
> };
>
> params.setParameter(HttpMethodParams.RETRY_HANDLER, myRetryHandler);
>
> Where params is a object of HttpParams class.
>
Yeah, right, but you would still have to write some code that actually
makes use of that retry handler like DefaultRequestDirector in HttpClient.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: making a client for downloading file through http
Posted by Håkon Sagehaug <Ha...@bccs.uib.no>.
Hi
Soemthing like
HttpRequestRetryHandler myRetryHandler = new HttpRequestRetryHandler() {
public boolean retryRequest(IOException exception,
int executionCount, HttpContext context) {
if (executionCount >= 5) {
// Do not retry if over max retry count
return false;
}
if (exception instanceof NoHttpResponseException) {
// Retry if the server dropped connection on us
return true;
}
if (exception instanceof SSLHandshakeException) {
// Do not retry on SSL handshake exception
return false;
}
HttpRequest request = (HttpRequest) context
.getAttribute(ExecutionContext.HTTP_REQUEST);
boolean idempotent = !(request instanceof
HttpEntityEnclosingRequest);
if (idempotent) {
// Retry if the request is considered idempotent
return true;
}
return false;
}
};
params.setParameter(HttpMethodParams.RETRY_HANDLER, myRetryHandler);
Where params is a object of HttpParams class.
Håkon
2009/11/6 Oleg Kalnichevski <ol...@apache.org>
> Håkon Sagehaug wrote:
>
>> Hi
>>
>> can one add a retry handler to the http core way of doing it also?
>>
>>
> Sure. HttpClient uses HttpCore internally.
>
> Oleg
>
>
> cheers, Håkon
>>
>> 2009/11/6 Oleg Kalnichevski <ol...@apache.org>
>>
>> Håkon Sagehaug wrote:
>>>
>>> Hi Oleg
>>>>
>>>> see inline
>>>>
>>>> 2009/11/6 Oleg Kalnichevski <ol...@apache.org>
>>>>
>>>> Håkon Sagehaug wrote:
>>>>
>>>>> Hi Oleg
>>>>>
>>>>>> Thanks for the tips, I think I'll go for the classic way. I followed
>>>>>> the
>>>>>> [1]
>>>>>> instroduction and I guess thats a good starting point.
>>>>>>
>>>>>>
>>>>>> Hi Håkon
>>>>>>
>>>>> Make sure you have also read this tutorial:
>>>>>
>>>>> http://hc.apache.org/httpcomponents-core/tutorial/html/index.html
>>>>>
>>>>> Yes, I've actually looked at this also
>>>>
>>>>
>>>> I get sometimes this error
>>>>>
>>>>> org.apache.http.NoHttpResponseException: The target server failed to
>>>>>> respond
>>>>>>
>>>>>> Waht can i do to avoid this, is it a timeout isssue. My code is mroe
>>>>>> or
>>>>>> less
>>>>>> that same setup as shown in [1].
>>>>>>
>>>>>>
>>>>>> HttpCore throws this exception if the target server fails to return
>>>>>>
>>>>> _any_
>>>>> response of what so ever and prematurely terminates the connection.
>>>>> This
>>>>> can
>>>>> happen, for instance, when the target server is being under heavy load
>>>>> and
>>>>> is unable to allocate a worker for a new connection, or due to
>>>>> unexpected
>>>>> error in the request processing code. However, there can be no 100%
>>>>> certainty about that. Usually the server logs is the only definitive
>>>>> source
>>>>> of information about that kind of problems.
>>>>>
>>>>> The strange thing is that if I used HTTPClient for the job of
>>>>>
>>>> downloading I
>>>> don't seem to have the problem. So just wondering if there where some
>>>> parameters one have to set in the http-core that is set in the
>>>> http-client
>>>> component. But I can check the server logs also
>>>>
>>>>
>>>> There is no special parameter. HttpClient simply employs a retry
>>> handler to
>>> re-try such requests automatically:
>>>
>>>
>>>
>>> http://hc.apache.org/httpcomponents-client/tutorial/html/fundamentals.html#d4e280
>>>
>>> Oleg
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>>
>>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
--
Håkon Sagehaug, Scientific Programmer
Parallab, Bergen Center for Computational Science (BCCS)
UNIFOB AS (University of Bergen Research Company)
Hakon.Sagehaug@bccs.uib.no, phone +47 55584125
Re: making a client for downloading file through http
Posted by Oleg Kalnichevski <ol...@apache.org>.
Håkon Sagehaug wrote:
> Hi
>
> can one add a retry handler to the http core way of doing it also?
>
Sure. HttpClient uses HttpCore internally.
Oleg
> cheers, Håkon
>
> 2009/11/6 Oleg Kalnichevski <ol...@apache.org>
>
>> Håkon Sagehaug wrote:
>>
>>> Hi Oleg
>>>
>>> see inline
>>>
>>> 2009/11/6 Oleg Kalnichevski <ol...@apache.org>
>>>
>>> Håkon Sagehaug wrote:
>>>> Hi Oleg
>>>>> Thanks for the tips, I think I'll go for the classic way. I followed the
>>>>> [1]
>>>>> instroduction and I guess thats a good starting point.
>>>>>
>>>>>
>>>>> Hi Håkon
>>>> Make sure you have also read this tutorial:
>>>>
>>>> http://hc.apache.org/httpcomponents-core/tutorial/html/index.html
>>>>
>>> Yes, I've actually looked at this also
>>>
>>>
>>>> I get sometimes this error
>>>>
>>>>> org.apache.http.NoHttpResponseException: The target server failed to
>>>>> respond
>>>>>
>>>>> Waht can i do to avoid this, is it a timeout isssue. My code is mroe or
>>>>> less
>>>>> that same setup as shown in [1].
>>>>>
>>>>>
>>>>> HttpCore throws this exception if the target server fails to return
>>>> _any_
>>>> response of what so ever and prematurely terminates the connection. This
>>>> can
>>>> happen, for instance, when the target server is being under heavy load
>>>> and
>>>> is unable to allocate a worker for a new connection, or due to unexpected
>>>> error in the request processing code. However, there can be no 100%
>>>> certainty about that. Usually the server logs is the only definitive
>>>> source
>>>> of information about that kind of problems.
>>>>
>>>> The strange thing is that if I used HTTPClient for the job of
>>> downloading I
>>> don't seem to have the problem. So just wondering if there where some
>>> parameters one have to set in the http-core that is set in the http-client
>>> component. But I can check the server logs also
>>>
>>>
>> There is no special parameter. HttpClient simply employs a retry handler to
>> re-try such requests automatically:
>>
>>
>> http://hc.apache.org/httpcomponents-client/tutorial/html/fundamentals.html#d4e280
>>
>> Oleg
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: making a client for downloading file through http
Posted by Håkon Sagehaug <Ha...@bccs.uib.no>.
Hi
can one add a retry handler to the http core way of doing it also?
cheers, Håkon
2009/11/6 Oleg Kalnichevski <ol...@apache.org>
> Håkon Sagehaug wrote:
>
>> Hi Oleg
>>
>> see inline
>>
>> 2009/11/6 Oleg Kalnichevski <ol...@apache.org>
>>
>> Håkon Sagehaug wrote:
>>>
>>> Hi Oleg
>>>>
>>>> Thanks for the tips, I think I'll go for the classic way. I followed the
>>>> [1]
>>>> instroduction and I guess thats a good starting point.
>>>>
>>>>
>>>> Hi Håkon
>>>
>>> Make sure you have also read this tutorial:
>>>
>>> http://hc.apache.org/httpcomponents-core/tutorial/html/index.html
>>>
>>
>> Yes, I've actually looked at this also
>>
>>
>>>
>>> I get sometimes this error
>>>
>>>> org.apache.http.NoHttpResponseException: The target server failed to
>>>> respond
>>>>
>>>> Waht can i do to avoid this, is it a timeout isssue. My code is mroe or
>>>> less
>>>> that same setup as shown in [1].
>>>>
>>>>
>>>> HttpCore throws this exception if the target server fails to return
>>> _any_
>>> response of what so ever and prematurely terminates the connection. This
>>> can
>>> happen, for instance, when the target server is being under heavy load
>>> and
>>> is unable to allocate a worker for a new connection, or due to unexpected
>>> error in the request processing code. However, there can be no 100%
>>> certainty about that. Usually the server logs is the only definitive
>>> source
>>> of information about that kind of problems.
>>>
>>> The strange thing is that if I used HTTPClient for the job of
>> downloading I
>> don't seem to have the problem. So just wondering if there where some
>> parameters one have to set in the http-core that is set in the http-client
>> component. But I can check the server logs also
>>
>>
> There is no special parameter. HttpClient simply employs a retry handler to
> re-try such requests automatically:
>
>
> http://hc.apache.org/httpcomponents-client/tutorial/html/fundamentals.html#d4e280
>
> Oleg
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
--
Håkon Sagehaug, Scientific Programmer
Parallab, Bergen Center for Computational Science (BCCS)
UNIFOB AS (University of Bergen Research Company)
Hakon.Sagehaug@bccs.uib.no, phone +47 55584125
Re: making a client for downloading file through http
Posted by Oleg Kalnichevski <ol...@apache.org>.
Håkon Sagehaug wrote:
> Hi Oleg
>
> see inline
>
> 2009/11/6 Oleg Kalnichevski <ol...@apache.org>
>
>> Håkon Sagehaug wrote:
>>
>>> Hi Oleg
>>>
>>> Thanks for the tips, I think I'll go for the classic way. I followed the
>>> [1]
>>> instroduction and I guess thats a good starting point.
>>>
>>>
>> Hi Håkon
>>
>> Make sure you have also read this tutorial:
>>
>> http://hc.apache.org/httpcomponents-core/tutorial/html/index.html
>
> Yes, I've actually looked at this also
>
>>
>>
>> I get sometimes this error
>>> org.apache.http.NoHttpResponseException: The target server failed to
>>> respond
>>>
>>> Waht can i do to avoid this, is it a timeout isssue. My code is mroe or
>>> less
>>> that same setup as shown in [1].
>>>
>>>
>> HttpCore throws this exception if the target server fails to return _any_
>> response of what so ever and prematurely terminates the connection. This can
>> happen, for instance, when the target server is being under heavy load and
>> is unable to allocate a worker for a new connection, or due to unexpected
>> error in the request processing code. However, there can be no 100%
>> certainty about that. Usually the server logs is the only definitive source
>> of information about that kind of problems.
>>
> The strange thing is that if I used HTTPClient for the job of downloading I
> don't seem to have the problem. So just wondering if there where some
> parameters one have to set in the http-core that is set in the http-client
> component. But I can check the server logs also
>
There is no special parameter. HttpClient simply employs a retry handler
to re-try such requests automatically:
http://hc.apache.org/httpcomponents-client/tutorial/html/fundamentals.html#d4e280
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: making a client for downloading file through http
Posted by Håkon Sagehaug <Ha...@bccs.uib.no>.
Hi Oleg
see inline
2009/11/6 Oleg Kalnichevski <ol...@apache.org>
> Håkon Sagehaug wrote:
>
>> Hi Oleg
>>
>> Thanks for the tips, I think I'll go for the classic way. I followed the
>> [1]
>> instroduction and I guess thats a good starting point.
>>
>>
> Hi Håkon
>
> Make sure you have also read this tutorial:
>
> http://hc.apache.org/httpcomponents-core/tutorial/html/index.html
Yes, I've actually looked at this also
>
>
>
> I get sometimes this error
>>
>> org.apache.http.NoHttpResponseException: The target server failed to
>> respond
>>
>> Waht can i do to avoid this, is it a timeout isssue. My code is mroe or
>> less
>> that same setup as shown in [1].
>>
>>
> HttpCore throws this exception if the target server fails to return _any_
> response of what so ever and prematurely terminates the connection. This can
> happen, for instance, when the target server is being under heavy load and
> is unable to allocate a worker for a new connection, or due to unexpected
> error in the request processing code. However, there can be no 100%
> certainty about that. Usually the server logs is the only definitive source
> of information about that kind of problems.
>
The strange thing is that if I used HTTPClient for the job of downloading I
don't seem to have the problem. So just wondering if there where some
parameters one have to set in the http-core that is set in the http-client
component. But I can check the server logs also
cheers, Håkon
>
> Cheers
>
> Oleg
>
>
>
>
> cheers, håkon
>>
>> [1] http://wiki.apache.org/HttpComponents/GeneralHttpCoreIntroduction
>>
>>
>>
>> 2009/11/6 Oleg Kalnichevski <ol...@apache.org>
>>
>> Håkon Sagehaug wrote:
>>>
>>> Hi all,
>>>>
>>>> I'm making a web service that should be bale to download files from ftp,
>>>> http etc. And now I-m making the http part and wondering the best way of
>>>> doing that using the httpcomponents libraries. The files that can be
>>>> downloaded can be small or large(100->Mb). The scenario is that the
>>>> client
>>>> gives the service a ftp or http url and the service downloads it to the
>>>> services local file system.
>>>> For now I've used the http core package for downloading files, without
>>>> connection re-use but will add that. So my question is what is the most
>>>> efficient way of doing this using the libraries.
>>>>
>>>> Hi Håkon
>>>
>>> I think HttpCore (classic) is a good choice as long as you do not mind
>>> managing connection persistence in your code.
>>>
>>>
>>> I've looked a little at nio
>>>
>>> component. What I want i guess is that the client calling the service
>>>> should
>>>> not wait until the download is complete before an response is given, but
>>>> just be notified that the download has been started. I guess one option
>>>> is
>>>> to do that in a thread, another is to use httpnio
>>>>
>>>>
>>>> I do not think HttpCore NIO is necessary in such a case. NIO is
>>> _significantly_ more complex and in my opinion offers little benefit for
>>> client side HTTP. HttpCore (classic) will only block until a response
>>> head
>>> is fully received. So, your could can examine the status code and headers
>>> before downloading the response body from the content input stream. I see
>>> little wrong with using a worker thread to download the response content
>>> as
>>> long as the number of worker threads remains low (<50 or so).
>>>
>>>
>>> Any tips or suggestions how to solve this?
>>>
>>>>
>>>> Hope this answers your question.
>>>
>>> Cheers
>>>
>>> Oleg
>>>
>>> cheers, Håkon
>>>
>>>>
>>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>>
>>>
>>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
--
Håkon Sagehaug, Scientific Programmer
Parallab, Bergen Center for Computational Science (BCCS)
UNIFOB AS (University of Bergen Research Company)
Hakon.Sagehaug@bccs.uib.no, phone +47 55584125
Re: making a client for downloading file through http
Posted by Oleg Kalnichevski <ol...@apache.org>.
Håkon Sagehaug wrote:
> Hi Oleg
>
> Thanks for the tips, I think I'll go for the classic way. I followed the [1]
> instroduction and I guess thats a good starting point.
>
Hi Håkon
Make sure you have also read this tutorial:
http://hc.apache.org/httpcomponents-core/tutorial/html/index.html
> I get sometimes this error
>
> org.apache.http.NoHttpResponseException: The target server failed to respond
>
> Waht can i do to avoid this, is it a timeout isssue. My code is mroe or less
> that same setup as shown in [1].
>
HttpCore throws this exception if the target server fails to return
_any_ response of what so ever and prematurely terminates the
connection. This can happen, for instance, when the target server is
being under heavy load and is unable to allocate a worker for a new
connection, or due to unexpected error in the request processing code.
However, there can be no 100% certainty about that. Usually the server
logs is the only definitive source of information about that kind of
problems.
Cheers
Oleg
> cheers, håkon
>
> [1] http://wiki.apache.org/HttpComponents/GeneralHttpCoreIntroduction
>
>
>
> 2009/11/6 Oleg Kalnichevski <ol...@apache.org>
>
>> Håkon Sagehaug wrote:
>>
>>> Hi all,
>>>
>>> I'm making a web service that should be bale to download files from ftp,
>>> http etc. And now I-m making the http part and wondering the best way of
>>> doing that using the httpcomponents libraries. The files that can be
>>> downloaded can be small or large(100->Mb). The scenario is that the
>>> client
>>> gives the service a ftp or http url and the service downloads it to the
>>> services local file system.
>>> For now I've used the http core package for downloading files, without
>>> connection re-use but will add that. So my question is what is the most
>>> efficient way of doing this using the libraries.
>>>
>> Hi Håkon
>>
>> I think HttpCore (classic) is a good choice as long as you do not mind
>> managing connection persistence in your code.
>>
>>
>> I've looked a little at nio
>>
>>> component. What I want i guess is that the client calling the service
>>> should
>>> not wait until the download is complete before an response is given, but
>>> just be notified that the download has been started. I guess one option is
>>> to do that in a thread, another is to use httpnio
>>>
>>>
>> I do not think HttpCore NIO is necessary in such a case. NIO is
>> _significantly_ more complex and in my opinion offers little benefit for
>> client side HTTP. HttpCore (classic) will only block until a response head
>> is fully received. So, your could can examine the status code and headers
>> before downloading the response body from the content input stream. I see
>> little wrong with using a worker thread to download the response content as
>> long as the number of worker threads remains low (<50 or so).
>>
>>
>> Any tips or suggestions how to solve this?
>>>
>> Hope this answers your question.
>>
>> Cheers
>>
>> Oleg
>>
>> cheers, Håkon
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: making a client for downloading file through http
Posted by Håkon Sagehaug <Ha...@bccs.uib.no>.
Hi Oleg
Thanks for the tips, I think I'll go for the classic way. I followed the [1]
instroduction and I guess thats a good starting point.
I get sometimes this error
org.apache.http.NoHttpResponseException: The target server failed to respond
Waht can i do to avoid this, is it a timeout isssue. My code is mroe or less
that same setup as shown in [1].
cheers, håkon
[1] http://wiki.apache.org/HttpComponents/GeneralHttpCoreIntroduction
2009/11/6 Oleg Kalnichevski <ol...@apache.org>
> Håkon Sagehaug wrote:
>
>> Hi all,
>>
>> I'm making a web service that should be bale to download files from ftp,
>> http etc. And now I-m making the http part and wondering the best way of
>> doing that using the httpcomponents libraries. The files that can be
>> downloaded can be small or large(100->Mb). The scenario is that the
>> client
>> gives the service a ftp or http url and the service downloads it to the
>> services local file system.
>> For now I've used the http core package for downloading files, without
>> connection re-use but will add that. So my question is what is the most
>> efficient way of doing this using the libraries.
>>
>
> Hi Håkon
>
> I think HttpCore (classic) is a good choice as long as you do not mind
> managing connection persistence in your code.
>
>
> I've looked a little at nio
>
>> component. What I want i guess is that the client calling the service
>> should
>> not wait until the download is complete before an response is given, but
>> just be notified that the download has been started. I guess one option is
>> to do that in a thread, another is to use httpnio
>>
>>
> I do not think HttpCore NIO is necessary in such a case. NIO is
> _significantly_ more complex and in my opinion offers little benefit for
> client side HTTP. HttpCore (classic) will only block until a response head
> is fully received. So, your could can examine the status code and headers
> before downloading the response body from the content input stream. I see
> little wrong with using a worker thread to download the response content as
> long as the number of worker threads remains low (<50 or so).
>
>
> Any tips or suggestions how to solve this?
>>
>>
> Hope this answers your question.
>
> Cheers
>
> Oleg
>
> cheers, Håkon
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
--
Håkon Sagehaug, Scientific Programmer
Parallab, Bergen Center for Computational Science (BCCS)
UNIFOB AS (University of Bergen Research Company)
Hakon.Sagehaug@bccs.uib.no, phone +47 55584125
Re: making a client for downloading file through http
Posted by Oleg Kalnichevski <ol...@apache.org>.
Håkon Sagehaug wrote:
> Hi all,
>
> I'm making a web service that should be bale to download files from ftp,
> http etc. And now I-m making the http part and wondering the best way of
> doing that using the httpcomponents libraries. The files that can be
> downloaded can be small or large(100->Mb). The scenario is that the client
> gives the service a ftp or http url and the service downloads it to the
> services local file system.
> For now I've used the http core package for downloading files, without
> connection re-use but will add that. So my question is what is the most
> efficient way of doing this using the libraries.
Hi Håkon
I think HttpCore (classic) is a good choice as long as you do not mind
managing connection persistence in your code.
I've looked a little at nio
> component. What I want i guess is that the client calling the service should
> not wait until the download is complete before an response is given, but
> just be notified that the download has been started. I guess one option is
> to do that in a thread, another is to use httpnio
>
I do not think HttpCore NIO is necessary in such a case. NIO is
_significantly_ more complex and in my opinion offers little benefit for
client side HTTP. HttpCore (classic) will only block until a response
head is fully received. So, your could can examine the status code and
headers before downloading the response body from the content input
stream. I see little wrong with using a worker thread to download the
response content as long as the number of worker threads remains low
(<50 or so).
> Any tips or suggestions how to solve this?
>
Hope this answers your question.
Cheers
Oleg
> cheers, Håkon
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org