HttpMethodDirector infinite loop?

[Jason Polites <ja...@gmail.com>]

Hi folks,
I am using Commons HttpClient 3.1 and have hit a problem which leads to an
infinite loop inside the HttpMethodDirector.

While attempting an authentication against Exchange 2007, we were hitting a
strange 401 error which obviously points to an authentication problem,
however all previous tests using our code have worked. (we are successfully
talking to Exchange 2000/2003/2007 using webdav).

Upon investigation, the following lines from
org.apache.commons.httpclient.HttpMethodDirector came into play:

192 if (isAuthenticationNeeded(method)) { 193 if
(processAuthenticationResponse(method)) { 194 LOG.debug("Retry
authentication"); 195 retry = true; 196 } 197 }

Exchange responds to our initial method with a 401, and commons then calls
the processAuthenticationResponse(method), which subsequently
callsprocessWWWAuthChallenge:

729 authstate.setAuthAttempted(true); 730 Credentials credentials =
this.state.getCredentials(authscope); 731 if (credentials == null) {
732 credentials
= promptForCredentials(authscheme, method.getParams(), authscope); ... } if
(credentials == null) { if (LOG.isInfoEnabled()) { LOG.info("No credentials
available for " + authscope); } return false; } else { return true; }

The promptForCredentials method is then called (line 732) which requires an
instance of a org.apache.commons.httpclient.auth.CredentialsProvider
registered in the HttpParams of the method we are executing.

In our original case, we hadn't registered such a provider however when we
did, our 401 error went away and our method executed successfully.

However, this case worked because our credentials were correct. In the event
that the credentials are in-fact faulty, or indeed the appropriate
permissions are not set on the server, this sequence of processing
authentication never completes, and loops forever.

processAuthenticationResponse returns true simply if there are credentials
present. If they are, then the "retry" flag (line 195) in the main
executeMethod method is set to true, but no retry attempt indicator is
incremented. Hence the method continues to loop forever.

We have "solved" this by making our CredentialsProvider implementation
maintain an internal record of the number of times it has been called,
however on face value the code Commons HttpClient code does not appear to
deal with credentials that fail continuously.

Of course we could expect that the server will eventually respond with a
403, indicating that our authentication attempts have failed too many times,
but in the case of Exchange it never does this and simply continues to
return a 401.

I'm not sure if 3.1 is still being supported, however given that the 4.0
release is still beta we will be using 3.1 for quite some time yet I
suspect.

Thanks.

Re: HttpMethodDirector infinite loop?

[Oleg Kalnichevski <ol...@apache.org>]

Jason Polites wrote:
> Thanks Mike.
> I confess I didn't read the JavaDoc very thoroughly.  Despite your concerns
> around caching of Credentials, in the specific case where this loop occurs
> the provider is indeed called every time, so maintaining a local count does
> the trick.
> 
> I look forward to migrating to v4.0 in the near future, so for now all is
> good.
> 
> Thanks again.
> 
> Jason.
> 

Folks,

CredentialsProvider in HC 3.x is utterly and irreparably broken. Avoid 
using it for anything other than interactive authentication.

HttpClient 4.0 offers a massively better API and a much, much, much 
cleaner code base. There is no point fixing HC 3.x

Oleg


> On Wed, Dec 10, 2008 at 2:55 AM, Michael Clark <mc...@apache.org> wrote:
> 
>> Jason Polites wrote:
>>
>>> We have "solved" this by making our CredentialsProvider
>>> implementation maintain an internal record of the number of times it
>>> has been called, however on face value the code Commons HttpClient
>>> code does not appear to deal with credentials that fail continuously.
>>>
>> Your solution actually does seem to be somewhat in line with the JavaDoc
>> for the CredentialsProvider interface: "It is a responsibility of the custom
>> credentials provider to keep track of authentication attempts and to ensure
>> that credentials known to be invalid are not retried."
>>
>> However, I see that the CredentialsProvider JavaDoc is also not very clear
>> about explaining the contract on how the CredentialsProvider will be used;
>> specifically: when and how often will the CredentialsProvider#getCredentials
>> method will be invoked?  This is not explained, and also there is this
>> comment to be worried about:
>>
>> "HttpClient will simply store the set of credentials returned by the custom
>> credentials provider in the http state object and will attempt to use these
>> credentials for all subsequent requests with the given authentication
>> scope."
>>
>> If caching were to happen, it would mean the CredentialsProvider wouldn't
>> be able to properly track the number of times the credentials it had
>> provided were actually used.
>>
>> A possibly more deterministic solution might be to call
>> HttpMethod#setDoAuthentication(false) on all the methods you execute. This
>> will (to the best of my knowledge) "turn off" HttpClient's automatic
>> handling of authentication challenges.  This moves the responsibility for
>> reacting to authentication challenges (e.g. 401s, 407s, ...) into your
>> application logic, where you could maintain and increment a counter.
>>
>> I agree, however, that some sort of max auth attempts counter internal to
>> HttpClient 3.x would be the nicest of all.  Unfortunately, it doesn't seem
>> to be there, and with everyone's energy now focused on HC 4.0, I am not sure
>> if you could expect such a feature to be implemented in the 3.x line.
>>  Someone more familiar with the project could probably answer this better
>> than I can.
>>
>> regards,
>>
>> Mike
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org

Re: HttpMethodDirector infinite loop?

[Jason Polites <ja...@gmail.com>]

Thanks Mike.
I confess I didn't read the JavaDoc very thoroughly.  Despite your concerns
around caching of Credentials, in the specific case where this loop occurs
the provider is indeed called every time, so maintaining a local count does
the trick.

I look forward to migrating to v4.0 in the near future, so for now all is
good.

Thanks again.

Jason.

On Wed, Dec 10, 2008 at 2:55 AM, Michael Clark <mc...@apache.org> wrote:

> Jason Polites wrote:
>
>> We have "solved" this by making our CredentialsProvider
>> implementation maintain an internal record of the number of times it
>> has been called, however on face value the code Commons HttpClient
>> code does not appear to deal with credentials that fail continuously.
>>
>
> Your solution actually does seem to be somewhat in line with the JavaDoc
> for the CredentialsProvider interface: "It is a responsibility of the custom
> credentials provider to keep track of authentication attempts and to ensure
> that credentials known to be invalid are not retried."
>
> However, I see that the CredentialsProvider JavaDoc is also not very clear
> about explaining the contract on how the CredentialsProvider will be used;
> specifically: when and how often will the CredentialsProvider#getCredentials
> method will be invoked?  This is not explained, and also there is this
> comment to be worried about:
>
> "HttpClient will simply store the set of credentials returned by the custom
> credentials provider in the http state object and will attempt to use these
> credentials for all subsequent requests with the given authentication
> scope."
>
> If caching were to happen, it would mean the CredentialsProvider wouldn't
> be able to properly track the number of times the credentials it had
> provided were actually used.
>
> A possibly more deterministic solution might be to call
> HttpMethod#setDoAuthentication(false) on all the methods you execute. This
> will (to the best of my knowledge) "turn off" HttpClient's automatic
> handling of authentication challenges.  This moves the responsibility for
> reacting to authentication challenges (e.g. 401s, 407s, ...) into your
> application logic, where you could maintain and increment a counter.
>
> I agree, however, that some sort of max auth attempts counter internal to
> HttpClient 3.x would be the nicest of all.  Unfortunately, it doesn't seem
> to be there, and with everyone's energy now focused on HC 4.0, I am not sure
> if you could expect such a feature to be implemented in the 3.x line.
>  Someone more familiar with the project could probably answer this better
> than I can.
>
> regards,
>
> Mike
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>

Re: HttpMethodDirector infinite loop?

[Michael Clark <mc...@apache.org>]

Jason Polites wrote:
> We have "solved" this by making our CredentialsProvider
> implementation maintain an internal record of the number of times it
> has been called, however on face value the code Commons HttpClient
> code does not appear to deal with credentials that fail continuously.

Your solution actually does seem to be somewhat in line with the JavaDoc 
for the CredentialsProvider interface: "It is a responsibility of the 
custom credentials provider to keep track of authentication attempts and 
to ensure that credentials known to be invalid are not retried."

However, I see that the CredentialsProvider JavaDoc is also not very 
clear about explaining the contract on how the CredentialsProvider will 
be used; specifically: when and how often will the 
CredentialsProvider#getCredentials method will be invoked?  This is not 
explained, and also there is this comment to be worried about:

"HttpClient will simply store the set of credentials returned by the 
custom credentials provider in the http state object and will attempt to 
use these credentials for all subsequent requests with the given 
authentication scope."

If caching were to happen, it would mean the CredentialsProvider 
wouldn't be able to properly track the number of times the credentials 
it had provided were actually used.

A possibly more deterministic solution might be to call 
HttpMethod#setDoAuthentication(false) on all the methods you execute. 
This will (to the best of my knowledge) "turn off" HttpClient's 
automatic handling of authentication challenges.  This moves the 
responsibility for reacting to authentication challenges (e.g. 401s, 
407s, ...) into your application logic, where you could maintain and 
increment a counter.

I agree, however, that some sort of max auth attempts counter internal 
to HttpClient 3.x would be the nicest of all.  Unfortunately, it doesn't 
seem to be there, and with everyone's energy now focused on HC 4.0, I am 
not sure if you could expect such a feature to be implemented in the 3.x 
line.  Someone more familiar with the project could probably answer this 
better than I can.

regards,

Mike


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org