You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by Karl Wright <da...@gmail.com> on 2013/02/15 16:35:27 UTC
Re: NTLM issues with 4.2.3
Jason and I have taken this conversation offline; we're going to try
out a couple of things and see what they do.
Karl
On Fri, Feb 15, 2013 at 10:00 AM, Jason Millard <js...@gmail.com> wrote:
> Yes I will definitely be willing to do this!
>
> Jason
>
>
> On Friday, February 15, 2013, Karl Wright wrote:
>>
>> Without the ability to duplicate the server-side setup here, the best
>> I can do is change what seems to be different and ask you to try it in
>> your environment, until it works there.
>>
>> The flag differences seem unimportant at first glance, but that is
>> where I would start. What I propose is that I create a number of
>> httpclient jars based on 4.2.3 and ask you to try each one of them in
>> turn. You would be downloading these from
>> http://people.apache.org/~kwright . Do you think this would work for
>> you?
>>
>> Karl
>>
>> On Fri, Feb 15, 2013 at 9:48 AM, Jason Millard <js...@gmail.com> wrote:
>> > On Fri, Feb 15, 2013 at 9:40 AM, Karl Wright <da...@gmail.com> wrote:
>> >> There's enough info in the capture data provided to tell me the answer
>> >> to the jcifs lmCompatibility question, I think. I'd still like to
>> >> know the answer to question (2) though.
>> >>
>> >> Thanks!
>> >> Karl
>> >>
>> >> On Fri, Feb 15, 2013 at 8:53 AM, Karl Wright <da...@gmail.com>
>> >> wrote:
>> >>> Yes - and more information too:
>> >>>
>> >>> (1) I need to know some details about how jcifs is being used when it
>> >>> is successful. Specifically, the question is whether you supply any
>> >>> -D jcifs configuration switches. If *no* switches or system property
>> >>> overrides are being supplied, that is also useful. Specifically, I'm
>> >>> looking for this one:
>> >>>
>> >>> LM_COMPATIBILITY = Config.getInt("jcifs.smb.lmCompatibility", 3);
>> >>>
>> >>> (2) I have tested the HttpClient NTLM code with all the local and
>> >>> domain security policies available on an Windows 2008 R2 setup, and
>> >>> found that it works on all. I will need to know what domain
>> >>> controller authentication policies have been selected to produce a
>> >>> failure.
>> >>>
>> >>> Karl
>> >>>
>> >>>
>> >>> On Fri, Feb 15, 2013 at 7:58 AM, Oleg Kalnichevski <ol...@apache.org>
>> >>> wrote:
>> >>>> Karl
>> >>>>
>> >>>> Do you think these captures could be useful? It appears we have
>> >>>> another
>> >>>> case where JCIFS works while our internal NTLM engine does not.
>> >>>>
>> >>>> Oleg
>> >>>>
>> >>>> -------- Forwarded Message --------
>> >>>> From: Jason Millard <js...@gmail.com>
>> >>>> Reply-to: "HttpClient User Discussion"
>> >>>> <ht...@hc.apache.org>
>> >>>> To: HttpClient User Discussion <ht...@hc.apache.org>
>> >>>> Subject: Re: NTLM issues with 4.2.3
>> >>>> Date: Thu, 14 Feb 2013 11:44:44 -0500
>> >>>>
>> >>>> On Thu, Feb 14, 2013 at 10:00 AM, Oleg Kalnichevski
>> >>>> <ol...@apache.org> wrote:
>> >>>>> On Wed, 2013-02-13 at 16:15 -0500, Jason Millard wrote:
>> >>>>>> Hello.
>> >>>>>>
>> >>>>>> I've been using previous versions of HttpClient forever using the
>> >>>>>> JCIFSEngine. I wanted to give 4.2.3 a try to see if it solved my
>> >>>>>> issues, but unfortunately I'm having the same problems.
>> >>>>>>
>> >>>>>> I was able to turn on debug logging and compare outputs. It's
>> >>>>>> almost
>> >>>>>> identical except for my final 200 vs 401 status code. Of course the
>> >>>>>> type 1, 2, 3 messages have different signatures.
>> >>>>>>
>> >>>>>> Since the messages have addresses that I don't really want public,
>> >>>>>> I
>> >>>>>> was wondering the best way to get help debugging.
>> >>>>>>
>> >>>>>> 2013/02/13 16:09:22:817 EST [DEBUG] headers - >> User-Agent:
>> >>>>>> Apache-HttpClient/4.2.3 (java 1.5)
>> >>>>>> 2013/02/13 16:09:22:817 EST [DEBUG] headers - >> Authorization:
>> >>>>>> NTLM xxxxxx
>> >>>>>> 2013/02/13 16:09:22:928 EST [DEBUG] wire - << "HTTP/1.1 401
>> >>>>>> Unauthorized[\r][\n]"
>> >>>>>> 2013/02/13 16:09:22:929 EST [DEBUG] wire - << "Content-Length:
>> >>>>>> 1539[\r][\n]"
>> >>>>>> 2013/02/13 16:09:22:929 EST [DEBUG] wire - << "Content-Type:
>> >>>>>> text/html[\r][\n]"
>> >>>>>> 2013/02/13 16:09:22:929 EST [DEBUG] wire - << "Server:
>> >>>>>> Microsoft-IIS/6.0[\r][\n]"
>> >>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: NTLM issues with 4.2.3
Posted by Karl Wright <da...@gmail.com>.
I think I'm pretty certain it is #1.
I'm going to check in fixes to this into both the branch and into
trunk, and flag the HTTPCLIENT-1315 ticket also as being potentially
fixed, since this could easily be the problem with that situation as
well.
Karl
On Sun, Feb 17, 2013 at 11:32 AM, Jason Millard <js...@gmail.com> wrote:
> Hello.
>
> Good news! I just tested the latest JAR and I'm glad to say it works
> great. I was able to retrieve the URL and receive a 200!
>
> Do you need any more wireshark logs to verify which fix 1 or 2 was the culprit?
>
> -- Jason
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: NTLM issues with 4.2.3
Posted by Karl Wright <da...@gmail.com>.
Hi Jason,
I've completed a thorough audit of the NTLM support found in JCIFS,
vs. the NTLM support coded in HttpClient. I did this by constructing
a tester which fed randomized authentication and Type 2 message
information to both pieces of software and comparing the resulting
Type 3 message fields. This audit yielded the following differences,
which I have corrected within the HttpClient I have placed at
http://people.apache.org/~kwright :
(1) Target vs. Domain. This is the change that is most likely to make
a difference for you and others. JCIFS was using the user-provided
domain name for constructing authentication cipher values, while
HttpClient was using the target field as returned from the server. In
many cases this would be the same, but not necessarily all. In your
case, the target value being returned differs from the domain value
being sent over in the Type 3 message, and would obviously mess up the
cipher values.
(2) Two distinct random client challenges, rather than one. JCIFS
uses a different random 8-byte challenge for the LMv2 response than it
does for the NTLMv2 response. Hard to see how this could matter,
except if the server side checks to be sure they are different.
(3) Case sensitivity, LM field. The hash that is returned in the LM
field for HttpClient was using the domain name as provided, while
JCIFS uses a value that is mapped to all upper case. Since your
domain names are all upper case, this is unlikely to be a factor in
your setup, however.
(4) SIGN flag interpretation. JCIFS provides a session key only if
the sign flag is set. HttpClient was always providing the session
key. But since we turned this off entirely during the experiments on
Friday, this also cannot be the issue in your case.
(5) Improperly formed NTLMv2 session key value. The value returned
for the session key for NTLMv2 mode was incorrect. Once again, this
cannot be the issue in your case.
There is one difference where I have NOT changed the behavior.
Specifically, the JCIFS implementation does not strip any
qualification suffix from the domain name, but the HttpClient
implementation does. As long as the domain names in the Type 3
messages in your captures look identical, this cannot be a factor.
However, I can certainly imagine cases where there could be a
divergence here; have to keep my eyes open for something appearing in
the future.
Please let me know what happens with this new code.
Karl
On Fri, Feb 15, 2013 at 3:42 PM, Karl Wright <da...@gmail.com> wrote:
> FWIW, it does not appear to be a flag issue.
> I'll be looking deeper this weekend to see if I can find anything
> wrong with the encryption logic. Odd that that should be the problem
> given the tests, but that seems to be where the research is pointing.
>
> Karl
>
> On Fri, Feb 15, 2013 at 10:35 AM, Karl Wright <da...@gmail.com> wrote:
>> Jason and I have taken this conversation offline; we're going to try
>> out a couple of things and see what they do.
>>
>> Karl
>>
>> On Fri, Feb 15, 2013 at 10:00 AM, Jason Millard <js...@gmail.com> wrote:
>>> Yes I will definitely be willing to do this!
>>>
>>> Jason
>>>
>>>
>>> On Friday, February 15, 2013, Karl Wright wrote:
>>>>
>>>> Without the ability to duplicate the server-side setup here, the best
>>>> I can do is change what seems to be different and ask you to try it in
>>>> your environment, until it works there.
>>>>
>>>> The flag differences seem unimportant at first glance, but that is
>>>> where I would start. What I propose is that I create a number of
>>>> httpclient jars based on 4.2.3 and ask you to try each one of them in
>>>> turn. You would be downloading these from
>>>> http://people.apache.org/~kwright . Do you think this would work for
>>>> you?
>>>>
>>>> Karl
>>>>
>>>> On Fri, Feb 15, 2013 at 9:48 AM, Jason Millard <js...@gmail.com> wrote:
>>>> > On Fri, Feb 15, 2013 at 9:40 AM, Karl Wright <da...@gmail.com> wrote:
>>>> >> There's enough info in the capture data provided to tell me the answer
>>>> >> to the jcifs lmCompatibility question, I think. I'd still like to
>>>> >> know the answer to question (2) though.
>>>> >>
>>>> >> Thanks!
>>>> >> Karl
>>>> >>
>>>> >> On Fri, Feb 15, 2013 at 8:53 AM, Karl Wright <da...@gmail.com>
>>>> >> wrote:
>>>> >>> Yes - and more information too:
>>>> >>>
>>>> >>> (1) I need to know some details about how jcifs is being used when it
>>>> >>> is successful. Specifically, the question is whether you supply any
>>>> >>> -D jcifs configuration switches. If *no* switches or system property
>>>> >>> overrides are being supplied, that is also useful. Specifically, I'm
>>>> >>> looking for this one:
>>>> >>>
>>>> >>> LM_COMPATIBILITY = Config.getInt("jcifs.smb.lmCompatibility", 3);
>>>> >>>
>>>> >>> (2) I have tested the HttpClient NTLM code with all the local and
>>>> >>> domain security policies available on an Windows 2008 R2 setup, and
>>>> >>> found that it works on all. I will need to know what domain
>>>> >>> controller authentication policies have been selected to produce a
>>>> >>> failure.
>>>> >>>
>>>> >>> Karl
>>>> >>>
>>>> >>>
>>>> >>> On Fri, Feb 15, 2013 at 7:58 AM, Oleg Kalnichevski <ol...@apache.org>
>>>> >>> wrote:
>>>> >>>> Karl
>>>> >>>>
>>>> >>>> Do you think these captures could be useful? It appears we have
>>>> >>>> another
>>>> >>>> case where JCIFS works while our internal NTLM engine does not.
>>>> >>>>
>>>> >>>> Oleg
>>>> >>>>
>>>> >>>> -------- Forwarded Message --------
>>>> >>>> From: Jason Millard <js...@gmail.com>
>>>> >>>> Reply-to: "HttpClient User Discussion"
>>>> >>>> <ht...@hc.apache.org>
>>>> >>>> To: HttpClient User Discussion <ht...@hc.apache.org>
>>>> >>>> Subject: Re: NTLM issues with 4.2.3
>>>> >>>> Date: Thu, 14 Feb 2013 11:44:44 -0500
>>>> >>>>
>>>> >>>> On Thu, Feb 14, 2013 at 10:00 AM, Oleg Kalnichevski
>>>> >>>> <ol...@apache.org> wrote:
>>>> >>>>> On Wed, 2013-02-13 at 16:15 -0500, Jason Millard wrote:
>>>> >>>>>> Hello.
>>>> >>>>>>
>>>> >>>>>> I've been using previous versions of HttpClient forever using the
>>>> >>>>>> JCIFSEngine. I wanted to give 4.2.3 a try to see if it solved my
>>>> >>>>>> issues, but unfortunately I'm having the same problems.
>>>> >>>>>>
>>>> >>>>>> I was able to turn on debug logging and compare outputs. It's
>>>> >>>>>> almost
>>>> >>>>>> identical except for my final 200 vs 401 status code. Of course the
>>>> >>>>>> type 1, 2, 3 messages have different signatures.
>>>> >>>>>>
>>>> >>>>>> Since the messages have addresses that I don't really want public,
>>>> >>>>>> I
>>>> >>>>>> was wondering the best way to get help debugging.
>>>> >>>>>>
>>>> >>>>>> 2013/02/13 16:09:22:817 EST [DEBUG] headers - >> User-Agent:
>>>> >>>>>> Apache-HttpClient/4.2.3 (java 1.5)
>>>> >>>>>> 2013/02/13 16:09:22:817 EST [DEBUG] headers - >> Authorization:
>>>> >>>>>> NTLM xxxxxx
>>>> >>>>>> 2013/02/13 16:09:22:928 EST [DEBUG] wire - << "HTTP/1.1 401
>>>> >>>>>> Unauthorized[\r][\n]"
>>>> >>>>>> 2013/02/13 16:09:22:929 EST [DEBUG] wire - << "Content-Length:
>>>> >>>>>> 1539[\r][\n]"
>>>> >>>>>> 2013/02/13 16:09:22:929 EST [DEBUG] wire - << "Content-Type:
>>>> >>>>>> text/html[\r][\n]"
>>>> >>>>>> 2013/02/13 16:09:22:929 EST [DEBUG] wire - << "Server:
>>>> >>>>>> Microsoft-IIS/6.0[\r][\n]"
>>>> >>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: NTLM issues with 4.2.3
Posted by Karl Wright <da...@gmail.com>.
FWIW, it does not appear to be a flag issue.
I'll be looking deeper this weekend to see if I can find anything
wrong with the encryption logic. Odd that that should be the problem
given the tests, but that seems to be where the research is pointing.
Karl
On Fri, Feb 15, 2013 at 10:35 AM, Karl Wright <da...@gmail.com> wrote:
> Jason and I have taken this conversation offline; we're going to try
> out a couple of things and see what they do.
>
> Karl
>
> On Fri, Feb 15, 2013 at 10:00 AM, Jason Millard <js...@gmail.com> wrote:
>> Yes I will definitely be willing to do this!
>>
>> Jason
>>
>>
>> On Friday, February 15, 2013, Karl Wright wrote:
>>>
>>> Without the ability to duplicate the server-side setup here, the best
>>> I can do is change what seems to be different and ask you to try it in
>>> your environment, until it works there.
>>>
>>> The flag differences seem unimportant at first glance, but that is
>>> where I would start. What I propose is that I create a number of
>>> httpclient jars based on 4.2.3 and ask you to try each one of them in
>>> turn. You would be downloading these from
>>> http://people.apache.org/~kwright . Do you think this would work for
>>> you?
>>>
>>> Karl
>>>
>>> On Fri, Feb 15, 2013 at 9:48 AM, Jason Millard <js...@gmail.com> wrote:
>>> > On Fri, Feb 15, 2013 at 9:40 AM, Karl Wright <da...@gmail.com> wrote:
>>> >> There's enough info in the capture data provided to tell me the answer
>>> >> to the jcifs lmCompatibility question, I think. I'd still like to
>>> >> know the answer to question (2) though.
>>> >>
>>> >> Thanks!
>>> >> Karl
>>> >>
>>> >> On Fri, Feb 15, 2013 at 8:53 AM, Karl Wright <da...@gmail.com>
>>> >> wrote:
>>> >>> Yes - and more information too:
>>> >>>
>>> >>> (1) I need to know some details about how jcifs is being used when it
>>> >>> is successful. Specifically, the question is whether you supply any
>>> >>> -D jcifs configuration switches. If *no* switches or system property
>>> >>> overrides are being supplied, that is also useful. Specifically, I'm
>>> >>> looking for this one:
>>> >>>
>>> >>> LM_COMPATIBILITY = Config.getInt("jcifs.smb.lmCompatibility", 3);
>>> >>>
>>> >>> (2) I have tested the HttpClient NTLM code with all the local and
>>> >>> domain security policies available on an Windows 2008 R2 setup, and
>>> >>> found that it works on all. I will need to know what domain
>>> >>> controller authentication policies have been selected to produce a
>>> >>> failure.
>>> >>>
>>> >>> Karl
>>> >>>
>>> >>>
>>> >>> On Fri, Feb 15, 2013 at 7:58 AM, Oleg Kalnichevski <ol...@apache.org>
>>> >>> wrote:
>>> >>>> Karl
>>> >>>>
>>> >>>> Do you think these captures could be useful? It appears we have
>>> >>>> another
>>> >>>> case where JCIFS works while our internal NTLM engine does not.
>>> >>>>
>>> >>>> Oleg
>>> >>>>
>>> >>>> -------- Forwarded Message --------
>>> >>>> From: Jason Millard <js...@gmail.com>
>>> >>>> Reply-to: "HttpClient User Discussion"
>>> >>>> <ht...@hc.apache.org>
>>> >>>> To: HttpClient User Discussion <ht...@hc.apache.org>
>>> >>>> Subject: Re: NTLM issues with 4.2.3
>>> >>>> Date: Thu, 14 Feb 2013 11:44:44 -0500
>>> >>>>
>>> >>>> On Thu, Feb 14, 2013 at 10:00 AM, Oleg Kalnichevski
>>> >>>> <ol...@apache.org> wrote:
>>> >>>>> On Wed, 2013-02-13 at 16:15 -0500, Jason Millard wrote:
>>> >>>>>> Hello.
>>> >>>>>>
>>> >>>>>> I've been using previous versions of HttpClient forever using the
>>> >>>>>> JCIFSEngine. I wanted to give 4.2.3 a try to see if it solved my
>>> >>>>>> issues, but unfortunately I'm having the same problems.
>>> >>>>>>
>>> >>>>>> I was able to turn on debug logging and compare outputs. It's
>>> >>>>>> almost
>>> >>>>>> identical except for my final 200 vs 401 status code. Of course the
>>> >>>>>> type 1, 2, 3 messages have different signatures.
>>> >>>>>>
>>> >>>>>> Since the messages have addresses that I don't really want public,
>>> >>>>>> I
>>> >>>>>> was wondering the best way to get help debugging.
>>> >>>>>>
>>> >>>>>> 2013/02/13 16:09:22:817 EST [DEBUG] headers - >> User-Agent:
>>> >>>>>> Apache-HttpClient/4.2.3 (java 1.5)
>>> >>>>>> 2013/02/13 16:09:22:817 EST [DEBUG] headers - >> Authorization:
>>> >>>>>> NTLM xxxxxx
>>> >>>>>> 2013/02/13 16:09:22:928 EST [DEBUG] wire - << "HTTP/1.1 401
>>> >>>>>> Unauthorized[\r][\n]"
>>> >>>>>> 2013/02/13 16:09:22:929 EST [DEBUG] wire - << "Content-Length:
>>> >>>>>> 1539[\r][\n]"
>>> >>>>>> 2013/02/13 16:09:22:929 EST [DEBUG] wire - << "Content-Type:
>>> >>>>>> text/html[\r][\n]"
>>> >>>>>> 2013/02/13 16:09:22:929 EST [DEBUG] wire - << "Server:
>>> >>>>>> Microsoft-IIS/6.0[\r][\n]"
>>> >>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org