[users@httpd] unicode in basic auth

[Milos Jakubicek <xj...@fi.muni.cz>]

Hi all,

I've maybe a very simple problem (I can't understand I didn't find a 
solution anywhere, so it must be really simple:) -- I need to use 
unicode characters (encoded in utf8) in basic auth username/password field.

I don't understand why it doesn't work now, I thought that there 
shouldn't be no problem: the utf8 string is encoded into base64 (i.e. 
plain ASCII) on client side and then back into utf8 on the server (which 
has locale en_US.utf8). But it doesn't work, characters with Czech 
diacritics are on the server side totally broken.

Does anybody have any idea what I'm doing wrong?

Thank you for your time,
Milos Jakubicek

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: [users@httpd] unicode in basic auth

[Milos Jakubicek <xj...@fi.muni.cz>]

Hi,

> If you are using Firefox as a browser, I strongly recommend that you get 
> an add-on like LiveHttpHeaders.  That will show you *exactly* what your 
> browser is sending to the server as HTTP headers with each request.

I should have read this earlier;) Meanwhile I've investigated the same 
but using wireshark...:)

Nevermind the result is that this is plain wrong done on the Firefox 
side (see this link for the current state in Firefox: 
https://bugzilla.mozilla.org/show_bug.cgi?id=436033).

E.g. in Opera it works without any problems -- I don't know whether they 
use utf8 as a hardcoded choice or they detect encoding from locale or 
from any other setting, but it works there. In Konqueror it doesn't work 
too.

I've not done any further research across other browsers, but the main 
result is that using non-latin1 (or maybe even non-ASCII) characters is 
completely unreliable and probably mostly unsupported on the browser 
side, which is really sad :(

--
Thank you all for your responses,
Milos



---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: [users@httpd] unicode in basic auth

[André Warnier <aw...@ice-sa.com>]

Milos Jakubicek wrote:
> 
> Unfortunately it's not so easy, let's see what I'm getting when I send 
> the following two strings "ěščřžýáíéňďťúůó" "ĚŠČŘŽÝÁÍÉŇĎŤÚŮÓ" as 
> username and password (delimited with "-"):

No, it's not easy, specially when you may have 2 or 3 translation steps 
going on in-between that you don't know..

But can you maybe try with something shorter ? :-)
For example, try typing "aša" for the user-id, and no password.
That should be 3 bytes in iso-8859-2 (or the equivalent Windows 8-bit 
codepage); but in UTF-8, the "š" will be at least 2 bytes, so the total 
should be at least 4 bytes.

But before you go any further :
I don't know if you are ever going to be able to find a solution to this 
issue, using Basic Authentication.  Maybe there is simply no solution, 
for the reasons given below.
So, if you want to avoid spending a lot of time maybe for nothing, you 
should consider seriously if you cannot do authentication in another 
way, for example using an html login form. That's because there are 
solutions for that, which work for UTF-8, and it is not less secure than 
basic authentication (which is not secure at all).

Think that maybe also different browsers handle this differently, if it 
is not explicitly defined in the HTTP RFC (2616?).
That may be another reason to switch to login forms.

This being said, if you want to persist :

If you are using Firefox as a browser, I strongly recommend that you get 
an add-on like LiveHttpHeaders.  That will show you *exactly* what your 
browser is sending to the server as HTTP headers with each request.

On the server, you should install a module that dumps the request 
headers to a logfile, as received.  That should exist in the standard 
Apache series. Unfortunately, when the server writes to its logfile, 
there might be some translation/encoding going on at that moment, which 
might or might not be dependent on the locale the server is running 
under.  That would confuse things even more.

So let's say that you type "aša" as the userid, and click OK.
Your browser is going to take *whatever it understood*, format this into 
the right form for Basic authentication, encode this as a Base64 string, 
and send this as the content of a Basic Authentication HTTP header to 
the server.  With LiveHttpHeaders, you will see what it sends (the Base64).

Now you need a program that decodes that Base64 back into a decoded 
string of bytes, and look at what it really is (3 bytes, 4 bytes,..).

Anyway, the server will decode the Base64 string it gets in the header, 
pick out of it the user-id, and try the authentication with that.
I don't think that, at that point (after decoding the Base64 encoding), 
the server is going to decode the resulting string any further, from/to 
UTF-8 or whatever. It will just use the *bytes* it gets after decoding 
the Base64 string.

As Eric pointed out before, there are no mechanisms defined in HTTP to 
indicate an encoding *for the HTTP headers*. They are just supposed to 
be iso-8859-1 (and maybe even just US-ASCII). That's because HTTP was 
invented by a guy in Western Europe.  He was English, but luckily for us 
Western Europeans, he was working in Geneva at the time, so his pals 
probably told him that he needed to support at least German and French 
too. So iso-8859-1 is the default charset under HTTP (and HTML).  Still 
today.  UTF-8 is not the default, and it needs to be declared 
explicitly.  But there is no way to do that for HTTP headers. (The 
situation is just about as confused regarding URLs.)

As for your secondary question (cannot believe nobody had that problem 
before..) : a lot of people do still have problems with issues like that 
all the time.  There exist all kinds of tricks and receipes to avoid, or 
work around, similar issues with html forms, or with filenames.

Yours is the first time I have heard it in relation with Basic 
Authentication and that browser built-in login dialog.




---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: [users@httpd] unicode in basic auth

[Milos Jakubicek <xj...@fi.muni.cz>]

Hi,
> This is another idea :
> Are you *sure* that the browser *really* understands what you 
> enter/paste in the userid/password fields of its built-in Basic 
> authentication dialog, as being UTF-8 ?
> Where is this specified ?

No I'm not -- I'd say it should use what is in my locale(?).
I also didn't find any info on this on the browsers side.

> Are you sure that the Base64 string that the server is receiving, really 
> corresponds to the byte sequence of what you entered, in the UTF-8 
> encoding ?  

Unfortunately it's not so easy, let's see what I'm getting when I send 
the following two strings "ěščřžýáíéňďťúůó" "ĚŠČŘŽÝÁÍÉŇĎŤÚŮÓ" as 
username and password (delimited with "-"):

"-a
Y~ЩАМИHeЗoС-`X}щамиGdзnс-"

The number of bytes is the same!:( I even tried to find out what 
conversion could produce such scrambled output, but didn't succeed in 
finding any. It even doesn't look like if it would cut the first byte 
from the utf8 character.

As there are also "normal" chars in the output (a,e,...) I also cannot 
try to "repair" the chars on the server.

I still can't believe there is no way how to do this, I mean, there must 
be many many people who need this -- or not?

Regards,
Milos

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: [users@httpd] unicode in basic auth

[André Warnier <aw...@ice-sa.com>]

Eric Covener wrote:
> On Fri, Oct 17, 2008 at 9:19 AM, Milos Jakubicek <xj...@fi.muni.cz> wrote:
>> Hi all,
>>
>> I've maybe a very simple problem (I can't understand I didn't find a
>> solution anywhere, so it must be really simple:) -- I need to use unicode
>> characters (encoded in utf8) in basic auth username/password field.
>>
>> I don't understand why it doesn't work now, I thought that there shouldn't
>> be no problem: the utf8 string is encoded into base64 (i.e. plain ASCII) on
>> client side and then back into utf8 on the server (which has locale
>> en_US.utf8). But it doesn't work, characters with Czech diacritics are on
>> the server side totally broken.
>>
>> Does anybody have any idea what I'm doing wrong?
> 
> It likely needs to match the _bytes_ that are stored in whichever
> AuthBasicProvider you're using -- Apache doesn't look at your username
> with utf-8 encoded character and match it to some other representation
> of the "same" characters.
> 
> There is no real facility where the browser can communicate the
> codepage of the bytes it sent in the authorization header,
> mod_authnz_ldap tries to guess based on the Accept-Language header.
> mod_authnz_ldap is a little unique in that it knows what to convert
> _to_ because the spec says things are stored in UTF-8.
> 

This is another idea :
Are you *sure* that the browser *really* understands what you 
enter/paste in the userid/password fields of its built-in Basic 
authentication dialog, as being UTF-8 ?
Where is this specified ?
Understand, I am not sure of this.
But I have been working in and around HTTP authentication for a few 
months now, looking at a lot of aspects of it.  I have not specially 
been looking at the problem you have above, but I cannot recall ever 
seeing any mention of UTF-8 and Basic authentication together, in all 
the hundreds of pages I must have read about this.

What I am saying is that your browser may very well be already 
interpreting whatever you type (which after all are only keystrokes) in 
iso-latin-2 or some Windows Codepage, and encoding *that* in Base64, and 
sending *that* to the server in the HTTP authentication header.
Are you sure that the Base64 string that the server is receiving, really 
corresponds to the byte sequence of what you entered, in the UTF-8 
encoding ?  It should be relatively easy to figure this out, because the 
length in bytes is not going to be the same in iso-latin-x or UTF-8.



---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: [users@httpd] unicode in basic auth

[Eric Covener <co...@gmail.com>]

On Fri, Oct 17, 2008 at 9:19 AM, Milos Jakubicek <xj...@fi.muni.cz> wrote:
> Hi all,
>
> I've maybe a very simple problem (I can't understand I didn't find a
> solution anywhere, so it must be really simple:) -- I need to use unicode
> characters (encoded in utf8) in basic auth username/password field.
>
> I don't understand why it doesn't work now, I thought that there shouldn't
> be no problem: the utf8 string is encoded into base64 (i.e. plain ASCII) on
> client side and then back into utf8 on the server (which has locale
> en_US.utf8). But it doesn't work, characters with Czech diacritics are on
> the server side totally broken.
>
> Does anybody have any idea what I'm doing wrong?

It likely needs to match the _bytes_ that are stored in whichever
AuthBasicProvider you're using -- Apache doesn't look at your username
with utf-8 encoded character and match it to some other representation
of the "same" characters.

There is no real facility where the browser can communicate the
codepage of the bytes it sent in the authorization header,
mod_authnz_ldap tries to guess based on the Accept-Language header.
mod_authnz_ldap is a little unique in that it knows what to convert
_to_ because the spec says things are stored in UTF-8.

-- 
Eric Covener
covener@gmail.com

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org