You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Carlos Alarcón <ca...@tyven.com> on 2009/02/06 11:27:13 UTC
[users@httpd] URL encodig - percent encoding
Hi, I am facing a problem which I haven't found any help over Internet
(maybe I did the wrong search):
I have:Apache/2.2.3 (CentOS), Centos 5.
I am trying to access specials URLs ( containing non ASCII chars)
So my browser (MZ or IE) converts URL http://rewrite.tyven/eñe.html into
http://rewrite.tyven/e%C3%B1e.html (which is ok, UTF-8 percent encoding
following RFC3986).
My apache gets that but seems to interpret that as ISO8859-1 so it tries
to retrieve file: eñe.html
I had been playing setting CentOS locales (/etc/sysconfig/i18n), right
now it is:
[root@xen3 ~]# locale
LANG=es_ES.UTF-8
LC_CTYPE="es_ES.UTF-8"
LC_NUMERIC="es_ES.UTF-8"
LC_TIME="es_ES.UTF-8"
LC_COLLATE="es_ES.UTF-8"
LC_MONETARY="es_ES.UTF-8"
LC_MESSAGES="es_ES.UTF-8"
LC_PAPER="es_ES.UTF-8"
LC_NAME="es_ES.UTF-8"
LC_ADDRESS="es_ES.UTF-8"
LC_TELEPHONE="es_ES.UTF-8"
LC_MEASUREMENT="es_ES.UTF-8"
LC_IDENTIFICATION="es_ES.UTF-8"
LC_ALL=
But I had test with just UTF-8, en_US-UTF-8, ...
in httpd.conf, I see AddDefaultCharset option, but I interpret it is
only to decode the pages content rather than the URLs. Anyway I had also
played with it without succeed.
I need it all be working UTF-8 to decode the URLs.
Any idea what might be wrong?
Regards
Re: [users@httpd] URL encodig - percent encoding
Posted by André Warnier <aw...@ice-sa.com>.
Carlos Alarcón wrote:
> Thanks a lot for the full explanation.
> You are totally right that it is my application the one that should keep
> consistency with the URL's it offers and the files they are mapped into.
> Actually this is my hole problem. It is supposed to be a service
> creation environment for web applications (accessible via web) where
> user decides the URLs he's going to uses for his pages using the
> browser, so depending on browser setup he might have it would drives to
> different URLs (imagine a browser using utf-8 and other using iso-8859).
>
> We will keep thinking on our best solution to this problem.
>
Hola Carlos.
The tips below are not foolproof, just a series of individual measures
that you can take to avoid more problems than necessary.
- on your systems filesystem, decide once and for all to encode your
filenames as UTF-8 (at least everything under your web document root).
- set your system's default locale to a UTF-8 locale, so that you do not
inadvertently create a file using the iso-8859 encoding.
Also make sure you set the user's locale that way.
- make sure that your webserver is set to use UTF-8 as the default
charset (for responses)
- create all your pages using a UTF-8 aware editor, and save them in the
UTF-8 encoding.
- add a proper
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8"> to
all html pages
- in your <form> tags, specify if possible :
<form .... enctype="multipart/form-data" accept-charset="UTF-8">
(ref : http://www.w3.org/TR/html4/interact/forms.html#form-data-set)
- in your forms, add a hidden field containing some string known to your
application, containing some characters (like n tilde which I cannot
type on this German keyboard) that will cause the string to be a
different "character length" when read as bytes (like iso-8859) or as
UTF-8 characters. Have your application check that string's length when
it receives the form's result, to make sure the string was sent as
UTF-8. Like :
<input type="hidden" name="check-charset" value="áéöüäíß">
which is 14 bytes, but only 7 Unicode characters.
- make sure your application expects all input to be Unicode/UTF-8, and
if a check of the previous field does not give a correct length, send an
error page back to the user telling them to get another browser.
Y buena suerte.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: Re: [users@httpd] URL encodig - percent encoding
Posted by Carlos Alarcón <ca...@tyven.com>.
Thanks a lot for the full explanation.
You are totally right that it is my application the one that should keep
consistency with the URL's it offers and the files they are mapped into.
Actually this is my hole problem. It is supposed to be a service
creation environment for web applications (accessible via web) where
user decides the URLs he's going to uses for his pages using the
browser, so depending on browser setup he might have it would drives to
different URLs (imagine a browser using utf-8 and other using iso-8859).
We will keep thinking on our best solution to this problem.
André Warnier escribió:
> <div class="moz-text-flowed" style="font-family: -moz-fixed">Carlos
> Alarcón wrote:
>> Hi, I am facing a problem which I haven't found any help over
>> Internet (maybe I did the wrong search):
>> I have:Apache/2.2.3 (CentOS), Centos 5.
>> I am trying to access specials URLs ( containing non ASCII chars)
>>
>> So my browser (MZ or IE) converts URL http://rewrite.tyven/eñe.html
>> into http://rewrite.tyven/e%C3%B1e.html (which is ok, UTF-8 percent
>> encoding following RFC3986).
>> My apache gets that but seems to interpret that as ISO8859-1 so it
>> tries to retrieve file: eñe.html
>>
> Hi Carlos.
> In my opinion, it is not you that is wrong, nor Apache.
> And it is not an OS or browser issue.
> It is just that the situation is quite confusing.
> And it is likely to remain quite confusing until (or if) both the HTTP
> protocol, and operating systems, adopt Unicode/UTF-8 as the default
> charset both for URLs and for filesystem directories and filenames.
>
> The full explanation is long, so here is a shortcut :
>
> The question is : how does a user (or browser) specify, in a URL, that
> he wants a file called "eñe.html", with the "ñ" meaning a single byte in
> the iso-8859-1 encoding ?
>
> In most cases, it should not matter. The user will rarely input this
> URL directly in the browser's URL bar. He will click on a link, that he
> has received previously in a page that came from your server.
> It is then the responsibility of the server, to make sure that the links
> in that page, when clicked on, requests the correct resource in the
> filesystem or otherwise.
>
> In the practice, it is thus so : if in your local filesystem you create
> filenames based on Unicode/UTF-8, then in the pages that you send to
> your users, you should use an appropriate encoding, so
> that when the user clicks on them, he gets the correct file.
>
> If your links are not links into the file system, but things that must
> be interpreted by some program at the server end (like when the user
> POSTs a form to a cgi-bin script), then it is the responsibility of
> the application to make sure that there is consistency between what
> the browser is sending, and how the application understands it.
> (There are several measures to achieve that, unfortunately nothing at
> the moment is really 100% foolproof).
>
> To put it another way : the browser is on the user side, and the user
> can decide what is sent in his URLs (it is always an option in the
> browser; the user can click or unclick the option "send URLs in UTF-8").
> (It may not even be a browser : think of wget and curl.)
>
> So basically when on the server side you get a HTTP request with a
> filepath, you don't know in which character-set/encoding it was sent,
> because there is not even a way available in a URL to say that.
> If the browser or other sends a URL that requests ""eñe.html" expressed
> in UTF-8, and on the server side there is no file with a name that
> matches that (byte-by-byte), then the browser should/will get a 404
> Not Found.
> If the browser or other sends a URL that requests ""eñe.html" expressed
> in UTF-8, and on the server side there is a file with a name that
> matches that, then the browser will get the requested page.
>
>
> Long explanation :
>
> The filesystem :
> As far as I know, Unix/Linux filesystems right now are what can be
> called "encoding-neutral". That means that the filenames for instance,
> are composed of "bytes", each byte being just one group of 8 bits,
> with a byte value 0-255 decimal.
>
> In other words, if a file in the filesystem has a name like (look at
> each sign here as an individual byte) "eñe.html", then that filename is
> 9 bytes long, and is composed of the 9 bytes having the hexadecimal
> values corresponding to those "characters"
> (try "ls -1 e*.html| od -t x1 -c" as Eric says).
>
> The fact that when you are on the server with a console, and you do a
> "ls -l *" in that directory, and you see this filename as "eñe.html",
> depends on the locale settings of the process which you use to view this
> directory. So if you see it as "eñe.html", then most probably your
> current locale is iso-8859-1.
>
> If you were to change your current locale to one based on UTF-8 (and
> adapt your terminal display accordingly), then (without changing the
> filename in the filesystem), you would see that same filename as
> "eñe.html". That is because then, the pair of consecutive bytes which
> before looked like "ñ", would be interpreted as the single Unicode
> character "n with tilde", correctly encoded in the UTF-8 encoding.
>
> For the same reason, if your current locale is based on UTF-8, and if
> you enter the command : "touch eñe.html", this would create a file whose
> name is 9 bytes long (the n tilde occupying 2 bytes), and which in the
> UTF-8 encoding would be read as "eñe.html". But then if you would change
> your locale to one based on the iso-8859-1 encoding (and your terminal
> accordingly), and you did the same "ls -l", you would see this filename
> again as "eñe.html".
>
> Inversely, now that your locale is based on iso-8859-1, and you do again
> a "touch eñe.html", then you would create a different file, whose name
> consists only of 8 characters (because the n tilde would now be
> translated to a single byte in the iso-8859-1 alphabet/encoding).
> If you then switch your locale again to a UTF-8 based locale, and do a
> "ls -l", you will see correctly the first file you created (whose name
> consists of 9 bytes), but you would see the second file you created
> probably as "e?e.html". That is because your current locale (UTF-8)
> would be trying to interpret that series of 8 bytes, and find out that
> the sequence starting at the second byte is a not a valid UTF-8
> encoding, so it would display this as a "?".
>
> Confusing, or clear ?
>
> Apache :
> URLs are not by default encoded as UTF-8. The definition of URLs in
> RFC 3986 says that (I summarise) any byte that cannot be represented
> by an ASCII character A-Za-z0-9 (and some more chars) should be
> "percent-encoded".
> But it never says that a URL *is* Unicode. Re-read part 2.5 carefully.
> (I am talking of the "file path" part; the DNS hostname part is
> different, and handled in RFC3490 (check PUNYCODE]).
>
> So, basically, it boils down to the fact that when Apache receives a
> URL, which when percent-decoded, has a filename at the end that consists
> of the 9 bytes "eñe.html", it is not going to translate that anymore,
> and it is going to look on the filesystem for a file that is named with
> those exact same 9 bytes.
>
> In other words, it is not Apache's problem that the user asked for a
> file named (in bytes) "eñe.html". Apache thinks that this is exactly
> what the browser wants, and looks for that.
> If on the filesystem, the filenames are encoded in iso-8859-1, then
> tough luck.
>
>
> </div>
>
Re: [users@httpd] URL encodig - percent encoding
Posted by André Warnier <aw...@ice-sa.com>.
Carlos Alarcón wrote:
> Hi, I am facing a problem which I haven't found any help over Internet
> (maybe I did the wrong search):
> I have:Apache/2.2.3 (CentOS), Centos 5.
> I am trying to access specials URLs ( containing non ASCII chars)
>
> So my browser (MZ or IE) converts URL http://rewrite.tyven/eñe.html into
> http://rewrite.tyven/e%C3%B1e.html (which is ok, UTF-8 percent encoding
> following RFC3986).
> My apache gets that but seems to interpret that as ISO8859-1 so it tries
> to retrieve file: eñe.html
>
Hi Carlos.
In my opinion, it is not you that is wrong, nor Apache.
And it is not an OS or browser issue.
It is just that the situation is quite confusing.
And it is likely to remain quite confusing until (or if) both the HTTP
protocol, and operating systems, adopt Unicode/UTF-8 as the default
charset both for URLs and for filesystem directories and filenames.
The full explanation is long, so here is a shortcut :
The question is : how does a user (or browser) specify, in a URL, that
he wants a file called "eñe.html", with the "ñ" meaning a single byte in
the iso-8859-1 encoding ?
In most cases, it should not matter. The user will rarely input this
URL directly in the browser's URL bar. He will click on a link, that he
has received previously in a page that came from your server.
It is then the responsibility of the server, to make sure that the links
in that page, when clicked on, requests the correct resource in the
filesystem or otherwise.
In the practice, it is thus so : if in your local filesystem you create
filenames based on Unicode/UTF-8, then in the pages that you send to
your users, you should use an appropriate encoding, so
that when the user clicks on them, he gets the correct file.
If your links are not links into the file system, but things that must
be interpreted by some program at the server end (like when the user
POSTs a form to a cgi-bin script), then it is the responsibility of the
application to make sure that there is consistency between what the
browser is sending, and how the application understands it.
(There are several measures to achieve that, unfortunately nothing at
the moment is really 100% foolproof).
To put it another way : the browser is on the user side, and the user
can decide what is sent in his URLs (it is always an option in the
browser; the user can click or unclick the option "send URLs in UTF-8").
(It may not even be a browser : think of wget and curl.)
So basically when on the server side you get a HTTP request with a
filepath, you don't know in which character-set/encoding it was sent,
because there is not even a way available in a URL to say that.
If the browser or other sends a URL that requests ""eñe.html" expressed
in UTF-8, and on the server side there is no file with a name that
matches that (byte-by-byte), then the browser should/will get a 404 Not
Found.
If the browser or other sends a URL that requests ""eñe.html" expressed
in UTF-8, and on the server side there is a file with a name that
matches that, then the browser will get the requested page.
Long explanation :
The filesystem :
As far as I know, Unix/Linux filesystems right now are what can be
called "encoding-neutral". That means that the filenames for instance,
are composed of "bytes", each byte being just one group of 8 bits, with
a byte value 0-255 decimal.
In other words, if a file in the filesystem has a name like (look at
each sign here as an individual byte) "eñe.html", then that filename is
9 bytes long, and is composed of the 9 bytes having the hexadecimal
values corresponding to those "characters"
(try "ls -1 e*.html| od -t x1 -c" as Eric says).
The fact that when you are on the server with a console, and you do a
"ls -l *" in that directory, and you see this filename as "eñe.html",
depends on the locale settings of the process which you use to view this
directory. So if you see it as "eñe.html", then most probably your
current locale is iso-8859-1.
If you were to change your current locale to one based on UTF-8 (and
adapt your terminal display accordingly), then (without changing the
filename in the filesystem), you would see that same filename as
"eñe.html". That is because then, the pair of consecutive bytes which
before looked like "ñ", would be interpreted as the single Unicode
character "n with tilde", correctly encoded in the UTF-8 encoding.
For the same reason, if your current locale is based on UTF-8, and if
you enter the command : "touch eñe.html", this would create a file whose
name is 9 bytes long (the n tilde occupying 2 bytes), and which in the
UTF-8 encoding would be read as "eñe.html". But then if you would change
your locale to one based on the iso-8859-1 encoding (and your terminal
accordingly), and you did the same "ls -l", you would see this filename
again as "eñe.html".
Inversely, now that your locale is based on iso-8859-1, and you do again
a "touch eñe.html", then you would create a different file, whose name
consists only of 8 characters (because the n tilde would now be
translated to a single byte in the iso-8859-1 alphabet/encoding).
If you then switch your locale again to a UTF-8 based locale, and do a
"ls -l", you will see correctly the first file you created (whose name
consists of 9 bytes), but you would see the second file you created
probably as "e?e.html". That is because your current locale (UTF-8)
would be trying to interpret that series of 8 bytes, and find out that
the sequence starting at the second byte is a not a valid UTF-8
encoding, so it would display this as a "?".
Confusing, or clear ?
Apache :
URLs are not by default encoded as UTF-8. The definition of URLs in RFC
3986 says that (I summarise) any byte that cannot be represented by an
ASCII character A-Za-z0-9 (and some more chars) should be
"percent-encoded".
But it never says that a URL *is* Unicode. Re-read part 2.5 carefully.
(I am talking of the "file path" part; the DNS hostname part is
different, and handled in RFC3490 (check PUNYCODE]).
So, basically, it boils down to the fact that when Apache receives a
URL, which when percent-decoded, has a filename at the end that consists
of the 9 bytes "eñe.html", it is not going to translate that anymore,
and it is going to look on the filesystem for a file that is named with
those exact same 9 bytes.
In other words, it is not Apache's problem that the user asked for a
file named (in bytes) "eñe.html". Apache thinks that this is exactly
what the browser wants, and looks for that.
If on the filesystem, the filenames are encoded in iso-8859-1, then
tough luck.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] URL encodig - percent encoding
Posted by Eric Covener <co...@gmail.com>.
On Fri, Feb 6, 2009 at 5:27 AM, Carlos Alarcón <ca...@tyven.com> wrote:
> Hi, I am facing a problem which I haven't found any help over Internet
> (maybe I did the wrong search):
> I have:Apache/2.2.3 (CentOS), Centos 5.
> I am trying to access specials URLs ( containing non ASCII chars)
>
> So my browser (MZ or IE) converts URL http://rewrite.tyven/eñe.html into
> http://rewrite.tyven/e%C3%B1e.html (which is ok, UTF-8 percent encoding
> following RFC3986).
> My apache gets that but seems to interpret that as ISO8859-1 so it tries to
> retrieve file: eñe.html
On unix, apache doesn't interpret it at all. It looks in the
filesystem for a file with the name containing the same bytes.
In your filesystem, you can view the bytes your filename was created
with via e.g.
ls -1 e*.html| od -t x1 -c
--
Eric Covener
covener@gmail.com
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org