You are viewing a plain text version of this content. The canonical link for it is here.
Posted to rivet-dev@tcl.apache.org by ta...@tohoku.iij.ad.jp on 2002/11/20 09:57:36 UTC
Japanized Rivet (and I hope more i18n ;-)
Hi, all.
Ok, Rivet can deal Japanese, Korean, Chinese language.
But for Japanese, it have still probrem.
In Japanese, there are 3 encodings (shiftjis, iso2022-jp,
euc-jp).
And some users use shiftjis, and the other users use iso2022-jp,
or euc-jp.
Rivet can deal only one encoding which is same as system encoding.
So if system encoding is euc-jp, shiftjis and iso2022-jp users will
get failer.
Rivet may detect encoding of string which inputed by user, and may
convert from correct encoding.
Here is a very add-hoc patch for it.
this patch requires auto-code detection library libwkf:
http://www.mysticwall.com/software/wkf/
I think this patch is too add-hoc.
TclWeb_StringToUtf() may have something like "HOOK".
HOOK means some tcl procedure call (if its procedure exists).
user (programmer) can write such procedure if he/she needs.
offcause such procedure returns encoding name for inputed strings.
I think such hook may good for non-Japanese users. Because
Rivet will be able to deal many encodings which is not same
as system encodings.
BEGIN>---8<CUT HERE>8---
begin 644 rivet-japanize.patch.gz
M'XL(")Y+VST"`W)I=F5T+6IA<&%N:7IE+G!A=&-H`.5::W/:R!+]C'_%K%-U
M%XS`>O`PL,D-ZR4)&R].^9'4K2T7)8O!*!82*PF2E-?__?8\]$0O0,F]5<L'
M8Z3I,S.G>[I[>F:FS^>HL;:1JQD-6]]@MVG9^L.I8VNG-YKQ"=\W%\&[R..C
M1J.1)5;YA&=H8FV0+"))[,MRORW"_Z)\5*_74S`C,E)?Z?5;/2;S^C5J2*(H
MR!*JDV]%1*]?'Z$CI)LN8N+3/]1'?'MU486?T\O[S^C$QL[:<`6D+50;G<QU
M`YOJ$@N\_17^:XT=ES3[JS8`K/H+?3[#<_3[\,-P,KH>3=\-S]\?U4,]O,7N
M1]6F'8Q-%]LK=*+3;XJ9U.E&M3/ZW,8>.A>ZXWZ?'@!^]!70G1+A7V##P1$M
MA#C:&:Z1@!1BY$"\Z/SWF:LYT^<QHP/0R7IYCQ-F?*"936`H!555Q+B&A@&@
M)0*.S$T^(#8WP&HQP'=8G6$['W-!VSD9N&E6&5":/]LM6PP3N(=XF*Y\:K;$
M8^04)H*:;)T8[>G)$4(GC3(_U"W+DB*<H;HLRX(D4:]<=B\`>%IHX=RN#$N=
MO>&./MN*<L)...@2>XWNH``"9&N2>T`4UM*3_$S44\%^$-6=7!<06
M<0Y...@LL>.9JZPE7F\ZVU[_YU4Z#-0)L"VECZC$8$ZK,IP4I7D$1@N*T(DK*=
M?C#8ZP4VC'-KN53-67(7NX4%)L>[N'9MW7RXL6[=>2*Q.9W4?=+0-N"-!6_V
M0N6*2AUICCP85OZX<C%\54>&X?O=:H*<-S-$^=\2_*C;[EHUR'K/DMZP9L0,
M"=`L-Z-65ZJVP$TM,0?V7F9GUUZKW7/L1,EN7SSK*^U0IMT1.I!G=X0>-?,7
MNJD9ZQE&QTQ\:C,>FHOC\,NE-9NRX4:?>]G]<9J-^TU_^?(X;RY>A=<M_@I&
M:"+`7AL84?@I^P%*>P%@NHG1S?G%I]&O'RXO+Q`,K?&*_EE9EL%FH]``)2D=
M;]V2CXW=-0"#Z/3R/6`]9[HVNG;*WP*$O=S1X1ET0-P3FR1Q3_J`_:_:MOIM
MRG(&=+)2[:5#'Z&72%U-7?7>P%-L0'9,V6.:9D22IF#:E,M6FT09B?A`R><2
M/I;9;#9I@""?N66CJ@[(X@#IZ!<4]-9X99).!JA>UVNL\5,:[Q4V8R+\B,DX
M$YR+1S[MX4_]K@DM!13U3$5P4N09FQ5]CJH_.:YM:LM5U=<`!Q2@087.!%I`
M\/`:U&#B_`EO64/_CK?UW_3CTK4:8[S=)F8+J[$KR#*EG`*0(<T-]0&]!)9K
M_.$3@6=/D31(Y;5";.L:NXP`XE^YC9%W,+XB/&]4@_$DP!C#9*>#HS3P+%"N
M`3*[9T2Z8)/T5X>[7%GWGP<)SS9_RG?TN?=3O`-:V&`RN/%:2W>)EL+B43H=
M828*0:5`\'ES#"8_/;=,376)E"P@CEX;9)%.Q=YZ+ZI,QJ.7T\KLK-NA7K+7
M+<U+EEO,R/"5^]<(?H#'/`-"94@;89LDBSZQ.WK(5-937--.GE((3+8P6@J*
M9[7Y[M`33O&)P>LDQX@J3]QXB<Z)\D'OP]4*^A\9>(E-E^G$F[/OWM(6/1_`
M[LL]LMYS4-*D.6G/3-G/WGY#9/MJI:0%>6@!*&W][5(7^?[+C2RS'MFDR8+4
MW7NU5<HSJ\(FY2^DN$GEFE-,DIO3_V`*J:MBOQ7AK0;DY3MLL"3CF=Q>7-1X
MW:--ETE7+BUN'59YS8A3Q>N//V"A*#23ETEXDO[)"Z68E;5H=J2(95A9Z0<Q
MB29WT'''#["_=H?8GR)%[._T!-TL=`?ICOFS"_RJ!L+SN:[I8%*H@9;6!B/7
M0@O562#:$3)4H(UL/\O>?/YC$RNB'879>TN2HO8.2L3F1K<MDRSRJ8-=MM/<
MP<$>>FZ4YE]W.6!)-6\V^0Z;?%`P8NH#0(\\GGUJENGJYAIG%'`K:26'J'<+
ML/T]8`6"8G')I`UH=M?;7:*<+A.["J4;8U.SK_#\W%I#`"`T#9)>@+C_`E"O
M:6B4HP'#TQT='.U/Y'Y#Z?:H<MJM<N)]2:>.:5:YQ[E=CG'VSNC\NV>^<<8M
M,9O:4._I\9A7]?)L;S&SMWS5SL*AHH\8">DY.`F=%Q?:[C0A&R#1R%9-9PX1
M9DC#&**#-V&5G+\=@V':.@E"#HE)8'N(%\81/90"`4RC$E%;2U'HY96V=":T
MV<:$QCKL8#3\,/[9@:B*D8KN=1>1,P0(F"9$ZGNLJ6MHHKO0PK'0<JTM(`K:
MF)S&H96M:JZNJ48S\VBNO#.D`P][MBR;M7:7*YY9D`']QN#0#*)3:IF.-'[4
MK!F>NHA\@=/Z\CA_"UTY[U7SLWX.SR[GO.BEFT3#'O[(!`'2`?;^>8FJX1<U
MDO9Q`39`>K@#K=BO\'L2%UC_+]'[\^GH]MRRKW\?7Z.__T:AY_11C20;S#HY
MX+&VZBGR,<?BY4T"&99-$=4=2Q9EN?%YE2,/8TJ2QVLM7W9X?3X>)TFKCJ;K
MOG#`!6OP4Y2KD'R(<UZ8]%CWW0)-10:>``^\7#,$J%)$;_Z@G@/%<\,:F[I;
M_1>QKHA5?&7KFEHQ;UGU>B+GP@*J.OJ#B6<UGC6!50G(Q^'K(JLOE-D7&7RA
M?@(_!>N&I=PK>#E;KZK!.9@0'L5'U5AC/HS:]DI[8V,<'22/IFQ=9H32\>1B
M/!D%,=*/JX>?*_N,;O6QWQEQV/5D^!0OCX#>)OA+4$W/2/M)APGG(#NC)1]\
M/,=N&7RP,63QF%V%J!;8R)'@TX:]EM1&]4ZW*\A^56RWG*GT.S:E7;&)1Y6M
MHQ!?.CO?R=`Q3:#6='"-5V&X<*J2()\NY^WG_(0EI/Q#E7/0S9L2+M[$%<)2
M)P:"3A@=7CK;[BJD#-`5.Z$RP)-_K)Q:7F)I;R2Q90-*U7&!C-33%-52O,*4
M(9\B%WAJ^+`VX*_]QOBKRW1-=)M_;T1[T%,NC=`WV3=&:)/=KXMLBTG]5J\O
MM8*[(BV9U-M;?ETLMO^!-#FZ^3%`FWO<C=VY@I#13]$"0@9$Z,)1&.57U<&=
M%DU(\BZ8I3"090?T)R28VY>'0F_2["#49!<[2!/K]-MG?44,[$#NLGHR?(7J
M>:$C^R&IP%WZ1]F1@(A&DX_3X=75\#_3R?"/$?&#R$M&GHO%ZY@2/7OP>A70
M@V'=JX9#;P>EQND82I9T*#[#3@Z$D`O[0[Z[1J0R=O^-/M(,4KWT=X***-+"
MI]C:AZAWH^%OHZOK<L@*%2<.XRL$5)PR$(>]!KTI2?;/'%0E=538U*X-6#+>
M!KN)?/9:])JKTO;+QC2`PHQ)R3YVVX%>OE#N:+7I.><,/7KY*G9PP>I0D0F1
M30Z+SK5HE3<*5!B`)P1/H>L]?%+\'"*X]Y)XS^/XF&</7@63L$2*1)REYW`]
M,,Z1?)?E?BM;)PKY](0"YY9TCE2H$./M2FGIG%3.N;W`=(ES/JXQ5IA[!;@!
MUP%,2B/\*C5>,V1T].0\.M[8UI*PRED1&/6#76R'7T8IRX(XW+YVY!UF;;L2
MWV`(&92ELRYAJ25&EU8H$QU=75U>[;R86-X8YV,/*AA0$8#]26B)-(RUY%Z$
M!"+XR;;,A\EZ.;0?@NG(`B*V$AA*!F?%&//N4QQ*F(?S??EJMR7"5[L;\!79
M--Q\6^'D(1",>QNKCSQT:9`[H3?CBQ$):OT\7Q3;*6:2M>V+8M+I4IR8K7&2
M,5[W8]Y&]G;;-$2U>U(Q$Y(\$SJF.YCC/:TH8:^V%RL9JRR-$AB."JW[_.0A
.<Y9'1_\%SW04`74Z````
`
end
END>---8<CUT HERE>8---
PS: For using this patch, "-DJAPANESE_HACK" to CFLAGS, and
"-lwkf" to linker flags.
Thanks.
---
Takeshi, Taguchi.
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Japanized Rivet (and I hope more i18n ;-)
Posted by Makoto Satoh <ma...@yahoo.co.jp>.
taguchi@tohoku.iij.ad.jp wrote:
> > Ok, to be clear... we are talking about users connecting to a
> > Rivet-based system with their web browsers? The problem being that
> > they might use some other kind of encoding from what you are using as
> > the system encoding?
>
> Yes.
Hi, I think that this would be overspecilization.
Although Japanese use 3 major character encodings, but we used to take
it for granted that we had to take care of which encoding to use.
In Java's world, the handling of 3 Japanese encodings is not perfect.
I think that this matter is not belonging to Rivet.
If you want to get POSTed data which encoding differ from the system
encoding, you could use some sort of CGI library or decode it from
the 'raw' input with your own library.
Any comment?
--
Makoto Satoh
<ma...@yahoo.co.jp>
http://yokohama.cool.ne.jp/greenhills/
__________________________________________________
Do You Yahoo!?
Yahoo! BB is Broadband by Yahoo! http://bb.yahoo.co.jp/
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Japanized Rivet (and I hope more i18n ;-)
Posted by ta...@tohoku.iij.ad.jp.
> Please let me know when you need 'AutoDetect' functionality. If you
> serve your 'form' page in EUC-JP, it would be assumed that the input
> data would be encoded using EUC-JP in most cases (I don't know the
> exception).
Actualy I've got such exception.
In last year, for debugging, I logged encoding which encoding was used
on my mail-form CGI which written tcl.
This script was written using euc-jp.
But some netscape4 user used euc-jp, and other netscape4 user used
shiftjis!
> # I don't want to raise a flame....
I also never want it.... ;-)
---
Taguchi,T.
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Japanized Rivet (and I hope more i18n ;-)
Posted by Makoto Satoh <ma...@yahoo.co.jp>.
taguchi@tohoku.iij.ad.jp wrote:
> Hi, Satoh-san.
>
> > If you want to get POSTed data which encoding differ from the system
> > encoding, you could use some sort of CGI library or decode it from
> > the 'raw' input with your own library.
>
> I want to do it on Rivet itselfs.
> Why should I use "other" library?
Rivet has to keep its simplicity, I think.
We can borrow the power of many library.
Please let me know when you need 'AutoDetect' functionality. If you
serve your 'form' page in EUC-JP, it would be assumed that the input
data would be encoded using EUC-JP in most cases (I don't know the
exception).
# I don't want to raise a flame....
--
Makoto Satoh
<ma...@yahoo.co.jp>
http://yokohama.cool.ne.jp/greenhills/
__________________________________________________
Do You Yahoo!?
Yahoo! BB is Broadband by Yahoo! http://bb.yahoo.co.jp/
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Japanized Rivet (and I hope more i18n ;-)
Posted by Makoto Satoh <ma...@yahoo.co.jp>.
davidw@dedasys.com (David N. Welton) wrote:
> So - it seems as if Netscape might send either one... is that just a
> netscape bug? If so, is it possible to determine what versions it
> occurs in? Does it occur with Mozilla?
I don't know whether this is a netscape bug. Does anyone know?
HTTP requests don't have any header about the charset so we could not
determine which encoding the client wish to receive.
As to such like Java's JISAutoDetect is not perfect against short
strings.
If Rivet will have auto-detection of the charsets, it will have to cover
all charsets over the world. So for Japanese users, they have to deal
with encodings for Japanese.
--
Makoto Satoh
<ma...@yahoo.co.jp>
http://yokohama.cool.ne.jp/greenhills/
__________________________________________________
Do You Yahoo!?
Yahoo! BB is Broadband by Yahoo! http://bb.yahoo.co.jp/
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Japanized Rivet (and I hope more i18n ;-)
Posted by "David N. Welton" <da...@dedasys.com>.
Makoto Satoh <ma...@yahoo.co.jp> writes:
> davidw@dedasys.com (David N. Welton) wrote:
> > > HTTP requests don't have any header about the charset so we could
> > > not determine which encoding the client wish to receive.
> >
> > # (ENVS) HTTP_ACCEPT_CHARSET = ISO-8859-1, utf-8;q=0.66, *;q=0.66
>
>
> Oops! My poor English!
>
> I meant that browsers don't send the encoding information of the
> form-data. Sorry. Or still do I confuse?
No, I understand now, thanks.
--
David N. Welton
Consulting: http://www.dedasys.com/
Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
Apache Tcl: http://tcl.apache.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Japanized Rivet (and I hope more i18n ;-)
Posted by "David N. Welton" <da...@dedasys.com>.
Makoto Satoh <ma...@yahoo.co.jp> writes:
> davidw@dedasys.com (David N. Welton) wrote:
> > So - it seems as if Netscape might send either one... is that
> > just a netscape bug? If so, is it possible to determine what
> > versions it occurs in? Does it occur with Mozilla?
> I don't know whether this is a netscape bug. Does anyone know?
The thing to do would be to test it out on mozilla, netscape and IE.
I will have to rely on you guys to do that though.
> HTTP requests don't have any header about the charset so we could
> not determine which encoding the client wish to receive.
# (ENVS) HTTP_ACCEPT_CHARSET = ISO-8859-1, utf-8;q=0.66, *;q=0.66
Try this page:
http://tcl.apache.org/mod_dtcl/examples.ttml
and see what you get for that variable.
> As to such like Java's JISAutoDetect is not perfect against short
> strings.
> If Rivet will have auto-detection of the charsets, it will have to
> cover all charsets over the world. So for Japanese users, they have
> to deal with encodings for Japanese.
As I said earlier, I don't think auto-detect bindings should be built
in to the core, but I would be happy to include them in the core
distribution, if we get a positive reaction to the idea, and it would
be useful to japanese users.
--
David N. Welton
Consulting: http://www.dedasys.com/
Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
Apache Tcl: http://tcl.apache.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Japanized Rivet (and I hope more i18n ;-)
Posted by "David N. Welton" <da...@dedasys.com>.
Makoto Satoh <ma...@yahoo.co.jp> writes:
> > I want to do it on Rivet itselfs. Why should I use "other"
> > library?
> Rivet has to keep its simplicity, I think.
Right - this is very central to our philosophy - where possible, don't
build it into the core. That means that people who don't need the
extra stuff don't have to have it.
If this is deemed very important to Japanese language usage, I would
be willing to include a Tcl extension, or library to make sure that
users have easy access to it.
> We can borrow the power of many library.
> Please let me know when you need 'AutoDetect' functionality. If you
> serve your 'form' page in EUC-JP, it would be assumed that the input
> data would be encoded using EUC-JP in most cases (I don't know the
> exception).
So - it seems as if Netscape might send either one... is that just a
netscape bug? If so, is it possible to determine what versions it
occurs in? Does it occur with Mozilla?
As for the
var get varname ?default? ?raw?
syntax... not bad, but it's not perfect either. What if you don't
want a default? Also... 'varname' itself can be in whatever language,
so maybe 'get' isn't the right place to have this. Maybe a seperate
command, 'var raw' that returns a list like this:
{var val} {foo bar} {shoo bee} {doo wah} {var otherval}
Then you can transform it all you want, in Tcl.
Does that make sense? I need to check and make sure that it's
feasible from the C side of things.
--
David N. Welton
Consulting: http://www.dedasys.com/
Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
Apache Tcl: http://tcl.apache.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Japanized Rivet (and I hope more i18n ;-)
Posted by ta...@tohoku.iij.ad.jp.
Hi, Satoh-san.
> If you want to get POSTed data which encoding differ from the system
> encoding, you could use some sort of CGI library or decode it from
> the 'raw' input with your own library.
I want to do it on Rivet itselfs.
Why should I use "other" library?
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Japanized Rivet (and I hope more i18n ;-)
Posted by ta...@tohoku.iij.ad.jp.
Hi, David.
> Ok, to be clear... we are talking about users connecting to a
> Rivet-based system with their web browsers? The problem being that
> they might use some other kind of encoding from what you are using as
> the system encoding?
Yes.
> Maybe we should also work this in with the 'source' arguments to
> var... something like a 'raw' that gives us a list of name/value pairs
> with no encoding/decoding... Or maybe the ability to specify the type
> of encoding to use?
I think so.
Rivet Tcl command "var get" may be as following:
var get varname ?default? ?raw?
....
if raw is specified, returns the value as
"binary" string.
var all ?raw?
(just like above)
Thanks.
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org
Re: Japanized Rivet (and I hope more i18n ;-)
Posted by "David N. Welton" <da...@dedasys.com>.
taguchi@tohoku.iij.ad.jp writes:
Still getting caught up from ApacheCon... Sorry for the delay.
> Ok, Rivet can deal Japanese, Korean, Chinese language. But for
> Japanese, it have still probrem.
> In Japanese, there are 3 encodings (shiftjis, iso2022-jp, euc-jp).
> And some users use shiftjis, and the other users use iso2022-jp, or
> euc-jp.
> Rivet can deal only one encoding which is same as system encoding.
> So if system encoding is euc-jp, shiftjis and iso2022-jp users will
> get failer.
Ok, to be clear... we are talking about users connecting to a
Rivet-based system with their web browsers? The problem being that
they might use some other kind of encoding from what you are using as
the system encoding?
> Rivet may detect encoding of string which inputed by user, and may
> convert from correct encoding.
> Here is a very add-hoc patch for it.
> this patch requires auto-code detection library libwkf:
> http://www.mysticwall.com/software/wkf/
> I think this patch is too add-hoc.
> TclWeb_StringToUtf() may have something like "HOOK". HOOK means
> some tcl procedure call (if its procedure exists). user
> (programmer) can write such procedure if he/she needs. offcause
> such procedure returns encoding name for inputed strings.
Hrm, I agree that adding that patch in isn't all that clean...
> I think such hook may good for non-Japanese users. Because Rivet
> will be able to deal many encodings which is not same as system
> encodings.
Tcl is already able to do encodings which aren't the system encoding.
We just need to figure out the best way to script it. Just to be
sure, can you confirm that you can transform something, say:
���ܸ�
to unicode, and then back into another encoding, using the 'encoding'
command?
Maybe we should also work this in with the 'source' arguments to
var... something like a 'raw' that gives us a list of name/value pairs
with no encoding/decoding... Or maybe the ability to specify the type
of encoding to use?
--
David N. Welton
Consulting: http://www.dedasys.com/
Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
Apache Tcl: http://tcl.apache.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: rivet-dev-unsubscribe@tcl.apache.org
For additional commands, e-mail: rivet-dev-help@tcl.apache.org