You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@trafodion.apache.org by "Liu, Yuan (Yuan)" <yu...@esgyn.cn> on 2017/06/27 06:53:59 UTC

How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Trafodioneers,

I created one table as below,
>>create table test_utf8(a varchar(6) character set utf8);

--- SQL operation complete.
>>showddl test_utf8;

CREATE TABLE TRAFODION.SEABASE.TEST_UTF8
  (
    A                                VARCHAR(6 CHARS) CHARACTER SET UTF8
      COLLATE DEFAULT DEFAULT NULL NOT SERIALIZED
  )
ATTRIBUTES ALIGNED FORMAT
;

Then I tried to insert Chinese characters into the table,
>>insert into test_utf8 values('我我我');

*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,18 BYTES,UTF8) Source Value:我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8).
>>insert into test_utf8 values('我我');

--- 1 row(s) inserted.

It seems that I can only insert 2 Chinese characters into the column, which is not as expected as I thought.

So I‘d want to ask how should I define the data type if I want to store N Chinese characters?

Another thing is that, from the error message, why the Source Type of ‘我我我’ is 18 bytes, utf8 while the target type is 24 bytes, utf8?


Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540

Re: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Posted by Qifan Chen <qi...@esgyn.com>.

Hi Hans,


Thanks for raising these points.  Particularly the 3rd one, I think the use of terminal charset as a common charset for all the literals and the entire query makes good sense.


For you 1st point, yes, sqlci is required to parser the introducer, and looks like it is handled already.


Copyright (c) 2015-2017 Esgyn Corporation

>>set param ?x 'abc';

>>set param ?y _utf8'abc';

>>set param ?z _abc'abc';


*** ERROR[15001] A syntax error occurred at or before:

set param ?z _abc'abc';

             ^ (14 characters from start of SQL statement)


Thanks a lot.


--Qifan

________________________________
From: Hans Zeller <ha...@esgyn.com>
Sent: Wednesday, June 28, 2017 1:28:44 PM
To: user@trafodion.incubator.apache.org
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Qifan,

I had the same question some time ago. People convinced me that this was not a good idea, for the following reasons:


  *   sqlci would then have to parse the SQL statement and find the places where we have these introducers.
  *   The user would have to type in the query, using different character sets. That is not easy to do. It is better to assume that all of the query is presented in a single character set.
  *   The ISO/ANSI SQL standard says in the <literal> section, Syntax Rule 10: "Each <character representation> is a character of the source language character set." I interpret this to say that all the characters in the SQL statement come from the same character set (called "source language character set" in the standard and "terminal character set" in sqlci).

Thanks,

Hans

From: Qifan Chen [mailto:qifan.chen@esgyn.com]
Sent: Wednesday, June 28, 2017 10:24 AM
To: user@trafodion.incubator.apache.org
Subject: Re: How many Chinese characters can be stored in varchar(n chars) character set utf8?


Hi Hans,



Thanks.  That explains the situation well.



I just wonder if the scope of the TERMINAL charset should be reduced in this case since the charset of the literal is already defined in the query.



Conceptually, the terminal charset is the character set of any literals without a character set prefix.



Thanks --Qifan

________________________________
From: Hans Zeller <ha...@esgyn.com>>
Sent: Wednesday, June 28, 2017 10:41:13 AM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Qifan,

As far as I remember, these character set introducers like _utf8 tell the compiler to cast the literal as a specific charset, but they have no influence on the terminal character set. sqlci would still convert the SQL string from ISO8859-1 (its default terminal charset) to UTF-8 or UCS-2 before passing it to the SQL compiler, where the _utf8 is being interpreted.

Thanks,

Hans

From: Qifan Chen [mailto:qifan.chen@esgyn.com]
Sent: Wednesday, June 28, 2017 5:49 AM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: Re: How many Chinese characters can be stored in varchar(n chars) character set utf8?


Hi,



I tried a variation of the query as follows.



>>create table test_utf8(a varchar(6) character set utf8);



--- SQL operation complete.



>>insert into test_utf8 values(_utf8'我我我');

..



*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,18 BYTES,UTF8) Source Value:我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8).



--- 0 row(s) inserted.





Since the character set of the string value is indicated as utf8 in the query,  conceptually there is no need to use TERMINAL_CHARSET to help define the char set of the string as utf8.



In other word, the above query should work without setting the terminal charset to utf8?



Thanks --Qifan

________________________________
From: Liu, Yuan (Yuan) <yu...@esgyn.cn>>
Sent: Tuesday, June 27, 2017 11:15:07 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Hans, very appreciated for your answer, now I am clear about this part. Thanks again.

Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540

From: Hans Zeller [mailto:hans.zeller@esgyn.com]
Sent: Wednesday, June 28, 2017 11:47 AM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Yuan, let me reply to both emails here to bring the thread back together.

First, thanks for the filing the JIRA, I hope we can fix this.

In the other thread you wrote the following:

As I know, one UTF8 character corresponds to 4 bytes, that means 7 UTF8 characaters is 7*4=28 bytes. This does no match with 21 BYTES, right? I think the value of 21 should be 28.

SQL>insert into test_utf8 values('我我我我我我我');

*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,21 BYTES,UTF8) Source Value:我我我我我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8). [2017-06-28 11:23:42]


  *   Since most Chinese characters use 3 bytes, the string '我我我我我我我' is 21 bytes long in UTF-8. Character literals always have a data type CHAR(n BYTES) CHARACTER SET UTF8, with n being the exact length of the literal. So, that part of the error message is correct.
  *   The method used to create the error message, ExRaiseDetailSqlError(), does not have enough information to decide whether the data types are VARCHAR(n) or VARCHAR(n BYTES), so it always prints the length in bytes. We would have to add two more parameters to this method to fix that. I can check how much effort that would be, hopefully it's ok. In this case, it prints VARCHAR(24 BYTES) instead of VARCHAR(6). Here we use the worst-case factor 4, since a column needs to be able to hold any characters.

Thank you,

Hans


From: Liu, Yuan (Yuan) [mailto:yuan.liu@esgyn.cn]
Sent: Tuesday, June 27, 2017 8:38 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Hans,

Thanks very much, I’ve created JIRA https://issues.apache.org/jira/browse/TRAFODION-2669 , I think it will be better if we change it a little bit, including the issue I mentioned in another email. It says source type is 21 bytes and target type is 24 bytes, but in reality the value can not be inserted.

Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540

From: Hans Zeller [mailto:hans.zeller@esgyn.com]
Sent: Wednesday, June 28, 2017 11:31 AM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Yuan, you made a very good point, the error message is misleading. It shows the target data type as CHAR(24 BYTES) when in reality it is CHAR(6). I think this could be made better (probably not perfect) by a change in method ExRaiseDetailSqlError() in file ExpError.cpp.

Do you want to file a JIRA for this? I can also do that if you like. You can assign the JIRA to me.

It might also be good to change the default terminal charset of sqlci to UTF-8. What do people think, should we make that incompatible change? It may be some work, not sure it's worth it.

Thanks,

Hans

From: Liu, Yuan (Yuan) [mailto:yuan.liu@esgyn.cn]
Sent: Tuesday, June 27, 2017 8:03 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

That is good to hear that, thanks, Hans~

Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540

From: Hans Zeller [mailto:hans.zeller@esgyn.com]
Sent: Wednesday, June 28, 2017 2:38 AM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi,

The problem is how sqlci interprets its input data. In your case, sqlci interprets the data as ISO8859-1, that makes the input 3*3 bytes or 9 ISO 8859-1 characters. It then translates those 9 (junk) characters into UTF-8, requiring 18 bytes, and those don't fit into the target table. Once you tell sqlci to interpret its input as UTF-8, it works. Also, you need the terminal that you are using to use UTF-8 (in my case that's a Linux xterm which is set to UTF-8):


Apache Trafodion Conversational Interface 2.2.0
Copyright (c) 2015-2017 Apache Software Foundation
>>create table test_utf8(a varchar(6) character set utf8);

--- SQL operation complete.
>>insert into test_utf8 values('我我我');

*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,18 BYTES,UTF8) Source Value:我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8).

--- 0 row(s) inserted.
>>set terminal_charset utf8;
>>insert into test_utf8 values('我我我');

--- 1 row(s) inserted.


Note the "set terminal_charset utf8" statement in sqlci that made it work.

When I ran trafci from a Linux prompt it worked - it seems trafci on Linux assumes its input is UTF-8.

My recommendation: a) use a terminal (e.g. xterm) that is set to UTF-8, b) if using sqlci, set the terminal charset of sqlci to utf8, c) type UTF-8 characters into the terminal.

Hans


From: Liu, Yuan (Yuan) [mailto:yuan.liu@esgyn.cn]
Sent: Tuesday, June 27, 2017 12:52 AM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hmm.. Especially the error message caused misunderstanding,

21 bytes can not be inserted into 24 bytes?

*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,21 BYTES,UTF8) Source Value:我我我我我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8). [2017-06-27 15:44:45]


Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540

From: Xu, Kai-Hua (Kevin) [mailto:kaihua.xu@esgyn.cn]
Sent: Tuesday, June 27, 2017 3:49 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

I got the same error on sqlci. I would suggest you use trafci. Ture, it an issue.
[cid:image001.jpg@01D2EFF9.A6761F60]

Best Regards,
Kevin Xu

From: Xu, Kai-Hua (Kevin)
Sent: Tuesday, June 27, 2017 3:41 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Yuan,

Curious, it works for me.
[cid:image002.jpg@01D2EFF9.A6761F60]

Best Regards,
Kevin Xu

From: Liu, Yuan (Yuan) [mailto:yuan.liu@esgyn.cn]
Sent: Tuesday, June 27, 2017 2:54 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Trafodioneers,

I created one table as below,
>>create table test_utf8(a varchar(6) character set utf8);

--- SQL operation complete.
>>showddl test_utf8;

CREATE TABLE TRAFODION.SEABASE.TEST_UTF8
  (
    A                                VARCHAR(6 CHARS) CHARACTER SET UTF8
      COLLATE DEFAULT DEFAULT NULL NOT SERIALIZED
  )
ATTRIBUTES ALIGNED FORMAT
;

Then I tried to insert Chinese characters into the table,
>>insert into test_utf8 values('我我我');

*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,18 BYTES,UTF8) Source Value:我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8).
>>insert into test_utf8 values('我我');

--- 1 row(s) inserted.

It seems that I can only insert 2 Chinese characters into the column, which is not as expected as I thought.

So I‘d want to ask how should I define the data type if I want to store N Chinese characters?

Another thing is that, from the error message, why the Source Type of ‘我我我’ is 18 bytes, utf8 while the target type is 24 bytes, utf8?


Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540

RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Posted by Hans Zeller <ha...@esgyn.com>.

Hi Qifan,

I had the same question some time ago. People convinced me that this was not a good idea, for the following reasons:


  *   sqlci would then have to parse the SQL statement and find the places where we have these introducers.
  *   The user would have to type in the query, using different character sets. That is not easy to do. It is better to assume that all of the query is presented in a single character set.
  *   The ISO/ANSI SQL standard says in the <literal> section, Syntax Rule 10: "Each <character representation> is a character of the source language character set." I interpret this to say that all the characters in the SQL statement come from the same character set (called "source language character set" in the standard and "terminal character set" in sqlci).

Thanks,

Hans

From: Qifan Chen [mailto:qifan.chen@esgyn.com]
Sent: Wednesday, June 28, 2017 10:24 AM
To: user@trafodion.incubator.apache.org
Subject: Re: How many Chinese characters can be stored in varchar(n chars) character set utf8?


Hi Hans,



Thanks.  That explains the situation well.



I just wonder if the scope of the TERMINAL charset should be reduced in this case since the charset of the literal is already defined in the query.



Conceptually, the terminal charset is the character set of any literals without a character set prefix.



Thanks --Qifan

________________________________
From: Hans Zeller <ha...@esgyn.com>>
Sent: Wednesday, June 28, 2017 10:41:13 AM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Qifan,

As far as I remember, these character set introducers like _utf8 tell the compiler to cast the literal as a specific charset, but they have no influence on the terminal character set. sqlci would still convert the SQL string from ISO8859-1 (its default terminal charset) to UTF-8 or UCS-2 before passing it to the SQL compiler, where the _utf8 is being interpreted.

Thanks,

Hans

From: Qifan Chen [mailto:qifan.chen@esgyn.com]
Sent: Wednesday, June 28, 2017 5:49 AM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: Re: How many Chinese characters can be stored in varchar(n chars) character set utf8?


Hi,



I tried a variation of the query as follows.



>>create table test_utf8(a varchar(6) character set utf8);



--- SQL operation complete.



>>insert into test_utf8 values(_utf8'我我我');

..



*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,18 BYTES,UTF8) Source Value:我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8).



--- 0 row(s) inserted.





Since the character set of the string value is indicated as utf8 in the query,  conceptually there is no need to use TERMINAL_CHARSET to help define the char set of the string as utf8.



In other word, the above query should work without setting the terminal charset to utf8?



Thanks --Qifan

________________________________
From: Liu, Yuan (Yuan) <yu...@esgyn.cn>>
Sent: Tuesday, June 27, 2017 11:15:07 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Hans, very appreciated for your answer, now I am clear about this part. Thanks again.

Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540

From: Hans Zeller [mailto:hans.zeller@esgyn.com]
Sent: Wednesday, June 28, 2017 11:47 AM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Yuan, let me reply to both emails here to bring the thread back together.

First, thanks for the filing the JIRA, I hope we can fix this.

In the other thread you wrote the following:

As I know, one UTF8 character corresponds to 4 bytes, that means 7 UTF8 characaters is 7*4=28 bytes. This does no match with 21 BYTES, right? I think the value of 21 should be 28.

SQL>insert into test_utf8 values('我我我我我我我');

*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,21 BYTES,UTF8) Source Value:我我我我我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8). [2017-06-28 11:23:42]


  *   Since most Chinese characters use 3 bytes, the string '我我我我我我我' is 21 bytes long in UTF-8. Character literals always have a data type CHAR(n BYTES) CHARACTER SET UTF8, with n being the exact length of the literal. So, that part of the error message is correct.
  *   The method used to create the error message, ExRaiseDetailSqlError(), does not have enough information to decide whether the data types are VARCHAR(n) or VARCHAR(n BYTES), so it always prints the length in bytes. We would have to add two more parameters to this method to fix that. I can check how much effort that would be, hopefully it's ok. In this case, it prints VARCHAR(24 BYTES) instead of VARCHAR(6). Here we use the worst-case factor 4, since a column needs to be able to hold any characters.

Thank you,

Hans


From: Liu, Yuan (Yuan) [mailto:yuan.liu@esgyn.cn]
Sent: Tuesday, June 27, 2017 8:38 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Hans,

Thanks very much, I’ve created JIRA https://issues.apache.org/jira/browse/TRAFODION-2669 , I think it will be better if we change it a little bit, including the issue I mentioned in another email. It says source type is 21 bytes and target type is 24 bytes, but in reality the value can not be inserted.

Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540

From: Hans Zeller [mailto:hans.zeller@esgyn.com]
Sent: Wednesday, June 28, 2017 11:31 AM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Yuan, you made a very good point, the error message is misleading. It shows the target data type as CHAR(24 BYTES) when in reality it is CHAR(6). I think this could be made better (probably not perfect) by a change in method ExRaiseDetailSqlError() in file ExpError.cpp.

Do you want to file a JIRA for this? I can also do that if you like. You can assign the JIRA to me.

It might also be good to change the default terminal charset of sqlci to UTF-8. What do people think, should we make that incompatible change? It may be some work, not sure it's worth it.

Thanks,

Hans

From: Liu, Yuan (Yuan) [mailto:yuan.liu@esgyn.cn]
Sent: Tuesday, June 27, 2017 8:03 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

That is good to hear that, thanks, Hans~

Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540

From: Hans Zeller [mailto:hans.zeller@esgyn.com]
Sent: Wednesday, June 28, 2017 2:38 AM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi,

The problem is how sqlci interprets its input data. In your case, sqlci interprets the data as ISO8859-1, that makes the input 3*3 bytes or 9 ISO 8859-1 characters. It then translates those 9 (junk) characters into UTF-8, requiring 18 bytes, and those don't fit into the target table. Once you tell sqlci to interpret its input as UTF-8, it works. Also, you need the terminal that you are using to use UTF-8 (in my case that's a Linux xterm which is set to UTF-8):


Apache Trafodion Conversational Interface 2.2.0
Copyright (c) 2015-2017 Apache Software Foundation
>>create table test_utf8(a varchar(6) character set utf8);

--- SQL operation complete.
>>insert into test_utf8 values('我我我');

*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,18 BYTES,UTF8) Source Value:我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8).

--- 0 row(s) inserted.
>>set terminal_charset utf8;
>>insert into test_utf8 values('我我我');

--- 1 row(s) inserted.


Note the "set terminal_charset utf8" statement in sqlci that made it work.

When I ran trafci from a Linux prompt it worked - it seems trafci on Linux assumes its input is UTF-8.

My recommendation: a) use a terminal (e.g. xterm) that is set to UTF-8, b) if using sqlci, set the terminal charset of sqlci to utf8, c) type UTF-8 characters into the terminal.

Hans


From: Liu, Yuan (Yuan) [mailto:yuan.liu@esgyn.cn]
Sent: Tuesday, June 27, 2017 12:52 AM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hmm.. Especially the error message caused misunderstanding,

21 bytes can not be inserted into 24 bytes?

*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,21 BYTES,UTF8) Source Value:我我我我我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8). [2017-06-27 15:44:45]


Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540

From: Xu, Kai-Hua (Kevin) [mailto:kaihua.xu@esgyn.cn]
Sent: Tuesday, June 27, 2017 3:49 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

I got the same error on sqlci. I would suggest you use trafci. Ture, it an issue.
[cid:image001.jpg@01D2EFF9.A6761F60]

Best Regards,
Kevin Xu

From: Xu, Kai-Hua (Kevin)
Sent: Tuesday, June 27, 2017 3:41 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Yuan,

Curious, it works for me.
[cid:image002.jpg@01D2EFF9.A6761F60]

Best Regards,
Kevin Xu

From: Liu, Yuan (Yuan) [mailto:yuan.liu@esgyn.cn]
Sent: Tuesday, June 27, 2017 2:54 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Trafodioneers,

I created one table as below,
>>create table test_utf8(a varchar(6) character set utf8);

--- SQL operation complete.
>>showddl test_utf8;

CREATE TABLE TRAFODION.SEABASE.TEST_UTF8
  (
    A                                VARCHAR(6 CHARS) CHARACTER SET UTF8
      COLLATE DEFAULT DEFAULT NULL NOT SERIALIZED
  )
ATTRIBUTES ALIGNED FORMAT
;

Then I tried to insert Chinese characters into the table,
>>insert into test_utf8 values('我我我');

*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,18 BYTES,UTF8) Source Value:我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8).
>>insert into test_utf8 values('我我');

--- 1 row(s) inserted.

It seems that I can only insert 2 Chinese characters into the column, which is not as expected as I thought.

So I‘d want to ask how should I define the data type if I want to store N Chinese characters?

Another thing is that, from the error message, why the Source Type of ‘我我我’ is 18 bytes, utf8 while the target type is 24 bytes, utf8?


Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540

Re: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Posted by Qifan Chen <qi...@esgyn.com>.

Hi Hans,


Thanks.  That explains the situation well.


I just wonder if the scope of the TERMINAL charset should be reduced in this case since the charset of the literal is already defined in the query.


Conceptually, the terminal charset is the character set of any literals without a character set prefix.


Thanks --Qifan

________________________________
From: Hans Zeller <ha...@esgyn.com>
Sent: Wednesday, June 28, 2017 10:41:13 AM
To: user@trafodion.incubator.apache.org
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Qifan,

As far as I remember, these character set introducers like _utf8 tell the compiler to cast the literal as a specific charset, but they have no influence on the terminal character set. sqlci would still convert the SQL string from ISO8859-1 (its default terminal charset) to UTF-8 or UCS-2 before passing it to the SQL compiler, where the _utf8 is being interpreted.

Thanks,

Hans

From: Qifan Chen [mailto:qifan.chen@esgyn.com]
Sent: Wednesday, June 28, 2017 5:49 AM
To: user@trafodion.incubator.apache.org
Subject: Re: How many Chinese characters can be stored in varchar(n chars) character set utf8?


Hi,



I tried a variation of the query as follows.



>>create table test_utf8(a varchar(6) character set utf8);



--- SQL operation complete.



>>insert into test_utf8 values(_utf8'我我我');

..



*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,18 BYTES,UTF8) Source Value:我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8).



--- 0 row(s) inserted.





Since the character set of the string value is indicated as utf8 in the query,  conceptually there is no need to use TERMINAL_CHARSET to help define the char set of the string as utf8.



In other word, the above query should work without setting the terminal charset to utf8?



Thanks --Qifan

________________________________
From: Liu, Yuan (Yuan) <yu...@esgyn.cn>>
Sent: Tuesday, June 27, 2017 11:15:07 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Hans, very appreciated for your answer, now I am clear about this part. Thanks again.

Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540

From: Hans Zeller [mailto:hans.zeller@esgyn.com]
Sent: Wednesday, June 28, 2017 11:47 AM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Yuan, let me reply to both emails here to bring the thread back together.

First, thanks for the filing the JIRA, I hope we can fix this.

In the other thread you wrote the following:

As I know, one UTF8 character corresponds to 4 bytes, that means 7 UTF8 characaters is 7*4=28 bytes. This does no match with 21 BYTES, right? I think the value of 21 should be 28.

SQL>insert into test_utf8 values('我我我我我我我');

*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,21 BYTES,UTF8) Source Value:我我我我我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8). [2017-06-28 11:23:42]


  *   Since most Chinese characters use 3 bytes, the string '我我我我我我我' is 21 bytes long in UTF-8. Character literals always have a data type CHAR(n BYTES) CHARACTER SET UTF8, with n being the exact length of the literal. So, that part of the error message is correct.
  *   The method used to create the error message, ExRaiseDetailSqlError(), does not have enough information to decide whether the data types are VARCHAR(n) or VARCHAR(n BYTES), so it always prints the length in bytes. We would have to add two more parameters to this method to fix that. I can check how much effort that would be, hopefully it's ok. In this case, it prints VARCHAR(24 BYTES) instead of VARCHAR(6). Here we use the worst-case factor 4, since a column needs to be able to hold any characters.

Thank you,

Hans


From: Liu, Yuan (Yuan) [mailto:yuan.liu@esgyn.cn]
Sent: Tuesday, June 27, 2017 8:38 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Hans,

Thanks very much, I’ve created JIRA https://issues.apache.org/jira/browse/TRAFODION-2669 , I think it will be better if we change it a little bit, including the issue I mentioned in another email. It says source type is 21 bytes and target type is 24 bytes, but in reality the value can not be inserted.

Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540

From: Hans Zeller [mailto:hans.zeller@esgyn.com]
Sent: Wednesday, June 28, 2017 11:31 AM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Yuan, you made a very good point, the error message is misleading. It shows the target data type as CHAR(24 BYTES) when in reality it is CHAR(6). I think this could be made better (probably not perfect) by a change in method ExRaiseDetailSqlError() in file ExpError.cpp.

Do you want to file a JIRA for this? I can also do that if you like. You can assign the JIRA to me.

It might also be good to change the default terminal charset of sqlci to UTF-8. What do people think, should we make that incompatible change? It may be some work, not sure it's worth it.

Thanks,

Hans

From: Liu, Yuan (Yuan) [mailto:yuan.liu@esgyn.cn]
Sent: Tuesday, June 27, 2017 8:03 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

That is good to hear that, thanks, Hans~

Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540

From: Hans Zeller [mailto:hans.zeller@esgyn.com]
Sent: Wednesday, June 28, 2017 2:38 AM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi,

The problem is how sqlci interprets its input data. In your case, sqlci interprets the data as ISO8859-1, that makes the input 3*3 bytes or 9 ISO 8859-1 characters. It then translates those 9 (junk) characters into UTF-8, requiring 18 bytes, and those don't fit into the target table. Once you tell sqlci to interpret its input as UTF-8, it works. Also, you need the terminal that you are using to use UTF-8 (in my case that's a Linux xterm which is set to UTF-8):


Apache Trafodion Conversational Interface 2.2.0
Copyright (c) 2015-2017 Apache Software Foundation
>>create table test_utf8(a varchar(6) character set utf8);

--- SQL operation complete.
>>insert into test_utf8 values('我我我');

*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,18 BYTES,UTF8) Source Value:我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8).

--- 0 row(s) inserted.
>>set terminal_charset utf8;
>>insert into test_utf8 values('我我我');

--- 1 row(s) inserted.


Note the "set terminal_charset utf8" statement in sqlci that made it work.

When I ran trafci from a Linux prompt it worked - it seems trafci on Linux assumes its input is UTF-8.

My recommendation: a) use a terminal (e.g. xterm) that is set to UTF-8, b) if using sqlci, set the terminal charset of sqlci to utf8, c) type UTF-8 characters into the terminal.

Hans


From: Liu, Yuan (Yuan) [mailto:yuan.liu@esgyn.cn]
Sent: Tuesday, June 27, 2017 12:52 AM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hmm.. Especially the error message caused misunderstanding,

21 bytes can not be inserted into 24 bytes?

*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,21 BYTES,UTF8) Source Value:我我我我我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8). [2017-06-27 15:44:45]


Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540

From: Xu, Kai-Hua (Kevin) [mailto:kaihua.xu@esgyn.cn]
Sent: Tuesday, June 27, 2017 3:49 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

I got the same error on sqlci. I would suggest you use trafci. Ture, it an issue.
[cid:image001.jpg@01D2EFEA.02276E50]

Best Regards,
Kevin Xu

From: Xu, Kai-Hua (Kevin)
Sent: Tuesday, June 27, 2017 3:41 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Yuan,

Curious, it works for me.
[cid:image002.jpg@01D2EFEA.02276E50]

Best Regards,
Kevin Xu

From: Liu, Yuan (Yuan) [mailto:yuan.liu@esgyn.cn]
Sent: Tuesday, June 27, 2017 2:54 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Trafodioneers,

I created one table as below,
>>create table test_utf8(a varchar(6) character set utf8);

--- SQL operation complete.
>>showddl test_utf8;

CREATE TABLE TRAFODION.SEABASE.TEST_UTF8
  (
    A                                VARCHAR(6 CHARS) CHARACTER SET UTF8
      COLLATE DEFAULT DEFAULT NULL NOT SERIALIZED
  )
ATTRIBUTES ALIGNED FORMAT
;

Then I tried to insert Chinese characters into the table,
>>insert into test_utf8 values('我我我');

*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,18 BYTES,UTF8) Source Value:我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8).
>>insert into test_utf8 values('我我');

--- 1 row(s) inserted.

It seems that I can only insert 2 Chinese characters into the column, which is not as expected as I thought.

So I‘d want to ask how should I define the data type if I want to store N Chinese characters?

Another thing is that, from the error message, why the Source Type of ‘我我我’ is 18 bytes, utf8 while the target type is 24 bytes, utf8?


Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540

RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Posted by Hans Zeller <ha...@esgyn.com>.

Hi Qifan,

As far as I remember, these character set introducers like _utf8 tell the compiler to cast the literal as a specific charset, but they have no influence on the terminal character set. sqlci would still convert the SQL string from ISO8859-1 (its default terminal charset) to UTF-8 or UCS-2 before passing it to the SQL compiler, where the _utf8 is being interpreted.

Thanks,

Hans

From: Qifan Chen [mailto:qifan.chen@esgyn.com]
Sent: Wednesday, June 28, 2017 5:49 AM
To: user@trafodion.incubator.apache.org
Subject: Re: How many Chinese characters can be stored in varchar(n chars) character set utf8?


Hi,



I tried a variation of the query as follows.



>>create table test_utf8(a varchar(6) character set utf8);



--- SQL operation complete.



>>insert into test_utf8 values(_utf8'我我我');

..



*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,18 BYTES,UTF8) Source Value:我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8).



--- 0 row(s) inserted.





Since the character set of the string value is indicated as utf8 in the query,  conceptually there is no need to use TERMINAL_CHARSET to help define the char set of the string as utf8.



In other word, the above query should work without setting the terminal charset to utf8?



Thanks --Qifan

________________________________
From: Liu, Yuan (Yuan) <yu...@esgyn.cn>>
Sent: Tuesday, June 27, 2017 11:15:07 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Hans, very appreciated for your answer, now I am clear about this part. Thanks again.

Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540

From: Hans Zeller [mailto:hans.zeller@esgyn.com]
Sent: Wednesday, June 28, 2017 11:47 AM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Yuan, let me reply to both emails here to bring the thread back together.

First, thanks for the filing the JIRA, I hope we can fix this.

In the other thread you wrote the following:

As I know, one UTF8 character corresponds to 4 bytes, that means 7 UTF8 characaters is 7*4=28 bytes. This does no match with 21 BYTES, right? I think the value of 21 should be 28.

SQL>insert into test_utf8 values('我我我我我我我');

*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,21 BYTES,UTF8) Source Value:我我我我我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8). [2017-06-28 11:23:42]


  *   Since most Chinese characters use 3 bytes, the string '我我我我我我我' is 21 bytes long in UTF-8. Character literals always have a data type CHAR(n BYTES) CHARACTER SET UTF8, with n being the exact length of the literal. So, that part of the error message is correct.
  *   The method used to create the error message, ExRaiseDetailSqlError(), does not have enough information to decide whether the data types are VARCHAR(n) or VARCHAR(n BYTES), so it always prints the length in bytes. We would have to add two more parameters to this method to fix that. I can check how much effort that would be, hopefully it's ok. In this case, it prints VARCHAR(24 BYTES) instead of VARCHAR(6). Here we use the worst-case factor 4, since a column needs to be able to hold any characters.

Thank you,

Hans


From: Liu, Yuan (Yuan) [mailto:yuan.liu@esgyn.cn]
Sent: Tuesday, June 27, 2017 8:38 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Hans,

Thanks very much, I’ve created JIRA https://issues.apache.org/jira/browse/TRAFODION-2669 , I think it will be better if we change it a little bit, including the issue I mentioned in another email. It says source type is 21 bytes and target type is 24 bytes, but in reality the value can not be inserted.

Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540

From: Hans Zeller [mailto:hans.zeller@esgyn.com]
Sent: Wednesday, June 28, 2017 11:31 AM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Yuan, you made a very good point, the error message is misleading. It shows the target data type as CHAR(24 BYTES) when in reality it is CHAR(6). I think this could be made better (probably not perfect) by a change in method ExRaiseDetailSqlError() in file ExpError.cpp.

Do you want to file a JIRA for this? I can also do that if you like. You can assign the JIRA to me.

It might also be good to change the default terminal charset of sqlci to UTF-8. What do people think, should we make that incompatible change? It may be some work, not sure it's worth it.

Thanks,

Hans

From: Liu, Yuan (Yuan) [mailto:yuan.liu@esgyn.cn]
Sent: Tuesday, June 27, 2017 8:03 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

That is good to hear that, thanks, Hans~

Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540

From: Hans Zeller [mailto:hans.zeller@esgyn.com]
Sent: Wednesday, June 28, 2017 2:38 AM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi,

The problem is how sqlci interprets its input data. In your case, sqlci interprets the data as ISO8859-1, that makes the input 3*3 bytes or 9 ISO 8859-1 characters. It then translates those 9 (junk) characters into UTF-8, requiring 18 bytes, and those don't fit into the target table. Once you tell sqlci to interpret its input as UTF-8, it works. Also, you need the terminal that you are using to use UTF-8 (in my case that's a Linux xterm which is set to UTF-8):


Apache Trafodion Conversational Interface 2.2.0
Copyright (c) 2015-2017 Apache Software Foundation
>>create table test_utf8(a varchar(6) character set utf8);

--- SQL operation complete.
>>insert into test_utf8 values('我我我');

*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,18 BYTES,UTF8) Source Value:我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8).

--- 0 row(s) inserted.
>>set terminal_charset utf8;
>>insert into test_utf8 values('我我我');

--- 1 row(s) inserted.


Note the "set terminal_charset utf8" statement in sqlci that made it work.

When I ran trafci from a Linux prompt it worked - it seems trafci on Linux assumes its input is UTF-8.

My recommendation: a) use a terminal (e.g. xterm) that is set to UTF-8, b) if using sqlci, set the terminal charset of sqlci to utf8, c) type UTF-8 characters into the terminal.

Hans


From: Liu, Yuan (Yuan) [mailto:yuan.liu@esgyn.cn]
Sent: Tuesday, June 27, 2017 12:52 AM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hmm.. Especially the error message caused misunderstanding,

21 bytes can not be inserted into 24 bytes?

*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,21 BYTES,UTF8) Source Value:我我我我我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8). [2017-06-27 15:44:45]


Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540

From: Xu, Kai-Hua (Kevin) [mailto:kaihua.xu@esgyn.cn]
Sent: Tuesday, June 27, 2017 3:49 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

I got the same error on sqlci. I would suggest you use trafci. Ture, it an issue.
[cid:image001.jpg@01D2EFEA.02276E50]

Best Regards,
Kevin Xu

From: Xu, Kai-Hua (Kevin)
Sent: Tuesday, June 27, 2017 3:41 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: RE: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Yuan,

Curious, it works for me.
[cid:image002.jpg@01D2EFEA.02276E50]

Best Regards,
Kevin Xu

From: Liu, Yuan (Yuan) [mailto:yuan.liu@esgyn.cn]
Sent: Tuesday, June 27, 2017 2:54 PM
To: user@trafodion.incubator.apache.org<ma...@trafodion.incubator.apache.org>
Subject: How many Chinese characters can be stored in varchar(n chars) character set utf8?

Hi Trafodioneers,

I created one table as below,
>>create table test_utf8(a varchar(6) character set utf8);

--- SQL operation complete.
>>showddl test_utf8;

CREATE TABLE TRAFODION.SEABASE.TEST_UTF8
  (
    A                                VARCHAR(6 CHARS) CHARACTER SET UTF8
      COLLATE DEFAULT DEFAULT NULL NOT SERIALIZED
  )
ATTRIBUTES ALIGNED FORMAT
;

Then I tried to insert Chinese characters into the table,
>>insert into test_utf8 values('我我我');

*** ERROR[8402] A string overflow occurred during the evaluation of a character expression. Conversion of Source Type:VARCHAR(REC_BYTE_V_ASCII,18 BYTES,UTF8) Source Value:我我我 to Target Type:VARCHAR(REC_BYTE_V_ASCII,24 BYTES,UTF8).
>>insert into test_utf8 values('我我');

--- 1 row(s) inserted.

It seems that I can only insert 2 Chinese characters into the column, which is not as expected as I thought.

So I‘d want to ask how should I define the data type if I want to store N Chinese characters?

Another thing is that, from the error message, why the Source Type of ‘我我我’ is 18 bytes, utf8 while the target type is 24 bytes, utf8?


Best regards,
Yuan
Email: yuan.liu@esgyn.cn<ma...@esgyn.cn>
Cellphone: (+86) 13671935540