You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by 程骥 <ji...@qq.com> on 2017/07/25 10:18:56 UTC
How can I set charset for flink sql？
My sql like this(contain a Chinese word)
Get exception when I submit the job to cluster.
Is there anyone tell me how to deal with it,thanks!
Re: How can I set charset for flink sql？
Posted by Timo Walther <tw...@apache.org>.
Hi,
currently Flink does not support this charset in a LIKE expression. This
is due to a limitation in the Apache Calcite library. Maybe you can open
an issue there.
The easiest solution for this is to implement your own scalar function,
that does a `string.contains("XXXX")`.
Here you can find some explanation how to implement a scalar function:
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/table/udfs.html#scalar-functions
I hope that helps.
Regards,
Timo
Am 25.07.17 um 12:18 schrieb 程骥:
> My sql like this(contain a Chinese word)
>
> Get exception when I submit the job to cluster.
>
> Is there anyone tell me how to deal with it,thanks!
Re: How can I set charset for flink sql？
Posted by Fabian Hueske <fh...@gmail.com>.
As Timo proposed, I would implement a Scalar user-defined function which
returns a boolean and use that instead of LIKE.
Have a look here [1].
Best, Fabian
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/table/udfs.html#scalar-functions
2017-07-26 3:47 GMT+02:00 Ted Yu <yu...@gmail.com>:
> Logged CALCITE-1903 for this bug.
>
> FYI
>
> On Tue, Jul 25, 2017 at 6:39 PM, 程骥 <ji...@qq.com> wrote:
>
>> OK，thanks for remind me.
>>
>> My sql like this(contain a Chinese word):
>>
>> SELECT
>> 'HIGH' AS LEVEL,
>> 'Firewall uplink bandwidth exception:greater than 10000' AS content,
>> `system.process.username`,
>> `system.process.memory.rss.bytes`
>> FROM
>> test
>> WHERE
>> `system.process.username` LIKE '%高危%'
>> AND
>> `system.process.memory.rss.bytes` > 10000
>>
>> Get exception when I submit the job to cluster.
>>
>> Caused by: org.apache.calcite.runtime.CalciteException: Failed to encode
>> '%高危%' in character set 'ISO-8859-1'
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> ~[na:1.8.0_45]
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
>> ~[na:1.8.0_45]
>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
>> Source) ~[na:1.8.0_45]
>> at java.lang.reflect.Constructor.newInstance(Unknown Source)
>> ~[na:1.8.0_45]
>> at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
>> ~[flink-table_2.11-1.3.1.jar:1.3.1]
>> at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:572)
>> ~[flink-table_2.11-1.3.1.jar:1.3.1]
>> at org.apache.calcite.util.NlsString.<init>(NlsString.java:81)
>> ~[flink-table_2.11-1.3.1.jar:1.3.1]
>> at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:864)
>> ~[flink-table_2.11-1.3.1.jar:1.3.1]
>> at org.apache.calcite.rex.RexBuilder.makeCharLiteral(RexBuilder.java:1051)
>> ~[flink-table_2.11-1.3.1.jar:1.3.1]
>> at org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convert
>> Literal(SqlNodeToRexConverterImpl.java:117)
>> ~[flink-table_2.11-1.3.1.jar:1.3.1]
>> at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4408)
>> ~[flink-table_2.11-1.3.1.jar:1.3.1]
>> at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:3787)
>> ~[flink-table_2.11-1.3.1.jar:1.3.1]
>> at org.apache.calcite.sql.SqlLiteral.accept(SqlLiteral.java:427)
>> ~[flink-table_2.11-1.3.1.jar:1.3.1]
>> at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.conv
>> ertExpression(SqlToRelConverter.java:4321) ~[flink-table_2.11-1.3.1.jar:1
>> .3.1]
>> at org.apache.calcite.sql2rel.StandardConvertletTable.convertEx
>> pressionList(StandardConvertletTable.java:968)
>> ~[flink-table_2.11-1.3.1.jar:1.3.1]
>> at org.apache.calcite.sql2rel.StandardConvertletTable.convertCa
>> ll(StandardConvertletTable.java:944) ~[flink-table_2.11-1.3.1.jar:1.3.1]
>> at org.apache.calcite.sql2rel.StandardConvertletTable.convertCa
>> ll(StandardConvertletTable.java:928) ~[flink-table_2.11-1.3.1.jar:1.3.1]
>> ... 50 common frames omitted
>>
>> Is there anyone tell me how to deal with it,thanks!
>>
>> ------------------ 原始邮件 ------------------
>> *发件人:* "Nico Kruber";<ni...@data-artisans.com>;
>> *发送时间:* 2017年7月25日(星期二) 晚上11:48
>> *收件人:* "user"<us...@flink.apache.org>;
>> *抄送:* "程骥"<ji...@qq.com>;
>> *主题:* Re: How can I set charset for flink sql？
>>
>> Please, for the sake of making your email searchable, do not post stack
>> traces
>> as screenshots but rather text into your email.
>>
>> On Tuesday, 25 July 2017 12:18:56 CEST 程骥 wrote:
>> > My sql like this(contain a Chinese word)
>> >
>> > Get exception when I submit the job to cluster.
>> >
>> >
>> >
>> > Is there anyone tell me how to deal with it,thanks!
>>
>>
>>
>
Re: How can I set charset for flink sql？
Posted by Ted Yu <yu...@gmail.com>.
Logged CALCITE-1903 for this bug.
FYI
On Tue, Jul 25, 2017 at 6:39 PM, 程骥 <ji...@qq.com> wrote:
> OK，thanks for remind me.
>
> My sql like this(contain a Chinese word):
>
> SELECT
> 'HIGH' AS LEVEL,
> 'Firewall uplink bandwidth exception:greater than 10000' AS content,
> `system.process.username`,
> `system.process.memory.rss.bytes`
> FROM
> test
> WHERE
> `system.process.username` LIKE '%高危%'
> AND
> `system.process.memory.rss.bytes` > 10000
>
> Get exception when I submit the job to cluster.
>
> Caused by: org.apache.calcite.runtime.CalciteException: Failed to encode
> '%高危%' in character set 'ISO-8859-1'
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> ~[na:1.8.0_45]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
> ~[na:1.8.0_45]
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
> Source) ~[na:1.8.0_45]
> at java.lang.reflect.Constructor.newInstance(Unknown Source)
> ~[na:1.8.0_45]
> at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463)
> ~[flink-table_2.11-1.3.1.jar:1.3.1]
> at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:572)
> ~[flink-table_2.11-1.3.1.jar:1.3.1]
> at org.apache.calcite.util.NlsString.<init>(NlsString.java:81)
> ~[flink-table_2.11-1.3.1.jar:1.3.1]
> at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:864)
> ~[flink-table_2.11-1.3.1.jar:1.3.1]
> at org.apache.calcite.rex.RexBuilder.makeCharLiteral(RexBuilder.java:1051)
> ~[flink-table_2.11-1.3.1.jar:1.3.1]
> at org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertLiteral(
> SqlNodeToRexConverterImpl.java:117) ~[flink-table_2.11-1.3.1.jar:1.3.1]
> at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.
> visit(SqlToRelConverter.java:4408) ~[flink-table_2.11-1.3.1.jar:1.3.1]
> at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.
> visit(SqlToRelConverter.java:3787) ~[flink-table_2.11-1.3.1.jar:1.3.1]
> at org.apache.calcite.sql.SqlLiteral.accept(SqlLiteral.java:427)
> ~[flink-table_2.11-1.3.1.jar:1.3.1]
> at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.
> convertExpression(SqlToRelConverter.java:4321)
> ~[flink-table_2.11-1.3.1.jar:1.3.1]
> at org.apache.calcite.sql2rel.StandardConvertletTable.
> convertExpressionList(StandardConvertletTable.java:968)
> ~[flink-table_2.11-1.3.1.jar:1.3.1]
> at org.apache.calcite.sql2rel.StandardConvertletTable.convertCall(
> StandardConvertletTable.java:944) ~[flink-table_2.11-1.3.1.jar:1.3.1]
> at org.apache.calcite.sql2rel.StandardConvertletTable.convertCall(
> StandardConvertletTable.java:928) ~[flink-table_2.11-1.3.1.jar:1.3.1]
> ... 50 common frames omitted
>
> Is there anyone tell me how to deal with it,thanks!
>
> ------------------ 原始邮件 ------------------
> *发件人:* "Nico Kruber";<ni...@data-artisans.com>;
> *发送时间:* 2017年7月25日(星期二) 晚上11:48
> *收件人:* "user"<us...@flink.apache.org>;
> *抄送:* "程骥"<ji...@qq.com>;
> *主题:* Re: How can I set charset for flink sql？
>
> Please, for the sake of making your email searchable, do not post stack
> traces
> as screenshots but rather text into your email.
>
> On Tuesday, 25 July 2017 12:18:56 CEST 程骥 wrote:
> > My sql like this(contain a Chinese word)
> >
> > Get exception when I submit the job to cluster.
> >
> >
> >
> > Is there anyone tell me how to deal with it,thanks!
>
>
>
回复： How can I set charset for flink sql？
Posted by 程骥 <ji...@qq.com>.
OK，thanks for remind me.
My sql like this(contain a Chinese word):
SELECT
'HIGH' AS LEVEL,
'Firewall uplink bandwidth exception:greater than 10000' AS content,
`system.process.username`,
`system.process.memory.rss.bytes`
FROM
test
WHERE
`system.process.username` LIKE '%高危%'
AND
`system.process.memory.rss.bytes` > 10000
Get exception when I submit the job to cluster.
Caused by: org.apache.calcite.runtime.CalciteException: Failed to encode '%高危%' in character set 'ISO-8859-1'
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_45]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[na:1.8.0_45]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[na:1.8.0_45]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[na:1.8.0_45]
at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:463) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:572) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.util.NlsString.<init>(NlsString.java:81) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.rex.RexBuilder.makeLiteral(RexBuilder.java:864) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.rex.RexBuilder.makeCharLiteral(RexBuilder.java:1051) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.SqlNodeToRexConverterImpl.convertLiteral(SqlNodeToRexConverterImpl.java:117) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4408) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:3787) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql.SqlLiteral.accept(SqlLiteral.java:427) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4321) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.StandardConvertletTable.convertExpressionList(StandardConvertletTable.java:968) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.StandardConvertletTable.convertCall(StandardConvertletTable.java:944) ~[flink-table_2.11-1.3.1.jar:1.3.1]
at org.apache.calcite.sql2rel.StandardConvertletTable.convertCall(StandardConvertletTable.java:928) ~[flink-table_2.11-1.3.1.jar:1.3.1]
... 50 common frames omitted
Is there anyone tell me how to deal with it,thanks!
------------------ 原始邮件 ------------------
发件人: "Nico Kruber";<ni...@data-artisans.com>;
发送时间: 2017年7月25日(星期二) 晚上11:48
收件人: "user"<us...@flink.apache.org>;
抄送: "程骥"<ji...@qq.com>;
主题: Re: How can I set charset for flink sql？
Please, for the sake of making your email searchable, do not post stack traces
as screenshots but rather text into your email.
On Tuesday, 25 July 2017 12:18:56 CEST 程骥 wrote:
> My sql like this(contain a Chinese word)
>
> Get exception when I submit the job to cluster.
>
>
>
> Is there anyone tell me how to deal with it,thanks!
Re: How can I set charset for flink sql？
Posted by Nico Kruber <ni...@data-artisans.com>.
Please, for the sake of making your email searchable, do not post stack traces
as screenshots but rather text into your email.
On Tuesday, 25 July 2017 12:18:56 CEST 程骥 wrote:
> My sql like this(contain a Chinese word)
>
> Get exception when I submit the job to cluster.
>
>
>
> Is there anyone tell me how to deal with it,thanks!