You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Yuan,Youjun" <yu...@baidu.com> on 2018/09/14 08:49:47 UTC
Conversion to relational algebra failed to preserve datatypes
Hi,
I am getting the following error while submitting job to a cluster, which seems failed to compare 2 RelDateTypes, though they seems identical (from the error message), and everything is OK if I run it locally.
I guess calcite failed to compare the first field named ts, of type TIMESTAMP(3), because:
* If I don't select ts, then everything goes fine
* If I cast ts to other type, like SELECT cast(ts AS TIMESTAMP), then everything is fine
* If I switch to EventTime, the issue also goes away. Currently it's ProcessTime
I am using Flink 1.4, and submitting job to a standalone cluster.
Below are the error:
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Caused by: org.apache.flink.client.program.ProgramInvocationException: The program caused an error:
at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:93)
at org.apache.flink.client.program.ClusterClient.getOptimizedPlan(ClusterClient.java:334)
at org.apache.flink.runtime.webmonitor.handlers.JarActionHandler.getJobGraphAndClassLoader(JarActionHandler.java:76)
at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleJsonRequest$0(JarRunHandler.java:69) ... 9 more
Caused by: java.lang.AssertionError: Conversion to relational algebra failed to preserve datatypes:
validated type:
RecordType(TIMESTAMP(3) NOT NULL ts, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" userId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" batchId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" projectId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" vehicleId, CHAR(5) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL field0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" rule0, DOUBLE threshold0, DOUBLE field_value0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" alarmId) NOT NULL
converted type:
RecordType(TIMESTAMP(3) NOT NULL ts, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" userId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" batchId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" projectId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" vehicleId, CHAR(5) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL field0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" rule0, DOUBLE threshold0, DOUBLE field_value0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" alarmId) NOT NULL
...
at org.apache.calcite.sql2rel.SqlToRelConverter.checkConvertedType(SqlToRelConverter.java:451)
at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:567)
at org.apache.flink.table.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:106)
----------------------------------------------------------------------------------------------------------------------------------------------------------------
thanks in advance,
youjun
答复: Conversion to relational algebra failed to preserve datatypes
Posted by "Yuan,Youjun" <yu...@baidu.com>.
Hi Timo,
I believe it has something to do with the number of field in the source. I can reproduce the issue with very simple SQL, like:
INSERT INTO IVC_ALARM_PUBLIC_OUT SELECT rowtime as ts FROM IVC_ALARM_PUBLIC_IN
Where rowtime actually represent proctime (I registered it as rowtime.proctime).
If I remove a few fields in the source, then everything goes fine.
Thanks
Youjun
发件人: Timo Walther <tw...@apache.org>
发送时间: Friday, September 14, 2018 9:17 PM
收件人: user@flink.apache.org
主题: Re: Conversion to relational algebra failed to preserve datatypes
Hi,
could you maybe post the query that caused the exception? I guess the exception is related to a time attribute [1] for the optimizer time attributes and timestamps make no difference however they have a slightly different data type that might have caused the error. I think is a bug that should be fixed, once we have more context.
Regards,
Timo
[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming.html#time-attributes
Am 14.09.18 um 10:49 schrieb Yuan,Youjun:
Hi,
I am getting the following error while submitting job to a cluster, which seems failed to compare 2 RelDateTypes, though they seems identical (from the error message), and everything is OK if I run it locally.
I guess calcite failed to compare the first field named ts, of type TIMESTAMP(3), because:
* If I don’t select ts, then everything goes fine
* If I cast ts to other type, like SELECT cast(ts AS TIMESTAMP), then everything is fine
* If I switch to EventTime, the issue also goes away. Currently it’s ProcessTime
I am using Flink 1.4, and submitting job to a standalone cluster.
Below are the error:
----------------------------------------------------------------------------------------------------------------------------------------------------------------
Caused by: org.apache.flink.client.program.ProgramInvocationException: The program caused an error:
at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:93)
at org.apache.flink.client.program.ClusterClient.getOptimizedPlan(ClusterClient.java:334)
at org.apache.flink.runtime.webmonitor.handlers.JarActionHandler.getJobGraphAndClassLoader(JarActionHandler.java:76)
at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleJsonRequest$0(JarRunHandler.java:69) ... 9 more
Caused by: java.lang.AssertionError: Conversion to relational algebra failed to preserve datatypes:
validated type:
RecordType(TIMESTAMP(3) NOT NULL ts, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" userId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" batchId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" projectId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" vehicleId, CHAR(5) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL field0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" rule0, DOUBLE threshold0, DOUBLE field_value0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" alarmId) NOT NULL
converted type:
RecordType(TIMESTAMP(3) NOT NULL ts, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" userId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" batchId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" projectId, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" vehicleId, CHAR(5) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL field0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" rule0, DOUBLE threshold0, DOUBLE field_value0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" alarmId) NOT NULL
...
at org.apache.calcite.sql2rel.SqlToRelConverter.checkConvertedType(SqlToRelConverter.java:451)
at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:567)
at org.apache.flink.table.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:106)
----------------------------------------------------------------------------------------------------------------------------------------------------------------
thanks in advance,
youjun
Re: Conversion to relational algebra failed to preserve datatypes
Posted by Timo Walther <tw...@apache.org>.
Hi,
could you maybe post the query that caused the exception? I guess the
exception is related to a time attribute [1] for the optimizer time
attributes and timestamps make no difference however they have a
slightly different data type that might have caused the error. I think
is a bug that should be fixed, once we have more context.
Regards,
Timo
[1]
https://ci.apache.org/projects/flink/flink-docs-master/dev/table/streaming.html#time-attributes
Am 14.09.18 um 10:49 schrieb Yuan,Youjun:
>
> Hi,
>
> I am getting the following error while submitting job to a cluster,
> which seems failed to compare 2 RelDateTypes, though they seems
> identical (from the error message), and everything is OK if I run it
> locally.
>
> I guess calcite failed to compare the first field named *ts*, of type
> *TIMESTAMP(3)*, because:
>
> * If I don’t select ts, then everything goes fine
> * If I cast ts to other type, like SELECT cast(ts AS TIMESTAMP),
> then everything is fine
> * If I switch to EventTime, the issue also goes away. Currently it’s
> ProcessTime
>
> I am using Flink 1.4, and submitting job to a standalone cluster.
>
> Below are the error:
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Caused by: org.apache.flink.client.program.ProgramInvocationException:
> The program caused an error:
>
> at
> org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:93)
>
>
> at
> org.apache.flink.client.program.ClusterClient.getOptimizedPlan(ClusterClient.java:334)
>
>
> at
> org.apache.flink.runtime.webmonitor.handlers.JarActionHandler.getJobGraphAndClassLoader(JarActionHandler.java:76)
>
>
> at
> org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleJsonRequest$0(JarRunHandler.java:69)
> ... 9 more
>
> Caused by: java.lang.AssertionError: Conversion to relational algebra
> failed to preserve datatypes:
>
> validated type:
>
> RecordType(TIMESTAMP(3) NOT NULL ts, VARCHAR(65536) CHARACTER SET
> "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" userId, VARCHAR(65536)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" batchId,
> VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" projectId, VARCHAR(65536) CHARACTER SET
> "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" vehicleId, CHAR(5)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL
> field0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" rule0, DOUBLE threshold0, DOUBLE
> field_value0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" alarmId) NOT NULL
>
> converted type:
>
> RecordType(TIMESTAMP(3) NOT NULL ts, VARCHAR(65536) CHARACTER SET
> "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" userId, VARCHAR(65536)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" batchId,
> VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" projectId, VARCHAR(65536) CHARACTER SET
> "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" vehicleId, CHAR(5)
> CHARACTER SET "UTF-16LE" COLLATE "ISO-8859-1$en_US$primary" NOT NULL
> field0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" rule0, DOUBLE threshold0, DOUBLE
> field_value0, VARCHAR(65536) CHARACTER SET "UTF-16LE" COLLATE
> "ISO-8859-1$en_US$primary" alarmId) NOT NULL
>
> ...
>
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.checkConvertedType(SqlToRelConverter.java:451)
>
>
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:567)
>
>
> at
> org.apache.flink.table.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:106)
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> thanks in advance,
>
> youjun
>