You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by Rui Li <li...@gmail.com> on 2020/09/03 02:31:16 UTC
Re: flink-sql-gateway hive ddl 语句parse报错 godfrey he 能帮忙看下吗？
不知道gateway模式是不是支持hive dialect。不过你确定设置方言的写法正确么？在SQL client里设置hive方言应该是
set table.sql-dialect=hive
On Thu, Sep 3, 2020 at 9:28 AM shougou <80...@qq.com> wrote:
> 使用flink-sql-gate-way 最新的master 版本，对应flink版本1.11.1
>
> 当使用 SET TABLE_SQL_DIALECT=hive 设置方言后
>
> 大概看了代码，感觉是SqlParser 的问题，是不是应该切到Flink Parser ？
>
> 执行hive ddl sql校验过不去，
> {
> "statement": "CREATE TABLE hive_table ( user_id STRING,
> order_amount
> DOUBLE ) PARTITIONED BY (dt STRING, hr STRING) STORED AS parquet
> TBLPROPERTIES ( 'partition.time-extractor.timestamp-pattern'='$dt
> $hr:00:00', 'sink.partition-commit.trigger'='partition-time',
> 'sink.partition-commit.delay'='1 h',
> 'sink.partition-commit.policy.kind'='metastore,success-file' );"
> }
>
> 报错信息如下：
> 2020-09-02 23:21:15,414 INFO
> com.ververica.flink.table.gateway.rest.session.Session [] - Session:
> ed388b8c8e094a09ffd69c4b58d463cd, run statement: CREATE TABLE hive_table
> (
> user_id STRING, order_amount DOUBLE ) PARTITIONED BY (dt STRING, hr
> STRING) STORED AS parquet TBLPROPERTIES (
> 'partition.time-extractor.timestamp-pattern'='$dt $hr:00:00',
> 'sink.partition-commit.trigger'='partition-time',
> 'sink.partition-commit.delay'='1 h',
> 'sink.partition-commit.policy.kind'='metastore,success-file' );
> 2020-09-02 23:21:15,418 ERROR
> com.ververica.flink.table.gateway.rest.session.Session [] - Session:
> ed388b8c8e094a09ffd69c4b58d463cd, Failed to parse statement: CREATE TABLE
> hive_table ( user_id STRING, order_amount DOUBLE ) PARTITIONED BY (dt
> STRING, hr STRING) STORED AS parquet TBLPROPERTIES (
> 'partition.time-extractor.timestamp-pattern'='$dt $hr:00:00',
> 'sink.partition-commit.trigger'='partition-time',
> 'sink.partition-commit.delay'='1 h',
> 'sink.partition-commit.policy.kind'='metastore,success-file' );
> 2020-09-02 23:21:15,445 ERROR
> com.ververica.flink.table.gateway.rest.handler.StatementExecuteHandler [] -
> Exception occurred in REST handler.
> org.apache.flink.runtime.rest.handler.RestHandlerException: Failed to parse
> statement.
> at
>
> com.ververica.flink.table.gateway.rest.handler.StatementExecuteHandler.handleRequest(StatementExecuteHandler.java:91)
> ~[flink-sql-gateway-0.2-SNAPSHOT.jar:?]
> at
>
> com.ververica.flink.table.gateway.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:77)
> ~[flink-sql-gateway-0.2-SNAPSHOT.jar:?]
> at
>
> com.ververica.flink.table.gateway.rest.handler.AbstractHandler.channelRead0(AbstractHandler.java:178)
> [flink-sql-gateway-0.2-SNAPSHOT.jar:?]
> at
>
> com.ververica.flink.table.gateway.rest.handler.AbstractHandler.channelRead0(AbstractHandler.java:75)
> [flink-sql-gateway-0.2-SNAPSHOT.jar:?]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:110)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:89)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:54)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:174)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:68)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> [flink-dist_2.11-1.11.1.jar:1.11.1]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
> Caused by: com.ververica.flink.table.gateway.utils.SqlGatewayException:
> Failed to parse statement.
> at
>
> com.ververica.flink.table.gateway.rest.session.Session.runStatement(Session.java:102)
> ~[flink-sql-gateway-0.2-SNAPSHOT.jar:?]
> at
>
> com.ververica.flink.table.gateway.rest.handler.StatementExecuteHandler.handleRequest(StatementExecuteHandler.java:81)
> ~[flink-sql-gateway-0.2-SNAPSHOT.jar:?]
> ... 43 more
> Caused by: org.apache.calcite.sql.parser.SqlParseException: Encountered
> "STRING" at line 1, column 88.
> Was expecting one of:
> ")" ...
> "," ...
>
> at
>
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.convertException(FlinkSqlParserImpl.java:416)
> ~[flink-table_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.normalizeException(FlinkSqlParserImpl.java:201)
> ~[flink-table_2.11-1.11.1.jar:1.11.1]
> at
> org.apache.calcite.sql.parser.SqlParser.handleException(SqlParser.java:148)
> ~[flink-table_2.11-1.11.1.jar:1.11.1]
> at
> org.apache.calcite.sql.parser.SqlParser.parseStmtList(SqlParser.java:203)
> ~[flink-table_2.11-1.11.1.jar:1.11.1]
> at
>
> com.ververica.flink.table.gateway.operation.SqlCommandParser.parseStmt(SqlCommandParser.java:115)
> ~[flink-sql-gateway-0.2-SNAPSHOT.jar:?]
> at
>
> com.ververica.flink.table.gateway.operation.SqlCommandParser.parse(SqlCommandParser.java:103)
> ~[flink-sql-gateway-0.2-SNAPSHOT.jar:?]
> at
>
> com.ververica.flink.table.gateway.rest.session.Session.runStatement(Session.java:93)
> ~[flink-sql-gateway-0.2-SNAPSHOT.jar:?]
> at
>
> com.ververica.flink.table.gateway.rest.handler.StatementExecuteHandler.handleRequest(StatementExecuteHandler.java:81)
> ~[flink-sql-gateway-0.2-SNAPSHOT.jar:?]
> ... 43 more
> Caused by: org.apache.flink.sql.parser.impl.ParseException: Encountered
> "STRING" at line 1, column 88.
> Was expecting one of:
> ")" ...
> "," ...
>
> at
>
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.generateParseException(FlinkSqlParserImpl.java:36086)
> ~[flink-table_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.jj_consume_token(FlinkSqlParserImpl.java:35900)
> ~[flink-table_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.ParenthesizedSimpleIdentifierList(FlinkSqlParserImpl.java:21398)
> ~[flink-table_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.SqlCreateTable(FlinkSqlParserImpl.java:5292)
> ~[flink-table_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.SqlCreateExtended(FlinkSqlParserImpl.java:6269)
> ~[flink-table_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.SqlCreate(FlinkSqlParserImpl.java:19047)
> ~[flink-table_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.SqlStmt(FlinkSqlParserImpl.java:3308)
> ~[flink-table_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.SqlStmtList(FlinkSqlParserImpl.java:2775)
> ~[flink-table_2.11-1.11.1.jar:1.11.1]
> at
>
> org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.parseSqlStmtList(FlinkSqlParserImpl.java:252)
> ~[flink-table_2.11-1.11.1.jar:1.11.1]
> at
> org.apache.calcite.sql.parser.SqlParser.parseStmtList(SqlParser.java:201)
> ~[flink-table_2.11-1.11.1.jar:1.11.1]
> at
>
> com.ververica.flink.table.gateway.operation.SqlCommandParser.parseStmt(SqlCommandParser.java:115)
> ~[flink-sql-gateway-0.2-SNAPSHOT.jar:?]
> at
>
> com.ververica.flink.table.gateway.operation.SqlCommandParser.parse(SqlCommandParser.java:103)
> ~[flink-sql-gateway-0.2-SNAPSHOT.jar:?]
> at
>
> com.ververica.flink.table.gateway.rest.session.Session.runStatement(Session.java:93)
> ~[flink-sql-gateway-0.2-SNAPSHOT.jar:?]
> at
>
> com.ververica.flink.table.gateway.rest.handler.StatementExecuteHandler.handleRequest(StatementExecuteHandler.java:81)
> ~[flink-sql-gateway-0.2-SNAPSHOT.jar:?]
> ... 43 more
>
>
>
>
> --
> Sent from: http://apache-flink.147419.n8.nabble.com/
>
--
Best regards!
Rui Li
Re: flink-sql-gateway hive ddl 语句parse报错 godfrey he 能帮忙看下吗？
Posted by shougou <80...@qq.com>.
TABLE_SQL_DIALECT这个key在tableConfig里对应的就是table.sql-dialect ，设置hive
dialect这个是没有问题的，需要在yml配置文件里加上configuration,我在tableenv里查看configuration的table.sql-dialect已经变成hive了，但是hive
DDL语句仍然过不了sqlparse。
--
Sent from: http://apache-flink.147419.n8.nabble.com/