You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by guanyq <dl...@163.com> on 2023/03/26 04:06:34 UTC
flink1.16 sql gateway hive2
本地启动了flink及hive在启动sql gateway时有以下异常，请问还需要其他什么操作么
./bin/sql-gateway.sh start-foreground -Dsql-gateway.endpoint.type=hiveserver2 -Dsql-gateway.endpoint.hiveserver2.catalog.hive-conf-dir=/usr/local/app/apache-hive-3.1.2-bin/conf
异常信息
Available factory identifiers are:
rest
at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:545) ~[flink-table-api-java-uber-1.16.0.jar:1.16.0]
at org.apache.flink.table.gateway.api.endpoint.SqlGatewayEndpointFactoryUtils.createSqlGatewayEndpoint(SqlGatewayEndpointFactoryUtils.java:65) ~[flink-table-api-java-uber-1.16.0.jar:1.16.0]
at org.apache.flink.table.gateway.SqlGateway.start(SqlGateway.java:72) [flink-sql-gateway-1.16.0.jar:1.16.0]
at org.apache.flink.table.gateway.SqlGateway.startSqlGateway(SqlGateway.java:118) [flink-sql-gateway-1.16.0.jar:1.16.0]
at org.apache.flink.table.gateway.SqlGateway.main(SqlGateway.java:98) [flink-sql-gateway-1.16.0.jar:1.16.0]
Exception in thread "main" org.apache.flink.table.gateway.api.utils.SqlGatewayException: Failed to start the endpoints.
at org.apache.flink.table.gateway.SqlGateway.start(SqlGateway.java:79)
at org.apache.flink.table.gateway.SqlGateway.startSqlGateway(SqlGateway.java:118)
at org.apache.flink.table.gateway.SqlGateway.main(SqlGateway.java:98)
Caused by: org.apache.flink.table.api.ValidationException: Could not find any factory for identifier 'hiveserver2' that implements 'SqlGatewayEndpointFactory' in the classpath.
Available factory identifiers are:
rest
at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:545)
at org.apache.flink.table.gateway.api.endpoint.SqlGatewayEndpointFactoryUtils.createSqlGatewayEndpoint(SqlGatewayEndpointFactoryUtils.java:65)
at org.apache.flink.table.gateway.SqlGateway.start(SqlGateway.java:72)
... 2 more
flink1.17.1版本 flink on yarn 提交无法获取配置文件
Posted by guanyq <dl...@163.com>.
/opt/flink/flink-1.17.1/bin/flink run-application -t yarn-application -yjm 1024m -ytm 1024m ./xx-1.0.jar ./config.properties以上提交命令制定的配置文件，为什么在容器内找配置文件？file /home/yarn/nm/usercache/root/appcache/application_1690773368385_0092/container_e183_1690773368385_0092_01_000001/./config.properties does not exist
flink1.10.1版本，读取日志目录问题
Posted by guanyq <dl...@163.com>.
请问下
flink如何监控并实时读取远程服务器的日志目录中所有日志文件内容
日志服务器(可以ssh连接，IP/用户名/密码)
Re:Re: flink1.16 sql gateway hive2
Posted by guanyq <dl...@163.com>.
感谢两位老师
在 2023-03-27 09:58:24，"Shengkai Fang" <fs...@gmail.com> 写道：
>方勇老师说的没错。我们在文档里面也加了如何配置 hiveserver2 endpoint 的文档[1]
>
>[1]
>https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/hive-compatibility/hiveserver2/#setting-up
>
>Shammon FY <zj...@gmail.com> 于2023年3月27日周一 08:41写道：
>
>> Hi
>>
>>
>> 如果要启动hiveserver2协议的gateway，需要将jar包flink-connector-hive_${scala.binary.version}放入到gateway的lib目录
>>
>> Best,
>> Shammon FY
>>
>>
>> On Sun, Mar 26, 2023 at 12:07 PM guanyq <dl...@163.com> wrote:
>>
>> > 本地启动了flink及hive在启动sql gateway时有以下异常，请问还需要其他什么操作么
>> > ./bin/sql-gateway.sh start-foreground
>> > -Dsql-gateway.endpoint.type=hiveserver2
>> >
>> -Dsql-gateway.endpoint.hiveserver2.catalog.hive-conf-dir=/usr/local/app/apache-hive-3.1.2-bin/conf
>> >
>> >
>> > 异常信息
>> >
>> > Available factory identifiers are:
>> > rest
>> > at
>> >
>> org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:545)
>> > ~[flink-table-api-java-uber-1.16.0.jar:1.16.0]
>> > at
>> >
>> org.apache.flink.table.gateway.api.endpoint.SqlGatewayEndpointFactoryUtils.createSqlGatewayEndpoint(SqlGatewayEndpointFactoryUtils.java:65)
>> > ~[flink-table-api-java-uber-1.16.0.jar:1.16.0]
>> > at org.apache.flink.table.gateway.SqlGateway.start(SqlGateway.java:72)
>> > [flink-sql-gateway-1.16.0.jar:1.16.0]
>> > at
>> >
>> org.apache.flink.table.gateway.SqlGateway.startSqlGateway(SqlGateway.java:118)
>> > [flink-sql-gateway-1.16.0.jar:1.16.0]
>> > at org.apache.flink.table.gateway.SqlGateway.main(SqlGateway.java:98)
>> > [flink-sql-gateway-1.16.0.jar:1.16.0]
>> > Exception in thread "main"
>> > org.apache.flink.table.gateway.api.utils.SqlGatewayException: Failed to
>> > start the endpoints.
>> > at org.apache.flink.table.gateway.SqlGateway.start(SqlGateway.java:79)
>> > at
>> >
>> org.apache.flink.table.gateway.SqlGateway.startSqlGateway(SqlGateway.java:118)
>> > at org.apache.flink.table.gateway.SqlGateway.main(SqlGateway.java:98)
>> > Caused by: org.apache.flink.table.api.ValidationException: Could not find
>> > any factory for identifier 'hiveserver2' that implements
>> > 'SqlGatewayEndpointFactory' in the classpath.
>> > Available factory identifiers are:
>> > rest
>> > at
>> >
>> org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:545)
>> > at
>> >
>> org.apache.flink.table.gateway.api.endpoint.SqlGatewayEndpointFactoryUtils.createSqlGatewayEndpoint(SqlGatewayEndpointFactoryUtils.java:65)
>> > at org.apache.flink.table.gateway.SqlGateway.start(SqlGateway.java:72)
>> > ... 2 more
>> >
>> >
>>
flink1.17.1版本
Posted by guanyq <dl...@163.com>.
请问，flink sql 能否通过sql语句将mysql表加载为flink 内存表
sql语句为多表关联
Re: flink1.17.1版本 MiniCluster is not yet running or has already been shut down.
Posted by Shammon FY <zj...@gmail.com>.
Hi,
运行的是哪个例子？从错误上看是在从MiniCluster获取结果的时候，MiniCluster被关闭了
Best,
Shammon FY
On Sat, Jul 22, 2023 at 3:25 PM guanyq <dl...@163.com> wrote:
> 本地IDEA运行 MiniCluster is not yet running or has already been shut down.
> 请问是什么原因，如何处理
>
>
>
>
> 15:19:27,511 INFO
> org.apache.flink.runtime.resourcemanager.ResourceManagerServiceImpl [] -
> Stopping resource manager service.
>
> 15:19:27,503 WARN
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher [] -
> Failed to get job status so we assume that the job has terminated. Some
> data might be lost.
>
> java.lang.IllegalStateException: MiniCluster is not yet running or has
> already been shut down.
>
> at org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)
> ~[flink-core-1.17.1.jar:1.17.1]
>
> at
> org.apache.flink.runtime.minicluster.MiniCluster.getDispatcherGatewayFuture(MiniCluster.java:1060)
> ~[flink-runtime-1.17.1.jar:1.17.1]
>
> at
> org.apache.flink.runtime.minicluster.MiniCluster.runDispatcherCommand(MiniCluster.java:933)
> ~[flink-runtime-1.17.1.jar:1.17.1]
>
> at
> org.apache.flink.runtime.minicluster.MiniCluster.getJobStatus(MiniCluster.java:857)
> ~[flink-runtime-1.17.1.jar:1.17.1]
>
> at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.getJobStatus(MiniClusterJobClient.java:91)
> ~[flink-runtime-1.17.1.jar:1.17.1]
>
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.isJobTerminated(CollectResultFetcher.java:210)
> [flink-streaming-java-1.17.1.jar:1.17.1]
>
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:118)
> [flink-streaming-java-1.17.1.jar:1.17.1]
>
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
> [flink-streaming-java-1.17.1.jar:1.17.1]
>
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> [flink-streaming-java-1.17.1.jar:1.17.1]
>
> at
> org.apache.flink.table.planner.connectors.CollectDynamicSink$CloseableRowIteratorWrapper.hasNext(CollectDynamicSink.java:222)
> [flink-table-planner_25e35ab8-6377-4c6a-a928-a9fe1ff9e7f4.jar:1.17.1]
>
> at
> org.apache.flink.table.utils.print.TableauStyle.print(TableauStyle.java:120)
> [flink-table-common-1.17.1.jar:1.17.1]
>
> at
> org.apache.flink.table.api.internal.TableResultImpl.print(TableResultImpl.java:153)
> [flink-table-api-java-1.17.1.jar:1.17.1]
flink1.17.1版本 MiniCluster is not yet running or has already been shut down.
Posted by guanyq <dl...@163.com>.
本地IDEA运行 MiniCluster is not yet running or has already been shut down. 请问是什么原因，如何处理
15:19:27,511 INFO org.apache.flink.runtime.resourcemanager.ResourceManagerServiceImpl [] - Stopping resource manager service.
15:19:27,503 WARN org.apache.flink.streaming.api.operators.collect.CollectResultFetcher [] - Failed to get job status so we assume that the job has terminated. Some data might be lost.
java.lang.IllegalStateException: MiniCluster is not yet running or has already been shut down.
at org.apache.flink.util.Preconditions.checkState(Preconditions.java:193) ~[flink-core-1.17.1.jar:1.17.1]
at org.apache.flink.runtime.minicluster.MiniCluster.getDispatcherGatewayFuture(MiniCluster.java:1060) ~[flink-runtime-1.17.1.jar:1.17.1]
at org.apache.flink.runtime.minicluster.MiniCluster.runDispatcherCommand(MiniCluster.java:933) ~[flink-runtime-1.17.1.jar:1.17.1]
at org.apache.flink.runtime.minicluster.MiniCluster.getJobStatus(MiniCluster.java:857) ~[flink-runtime-1.17.1.jar:1.17.1]
at org.apache.flink.runtime.minicluster.MiniClusterJobClient.getJobStatus(MiniClusterJobClient.java:91) ~[flink-runtime-1.17.1.jar:1.17.1]
at org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.isJobTerminated(CollectResultFetcher.java:210) [flink-streaming-java-1.17.1.jar:1.17.1]
at org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:118) [flink-streaming-java-1.17.1.jar:1.17.1]
at org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106) [flink-streaming-java-1.17.1.jar:1.17.1]
at org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80) [flink-streaming-java-1.17.1.jar:1.17.1]
at org.apache.flink.table.planner.connectors.CollectDynamicSink$CloseableRowIteratorWrapper.hasNext(CollectDynamicSink.java:222) [flink-table-planner_25e35ab8-6377-4c6a-a928-a9fe1ff9e7f4.jar:1.17.1]
at org.apache.flink.table.utils.print.TableauStyle.print(TableauStyle.java:120) [flink-table-common-1.17.1.jar:1.17.1]
at org.apache.flink.table.api.internal.TableResultImpl.print(TableResultImpl.java:153) [flink-table-api-java-1.17.1.jar:1.17.1]
Re: flink1.16 sql gateway hive2
Posted by Shengkai Fang <fs...@gmail.com>.
方勇老师说的没错。我们在文档里面也加了如何配置 hiveserver2 endpoint 的文档[1]
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/hive-compatibility/hiveserver2/#setting-up
Shammon FY <zj...@gmail.com> 于2023年3月27日周一 08:41写道：
> Hi
>
>
> 如果要启动hiveserver2协议的gateway，需要将jar包flink-connector-hive_${scala.binary.version}放入到gateway的lib目录
>
> Best,
> Shammon FY
>
>
> On Sun, Mar 26, 2023 at 12:07 PM guanyq <dl...@163.com> wrote:
>
> > 本地启动了flink及hive在启动sql gateway时有以下异常，请问还需要其他什么操作么
> > ./bin/sql-gateway.sh start-foreground
> > -Dsql-gateway.endpoint.type=hiveserver2
> >
> -Dsql-gateway.endpoint.hiveserver2.catalog.hive-conf-dir=/usr/local/app/apache-hive-3.1.2-bin/conf
> >
> >
> > 异常信息
> >
> > Available factory identifiers are:
> > rest
> > at
> >
> org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:545)
> > ~[flink-table-api-java-uber-1.16.0.jar:1.16.0]
> > at
> >
> org.apache.flink.table.gateway.api.endpoint.SqlGatewayEndpointFactoryUtils.createSqlGatewayEndpoint(SqlGatewayEndpointFactoryUtils.java:65)
> > ~[flink-table-api-java-uber-1.16.0.jar:1.16.0]
> > at org.apache.flink.table.gateway.SqlGateway.start(SqlGateway.java:72)
> > [flink-sql-gateway-1.16.0.jar:1.16.0]
> > at
> >
> org.apache.flink.table.gateway.SqlGateway.startSqlGateway(SqlGateway.java:118)
> > [flink-sql-gateway-1.16.0.jar:1.16.0]
> > at org.apache.flink.table.gateway.SqlGateway.main(SqlGateway.java:98)
> > [flink-sql-gateway-1.16.0.jar:1.16.0]
> > Exception in thread "main"
> > org.apache.flink.table.gateway.api.utils.SqlGatewayException: Failed to
> > start the endpoints.
> > at org.apache.flink.table.gateway.SqlGateway.start(SqlGateway.java:79)
> > at
> >
> org.apache.flink.table.gateway.SqlGateway.startSqlGateway(SqlGateway.java:118)
> > at org.apache.flink.table.gateway.SqlGateway.main(SqlGateway.java:98)
> > Caused by: org.apache.flink.table.api.ValidationException: Could not find
> > any factory for identifier 'hiveserver2' that implements
> > 'SqlGatewayEndpointFactory' in the classpath.
> > Available factory identifiers are:
> > rest
> > at
> >
> org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:545)
> > at
> >
> org.apache.flink.table.gateway.api.endpoint.SqlGatewayEndpointFactoryUtils.createSqlGatewayEndpoint(SqlGatewayEndpointFactoryUtils.java:65)
> > at org.apache.flink.table.gateway.SqlGateway.start(SqlGateway.java:72)
> > ... 2 more
> >
> >
>
Re: flink1.16 sql gateway hive2
Posted by Shammon FY <zj...@gmail.com>.
Hi
如果要启动hiveserver2协议的gateway，需要将jar包flink-connector-hive_${scala.binary.version}放入到gateway的lib目录
Best,
Shammon FY
On Sun, Mar 26, 2023 at 12:07 PM guanyq <dl...@163.com> wrote:
> 本地启动了flink及hive在启动sql gateway时有以下异常，请问还需要其他什么操作么
> ./bin/sql-gateway.sh start-foreground
> -Dsql-gateway.endpoint.type=hiveserver2
> -Dsql-gateway.endpoint.hiveserver2.catalog.hive-conf-dir=/usr/local/app/apache-hive-3.1.2-bin/conf
>
>
> 异常信息
>
> Available factory identifiers are:
> rest
> at
> org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:545)
> ~[flink-table-api-java-uber-1.16.0.jar:1.16.0]
> at
> org.apache.flink.table.gateway.api.endpoint.SqlGatewayEndpointFactoryUtils.createSqlGatewayEndpoint(SqlGatewayEndpointFactoryUtils.java:65)
> ~[flink-table-api-java-uber-1.16.0.jar:1.16.0]
> at org.apache.flink.table.gateway.SqlGateway.start(SqlGateway.java:72)
> [flink-sql-gateway-1.16.0.jar:1.16.0]
> at
> org.apache.flink.table.gateway.SqlGateway.startSqlGateway(SqlGateway.java:118)
> [flink-sql-gateway-1.16.0.jar:1.16.0]
> at org.apache.flink.table.gateway.SqlGateway.main(SqlGateway.java:98)
> [flink-sql-gateway-1.16.0.jar:1.16.0]
> Exception in thread "main"
> org.apache.flink.table.gateway.api.utils.SqlGatewayException: Failed to
> start the endpoints.
> at org.apache.flink.table.gateway.SqlGateway.start(SqlGateway.java:79)
> at
> org.apache.flink.table.gateway.SqlGateway.startSqlGateway(SqlGateway.java:118)
> at org.apache.flink.table.gateway.SqlGateway.main(SqlGateway.java:98)
> Caused by: org.apache.flink.table.api.ValidationException: Could not find
> any factory for identifier 'hiveserver2' that implements
> 'SqlGatewayEndpointFactory' in the classpath.
> Available factory identifiers are:
> rest
> at
> org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:545)
> at
> org.apache.flink.table.gateway.api.endpoint.SqlGatewayEndpointFactoryUtils.createSqlGatewayEndpoint(SqlGatewayEndpointFactoryUtils.java:65)
> at org.apache.flink.table.gateway.SqlGateway.start(SqlGateway.java:72)
> ... 2 more
>
>