You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by "xie_guoguo@163.com" <xi...@163.com> on 2021/07/14 01:39:53 UTC
flink时态表：两个Hbase左关联有报错情况
您好，有关flinkSQL时态表左关联时遇到了问题。
具体场景：
两个Hbase表做时态表关联，左表定义了处理时间，右表原本不变(rowkey,cf)，FLINK版本1.13.0，报错情况如下，麻烦问一下大家又遇到类似问题嘛，该怎么处理！
2021-07-14 09:22:20.592 WARN org.apache.flink.runtime.taskmanager.Task --- 2021-07-14 09:22:20.596 WARN org.apache.flink.runtime.taskmanager.Task --- LookupJoin(table=[default_catalog.default_database.hbase_source_pollution_dwb_enterprise_wodhz], joinType=[LeftOuterJoin], async=[true], lookup=[code=$f4], select=[code, data1, data2, p, $f4, code0, data]) -> Calc(select=[code, ROW(,....,data.activ) -> NotNullEnforcer(fields=[EXPR$1, EXPR$2, p, EXPR$4]) (3/3)#3 (4ada188e117c67ccd9bd6488ae95216a) switched from RUNNING to FAILED with failure cause: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: org/apache/flink/hbase/shaded/org/apache/commons/io/IOUtils
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928
at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.quiesceTimeServiceAndCloseOperator(StreamOperatorWrapper.java:168)
at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.close(StreamOperatorWrapper.java:131)
at org.apache.flink.streaming.runtime.tasks.OperatorChain.closeOperators(OperatorChain.java:448)
at org.apache.flink.streaming.runtime.tasks.StreamTask.afterInvoke(StreamTask.java:671)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:629)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:776)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/hbase/shaded/org/apache/commons/io/IOUtils
at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.AsyncConnectionImpl.close(AsyncConnectionImpl.java:193)
at org.apache.flink.connector.hbase2.source.HBaseRowDataAsyncLookupFunction.close(HBaseRowDataAsyncLookupFunction.java:251)
at LookupFunction$3.close(Unknown Source
ps：同样的代码，左表换成Kafka，能够正常运行。网上搜索了一下相关报错，好像时缺包，目前pom文件是有hbase-client、hbase-commen、flink-sql-connector-hbase。
Sincerely，
xie_guoguo@163.com
Re: flink时态表：两个Hbase左关联有报错情况
Posted by zhisheng <zh...@gmail.com>.
是不是加了 'lookup.async' = 'true'，当 rowkey 为 null 的时候会出现这个问题
https://issues.apache.org/jira/browse/FLINK-24528
Michael Ran <gr...@163.com> 于2021年7月23日周五 上午10:44写道：
> java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError:
> org/apache/flink/hbase/shaded/org/apache/commons/io/IOUtils<br/><br/>缺jar
> 在 2021-07-14 09:39:53，"xie_guoguo@163.com" <xi...@163.com> 写道：
> >您好，有关flinkSQL时态表左关联时遇到了问题。
> >具体场景：
> >
> 两个Hbase表做时态表关联，左表定义了处理时间，右表原本不变(rowkey,cf)，FLINK版本1.13.0，报错情况如下，麻烦问一下大家又遇到类似问题嘛，该怎么处理！
> >
> >2021-07-14 09:22:20.592 WARN org.apache.flink.runtime.taskmanager.Task
> --- 2021-07-14 09:22:20.596 WARN
> org.apache.flink.runtime.taskmanager.Task ---
> LookupJoin(table=[default_catalog.default_database.hbase_source_pollution_dwb_enterprise_wodhz],
> joinType=[LeftOuterJoin], async=[true], lookup=[code=$f4], select=[code,
> data1, data2, p, $f4, code0, data]) -> Calc(select=[code,
> ROW(,....,data.activ) -> NotNullEnforcer(fields=[EXPR$1, EXPR$2, p,
> EXPR$4]) (3/3)#3 (4ada188e117c67ccd9bd6488ae95216a) switched from RUNNING
> to FAILED with failure cause: java.util.concurrent.ExecutionException:
> java.lang.NoClassDefFoundError:
> org/apache/flink/hbase/shaded/org/apache/commons/io/IOUtils
> >at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> >at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928
> >at
> org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.quiesceTimeServiceAndCloseOperator(StreamOperatorWrapper.java:168)
> >at
> org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.close(StreamOperatorWrapper.java:131)
> >at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.closeOperators(OperatorChain.java:448)
> >at
> org.apache.flink.streaming.runtime.tasks.StreamTask.afterInvoke(StreamTask.java:671)
> >at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:629)
> >at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:776)
> >at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563)
> >at java.lang.Thread.run(Thread.java:748)
> >Caused by: java.lang.NoClassDefFoundError:
> org/apache/flink/hbase/shaded/org/apache/commons/io/IOUtils
> >at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.AsyncConnectionImpl.close(AsyncConnectionImpl.java:193)
> >at
> org.apache.flink.connector.hbase2.source.HBaseRowDataAsyncLookupFunction.close(HBaseRowDataAsyncLookupFunction.java:251)
> >at LookupFunction$3.close(Unknown Source
> >
>
> >ps：同样的代码，左表换成Kafka，能够正常运行。网上搜索了一下相关报错，好像时缺包，目前pom文件是有hbase-client、hbase-commen、flink-sql-connector-hbase。
> >
> >
> >
> >Sincerely，
> >xie_guoguo@163.com
>
Re:flink时态表：两个Hbase左关联有报错情况
Posted by Michael Ran <gr...@163.com>.
java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: org/apache/flink/hbase/shaded/org/apache/commons/io/IOUtils<br/><br/>缺jar
在 2021-07-14 09:39:53，"xie_guoguo@163.com" <xi...@163.com> 写道：
>您好，有关flinkSQL时态表左关联时遇到了问题。
>具体场景：
> 两个Hbase表做时态表关联，左表定义了处理时间，右表原本不变(rowkey,cf)，FLINK版本1.13.0，报错情况如下，麻烦问一下大家又遇到类似问题嘛，该怎么处理！
>
>2021-07-14 09:22:20.592 WARN org.apache.flink.runtime.taskmanager.Task --- 2021-07-14 09:22:20.596 WARN org.apache.flink.runtime.taskmanager.Task --- LookupJoin(table=[default_catalog.default_database.hbase_source_pollution_dwb_enterprise_wodhz], joinType=[LeftOuterJoin], async=[true], lookup=[code=$f4], select=[code, data1, data2, p, $f4, code0, data]) -> Calc(select=[code, ROW(,....,data.activ) -> NotNullEnforcer(fields=[EXPR$1, EXPR$2, p, EXPR$4]) (3/3)#3 (4ada188e117c67ccd9bd6488ae95216a) switched from RUNNING to FAILED with failure cause: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: org/apache/flink/hbase/shaded/org/apache/commons/io/IOUtils
>at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
>at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928
>at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.quiesceTimeServiceAndCloseOperator(StreamOperatorWrapper.java:168)
>at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.close(StreamOperatorWrapper.java:131)
>at org.apache.flink.streaming.runtime.tasks.OperatorChain.closeOperators(OperatorChain.java:448)
>at org.apache.flink.streaming.runtime.tasks.StreamTask.afterInvoke(StreamTask.java:671)
>at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:629)
>at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:776)
>at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563)
>at java.lang.Thread.run(Thread.java:748)
>Caused by: java.lang.NoClassDefFoundError: org/apache/flink/hbase/shaded/org/apache/commons/io/IOUtils
>at org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.AsyncConnectionImpl.close(AsyncConnectionImpl.java:193)
>at org.apache.flink.connector.hbase2.source.HBaseRowDataAsyncLookupFunction.close(HBaseRowDataAsyncLookupFunction.java:251)
>at LookupFunction$3.close(Unknown Source
>
>ps：同样的代码，左表换成Kafka，能够正常运行。网上搜索了一下相关报错，好像时缺包，目前pom文件是有hbase-client、hbase-commen、flink-sql-connector-hbase。
>
>
>
>Sincerely，
>xie_guoguo@163.com
Re: flink时态表：两个Hbase左关联有报错情况
Posted by Caizhi Weng <ts...@gmail.com>.
Hi！
flink classpath 下是否有 hbase connector 相关的 jar 包呢？如果用户代码里没有用到 hbase
相关的类和方法，其实用户代码的 pom 文件里是不需要这些依赖的。
xie_guoguo@163.com <xi...@163.com> 于2021年7月14日周三 上午9:43写道：
> 您好，有关flinkSQL时态表左关联时遇到了问题。
> 具体场景：
>
> 两个Hbase表做时态表关联，左表定义了处理时间，右表原本不变(rowkey,cf)，FLINK版本1.13.0，报错情况如下，麻烦问一下大家又遇到类似问题嘛，该怎么处理！
>
> 2021-07-14 09:22:20.592 WARN org.apache.flink.runtime.taskmanager.Task
> --- 2021-07-14 09:22:20.596 WARN
> org.apache.flink.runtime.taskmanager.Task ---
> LookupJoin(table=[default_catalog.default_database.hbase_source_pollution_dwb_enterprise_wodhz],
> joinType=[LeftOuterJoin], async=[true], lookup=[code=$f4], select=[code,
> data1, data2, p, $f4, code0, data]) -> Calc(select=[code,
> ROW(,....,data.activ) -> NotNullEnforcer(fields=[EXPR$1, EXPR$2, p,
> EXPR$4]) (3/3)#3 (4ada188e117c67ccd9bd6488ae95216a) switched from RUNNING
> to FAILED with failure cause: java.util.concurrent.ExecutionException:
> java.lang.NoClassDefFoundError:
> org/apache/flink/hbase/shaded/org/apache/commons/io/IOUtils
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928
> at
> org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.quiesceTimeServiceAndCloseOperator(StreamOperatorWrapper.java:168)
> at
> org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.close(StreamOperatorWrapper.java:131)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.closeOperators(OperatorChain.java:448)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.afterInvoke(StreamTask.java:671)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:629)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:776)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/flink/hbase/shaded/org/apache/commons/io/IOUtils
> at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.AsyncConnectionImpl.close(AsyncConnectionImpl.java:193)
> at
> org.apache.flink.connector.hbase2.source.HBaseRowDataAsyncLookupFunction.close(HBaseRowDataAsyncLookupFunction.java:251)
> at LookupFunction$3.close(Unknown Source
>
>
> ps：同样的代码，左表换成Kafka，能够正常运行。网上搜索了一下相关报错，好像时缺包，目前pom文件是有hbase-client、hbase-commen、flink-sql-connector-hbase。
>
>
>
> Sincerely，
> xie_guoguo@163.com
>