You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@linkis.apache.org by rita <ri...@163.com> on 2022/10/12 02:08:40 UTC
[DISCUSS] An error occurs when data class is submitted to yarn
Dear:
Wechat group "Apache Linkis community Development group "chat records are as follows: 微信群"Apache Linkis 社区开发群"的聊天记录如下:
-- -- -- 2022-10-9 -- -- -- -
Zosimer 23:54
@peacewong@WDS
Zosimer 23:54
The message says the class cannot be found
-- -- -- 2022-10-10 -- -- -- -
peacewong@WDS 00:34
You can look at the stderr log and remove the prompt for conflicting log4j. This problem is included in the FAQ.
Zosimer 10:24
I don't think so
Zosimer 10:24
I got rid of that and I still made that mistake
Zosimer 10:25
select * and select count are both fine, so join two tables with an error
Zosimer 10:26
It is not committed to yarn
Zosimer 10:26
An error is reported before committing to yarn
peacewong@WDS 10:28
Let's put this jar in there, okay?
Zosimer 11:29
@peacewong@WDS Everything has been tried
Zosimer 11:29
Various permutations and combinations
Zosimer 11:29
[his face]
Yang Shiyue pleases,
There should be a configuration of hiveengineconfiguration hive_lib_home
Yang Shiyue now
You can configure this parameter or disable maplocaltask of hive
China 11:42
Probably the same question I had last time
China 11:42
Less a jar
China"
this
China 11:44
@Zosimer try this jar in lib
Zosimer 11:50
What exactly is this parameter
Zosimer 11:51
Do you remember
Zosimer 11:51
If I close this, will there be no progress bar
Yang Shiyue 11:51
I won't
Zosimer 11:52
What is this specific parameter
Zosimer 11:52
What are the parameter names
Zosimer 11:53
I didn't find it on Baidu
Yang Shiyue 11:53
Search hive to disable mapjoin
Zosimer 11:54
So this is going to close mapjoin, right?
Yang Shiyue 11:54
Yes.
Zosimer 11:55
Can be solved by jar package, this kind of turn off automatic mapjoin is not performance impact
Yang Shiyue now
So let's do it this way
Zosimer 12:02
Does this parameter have a specific name? Is hive_lib_home the full name
Yang Shiyue 12:04
You can see the code, specific I also forgot [facepalm] set cost to hive directory should be good
Zosimer 12:05
this
Yang Shiyue 12:05
Yes, hive.lib should be that
Zosimer 12:06
This should be unified to make a parameter
Zosimer 12:06
Configure it in the configuration file
Zosimer 12:07
DWS at the beginning of
Zosimer 14:32
All right, thank you
————— 2022-10-9 —————
Zosimer 23:54
@peacewong@WDS
Zosimer 23:54
提示说类找不到
————— 2022-10-10 —————
peacewong@WDS 00:34
可以看看stderr的日志下，把提示的冲突的log4j移除下，常见问题里面有这个问题的。
Zosimer 10:24
好像不行
Zosimer 10:24
我去掉了还是报这个错
Zosimer 10:25
很奇怪，select * 和select count 都没问题，两张表join就会报错
Zosimer 10:26
而且不会提交到yarn
Zosimer 10:26
在提交到yarn之前就报错
peacewong@WDS 10:28
把这个jar加进来看看？
Zosimer 11:29
@peacewong@WDS 各种办法都试了
Zosimer 11:29
各种排列组合
Zosimer 11:29
[捂脸]
杨峙岳 11:36
应该有个参数的，hiveengineconfiguration的hive_lib_home
杨峙岳 11:37
这个可以配置一下这个参数，或者关闭hive的maplocaltask
华 11:42
应该跟我上次那个问题差不多
华 11:42
还少个jar
华 11:43
这个
华 11:44
@Zosimer 把这jar放到lib试试
Zosimer 11:50
这个具体什么参数
Zosimer 11:51
记得吗
Zosimer 11:51
如果关闭这个是不是会没有进度条
杨峙岳 11:51
不会的
Zosimer 11:52
这个具体参数是什么
Zosimer 11:52
参数名称是啥
Zosimer 11:53
我百度搜没搜到
杨峙岳 11:53
直接搜 hive关闭mapjoin
Zosimer 11:54
这个就是关闭mapjoin？
杨峙岳 11:54
是的
Zosimer 11:55
能不能通过jar包的方式解决啊，这种关闭自动mapjoin是不是有性能方面的影响
杨峙岳 11:55
那就用这个方法吧
Zosimer 12:02
这个参数有具体的名称吗？，就是hive_lib_home是全名吗
杨峙岳 12:04
可以看下代码，具体我也忘了[捂脸] 设置成本地hive的目录应该就好
Zosimer 12:05
这个
杨峙岳 12:05
是的，hive.lib 应该就是这个
Zosimer 12:06
@杨峙岳 这个应该统一搞个参数
Zosimer 12:06
在配置文件里面配置
Zosimer 12:07
dws开头的
Zosimer 14:32
可以了，感谢