[DISCUSS]Spark 2.4.3 plugin is also related to the hadoop version？

[Andy <an...@163.com>]

————— 2022-5-19 —————

Mr. Lightning 10:10
Guys, please tell me if the spark 2.4.3 plugin is also related to the hadoop version.
Now I can run the spark-sql command on the ecm node, and the job can also go to yarn, but the linkis cli fails to run, and the spark context report fails.
Does this mean I need to recompile the spark engine?

Mr. Lightning 10:11
linkis cli jobs can be submitted to yarn

Mr. Lightning 10:14
error initializing sparkcontext

peacewong@WDS 10:15
There should be no difference between the two, but the submission entry is different

Mr. Lightning 10:18
i try again
spark-sql -e

Mr. Lightning 10:23
I can run spark-sql on both nodes, but I can't run it successfully with linkis-cli-spark-sql

Mr. Lightning 10:23
Failed to initialize engine

peacewong@WDS 10:37
Brother Hua Jin, you can send the log of ec to see

peacewong@WDS 10:38
You can consider removing the cache and changing it to configurable. The default configuration is to use the cache. You can come to an issue and pr

Mr. Lightning 10:52
I read it. I probably know that our yarn cluster doesn't know about my client [Wang Chai]

Mr. Lightning 10:53
I will try again later thank you peace

peacewong@WDS 10:53
[shake hands]



—————  2022-5-19  —————

闪电先生 10:10
各位大佬 请教一下 spark 2.4.3那个插件，是不是也是和hadoop版本有关的。
我现在spark-sql命令在ecm节点上面是可以跑的，作业也可以到yarn，但是用linkis cli跑不成功，报spark上下文失败。
这是说明我要重新编译spark引擎吗？

闪电先生 10:11
linkis cli作业可以提交到yarn

闪电先生 10:14
error initializing sparkcontext

peacewong@WDS 10:15
这两个应该没有区别来着，只是提交入口不一样

闪电先生 10:18
我再试一下
spark-sql -e

闪电先生 10:23
我二个节点spark-sql都可以跑，用linkis-cli-spark-sql跑不成功

闪电先生 10:23
初始化引擎失败

peacewong@WDS 10:37
华金兄可以把ec的日志发下看看

peacewong@WDS 10:38
可以考虑把cache移除下，改成可以配置的，默认配置走缓存。可以来个issue和pr下

闪电先生 10:52
我看了。我大概知道了 我们的yarn集群不知道我的这个客户端[旺柴]

闪电先生 10:53
我等会再试试 谢谢和平

peacewong@WDS 10:53
[握手]


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For additional commands, e-mail: dev-help@linkis.apache.org