[DISCUSS]Spark Engine Enable Problem

[Chen Xia <ca...@apache.org>]

kinghao 3-6 22:48:31
@奇世豪 Brother Shihao, please help me to read this, I got an error when
submitting the data_calc task

Xi Shihao 3-6 22:49:35
You need to put the mysql driver in the spark ec package

kinghao 3-6 22:49:36
I see that linkis-computation-engineconn-1.3.2-SNAPSHOT.jar is in the
spark plug-in. I copied the jdbc driver package to the spark plug-in
and still reported this error.

Xi Shihao 3-6 22:54:23
I don’t have this error here, can you see the list of jar packages?

kinghao 3-6 23:57:59
@奇世豪 After repackaging lib.zip, I executed the task and reported the
following error, please help me


Xi Shihao 3-7 07:34:44
Check this to see if the version of the package is inconsistent common-lang3

Xi Shihao 3-7 08:12:35
@赵金福 Post the complete background log, stdout stderr

Xi Shihao 3-7 10:53:15
I see that the commons-lang3 referenced in the system is this,
Check if there are different versions in your spark startup package or elsewhere

Xi Shihao 3-7 10:53:21
Maybe it's the conflict

Xi Shihao 3-7 10:53:53
It is the jar package referenced when starting the spark executor

kinghao 3-7 11:05:14
ok, let me check
kinghao 3-7 11:32:31
does have a low version

kinghao 3-7 11:32:56
Can you run tasks normally there?

Xi Shihao 3-7 11:59:19
We are normal here

Xi Shihao 3-7 12:51:29
I have seen this package for a long time

kinghao 3-7 13:07:10
It's in the bag above
-----
kinghao 3-6 22:48:31
@郗世豪  世豪兄，帮忙看下这个，提交data_calc任务的时候报的错

郗世豪 3-6 22:49:35
需要把mysql驱动放在spark ec 包里

kinghao 3-6 22:49:36
我看linkis-computation-engineconn-1.3.2-SNAPSHOT.jar这个是spark插件里面的，我把jdbc驱动包拷贝到spark插件里面还是报这个错

郗世豪  3-6 22:54:23
我这里没有这个报错呢，能看看jar包列表吗

kinghao 3-6 23:57:59
@郗世豪  我重新打包 lib.zip 后执行任务报下面的错，麻烦帮看下


郗世豪 3-7 07:34:44
这个看一下是不是包的版本不一致common-lang3

郗世豪 3-7 08:12:35
@郝金福 完整的后台日志发一下吧，stdout stderr

郗世豪 3-7 10:53:15
我看系统里引用的这个commons-lang3 是这个，
你看看你们spark启动包或者其他地方是不是有不同版本

郗世豪 3-7 10:53:21
可能是这个冲突了

郗世豪 3-7 10:53:53
就是启动spark executor的时候引用的jar包

kinghao 3-7 11:05:14
好的我排查下
kinghao 3-7 11:32:31
确实有一个低版本

kinghao 3-7 11:32:56
你那里可以正常跑任务吗

郗世豪 3-7 11:59:19
我们这里正常的

郗世豪 3-7 12:51:29
这个包我看加了好久了

kinghao 3-7 13:07:10
是上面那个包里面的


Best Regards!
Chen Xia.