You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@linkis.apache.org by GitBox <gi...@apache.org> on 2022/04/12 03:47:26 UTC
[GitHub] [incubator-linkis] shenbinglife opened a new issue, #1952: [Bug] Python exec failed, workdir log shows start spark engine and guava jar conflict !!!
shenbinglife opened a new issue, #1952:
URL: https://github.com/apache/incubator-linkis/issues/1952
### Search before asking
- [X] I searched the [issues](https://github.com/apache/incubator-linkis/issues) and found no similar issues.
### Linkis Component
linkis-cg-engineConnplugin
### What happened + What you expected to happen
1. Exec python in Scripts
```python
print("asdf")
```
2. page return failed， notice
```
Job with execId-IDE_root_spark_1 and subJobId : 12 from orchestrator completed with state ErrorExecuteResponse(21304, Task is Failed,errorMsg: errCode: 12003 ,desc: cq-test-dtp-cdh-10-3-104-43:9101_2 Failed to async get EngineNode AMErrorException: errCode: 30002 ,desc: ServiceInstance(linkis-cg-engineconn, cq-test-dtp-cdh-10-3-104-43:34171) ticketID:ee7f2405-9e34-4e36-a1d4-0fa0fc8942d2 初始化引擎失败,原因: Failed to start EngineConn, reason: ECMErrorException: errCode: 10000 ,desc: Failed to init DefaultEngineConn(ShuttingDown, ee7f2405-9e34-4e36-a1d4-0fa0fc8942d2, org.apache.linkis.manager.engineplugin.common.resource.UserNodeResource@1bd1844d, [[key: userCreator, value: {"creator":"IDE","user":"root"}, str: root-IDE], [key: engineType, value: {"engineType":"spark","version":"2.4.3"}, str: spark-2.4.3]], EngineConnCreationDescImpl(dssmark_id: mark_2,null,{wds.linkis.rm.yarnqueue.memory.max=300G, wds.linkis.rm.client.core.max=10, wds.linkis.rm.instance=10, wds.linkis.rm
.yarnqueue.instance.max=30, wds.linkis.rm.yarnqueue.cores.max=150, wds.linkis.rm.client.memory.max=20G, wds.linkis.rm.yarnqueue=default}), org.apache.linkis.ecm.core.engineconn.EngineConnInfo@33e362f3, null, org.apache.linkis.ecm.core.launch.EngineConnLaunchRunnerImpl@6bcd8032, ServiceInstance(linkis-cg-engineconn, cq-test-dtp-cdh-10-3-104-43:34171), null), status shutting down ,ip: cq-test-dtp-cdh-10-3-104-43 ,port: 9102 ,serviceKind: linkis-cg-engineconnmanager
You can go to this path(/appcom/tmp/root/workDir/ee7f2405-9e34-4e36-a1d4-0fa0fc8942d2/logs) to find the reason or ask the administrator for help ,ip: cq-test-dtp-cdh-10-3-104-43 ,port: 9101 ,serviceKind: linkis-cg-linkismanager ,ip: cq-test-dtp-cdh-10-3-104-43 ,port: 9104 ,serviceKind: linkis-cg-entrance,null)
```
3. see workDir log , find guava conflict
```
The following method did not exist:
com.google.common.cache.CacheBuilder.recordStats()Lcom/google/common/cache/CacheBuilder;
The method's class, com.google.common.cache.CacheBuilder, is available from the following locations:
jar:file:/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/jars/guava-11.0.2.jar!/com/google/common/cache/CacheBuilder.class
jar:file:/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/spark/hive/hive-exec-2.1.1-cdh6.3.0.jar!/com/google/common/cache/CacheBuilder.class
The class hierarchy was loaded from the following locations:
com.google.common.cache.CacheBuilder: file:/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/jars/guava-11.0.2.jar
```
4. see workDir engineConnExec.sh, And why it's a spark job? I am running python
```shell
spark-submit --master yarn --deploy-mode client --name Linkis-EngineConn-Spark_IDE --jars ,${UDF_JARS} --conf spark.driver.extraJavaOptions=" -server -XX:+UseG1GC -XX:MaxPermSize=250m -XX:PermSize=128m -Xloggc:${LOG_DIRS}/gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Dwds.linkis.server.conf=linkis-engineconn.properties -Dwds.linkis.gateway.url=http://****:9201 -Dlogging.file=log4j2-engineconn.xml -DTICKET_ID=7de696ac-ae00-4478-bd49-62a2b4f78a76 -Djava.io.tmpdir=${TEMP_DIRS} -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=${RANDOM_PORT}" --driver-memory 2g --driver-class-path :${CLASSPATH} --driver-cores 1 --executor-memory 4g --executor-cores 2 --num-executors 3 --queue default --class org.apache.linkis.engineconn.launch.EngineConnServer 1> ${LOG_DIRS}/stdout 2>> ${LOG_DIRS}/stderr ${PWD}/lib/linkis-engineplugin-spark-1.0.3.jar --engineconn-conf wds.linkis.rm.instance=10 --engineconn-conf label.userCreator=root-IDE --engineconn-conf tick
etId=7de696ac-ae00-4478-bd49-62a2b4f78a76 --engineconn-conf wds.linkis.rm.yarnqueue.memory.max=300G --engineconn-conf label.engineType=spark-2.4.3 --engineconn-conf wds.linkis.rm.yarnqueue.instance.max=30 --engineconn-conf wds.linkis.rm.client.memory.max=20G --engineconn-conf wds.linkis.rm.client.core.max=10 --engineconn-conf wds.linkis.rm.yarnqueue.cores.max=150 --engineconn-conf user=root --engineconn-conf wds.linkis.rm.yarnqueue=default --spring-conf eureka.client.serviceUrl.defaultZone=http://****:20303/eureka/ --spring-conf logging.config=classpath:log4j2-engineconn.xml --spring-conf spring.profiles.active=engineconn --spring-conf server.port=46287 --spring-conf spring.application.name=linkis-cg-engineconn
```
### Relevent platform
DSS 1.0.1
Linkis 1.0.3
CDH6.3.0
### Reproduction script
see before
### Anything else
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For additional commands, e-mail: dev-help@linkis.apache.org
[GitHub] [incubator-linkis] casionone commented on issue #1952: [Bug] Python exec failed, workdir log shows start spark engine and guava jar conflict !!!
Posted by GitBox <gi...@apache.org>.
casionone commented on issue #1952:
URL: https://github.com/apache/incubator-linkis/issues/1952#issuecomment-1096010082
Try the following steps to see if it can be solved.
gauva版本问题
Correct the classpath of your application so that it contains a single, compatible version of com.google.common.cache.CacheBuilder
拷贝至spark 的lib下
cp lib/linkis-computation-governance/linkis-client/linkis-cli/guava-xx-jre.jar lib/linkis-engineconn-plugins/spark/dist/v2.4.3/lib
cp lib/linkis-commons/public-module/failureaccess-xx.jar lib/linkis-engineconn-plugins/spark/dist/v2.4.3/lib
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For additional commands, e-mail: dev-help@linkis.apache.org
[GitHub] [incubator-linkis] casionone commented on issue #1952: [Bug] Python exec failed, workdir log shows start spark engine and guava jar conflict !!!
Posted by GitBox <gi...@apache.org>.
casionone commented on issue #1952:
URL: https://github.com/apache/incubator-linkis/issues/1952#issuecomment-1099805475
> 根据你的方法解决了。
>
> 执行python脚本为什么启动了spark？ 这应该不是必要的
确认下scripts中 新建脚本时 类型选择的是Python(后缀为.python)还是PythonSpark(后缀为.py)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For additional commands, e-mail: dev-help@linkis.apache.org
[GitHub] [incubator-linkis] casionone closed issue #1952: [Bug] Python exec failed, workdir log shows start spark engine and guava jar conflict !!!
Posted by GitBox <gi...@apache.org>.
casionone closed issue #1952: [Bug] Python exec failed, workdir log shows start spark engine and guava jar conflict !!!
URL: https://github.com/apache/incubator-linkis/issues/1952
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For additional commands, e-mail: dev-help@linkis.apache.org
[GitHub] [incubator-linkis] shenbinglife commented on issue #1952: [Bug] Python exec failed, workdir log shows start spark engine and guava jar conflict !!!
Posted by GitBox <gi...@apache.org>.
shenbinglife commented on issue #1952:
URL: https://github.com/apache/incubator-linkis/issues/1952#issuecomment-1096231788
根据你的方法解决了。
新的错误：
initialize python executor failed, please ask administrator for help! java.io.IOException: Cannot run program "/opt/cloudera/anaconda3//bin/python": error=2, No such file or directory
为什么会从/opt/cloudera/anaconda3 目录下识别python呢？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For additional commands, e-mail: dev-help@linkis.apache.org