You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@linkis.apache.org by GitBox <gi...@apache.org> on 2022/08/23 13:52:45 UTC
[GitHub] [incubator-linkis] 2018yinjian opened a new issue, #2963: Failed to request external resourceRMErrorException: errCode: 11007 获取主Yarn resourcemanager异常
2018yinjian opened a new issue, #2963:
URL: https://github.com/apache/incubator-linkis/issues/2963
### Before asking
- [X] I have read the [How to ask for help](https://linkis.apache.org/community/how-to-ask-for-help) (我已阅读[如何寻求帮助](https://linkis.apache.org/zh-CN/community/how-to-ask-for-help)指引文档).
### Your environment
<!-- Please describe the linkis version you are using and basic environment information -->
<!-- 请描述您使用的linkis版本和基本环境信息 -->
- Linkis version used: 1.1.2
- Environment name and version:
- 阿里云-E-mapreduc
- hadoop 3.2.1
- hive-3.1.2
- spark-2.4.7
- scala-2.11.12
- jdk 1.8.0_252
- dss 1.1.0
- linkis 1.1.1
### Describe your questions
<!-- Please describe your questions -->
<!-- 请描述您的问题 -->
1，执行pyspark报错（Yarn地址能够直接访问）
ERROR Job with execId-IDE_root_spark_4 + subJobId : 51 execute failed,21304, Task is Failed,errorMsg: errCode: 12003 ,desc: emr-header-3.cluster-254539:9101_0 Failed to async get EngineNode MatchError: LinkisException{errCode=11006, desc='Failed to request external resourceRMErrorException: errCode: 11007 ,desc: Get active Yarn resourcemanager from : http://172.17.217.248:8088 exception.(从 http://172.17.217.248:8088 获取主Yarn resourcemanager异常) ,ip: emr-header-3.cluster-254539 ,port: 9101 ,serviceKind: linkis-cg-linkismanager', ip='emr-header-3.cluster-254539', port=9101, serviceKind='linkis-cg-linkismanager'} (of class org.apache.linkis.manager.common.exception.RMErrorException) ,ip: emr-header-3.cluster-254539 ,port: 9104 ,serviceKind: linkis-cg-entrance
2，通过linkis-cli的方式报错（Yarn地址能够直接访问）：
[root@emr-header-3 linkis]# sh ./bin/linkis-cli -engineType spark-2.4.7 -codeType sql -code "select * from stg.stg_base_area;" -submitUser hadoop -proxyUser hadoop
=====Java Start Command=====
exec /usr/lib/jvm/java-1.8.0/bin/java -server -Xms32m -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/DataSphere/linkis/logs/linkis-cli -XX:ErrorFile=/opt/DataSphere/linkis/logs/linkis-cli/ps_err_pid%p.log -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80 -XX:+DisableExplicitGC -classpath /opt/DataSphere/linkis/conf/linkis-cli:/opt/DataSphere/linkis/lib/linkis-computation-governance/linkis-client/linkis-cli/*:/opt/DataSphere/linkis/lib/linkis-commons/public-module/*: -Dconf.root=/opt/DataSphere/linkis/conf/linkis-cli -Dconf.file=linkis-cli.properties -Dlog.path=/opt/DataSphere/linkis/logs/linkis-cli -Dlog.file=linkis-client.root.log.20220823214054389933576 org.apache.linkis.cli.application.LinkisClientApplication '-engineType spark-2.4.7 -codeType sql -code select * from stg.stg_base_area; -submitUser hadoop -proxyUser hadoop'
[INFO] LogFile path: /opt/DataSphere/linkis/logs/linkis-cli/linkis-client.root.log.20220823214054389933576
[INFO] User does not provide usr-configuration file. Will use default config
[INFO] connecting to linkis gateway:http://192.168.2.247:9100
JobId:52
TaskId:52
ExecId:exec_id018032linkis-cg-entranceemr-header-3.cluster-254539:9104LINKISCLI_hadoop_spark_1
[INFO] Job is successfully submitted!
2022-08-23 21:40:55.040 INFO Program is substituting variables for you
2022-08-23 21:40:55.040 INFO Variables substitution ended successfully
2022-08-23 21:40:55.040 WARN The code you submit will not be limited by the limit
Job with jobId : LINKISCLI_hadoop_spark_1 and execID : LINKISCLI_hadoop_spark_1 submitted
2022-08-23 21:40:55.040 INFO You have submitted a new job, script code (after variable substitution) is
************************************SCRIPT CODE************************************
select * from stg.stg_base_area
************************************SCRIPT CODE************************************
2022-08-23 21:40:55.040 INFO Your job is accepted, jobID is LINKISCLI_hadoop_spark_1 and jobReqId is 52 in ServiceInstance(linkis-cg-entrance, emr-header-3.cluster-254539:9104). Please wait it to be scheduled
job is scheduled.
2022-08-23 21:40:55.040 INFO Your job is Scheduled. Please wait it to run.
Your job is being scheduled by orchestrator.
2022-08-23 21:40:55.040 INFO job is running.
2022-08-23 21:40:55.040 INFO Your job is Running now. Please wait it to complete.
2022-08-23 21:40:55.040 INFO Job with jobGroupId : 52 and subJobId : 52 was submitted to Orchestrator.
2022-08-23 21:40:55.040 INFO Background is starting a new engine for you,execId astJob_6_codeExec_6 mark id is mark_6, it may take several seconds, please wait
2022-08-23 21:40:56.040 ERROR Job with execId-LINKISCLI_hadoop_spark_1 + subJobId : 52 execute failed,21304, Task is Failed,errorMsg: errCode: 12003 ,desc: emr-header-3.cluster-254539:9101_1 Failed to async get EngineNode MatchError: LinkisException{errCode=11006, desc='Failed to request external resourceRMErrorException: errCode: 11007 ,desc: Get active Yarn resourcemanager from : http://192.168.2.248:8088 exception.(从 http://192.168.2.248:8088 获取主Yarn resourcemanager异常) ,ip: emr-header-3.cluster-254539 ,port: 9101 ,serviceKind: linkis-cg-linkismanager', ip='emr-header-3.cluster-254539', port=9101, serviceKind='linkis-cg-linkismanager'} (of class org.apache.linkis.manager.common.exception.RMErrorException) ,ip: emr-header-3.cluster-254539 ,port: 9104 ,serviceKind: linkis-cg-entrance
2022-08-23 21:40:56.040 INFO job is completed.
2022-08-23 21:40:56.040 INFO Task creation time(任务创建时间): 2022-08-23 21:40:55, Task scheduling time(任务调度时间): 2022-08-23 21:40:55, Task start time(任务开始时间): 2022-08-23 21:40:55, Missionend time(任务结束时间): 2022-08-23 21:40:56
2022-08-23 21:40:56.040 INFO Your mission(您的任务) 52 The total time spent is(总耗时时间为): 365 ms
2022-08-23 21:40:56.040 INFO Sorry. Your job completed with a status Failed. You can view logs for the reason.
[INFO] Job status is not success but 'FAILED'. Will not try to retrieve any Result
[INFO] Job failed! Will not try get execute result.
============Result:================
TaskId:52
ExecId: exec_id018032linkis-cg-entranceemr-header-3.cluster-254539:9104LINKISCLI_hadoop_spark_1
User:hadoop
Current job status:FAILED
extraMsg:
errDesc: 21304, Task is Failed,errorMsg: errCode: 12003 ,desc: emr-header-3.cluster-254539:9101_1 Failed to async get EngineNode MatchError: LinkisException{errCode=11006, desc='Failed to request external resourceRMErrorException: errCode: 11007 ,desc: Get active
[INFO] JobStatus is not 'success'. Will not retrieve result-set.
############Execute Error!!!########
3，数据库表：linkis_cg_rm_external_resource_provider
Yarn
default
{
"rmWebAddress": "http://192.168.2.248:8088",
"hadoopVersion":"3.2.1",
"authorEnable":false,
"user":"hadoop",
"pwd":"hadoop"
}
### Eureka service list
<!-- You can directly copy and paste the screenshot of the list of registered service instances displayed on the Eureka page here -->
<!-- 可以将Eureka页面显示的注册服务实例列表的的截图 直接复制并粘贴到这里。 粘贴后，会自动生成如下格式的url-->
### Some logs info or acctch file
<!-- Please add the log in the following format, the key error log must be pasted in the code style, and the text as much as possible, the detailed log or other files can be dragged and uploaded as an attachment -->
<!-- 请按如下格式 添加日志，关键错误日志请贴到[````log][````]之间，尽可能贴文字，详细日志或则其它文件，可以拖拽上传作为附件 -->
linkis-cg-linkismanager.log:
<!--日志文字贴到这里-->
2022-08-23 21:04:19.360 [WARN ] [AskEngineService-Thread-1 ] o.a.l.r.e.s.i.ExternalResourceServiceImpl (180) [retry] - failed to request external resource provider, retryNum 0 org.apache.linkis.manager.common.exception.RMErrorException: errCode: 11007 ,desc: Get active Yarn resourcemanager from : http://192.168.2.248:8088 exception.(从 http://192.168.2.248:8088 获取主Yarn resourcemanager异常) ,ip: emr-header-3.cluster-254539 ,port: 9101 ,serviceKind: linkis-cg-linkismanager
log file:
<!-- 拖拽上传后，会自动生成如下示例格式的附件url -->
[linkis-cg-linkismanager.log](https://github.com/apache/incubator-linkis/files/9402960/linkis-cg-linkismanager.log)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [incubator-linkis] casionone closed issue #2963: Failed to request external resourceRMErrorException: errCode: 11007 获取主Yarn resourcemanager异常
Posted by GitBox <gi...@apache.org>.
casionone closed issue #2963: Failed to request external resourceRMErrorException: errCode: 11007 获取主Yarn resourcemanager异常
URL: https://github.com/apache/incubator-linkis/issues/2963
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [incubator-linkis] utopianet commented on issue #2963: Failed to request external resourceRMErrorException: errCode: 11007 获取主Yarn resourcemanager异常
Posted by GitBox <gi...@apache.org>.
utopianet commented on issue #2963:
URL: https://github.com/apache/incubator-linkis/issues/2963#issuecomment-1242603948
请参考以下文章
https://blog.csdn.net/weixin_30305735/article/details/102137099
Simple认证是根据执行的用户名、用户组进行验证的，无需任何配置和代码。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [incubator-linkis] 2018yinjian commented on issue #2963: Failed to request external resourceRMErrorException: errCode: 11007 获取主Yarn resourcemanager异常
Posted by GitBox <gi...@apache.org>.
2018yinjian commented on issue #2963:
URL: https://github.com/apache/incubator-linkis/issues/2963#issuecomment-1226702364
> ```
> 2022-08-23 21:04:19.371 [ERROR] [AskEngineService-Thread-1 ] o.a.l.r.e.y.YarnResourceRequester (280) [apply] - Get Yarn resourcemanager info error, No content to map due to end-of-input
> at [Source: (String)""; line: 1, column: 0] com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input
> ```
>
> please try to excute `curl http://192.168.2.248:8088/ws/v1/cluster/info `，the response is ok?
[root@emr-header-3 zmn]# curl http://192.168.2.248:8088/ws/v1/cluster/info
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 401 Authentication required</title>
</head>
<body><h2>HTTP ERROR 401</h2>
<p>Problem accessing /ws/v1/cluster/info. Reason:
<pre> Authentication required</pre></p>
</body>
</html>
yarn启用了Simple认证，需要加上认证用户
[root@emr-header-3 zmn]# curl http://192.168.2.248:8088/ws/v1/cluster/info?user.name=hadoop
{"clusterInfo":{"id":1640111027208,"startedOn":1640111027208,"state":"STARTED","haState":"ACTIVE","rmStateStoreName":"org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore","resourceManagerVersion":"3.2.1","resourceManagerBuildVersion":"3.2.1 from 79f54210e4619179b3a068369aba61e816eb1809 by jenkins source checksum 3120421249ad6ad216e5915e1442e18","resourceManagerVersionBuiltOn":"2021-03-26T12:16Z","hadoopVersion":"3.2.1","hadoopBuildVersion":"3.2.1 from 79f54210e4619179b3a068369aba61e816eb1809 by jenkins source checksum a727b26fa21579ad1b194bc17821d8","hadoopVersionBuiltOn":"2021-03-26T12:13Z","haZooKeeperConnectionState":"CONNECTED"}}
1，http://192.168.2.248:8088/cluster?user.name=hadoop url上带上用户也会报同样的错误
2，{
"rmWebAddress": "http://192.168.2.248:8088",
"hadoopVersion": "3.2.1",
"authorEnable":true,
"user":"hadoop",
"pwd":"hadoop",
"kerberosEnable":false,
"principalName":"@PRINCIPAL_NAME",
"keytabPath":"@KEYTAB_PATH",
"krb5Path":"@KRB5_PATH"
}
authorEnable改成true配置用户，也会存在同样的问题
3，请问认证的信息应该配置在哪里才会生效
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [incubator-linkis] 2018yinjian commented on issue #2963: Failed to request external resourceRMErrorException: errCode: 11007 获取主Yarn resourcemanager异常
Posted by GitBox <gi...@apache.org>.
2018yinjian commented on issue #2963:
URL: https://github.com/apache/incubator-linkis/issues/2963#issuecomment-1226696522
[root@emr-header-3 zmn]# curl http://192.168.2.248:8088/ws/v1/cluster/info
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 401 Authentication required</title>
</head>
<body><h2>HTTP ERROR 401</h2>
<p>Problem accessing /ws/v1/cluster/info. Reason:
<pre> Authentication required</pre></p>
</body>
</html>
yarn启用了Simple认证，需要加上认证用户
[root@emr-header-3 zmn]# curl http://192.168.2.248:8088/ws/v1/cluster/info?user.name=hadoop
{"clusterInfo":{"id":1640111027208,"startedOn":1640111027208,"state":"STARTED","haState":"ACTIVE","rmStateStoreName":"org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore","resourceManagerVersion":"3.2.1","resourceManagerBuildVersion":"3.2.1 from 79f54210e4619179b3a068369aba61e816eb1809 by jenkins source checksum 3120421249ad6ad216e5915e1442e18","resourceManagerVersionBuiltOn":"2021-03-26T12:16Z","hadoopVersion":"3.2.1","hadoopBuildVersion":"3.2.1 from 79f54210e4619179b3a068369aba61e816eb1809 by jenkins source checksum a727b26fa21579ad1b194bc17821d8","hadoopVersionBuiltOn":"2021-03-26T12:13Z","haZooKeeperConnectionState":"CONNECTED"}}
1，http://192.168.2.248:8088/cluster?user.name=hadoop url上带上用户也会报同样的错误
2，{
"rmWebAddress": "http://172.17.217.248:8088",
"hadoopVersion": "3.2.1",
"authorEnable":true,
"user":"hadoop",
"pwd":"hadoop",
"kerberosEnable":false,
"principalName":"@PRINCIPAL_NAME",
"keytabPath":"@KEYTAB_PATH",
"krb5Path":"@KRB5_PATH"
}
authorEnable改成true配置用户，也会存在同样的问题
3，请问认证的信息应该配置在哪里才会生效
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [incubator-linkis] github-actions[bot] commented on issue #2963: Failed to request external resourceRMErrorException: errCode: 11007 获取主Yarn resourcemanager异常
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #2963:
URL: https://github.com/apache/incubator-linkis/issues/2963#issuecomment-1224109659
:blush: Welcome to the Apache Linkis (incubating) community!!
We are glad that you are contributing by opening this issue.
Please make sure to include all the relevant context.
We will be here shortly.
If you are interested in contributing to our website project, please let us know!
You can check out our contributing guide on
:point_right: [How to Participate in Project Contribution](https://linkis.apache.org/community/how-to-contribute).
WeChat Group:
Mailing Lists:
|name|description|Subscribe|Unsubscribe|archive|
|:-----|:--------|:------|:-------|:-----|
| [dev@linkis.apache.org](mailto:dev@linkis.apache.org) | community activity information | [subscribe](mailto:dev-subscribe@linkis.apache.org) | [unsubscribe](mailto:dev-unsubscribe@linkis.apache.org) | [archive](http://mail-archives.apache.org/mod_mbox/linkis-dev) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [incubator-linkis] casionone commented on issue #2963: Failed to request external resourceRMErrorException: errCode: 11007 获取主Yarn resourcemanager异常
Posted by GitBox <gi...@apache.org>.
casionone commented on issue #2963:
URL: https://github.com/apache/incubator-linkis/issues/2963#issuecomment-1225689732
```
2022-08-23 21:04:19.371 [ERROR] [AskEngineService-Thread-1 ] o.a.l.r.e.y.YarnResourceRequester (280) [apply] - Get Yarn resourcemanager info error, No content to map due to end-of-input
at [Source: (String)""; line: 1, column: 0] com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org