You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@linkis.apache.org by GitBox <gi...@apache.org> on 2022/09/01 04:08:54 UTC

[GitHub] [incubator-linkis] 2018yinjian opened a new issue, #3186: spark任务无法提交任务到yarn值

2018yinjian opened a new issue, #3186:
URL: https://github.com/apache/incubator-linkis/issues/3186

   ### Before asking
   
   - [X] I have read the [How to ask for help](https://linkis.apache.org/community/how-to-ask-for-help) (我已阅读[如何寻求帮助](https://linkis.apache.org/zh-CN/community/how-to-ask-for-help)指引文档).
   
   
   ### Your environment
   
   <!-- Please describe the linkis version you are using and basic environment information -->
   <!-- 请描述您使用的linkis版本和基本环境信息 -->
   - Linkis version used: 1.1.1
   - Environment name and version:
       - E-MapReduce
       - hadoop 3.2.1
       - hive-3.1.2
       - spark-2.4.7
       - scala-2.11.12
       - jdk 1.8.0_250
       - ....
   
   
   ### Describe your questions
   
   <!-- Please describe your questions -->
   <!-- 请描述您的问题 -->
   执行spark任务前台无报错信息,执行大约10分钟后台有报错信息
   ![image](https://user-images.githubusercontent.com/52166229/187826772-7d58db04-a83a-4da4-b029-c413508c7309.png)
   
   无明显报错,通过查看/appcom/tmp/root/20220901/spark/2ad67cc0-0f01-4e09-9cef-0029570b40e8/logs/stdout
   ![image](https://user-images.githubusercontent.com/52166229/187826874-c4e2c386-ff48-4993-93e4-3b3a365ae1c1.png)
   通过日志spark能够创建好,到提交yarn的时候就提不上去,感觉是哪里配置有问题
   ![b89559786c0f7074e40780e0c4dc547](https://user-images.githubusercontent.com/52166229/187827029-def60324-ed87-4d7d-b964-e2a565e656eb.jpg)
   连接yarn的源码我在url后面添加了用户信息,之前配置到表里重启了也无法连接到yarn,url添加了用户信息后能够获取到yarn的info
   hql能执行,现在就是spark任务提交不上去,麻烦大佬看下是哪里配置存在问题
   
   ### Eureka service list
   
   <!-- You can directly copy and paste the screenshot of the list of registered service instances displayed on the Eureka page here -->
   <!--  可以将Eureka页面显示的注册服务实例列表的的截图 直接复制并粘贴到这里。 粘贴后,会自动生成如下格式的url-->
   ![image](https://user-images.githubusercontent.com/52166229/187827305-8cd60fec-6de5-49dd-833e-0a78de921e0d.png)
   
   
   ### Some logs info or acctch file
   
   <!-- Please add the log in the following format, the key error log must be pasted in the code style, and the text as much as possible, the detailed log or other files can be dragged and uploaded as an attachment -->
   <!-- 请按如下格式 添加日志,关键错误日志请贴到[````log][````]之间,尽可能贴文字,详细日志或则其它文件,可以拖拽上传作为附件 -->
   
   
   <!--日志文字贴到这里-->
   2022-09-01 11:58:37.986 INFO  [main] org.apache.hadoop.io.retry.RetryInvocationHandler 411 log - java.net.ConnectException: Your endpoint configuration is wrong; For more details see:  http://wiki.apache.org/hadoop/UnsetHostnameOrPort, while invoking ApplicationClientProtocolPBClientImpl.getClusterMetrics over null after 15 failover attempts. Trying to failover after sleeping for 20832ms.
   2022-09-01 11:59:08.824 INFO  [main] org.apache.hadoop.io.retry.RetryInvocationHandler 411 log - java.net.ConnectException: Your endpoint configuration is wrong; For more details see:  http://wiki.apache.org/hadoop/UnsetHostnameOrPort, while invoking ApplicationClientProtocolPBClientImpl.getClusterMetrics over null after 16 failover attempts. Trying to failover after sleeping for 30355ms.
   2022-09-01 11:59:27.440 INFO  [Linkis-Default-Scheduler-Thread-16] org.apache.linkis.engineconn.computation.executor.upstream.ECTaskEntranceMonitor 41 info - stopping upstream monitor
   2022-09-01 11:59:27.441 INFO  [Thread-27] com.netflix.loadbalancer.PollingServerListUpdater 53 run - Shutting down the Executor Pool for PollingServerListUpdater
   2022-09-01 11:59:27.442 INFO  [SpringContextShutdownHook] com.netflix.discovery.DiscoveryClient 1352 notify - Saw local status change event StatusChangeEvent [timestamp=1662004767442, current=DOWN, previous=UP]
   2022-09-01 11:59:27.442 INFO  [DiscoveryClient-InstanceInfoReplicator-0] com.netflix.discovery.DiscoveryClient 873 register - DiscoveryClient_LINKIS-CG-ENGINECONN/emr-header-3.cluster-254539:linkis-cg-engineconn:13859: registering service...
   2022-09-01 11:59:27.443 INFO  [SpringContextShutdownHook] org.apache.linkis.engineconn.acessible.executor.service.DefaultAccessibleService 41 info - executorShutDownHook  start to execute.
   2022-09-01 11:59:27.443 WARN  [SpringContextShutdownHook] org.apache.linkis.engineconn.acessible.executor.service.DefaultAccessibleService 50 warn - EngineConn not ready, do not shutdown
   2022-09-01 11:59:27.444 INFO  [shutdown-hook-0] org.apache.spark.storage.DiskBlockManager 54 logInfo - Shutdown hook called
   2022-09-01 11:59:27.446 INFO  [DiscoveryClient-InstanceInfoReplicator-0] com.netflix.discovery.DiscoveryClient 882 register - DiscoveryClient_LINKIS-CG-ENGINECONN/emr-header-3.cluster-254539:linkis-cg-engineconn:13859 - registration status: 204
   2022-09-01 11:59:27.452 INFO  [shutdown-hook-0] org.apache.spark.util.ShutdownHookManager 54 logInfo - Shutdown hook called
   2022-09-01 11:59:27.454 INFO  [shutdown-hook-0] org.apache.spark.util.ShutdownHookManager 54 logInfo - Deleting directory /appcom/tmp/root/20220901/spark/2ad67cc0-0f01-4e09-9cef-0029570b40e8/tmp/spark-93642eb4-b2be-46ed-9e1e-62e470e96b94
   2022-09-01 11:59:27.455 INFO  [SpringContextShutdownHook] org.apache.linkis.engineconn.acessible.executor.service.DefaultAccessibleService 41 info - executorShutDownHook  start to execute.
   2022-09-01 11:59:27.455 WARN  [SpringContextShutdownHook] org.apache.linkis.engineconn.acessible.executor.service.DefaultAccessibleService 50 warn - EngineConn not ready, do not shutdown
   2022-09-01 11:59:27.459 INFO  [shutdown-hook-0] org.apache.spark.util.ShutdownHookManager 54 logInfo - Deleting directory /appcom/tmp/root/20220901/spark/2ad67cc0-0f01-4e09-9cef-0029570b40e8/tmp/spark-efa1211b-31dd-44e1-9b33-f4fdcc987f64
   2022-09-01 11:59:27.462 INFO  [SpringContextShutdownHook] org.apache.linkis.engineconn.acessible.executor.service.DefaultAccessibleService 41 info - executorShutDownHook  start to execute.
   2022-09-01 11:59:27.462 WARN  [SpringContextShutdownHook] org.apache.linkis.engineconn.acessible.executor.service.DefaultAccessibleService 50 warn - EngineConn not ready, do not shutdown
   2022-09-01 11:59:27.464 INFO  [SpringContextShutdownHook] com.netflix.util.concurrent.ShutdownEnabledTimer 67 cancel - Shutdown hook removed for: NFLoadBalancer-PingTimer-linkis-cg-engineconnmanager
   2022-09-01 11:59:27.464 INFO  [shutdown-hook-0] org.apache.spark.util.ShutdownHookManager 54 logInfo - Deleting directory /appcom/tmp/root/20220901/spark/2ad67cc0-0f01-4e09-9cef-0029570b40e8/tmp/spark-efa1211b-31dd-44e1-9b33-f4fdcc987f64/userFiles-b438351e-d0cb-4abf-a954-3f430ffa50b7
   2022-09-01 11:59:27.464 INFO  [SpringContextShutdownHook] com.netflix.util.concurrent.ShutdownEnabledTimer 72 cancel - Exception caught (might be ok if at shutdown) java.lang.IllegalStateException: Shutdown in progress
           at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[?:1.8.0_252]
           at java.lang.Runtime.removeShutdownHook(Runtime.java:239) ~[?:1.8.0_252]
           at com.netflix.util.concurrent.ShutdownEnabledTimer.cancel(ShutdownEnabledTimer.java:70) ~[netflix-commons-util-0.3.0.jar:0.3.0]
           at com.netflix.loadbalancer.BaseLoadBalancer.cancelPingTask(BaseLoadBalancer.java:632) ~[ribbon-loadbalancer-2.3.0.jar:2.3.0]
           at com.netflix.loadbalancer.BaseLoadBalancer.shutdown(BaseLoadBalancer.java:883) ~[ribbon-loadbalancer-2.3.0.jar:2.3.0]
           at com.netflix.loadbalancer.DynamicServerListLoadBalancer.shutdown(DynamicServerListLoadBalancer.java:285) ~[ribbon-loadbalancer-2.3.0.jar:2.3.0]
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252]
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252]
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
           at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
           at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:280) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE]
           at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:214) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE]
           at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE]
           at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE]
           at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1092) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE]
           at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE]
           at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1085) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE]
           at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1061) ~[spring-context-5.2.15.RELEASE.jar:5.2.15.RELEASE]
           at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1030) ~[spring-context-5.2.15.RELEASE.jar:5.2.15.RELEASE]
           at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:979) ~[spring-context-5.2.15.RELEASE.jar:5.2.15.RELEASE]
           at org.springframework.cloud.context.named.NamedContextFactory.destroy(NamedContextFactory.java:93) ~[spring-cloud-context-2.2.9.RELEASE.jar:2.2.9.RELEASE]
           at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:199) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE]
           at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE]
           at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE]
           at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1092) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE]
           at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE]
           at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1085) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE]
           at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1061) ~[spring-context-5.2.15.RELEASE.jar:5.2.15.RELEASE]
           at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1030) ~[spring-context-5.2.15.RELEASE.jar:5.2.15.RELEASE]
           at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.doClose(ServletWebServerApplicationContext.java:170) ~[spring-boot-2.3.12.RELEASE.jar:2.3.12.RELEASE]
           at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:949) ~[spring-context-5.2.15.RELEASE.jar:5.2.15.RELEASE]
   
   2022-09-01 11:59:27.472 INFO  [shutdown-hook-0] org.apache.spark.util.ShutdownHookManager 54 logInfo - Deleting directory /appcom/tmp/root/20220901/spark/2ad67cc0-0f01-4e09-9cef-0029570b40e8/tmp/repl-025d4d81-253b-4e91-8cfc-b4470ef694c7
   2022-09-01 11:59:27.475 INFO  [SpringContextShutdownHook] com.netflix.discovery.DiscoveryClient 935 shutdown - Shutting down DiscoveryClient ...
   2022-09-01 11:59:27.653 INFO  [Linkis-Default-Scheduler-Thread-7] org.apache.linkis.engineconn.computation.executor.upstream.ECTaskEntranceMonitor 41 info - stopping upstream monitor
   2022-09-01 11:59:30.477 INFO  [SpringContextShutdownHook] com.netflix.discovery.DiscoveryClient 971 unregister - Unregistering ...
   2022-09-01 11:59:30.481 INFO  [SpringContextShutdownHook] com.netflix.discovery.DiscoveryClient 973 unregister - DiscoveryClient_LINKIS-CG-ENGINECONN/emr-header-3.cluster-254539:linkis-cg-engineconn:13859 - deregister  status: 200
   2022-09-01 11:59:30.490 INFO  [SpringContextShutdownHook] com.netflix.discovery.DiscoveryClient 960 shutdown - Completed shut down of DiscoveryClient
   ````
   log file:
    <!-- 拖拽上传后,会自动生成如下示例格式的附件url -->
   这是/appcom/tmp/root/20220901/spark/2ad67cc0-0f01-4e09-9cef-0029570b40e8/logs/stdout日志
   
   [stdout.txt](https://github.com/apache/incubator-linkis/files/9466755/stdout.txt)
   
   [linkis-ps-cs.log](https://github.com/apache/incubator-linkis/files/9466756/linkis-ps-cs.log)
   [linkis-cg-entrance.log](https://github.com/apache/incubator-linkis/files/9466757/linkis-cg-entrance.log)
   
   
   [linkis-cg-engineconnmanager.log](https://github.com/apache/incubator-linkis/files/9466759/linkis-cg-engineconnmanager.log)
   
   
   [linkis-cg-linkismanager.log](https://github.com/apache/incubator-linkis/files/9466760/linkis-cg-linkismanager.log)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org


[GitHub] [incubator-linkis] casionone closed issue #3186: spark任务无法提交任务到yarn值

Posted by GitBox <gi...@apache.org>.
casionone closed issue #3186: spark任务无法提交任务到yarn值
URL: https://github.com/apache/incubator-linkis/issues/3186


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org


[GitHub] [incubator-linkis] utopianet commented on issue #3186: spark任务无法提交任务到yarn值

Posted by GitBox <gi...@apache.org>.
utopianet commented on issue #3186:
URL: https://github.com/apache/incubator-linkis/issues/3186#issuecomment-1242603276

   请确保每一个作业提交节点、执行节点的hostname的yarn集群中每一台机器的/etc/hosts中都进行了配置。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org


[GitHub] [incubator-linkis] casionone commented on issue #3186: spark任务无法提交任务到yarn值

Posted by GitBox <gi...@apache.org>.
casionone commented on issue #3186:
URL: https://github.com/apache/incubator-linkis/issues/3186#issuecomment-1235476294

   check query yarn resource info is ok 
   https://linkis.apache.org/zh-CN/docs/1.2.0/deployment/quick-deploy#8-%E5%B8%B8%E8%A7%81%E5%BC%82%E5%B8%B8%E9%97%AE%E9%A2%98%E6%8E%92%E6%9F%A5%E6%8C%87%E5%BC%95 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org


[GitHub] [incubator-linkis] github-actions[bot] commented on issue #3186: spark任务无法提交任务到yarn值

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #3186:
URL: https://github.com/apache/incubator-linkis/issues/3186#issuecomment-1233714127

   :blush:  Welcome to the Apache Linkis (incubating) community!!
   We are glad that you are contributing by opening this issue.
   
   Please make sure to include all the relevant context.
   We will be here shortly.
   
   If you are interested in contributing to our website project, please let us know!
   You can check out our contributing guide on 
    :point_right:  [How to Participate in Project Contribution](https://linkis.apache.org/community/how-to-contribute).
    
   WeChat  Group:
   
   ![image](https://linkis.apache.org/Images/wedatasphere_contact_01.png)
   Mailing Lists:
   |name|description|Subscribe|Unsubscribe|archive|
   |:-----|:--------|:------|:-------|:-----|
   | [dev@linkis.apache.org](mailto:dev@linkis.apache.org) | community activity information | [subscribe](mailto:dev-subscribe@linkis.apache.org) | [unsubscribe](mailto:dev-unsubscribe@linkis.apache.org) | [archive](http://mail-archives.apache.org/mod_mbox/linkis-dev) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org