You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2020/04/22 03:13:37 UTC
[GitHub] [incubator-dolphinscheduler] 743294668 opened a new issue #2491: [QUESTION] The script executes successfully and returns 0, but the shell type node shows that the execution failed
743294668 opened a new issue #2491:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2491
RT
**Describe the question**
I used a shell node to execute a sqoop command successfully and finally printed out the command execution return value. It returned 0 but the web showed that the execution failed. What is the problem, how does the DP shell node determine whether the script is successfully executed?
**Which version of DolphinScheduler:**
-[1.2.0]
**Additional context**
Hadoop
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] 743294668 edited a comment on issue #2491: org.apache.dolphinscheduler.common.utils.HttpUtils:[73] - null: Name or service not known java.net.UnknownHostException: null: Name or service not known
Posted by GitBox <gi...@apache.org>.
743294668 edited a comment on issue #2491:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2491#issuecomment-618129628
想问下解决方法是什么？谢谢
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] dukeshiny commented on issue #2491: org.apache.dolphinscheduler.common.utils.HttpUtils:[73] - null: Name or service not known java.net.UnknownHostException: null: Name or service not known
Posted by GitBox <gi...@apache.org>.
dukeshiny commented on issue #2491:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2491#issuecomment-624625355
After I set yarn.resourcemanager.ha.rm.ids=cdh201,cdh202,another error happens,even if the sqoop task succeeded!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] gabrywu commented on issue #2491: org.apache.dolphinscheduler.common.utils.HttpUtils:[73] - null: Name or service not known java.net.UnknownHostException: null: Name or service not known
Posted by GitBox <gi...@apache.org>.
gabrywu commented on issue #2491:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2491#issuecomment-639277851
When running the shell script ,worker find strings like application_1587605547992_0003.
Then worker think the shell script startup a yarn job , it will monitor the status of the application usig HttpUtils . However missing some config , like yarn.application.status.address
yarn.resourcemanager.ha.rm.ids, error throw .
DolphinScheduler developers already talked about this ,you can subscribe the dev mail list, and can find what you want to know.
I don't agree with this mechanism actually . Maybe you can give better suggestions
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] 743294668 commented on issue #2491: org.apache.dolphinscheduler.common.utils.HttpUtils:[73] - null: Name or service not known java.net.UnknownHostException: null: Name or service not known
Posted by GitBox <gi...@apache.org>.
743294668 commented on issue #2491:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2491#issuecomment-618824673
Problem solving：Need Yarn configuration in /opt/dolphinscheduler/install/conf/common/hadoop：
yarn.application.status.address
yarn.resourcemanager.ha.rm.ids
https://cloud.tencent.com/developer/article/1596505
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] 743294668 commented on issue #2491: org.apache.dolphinscheduler.common.utils.HttpUtils:[73] - null: Name or service not known java.net.UnknownHostException: null: Name or service not known
Posted by GitBox <gi...@apache.org>.
743294668 commented on issue #2491:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2491#issuecomment-618128799
[INFO] 2020-04-23 09:37:29.121 - [taskAppId=TASK-48-176-302]:[231] - process id is 742
[INFO] 2020-04-23 09:37:29.122 - [taskAppId=TASK-48-176-302]:[432] - find app id: application_1587605547992_0003
[INFO] 2020-04-23 09:37:29.122 - [taskAppId=TASK-48-176-302]:[236] - yarn log url:application_1587605547992_0003
[ERROR] 2020-04-23 09:37:29.229 org.apache.dolphinscheduler.common.utils.HttpUtils:[73] - null: Name or service not known
java.net.UnknownHostException: null: Name or service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:111)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.dolphinscheduler.common.utils.HttpUtils.get(HttpUtils.java:60)
at org.apache.dolphinscheduler.common.utils.HadoopUtils.getApplicationStatus(HadoopUtils.java:377)
at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.isSuccessOfYarnState(AbstractCommandExecutor.java:396)
at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.updateState(AbstractCommandExecutor.java:241)
at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.run(AbstractCommandExecutor.java:170)
at org.apache.dolphinscheduler.server.worker.task.shell.ShellTask.handle(ShellTask.java:105)
at org.apache.dolphinscheduler.server.worker.runner.TaskScheduleThread.run(TaskScheduleThread.java:142)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[ERROR] 2020-04-23 09:37:29.229 - [taskAppId=TASK-48-176-302]:[410] - yarn applications: [application_1587605547992_0003] status failed : null
java.lang.NullPointerException: null
at org.apache.dolphinscheduler.common.utils.HadoopUtils.getApplicationStatus(HadoopUtils.java:380)
at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.isSuccessOfYarnState(AbstractCommandExecutor.java:396)
at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.updateState(AbstractCommandExecutor.java:241)
at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.run(AbstractCommandExecutor.java:170)
at org.apache.dolphinscheduler.server.worker.task.shell.ShellTask.handle(ShellTask.java:105)
at org.apache.dolphinscheduler.server.worker.runner.TaskScheduleThread.run(TaskScheduleThread.java:142)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[INFO] 2020-04-23 09:37:29.229 org.apache.dolphinscheduler.server.worker.runner.TaskScheduleThread:[156] - task instance id : 302,task final status : FAILURE
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] dukeshiny edited a comment on issue #2491: org.apache.dolphinscheduler.common.utils.HttpUtils:[73] - null: Name or service not known java.net.UnknownHostException: null: Name or service not known
Posted by GitBox <gi...@apache.org>.
dukeshiny edited a comment on issue #2491:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2491#issuecomment-624625355
After I set yarn.resourcemanager.ha.rm.ids=cdh201,cdh202,another error happens,even if the sqoop task succeeded!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] 743294668 edited a comment on issue #2491: org.apache.dolphinscheduler.common.utils.HttpUtils:[73] - null: Name or service not known java.net.UnknownHostException: null: Name or service not known
Posted by GitBox <gi...@apache.org>.
743294668 edited a comment on issue #2491:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2491#issuecomment-618129628
@lslone我看到你也遇到过相同的问题并解决了，想问下你的方法是什么？谢谢
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] 743294668 commented on issue #2491: org.apache.dolphinscheduler.common.utils.HttpUtils:[73] - null: Name or service not known java.net.UnknownHostException: null: Name or service not known
Posted by GitBox <gi...@apache.org>.
743294668 commented on issue #2491:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2491#issuecomment-618129628
@lslone
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] gabrywu edited a comment on issue #2491: org.apache.dolphinscheduler.common.utils.HttpUtils:[73] - null: Name or service not known java.net.UnknownHostException: null: Name or service not known
Posted by GitBox <gi...@apache.org>.
gabrywu edited a comment on issue #2491:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2491#issuecomment-639277851
When running the shell script ,worker find strings like application_1587605547992_0003.
Then worker think the shell script startup a yarn job , it will monitor the status of the application usig HttpUtils . However missing some config , like yarn.application.status.address
yarn.resourcemanager.ha.rm.ids, then errors thrown .
DolphinScheduler developers already talked about this ,you can subscribe the dev mail list, and can find what you want to know.
I don't agree with this mechanism actually . Maybe you can give better suggestions
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org