You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2020/03/26 08:22:41 UTC
[GitHub] [incubator-dolphinscheduler] waniuwa opened a new issue #2316: [BUG] Submit spark task report error: File does not exist：hdfs:xxx (通过spark node提交spark任务报错：File does not exist：hdfs:xxx)
waniuwa opened a new issue #2316: [BUG] Submit spark task report error: File does not exist：hdfs:xxx (通过spark node提交spark任务报错：File does not exist：hdfs:xxx)
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2316
In the spark node submission task, configure -- files application.conf in other parameters. Resources also specify files, but the runtime prompts that there is no corresponding application.conf file on HDFS. It works normally to submit spark tasks through the shell.
------------------------------------------------------------------------------
在spark node提交任务，在其他参数里配置--files application.conf ，资源也指定了文件，但运行时提示hdfs上不存在对应的application.conf 文件。通过shell提交spark任务倒是可以正常运行。
[INFO] 2020-03-26 15:49:47.868 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:47 INFO security.UserGroupInformation: Login successful for user bigdata@TEST1.COM using keytab file /opt/keytab/bigdata.keytab
[INFO] 2020-03-26 15:49:49.042 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:48 INFO yarn.Client: Attempting to login to the Kerberos using principal: bigdata@TEST1.COM and keytab: /opt/keytab/bigdata.keytab
20/03/26 15:49:48 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm154
20/03/26 15:49:48 INFO yarn.Client: Requesting a new application from cluster with 6 NodeManagers
20/03/26 15:49:48 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (41121 MB per container)
20/03/26 15:49:48 INFO yarn.Client: Will allocate AM container, with 3456 MB memory including 384 MB overhead
20/03/26 15:49:48 INFO yarn.Client: Setting up container launch context for our AM
20/03/26 15:49:48 INFO yarn.Client: Setting up the launch environment for our AM container
20/03/26 15:49:49 INFO yarn.Client: Credentials file set to: credentials-faa7e04b-f09e-40a4-ac42-58ab0f7c9003
[INFO] 2020-03-26 15:49:50.218 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:49 INFO yarn.YarnSparkHadoopUtil: getting token for namenode: hdfs://bigdata0:8020/user/bigdata/.sparkStaging/application_1585031466521_0129
20/03/26 15:49:49 INFO hdfs.DFSClient: Created token for bigdata: HDFS_DELEGATION_TOKEN owner=bigdata@TEST1.COM, renewer=bigdata, realUser=, issueDate=1585208989034, maxDate=1585813789034, sequenceNumber=117961, masterKeyId=387 on xxxx:8020
20/03/26 15:49:49 INFO yarn.Client: Renewal Interval set to 86400048
20/03/26 15:49:49 INFO yarn.Client: Preparing resources for our AM container
20/03/26 15:49:49 INFO yarn.YarnSparkHadoopUtil: getting token for namenode: hdfs://bigdata0:8020/user/bigdata/.sparkStaging/application_1585031466521_0129
20/03/26 15:49:49 INFO hdfs.DFSClient: Created token for bigdata: HDFS_DELEGATION_TOKEN owner=bigdata@TEST1.COM, renewer=yarn, realUser=, issueDate=1585208989092, maxDate=1585813789092, sequenceNumber=117962, masterKeyId=387 on xxxx:8020
20/03/26 15:49:50 INFO hive.metastore: Trying to connect to metastore with URI thrift://bigdata0:9083
[INFO] 2020-03-26 15:49:51.223 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:50 INFO hive.metastore: Opened a connection to metastore, current connections: 1
20/03/26 15:49:50 INFO hive.metastore: Connected to metastore.
20/03/26 15:49:50 INFO hive.metastore: Closed a connection to metastore, current connections: 0
20/03/26 15:49:50 INFO yarn.Client: To enable the AM to login from keytab, credentials are being copied over to the AM via the YARN Secure Distributed Cache.
20/03/26 15:49:50 INFO yarn.Client: Uploading resource file:/opt/keytab/bigdata.keytab -> hdfs://bigdata0:8020/user/bigdata/.sparkStaging/application_1585031466521_0129/bigdata.keytab
20/03/26 15:49:50 INFO yarn.Client: Uploading resource file:/tmp/dolphinscheduler/exec/process/3/7/107/164/offline-analysis-2.jar -> hdfs://bigdata0:8020/user/bigdata/.sparkStaging/application_1585031466521_0129/offline-analysis-2.jar
20/03/26 15:49:51 INFO yarn.Client: Uploading resource file:/tmp/dolphinscheduler/exec/process/3/7/107/164/application.conf -> hdfs://bigdata0:8020/user/bigdata/.sparkStaging/application_1585031466521_0129/application.conf
20/03/26 15:49:51 INFO yarn.Client: Uploading resource file:/tmp/spark-4499381f-3954-4240-9edf-cefc3eb58f5a/__spark_conf__2004243660860313056.zip -> hdfs://bigdata0:8020/user/bigdata/.sparkStaging/application_1585031466521_0129/__spark_conf__2004243660860313056.zip
20/03/26 15:49:51 INFO spark.SecurityManager: Changing view acls to: rootadmin,bigdata
[INFO] 2020-03-26 15:49:52.478 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:51 INFO spark.SecurityManager: Changing modify acls to: rootadmin,bigdata
20/03/26 15:49:51 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(rootadmin, bigdata); users with modify permissions: Set(rootadmin, bigdata)
20/03/26 15:49:51 INFO yarn.Client: Submitting application 129 to ResourceManager
20/03/26 15:49:51 INFO impl.YarnClientImpl: Submitted application application_1585031466521_0129
20/03/26 15:49:52 INFO yarn.Client: Application report for application_1585031466521_0129 (state: ACCEPTED)
[INFO] 2020-03-26 15:49:53.484 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:52 INFO yarn.Client:
client token: Token { kind: YARN_CLIENT_TOKEN, service: }
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.users.bigdata
queue: root.users.bigdata
start time: 1585208991258
final status: UNDEFINED
tracking URL: http://bigdata3:8778/proxy/application_1585031466521_0129/
user: bigdata
20/03/26 15:49:53 INFO yarn.Client: Application report for application_1585031466521_0129 (state: ACCEPTED)
[INFO] 2020-03-26 15:49:54.486 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:54 INFO yarn.Client: Application report for application_1585031466521_0129 (state: ACCEPTED)
[INFO] 2020-03-26 15:49:55.487 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:55 INFO yarn.Client: Application report for application_1585031466521_0129 (state: ACCEPTED)
[INFO] 2020-03-26 15:49:56.489 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:56 INFO yarn.Client: Application report for application_1585031466521_0129 (state: ACCEPTED)
[INFO] 2020-03-26 15:49:57.490 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:57 INFO yarn.Client: Application report for application_1585031466521_0129 (state: ACCEPTED)
[INFO] 2020-03-26 15:49:58.492 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:58 INFO yarn.Client: Application report for application_1585031466521_0129 (state: ACCEPTED)
[INFO] 2020-03-26 15:49:59.494 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:49:59 INFO yarn.Client: Application report for application_1585031466521_0129 (state: ACCEPTED)
[INFO] 2020-03-26 15:50:00.498 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:50:00 INFO yarn.Client: Application report for application_1585031466521_0129 (state: ACCEPTED)
[INFO] 2020-03-26 15:50:01.499 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:50:01 INFO yarn.Client: Application report for application_1585031466521_0129 (state: ACCEPTED)
[INFO] 2020-03-26 15:50:02.501 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:50:02 INFO yarn.Client: Application report for application_1585031466521_0129 (state: FAILED)
[INFO] 2020-03-26 15:50:02.908 - [taskAppId=TASK-7-107-164]:[106] - -> 20/03/26 15:50:02 INFO yarn.Client:
client token: N/A
diagnostics: Application application_1585031466521_0129 failed 2 times due to AM Container for appattempt_1585031466521_0129_000002 exited with exitCode: -1000
For more detailed output, check application tracking page:http://bigdata3:8778/proxy/application_1585031466521_0129/Then, click on links to logs of each attempt.
Diagnostics: File does not exist: hdfs://bigdata0:8020/user/bigdata/.sparkStaging/application_1585031466521_0129/application.conf
java.io.FileNotFoundException: File does not exist: hdfs://bigdata0:8020/user/bigdata/.sparkStaging/application_1585031466521_0129/application.conf
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1269)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1261)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1261)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:364)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:361)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.users.bigdata
start time: 1585208991258
final status: FAILED
tracking URL: http://bigdata3:8778/cluster/app/application_1585031466521_0129
user: bigdata
Exception in thread "main" org.apache.spark.SparkException: Application application_1585031466521_0129 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1025)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1072)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:730)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/03/26 15:50:02 INFO util.ShutdownHookManager: Shutdown hook called
20/03/26 15:50:02 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-4499381f-3954-4240-9edf-cefc3eb58f5a
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-dolphinscheduler] dailidong commented on issue #2316: [BUG] Submit spark task report error: File does not exist：hdfs:xxx (通过spark node提交spark任务报错：File does not exist：hdfs:xxx)
Posted by GitBox <gi...@apache.org>.
dailidong commented on issue #2316: [BUG] Submit spark task report error: File does not exist：hdfs:xxx (通过spark node提交spark任务报错：File does not exist：hdfs:xxx)
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2316#issuecomment-605387112
thanks for report this issue
@xingchun-chen ,please verify this problem when you free
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [dolphinscheduler] lenboo commented on issue #2316: [BUG] Submit spark task report error: File does not exist：hdfs:xxx (通过spark node提交spark任务报错：File does not exist：hdfs:xxx)
Posted by GitBox <gi...@apache.org>.
lenboo commented on issue #2316:
URL: https://github.com/apache/dolphinscheduler/issues/2316#issuecomment-1005358509
@waniuwa Please add more details about the spark task to fix the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] lenboo closed issue #2316: [BUG] Submit spark task report error: File does not exist：hdfs:xxx (通过spark node提交spark任务报错：File does not exist：hdfs:xxx)
Posted by GitBox <gi...@apache.org>.
lenboo closed issue #2316:
URL: https://github.com/apache/dolphinscheduler/issues/2316
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org