You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/06/22 02:35:35 UTC
[GitHub] [dolphinscheduler] lyn916 opened a new issue #5672: [Question] Shell failed to call hive to execute DDL
lyn916 opened a new issue #5672:
URL: https://github.com/apache/dolphinscheduler/issues/5672
**Which version of DolphinScheduler:**
-[1.3.6]
**Additional context**
org.apache.hadoop.hive.ql.exec.DDLTask. No such file or directory
**Requirement or improvement**
- when i use shell merge the hive orc partitions, FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. No such file or directory
```
2021-06-22 01:00:00.858 INFO - [taskAppId=TASK-10-168-172]:[160] - raw script : #!/bin/bash
source /etc/profile
date=`date -d "-2 days" +%Y%m%d`
hive -e "
ALTER TABLE monitor_data_1 PARTITION(level='0',day='${date}') CONCATENATE;
ALTER TABLE monitor_data_1 PARTITION(level='1',day='${date}') CONCATENATE;
"
2021-06-22 01:00:00.858 INFO - [taskAppId=TASK-10-168-172]:[161] - task execute path : /var/lib/dolphinscheduler/exec/process/1/10/168/172
2021-06-22 01:00:00.858 INFO - [taskAppId=TASK-10-168-172]:[87] - tenantCode user:bde, task dir:10_168_172
2021-06-22 01:00:00.859 INFO - [taskAppId=TASK-10-168-172]:[92] - create command file:/var/lib/dolphinscheduler/exec/process/1/10/168/172/10_168_172.command
2021-06-22 01:00:00.859 INFO - [taskAppId=TASK-10-168-172]:[111] - command : #!/bin/sh
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /usr/local/apache-dolphinscheduler-1.3.6-bin/conf/env/dolphinscheduler_env.sh
/var/lib/dolphinscheduler/exec/process/1/10/168/172/10_168_172_node.sh
2021-06-22 01:00:00.861 INFO - [taskAppId=TASK-10-168-172]:[327] - task run command:
sudo -u bde sh /var/lib/dolphinscheduler/exec/process/1/10/168/172/10_168_172.command
2021-06-22 01:00:00.861 INFO - [taskAppId=TASK-10-168-172]:[208] - process start, process id is: 6988
2021-06-22 01:00:10.935 INFO - [taskAppId=TASK-10-168-172]:[129] - ->
Logging initialized using configuration in file:/usr/local/apache-hive-2.3.7-bin/conf/hive-log4j2.properties Async: true
2021-06-22 01:00:18.981 INFO - [taskAppId=TASK-10-168-172]:[129] - -> Job Submission failed with exception 'org.apache.hadoop.io.nativeio.NativeIOException(No such file or directory)'
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. No such file or directory
2021-06-22 01:00:19.187 INFO - [taskAppId=TASK-10-168-172]:[217] - process has exited, execute path:/var/lib/dolphinscheduler/exec/process/1/10/168/172, processId:6988 ,exitStatusCode:0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] closed issue #5672: [Question] Shell failed to call hive to execute DDL
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #5672:
URL: https://github.com/apache/dolphinscheduler/issues/5672
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #5672: [Question] Shell failed to call hive to execute DDL
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #5672:
URL: https://github.com/apache/dolphinscheduler/issues/5672#issuecomment-865477444
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Tandoy commented on issue #5672: [Question] Shell failed to call hive to execute DDL
Posted by GitBox <gi...@apache.org>.
Tandoy commented on issue #5672:
URL: https://github.com/apache/dolphinscheduler/issues/5672#issuecomment-866646246
Does your shell script not rely on dolphinscheduler to execute normally on the cluster?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #5672: [Question] Shell failed to call hive to execute DDL
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #5672:
URL: https://github.com/apache/dolphinscheduler/issues/5672#issuecomment-1022720551
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #5672: [Question] Shell failed to call hive to execute DDL
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #5672:
URL: https://github.com/apache/dolphinscheduler/issues/5672#issuecomment-1028478434
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #5672: [Question] Shell failed to call hive to execute DDL
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #5672:
URL: https://github.com/apache/dolphinscheduler/issues/5672#issuecomment-865477444
**Which version of DolphinScheduler:**
-[1.3.6]
**Additional context**
org.apache.hadoop.hive.ql.exec.DDLTask. No such file or directory
**Requirement or improvement**
- when i use shell merge the hive orc partitions, FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. No such file or directory
```
2021-06-22 01:00:00.858 INFO - [taskAppId=TASK-10-168-172]:[160] - raw script : #!/bin/bash
source /etc/profile
date=`date -d "-2 days" +%Y%m%d`
hive -e "
ALTER TABLE monitor_data_1 PARTITION(level='0',day='${date}') CONCATENATE;
ALTER TABLE monitor_data_1 PARTITION(level='1',day='${date}') CONCATENATE;
"
2021-06-22 01:00:00.858 INFO - [taskAppId=TASK-10-168-172]:[161] - task execute path : /var/lib/dolphinscheduler/exec/process/1/10/168/172
2021-06-22 01:00:00.858 INFO - [taskAppId=TASK-10-168-172]:[87] - tenantCode user:bde, task dir:10_168_172
2021-06-22 01:00:00.859 INFO - [taskAppId=TASK-10-168-172]:[92] - create command file:/var/lib/dolphinscheduler/exec/process/1/10/168/172/10_168_172.command
2021-06-22 01:00:00.859 INFO - [taskAppId=TASK-10-168-172]:[111] - command : #!/bin/sh
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /usr/local/apache-dolphinscheduler-1.3.6-bin/conf/env/dolphinscheduler_env.sh
/var/lib/dolphinscheduler/exec/process/1/10/168/172/10_168_172_node.sh
2021-06-22 01:00:00.861 INFO - [taskAppId=TASK-10-168-172]:[327] - task run command:
sudo -u bde sh /var/lib/dolphinscheduler/exec/process/1/10/168/172/10_168_172.command
2021-06-22 01:00:00.861 INFO - [taskAppId=TASK-10-168-172]:[208] - process start, process id is: 6988
2021-06-22 01:00:10.935 INFO - [taskAppId=TASK-10-168-172]:[129] - ->
Logging initialized using configuration in file:/usr/local/apache-hive-2.3.7-bin/conf/hive-log4j2.properties Async: true
2021-06-22 01:00:18.981 INFO - [taskAppId=TASK-10-168-172]:[129] - -> Job Submission failed with exception 'org.apache.hadoop.io.nativeio.NativeIOException(No such file or directory)'
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. No such file or directory
2021-06-22 01:00:19.187 INFO - [taskAppId=TASK-10-168-172]:[217] - process has exited, execute path:/var/lib/dolphinscheduler/exec/process/1/10/168/172, processId:6988 ,exitStatusCode:0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] lyn916 commented on issue #5672: [Question] Shell failed to call hive to execute DDL
Posted by GitBox <gi...@apache.org>.
lyn916 commented on issue #5672:
URL: https://github.com/apache/dolphinscheduler/issues/5672#issuecomment-865504706
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] lyn916 commented on issue #5672: [Question] Shell failed to call hive to execute DDL
Posted by GitBox <gi...@apache.org>.
lyn916 commented on issue #5672:
URL: https://github.com/apache/dolphinscheduler/issues/5672#issuecomment-865504706
i have partiton ，but log say ‘is partitioned but partition not found‘.
`Logging initialized using configuration in file:/usr/local/apache-hive-2.3.7-bin/conf/hive-log4j2.properties Async: true
FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException: source table default.monitor_data_1 is partitioned but partition not found.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] lyn916 commented on issue #5672: [Question] Shell failed to call hive to execute DDL
Posted by GitBox <gi...@apache.org>.
lyn916 commented on issue #5672:
URL: https://github.com/apache/dolphinscheduler/issues/5672#issuecomment-865614767
```2021-06-22 11:50:09.919 WARN [org.apache.hadoop.mapreduce.JobResourceUploader] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2021-06-22 11:50:09.926 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Cleaning up the staging area file:/var/lib/hdfs/tmp/mapred/staging/bde808230759/.staging/job_local808230759_0001
2021-06-22 11:50:09.929 ERROR [org.apache.hadoop.hive.ql.exec.Task] - Job Submission failed with exception 'org.apache.hadoop.io.nativeio.NativeIOException(No such file or directory)'
ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:230)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:729)
at org.apache.hadoop.fs.ChecksumFileSystem$1.apply(ChecksumFileSystem.java:505)
at org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:486)
at org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:508)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:601)
at org.apache.hadoop.mapreduce.JobResourceUploader.uploadFiles(JobResourceUploader.java:94)
at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:95)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:190)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
at org.apache.hadoop.hive.ql.io.merge.MergeFileTask.execute(MergeFileTask.java:156)
at org.apache.hadoop.hive.ql.exec.DDLTask.mergeFiles(DDLTask.java:754)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:553)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:787)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
2021-06-22 11:50:09.936 ERROR [org.apache.hadoop.hive.ql.Driver] - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. No such file or directory
2021-06-22 11:50:09.936 INFO [org.apache.hadoop.hive.ql.Driver] - Completed executing command(queryId=bde_20210622115005_532b5d24-6e19-4f5a-9f5a-427c7ad0914d); Time taken: 0.64 seconds
2021-06-22 11:50:10.011 INFO [org.apache.hadoop.hive.conf.HiveConf] - Using the default value passed in for log id: faed4357-6e4d-4cf6-a2b2-0b5ac225ad3f
2021-06-22 11:50:10.011 INFO [org.apache.hadoop.hive.ql.session.SessionState] - Resetting thread name to main
2021-06-22 11:50:10.012 INFO [org.apache.hadoop.hive.conf.HiveConf] - Using the default value passed in for log id: faed4357-6e4d-4cf6-a2b2-0b5ac225ad3f
2021-06-22 11:50:10.022 INFO [org.apache.hadoop.hive.ql.session.SessionState] - Deleted directory: /tmp/hive/bde/faed4357-6e4d-4cf6-a2b2-0b5ac225ad3f on fs with scheme hdfs
2021-06-22 11:50:10.023 INFO [org.apache.hadoop.hive.ql.session.SessionState] - Deleted directory: /tmp/bde/faed4357-6e4d-4cf6-a2b2-0b5ac225ad3f on fs with scheme file
2021-06-22 11:50:10.023 INFO [hive.metastore] - Closed a connection to metastore, current connections: 2
2021-06-22 11:50:10.261 INFO [org.apache.zookeeper.ZooKeeper] - Session: 0xab7a1fb342aa0058 closed
2021-06-22 11:50:10.261 INFO [org.apache.zookeeper.ClientCnxn] - EventThread shut down for session: 0xab7a1fb342aa0058
2021-06-22 11:50:10.262 INFO [CuratorFrameworkSingleton] - Closing ZooKeeper client.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #5672: [Question] Shell failed to call hive to execute DDL
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #5672:
URL: https://github.com/apache/dolphinscheduler/issues/5672#issuecomment-865477511
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/zh-cn/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org