You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/06/28 12:26:38 UTC
[GitHub] [dolphinscheduler] EtachGu opened a new issue #5709: [Bug][Worker] Sqoop task create file permission denied
EtachGu opened a new issue #5709:
URL: https://github.com/apache/dolphinscheduler/issues/5709
**For better global communication, Please describe it in English. If you feel the description in English is not clear, then you can append description in Chinese(just for Mandarin(CN)), thx! **
**Describe the bug**
A clear and concise description of what the bug is.
A sqoop task create file permission denied
**To Reproduce**
log file, mysql url has been desensitization
```log
[INFO] 2021-06-28 20:06:42.543 - [taskAppId=TASK-3-66-67]:[115] - create dir success /tmp/dolphinscheduler/exec/process/1/3/66/67
[INFO] 2021-06-28 20:06:42.599 - [taskAppId=TASK-3-66-67]:[115] - create linux os user : tenet1
[INFO] 2021-06-28 20:06:42.600 - [taskAppId=TASK-3-66-67]:[115] - execute cmd : sudo useradd -g root
tenet1
[INFO] 2021-06-28 20:06:42.761 - [taskAppId=TASK-3-66-67]:[115] - create user name success tenet1
[INFO] 2021-06-28 20:06:42.885 - [taskAppId=TASK-3-66-67]:[57] - sqoop task params {"customShell":"/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin/sqoop import-all-tables \\\n --connect \"jdbc:mysql://xxx:3306/record?allowMultiQueries=true&useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=GMT%2B8&allowPublicKeyRetrieval=true\" \\\n --username root \\\n --password xxx \\\n --hive-import \\\n --hive-database ods_record2 \\\n --hive-overwrite \\\n --create-hive-table \\\n -m 5","jobType":"CUSTOM","localParams":[]}
[INFO] 2021-06-28 20:06:42.903 - [taskAppId=TASK-3-66-67]:[84] - sqoop script: /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin/sqoop import-all-tables \
--connect "jdbc:mysql://xxx:3306/platform_record?allowMultiQueries=true&useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=GMT%2B8&allowPublicKeyRetrieval=true" \
--username root \
--password xxx \
--hive-import \
--hive-database ods_record2 \
--hive-overwrite \
--create-hive-table \
-m 5
[INFO] 2021-06-28 20:06:42.905 - [taskAppId=TASK-3-66-67]:[87] - tenantCode user:tenet1, task dir:3_66_67
[INFO] 2021-06-28 20:06:42.909 - [taskAppId=TASK-3-66-67]:[92] - create command file:/tmp/dolphinscheduler/exec/process/1/3/66/67/3_66_67.command
[INFO] 2021-06-28 20:06:42.910 - [taskAppId=TASK-3-66-67]:[111] - command : #!/bin/sh
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /opt/apache-dolphinscheduler-1.3.6-bin/conf/env/dolphinscheduler_env.sh
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin/sqoop import-all-tables \
--connect "jdbc:mysql://xxx:3306/platform_record?allowMultiQueries=true&useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=GMT%2B8&allowPublicKeyRetrieval=true" \
--username root \
--password xxx \
--hive-import \
--hive-database ods_record2 \
--hive-overwrite \
--create-hive-table \
-m 5
[INFO] 2021-06-28 20:06:42.920 - [taskAppId=TASK-3-66-67]:[327] - task run command:
sudo -u tenet1 sh /tmp/dolphinscheduler/exec/process/1/3/66/67/3_66_67.command
[INFO] 2021-06-28 20:06:42.926 - [taskAppId=TASK-3-66-67]:[208] - process start, process id is: 153
[INFO] 2021-06-28 20:06:43.934 - [taskAppId=TASK-3-66-67]:[129] - -> Warning: /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
[INFO] 2021-06-28 20:06:44.937 - [taskAppId=TASK-3-66-67]:[129] - -> WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
2021-06-28 20:06:44,872 INFO [main] sqoop.Sqoop (Sqoop.java:<init>(94)) - Running Sqoop version: 1.4.7-cdh6.3.2
[INFO] 2021-06-28 20:06:45.939 - [taskAppId=TASK-3-66-67]:[129] - -> 2021-06-28 20:06:45,022 WARN [main] tool.BaseSqoopTool (BaseSqoopTool.java:applyCredentialsOptions(1179)) - Setting your password on the command-line is insecure. Consider using -P instead.
2021-06-28 20:06:45,031 INFO [main] tool.BaseSqoopTool (BaseSqoopTool.java:validateOutputFormatOptions(1550)) - Using Hive-specific delimiters for output. You can override
2021-06-28 20:06:45,031 INFO [main] tool.BaseSqoopTool (BaseSqoopTool.java:validateOutputFormatOptions(1551)) - delimiters with --fields-terminated-by, etc.
2021-06-28 20:06:45,163 INFO [main] manager.MySQLManager (MySQLManager.java:initOptionDefaults(68)) - Preparing to use a MySQL streaming resultset.
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
2021-06-28 20:06:45,701 INFO [main] tool.CodeGenTool (CodeGenTool.java:generateORM(91)) - Beginning code generation
2021-06-28 20:06:45,733 INFO [main] manager.SqlManager (SqlManager.java:execute(775)) - Executing SQL statement: SELECT t.* FROM `api_time_statistics` AS t LIMIT 1
2021-06-28 20:06:45,785 INFO [main] manager.SqlManager (SqlManager.java:execute(775)) - Executing SQL statement: SELECT t.* FROM `api_time_statistics` AS t LIMIT 1
2021-06-28 20:06:45,799 INFO [main] orm.CompilationManager (CompilationManager.java:findHadoopJars(100)) - HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce
[INFO] 2021-06-28 20:06:47.943 - [taskAppId=TASK-3-66-67]:[129] - -> 2021-06-28 20:06:47,874 ERROR [main] orm.CompilationManager (CompilationManager.java:compile(255)) - Could not rename /tmp/sqoop-tenet1/compile/c7a2c509fe2a239a24844297b4d03f00/api_time_statistics.java to /tmp/dolphinscheduler/exec/process/1/3/66/67/./api_time_statistics.java. Error: /tmp/dolphinscheduler/exec/process/1/3/66/67/./api_time_statistics.java (Permission denied)
2021-06-28 20:06:47,874 INFO [main] orm.CompilationManager (CompilationManager.java:jar(363)) - Writing jar file: /tmp/sqoop-tenet1/compile/c7a2c509fe2a239a24844297b4d03f00/api_time_statistics.jar
2021-06-28 20:06:47,906 WARN [main] manager.MySQLManager (MySQLManager.java:importTable(113)) - It looks like you are importing from mysql.
2021-06-28 20:06:47,906 WARN [main] manager.MySQLManager (MySQLManager.java:importTable(114)) - This transfer can be faster! Use the --direct
2021-06-28 20:06:47,906 WARN [main] manager.MySQLManager (MySQLManager.java:importTable(115)) - option to exercise a MySQL-specific fast path.
2021-06-28 20:06:47,907 INFO [main] manager.MySQLManager (MySQLManager.java:checkDateTimeBehavior(204)) - Setting zero DATETIME behavior to convertToNull (mysql)
2021-06-28 20:06:47,919 INFO [main] mapreduce.ImportJobBase (ImportJobBase.java:runImport(241)) - Beginning import of api_time_statistics
[INFO] 2021-06-28 20:06:48.944 - [taskAppId=TASK-3-66-67]:[129] - -> 2021-06-28 20:06:48,312 INFO [main] Configuration.deprecation (Configuration.java:logDeprecation(1381)) - mapred.jar is deprecated. Instead, use mapreduce.job.jar
2021-06-28 20:06:48,627 WARN [main] impl.MetricsConfig (MetricsConfig.java:loadFirst(133)) - Cannot locate configuration: tried hadoop-metrics2-jobtracker.properties,hadoop-metrics2.properties
2021-06-28 20:06:48,717 INFO [main] impl.MetricsSystemImpl (MetricsSystemImpl.java:startTimer(374)) - Scheduled Metric snapshot period at 10 second(s).
2021-06-28 20:06:48,720 INFO [main] impl.MetricsSystemImpl (MetricsSystemImpl.java:start(191)) - JobTracker metrics system started
2021-06-28 20:06:48,866 INFO [main] db.DBInputFormat (DBInputFormat.java:setTxIsolation(192)) - Using read commited transaction isolation
2021-06-28 20:06:48,868 INFO [main] db.DataDrivenDBInputFormat (DataDrivenDBInputFormat.java:getSplits(167)) - BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `api_time_statistics`
2021-06-28 20:06:48,873 INFO [main] db.IntegerSplitter (IntegerSplitter.java:split(145)) - Split size: 23; Num splits: 5 from: 1 to: 117
2021-06-28 20:06:48,894 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(202)) - number of splits:5
[INFO] 2021-06-28 20:06:49.946 - [taskAppId=TASK-3-66-67]:[129] - -> 2021-06-28 20:06:49,033 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(298)) - Submitting tokens for job: job_local2104280907_0001
2021-06-28 20:06:49,035 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(299)) - Executing with tokens: []
2021-06-28 20:06:49,307 INFO [main] mapred.LocalDistributedCacheManager (LocalDistributedCacheManager.java:symlink(201)) - Creating symlink: /tmp/hadoop-tenet1/mapred/local/1624882009141/libjars <- /tmp/dolphinscheduler/exec/process/1/3/66/67/libjars/*
2021-06-28 20:06:49,340 WARN [main] fs.FileUtil (FileUtil.java:symLink(1054)) - Command 'ln -s /tmp/hadoop-tenet1/mapred/local/1624882009141/libjars /tmp/dolphinscheduler/exec/process/1/3/66/67/libjars/*' failed 1 with: ln: failed to create symbolic link '/tmp/dolphinscheduler/exec/process/1/3/66/67/libjars/*': No such file or directory
2021-06-28 20:06:49,341 WARN [main] mapred.LocalDistributedCacheManager (LocalDistributedCacheManager.java:symlink(203)) - Failed to create symlink: /tmp/hadoop-tenet1/mapred/local/1624882009141/libjars <- /tmp/dolphinscheduler/exec/process/1/3/66/67/libjars/*
2021-06-28 20:06:49,341 INFO [main] mapred.LocalDistributedCacheManager (LocalDistributedCacheManager.java:setup(165)) - Localized file:/tmp/hadoop/mapred/staging/tenet12104280907/.staging/job_local2104280907_0001/libjars as file:/tmp/hadoop-tenet1/mapred/local/1624882009141/libjars
2021-06-28 20:06:49,483 INFO [main] mapreduce.Job (Job.java:submit(1574)) - The url to track the job: http://localhost:8080/
2021-06-28 20:06:49,484 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1619)) - Running job: job_local2104280907_0001
2021-06-28 20:06:49,491 INFO [Thread-6] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(501)) - OutputCommitter set in config null
2021-06-28 20:06:49,506 INFO [Thread-6] output.FileOutputCommitter (FileOutputCommitter.java:<init>(129)) - File Output Committer Algorithm version is 2
2021-06-28 20:06:49,506 INFO [Thread-6] output.FileOutputCommitter (FileOutputCommitter.java:<init>(144)) - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-06-28 20:06:49,507 INFO [Thread-6] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(519)) - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2021-06-28 20:06:49,511 ERROR [Thread-6] output.FileOutputCommitter (FileOutputCommitter.java:setupJob(344)) - Mkdirs failed to create file:/tmp/dolphinscheduler/exec/process/1/3/66/67/api_time_statistics/_temporary/0
2021-06-28 20:06:49,561 INFO [Thread-6] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(478)) - Waiting for map tasks
2021-06-28 20:06:49,564 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(252)) - Starting task: attempt_local2104280907_0001_m_000000_0
2021-06-28 20:06:49,602 INFO [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(129)) - File Output Committer Algorithm version is 2
2021-06-28 20:06:49,602 INFO [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(144)) - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-06-28 20:06:49,637 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(625)) - Using ResourceCalculatorProcessTree : [ ]
2021-06-28 20:06:49,648 INFO [LocalJobRunner Map Task Executor #0] db.DBInputFormat (DBInputFormat.java:setTxIsolation(192)) - Using read commited transaction isolation
2021-06-28 20:06:49,657 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(768)) - Processing split: `id` >= 1 AND `id` < 25
2021-06-28 20:06:49,673 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(252)) - Starting task: attempt_local2104280907_0001_m_000001_0
2021-06-28 20:06:49,677 INFO [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(129)) - File Output Committer Algorithm version is 2
2021-06-28 20:06:49,677 INFO [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(144)) - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-06-28 20:06:49,678 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(625)) - Using ResourceCalculatorProcessTree : [ ]
2021-06-28 20:06:49,684 INFO [LocalJobRunner Map Task Executor #0] db.DBInputFormat (DBInputFormat.java:setTxIsolation(192)) - Using read commited transaction isolation
2021-06-28 20:06:49,687 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(768)) - Processing split: `id` >= 25 AND `id` < 48
2021-06-28 20:06:49,689 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(252)) - Starting task: attempt_local2104280907_0001_m_000002_0
2021-06-28 20:06:49,693 INFO [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(129)) - File Output Committer Algorithm version is 2
2021-06-28 20:06:49,694 INFO [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(144)) - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-06-28 20:06:49,694 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(625)) - Using ResourceCalculatorProcessTree : [ ]
2021-06-28 20:06:49,702 INFO [LocalJobRunner Map Task Executor #0] db.DBInputFormat (DBInputFormat.java:setTxIsolation(192)) - Using read commited transaction isolation
2021-06-28 20:06:49,704 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(768)) - Processing split: `id` >= 48 AND `id` < 71
2021-06-28 20:06:49,706 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(252)) - Starting task: attempt_local2104280907_0001_m_000003_0
2021-06-28 20:06:49,708 INFO [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(129)) - File Output Committer Algorithm version is 2
2021-06-28 20:06:49,708 INFO [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(144)) - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-06-28 20:06:49,709 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(625)) - Using ResourceCalculatorProcessTree : [ ]
2021-06-28 20:06:49,716 INFO [LocalJobRunner Map Task Executor #0] db.DBInputFormat (DBInputFormat.java:setTxIsolation(192)) - Using read commited transaction isolation
2021-06-28 20:06:49,719 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(768)) - Processing split: `id` >= 71 AND `id` < 94
2021-06-28 20:06:49,723 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(252)) - Starting task: attempt_local2104280907_0001_m_000004_0
2021-06-28 20:06:49,726 INFO [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(129)) - File Output Committer Algorithm version is 2
2021-06-28 20:06:49,726 INFO [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(144)) - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2021-06-28 20:06:49,726 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(625)) - Using ResourceCalculatorProcessTree : [ ]
2021-06-28 20:06:49,733 INFO [LocalJobRunner Map Task Executor #0] db.DBInputFormat (DBInputFormat.java:setTxIsolation(192)) - Using read commited transaction isolation
2021-06-28 20:06:49,735 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(768)) - Processing split: `id` >= 94 AND `id` <= 117
2021-06-28 20:06:49,738 INFO [Thread-6] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(486)) - map task executor complete.
2021-06-28 20:06:49,739 WARN [Thread-6] mapred.LocalJobRunner (LocalJobRunner.java:run(590)) - job_local2104280907_0001
java.lang.Exception: java.io.IOException: Mkdirs failed to create file:/tmp/dolphinscheduler/exec/process/1/3/66/67/api_time_statistics/_temporary/0/_temporary/attempt_local2104280907_0001_m_000000_0 (exists=false, cwd=file:/tmp/dolphinscheduler/exec/process/1/3/66/67)
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
Caused by: java.io.IOException: Mkdirs failed to create file:/tmp/dolphinscheduler/exec/process/1/3/66/67/api_time_statistics/_temporary/0/_temporary/attempt_local2104280907_0001_m_000000_0 (exists=false, cwd=file:/tmp/dolphinscheduler/exec/process/1/3/66/67)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:458)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:443)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1052)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1032)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:921)
at org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.getFSDataOutputStream(RawKeyTextOutputFormat.java:44)
at org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.getOutputStream(RawKeyTextOutputFormat.java:63)
at org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.getRecordWriter(RawKeyTextOutputFormat.java:74)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:659)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:779)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[INFO] 2021-06-28 20:06:50.605 - [taskAppId=TASK-3-66-67]:[217] - process has exited, execute path:/tmp/dolphinscheduler/exec/process/1/3/66/67, processId:153 ,exitStatusCode:0
[INFO] 2021-06-28 20:06:50.949 - [taskAppId=TASK-3-66-67]:[129] - -> 2021-06-28 20:06:50,493 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1640)) - Job job_local2104280907_0001 running in uber mode : false
2021-06-28 20:06:50,496 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1647)) - map 0% reduce 0%
2021-06-28 20:06:50,499 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1660)) - Job job_local2104280907_0001 failed with state FAILED due to: NA
2021-06-28 20:06:50,519 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1665)) - Counters: 0
2021-06-28 20:06:50,528 WARN [main] mapreduce.Counters (AbstractCounters.java:getGroup(235)) - Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
2021-06-28 20:06:50,530 INFO [main] mapreduce.ImportJobBase (ImportJobBase.java:runJob(190)) - Transferred 0 bytes in 1.9577 seconds (0 bytes/sec)
2021-06-28 20:06:50,531 WARN [main] mapreduce.Counters (AbstractCounters.java:getGroup(235)) - Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2021-06-28 20:06:50,531 INFO [main] mapreduce.ImportJobBase (ImportJobBase.java:runJob(192)) - Retrieved 0 records.
2021-06-28 20:06:50,533 ERROR [main] tool.ImportAllTablesTool (ImportAllTablesTool.java:run(111)) - Error during import: Import job failed!
```
**Expected behavior**
It should work successed
**Which version of Dolphin Scheduler:**
-[1.3.6]
**Additional context**
I tried to add `umask 0002` in `/etc/bash.bashrc`, the root user `umask -p` is `0002`, but the permission of directory created by taskAppId=TASK-3-66-67 still `drwxr-xr-x`
which cause above error, please tell how to solve this problem，Thank you in advance
**Requirement or improvement**
- The process directory should created by tenet， the owner should be the tenet who exec command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #5709: [Bug][Worker] Sqoop task create file permission denied
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #5709:
URL: https://github.com/apache/dolphinscheduler/issues/5709#issuecomment-869641707
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/zh-cn/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EtachGu edited a comment on issue #5709: [Bug][Worker] Sqoop task create file permission denied
Posted by GitBox <gi...@apache.org>.
EtachGu edited a comment on issue #5709:
URL: https://github.com/apache/dolphinscheduler/issues/5709#issuecomment-870470495
有没有人看一下，要不用中文说一遍，Sqoop任务出错，提示无权限修改创建文件或修改文件。
定位原因：
`FileUtils.java`
```java
/**
* create directory and user
* @param execLocalPath execute local path
* @param userName user name
* @throws IOException errors
*/
public static void createWorkDirAndUserIfAbsent(String execLocalPath, String userName) throws IOException {
//if work dir exists, first delete
File execLocalPathFile = new File(execLocalPath);
if (execLocalPathFile.exists()) {
org.apache.commons.io.FileUtils.forceDelete(execLocalPathFile);
}
//create work dir
org.apache.commons.io.FileUtils.forceMkdir(execLocalPathFile); // 这个地方创建的文件夹默认权限 drwxr-xr-x
String mkdirLog = "create dir success " + execLocalPath;
LoggerUtils.logInfo(Optional.ofNullable(logger), mkdirLog);
LoggerUtils.logInfo(Optional.ofNullable(taskLoggerThreadLocal.get()), mkdirLog);
//if not exists this user,then create
OSUtils.taskLoggerThreadLocal.set(taskLoggerThreadLocal.get());
try {
if (!OSUtils.getUserList().contains(userName)) {
boolean isSuccessCreateUser = OSUtils.createUser(userName);
String infoLog;
if (isSuccessCreateUser) {
infoLog = String.format("create user name success %s", userName);
} else {
infoLog = String.format("create user name fail %s", userName);
}
LoggerUtils.logInfo(Optional.ofNullable(logger), infoLog);
LoggerUtils.logInfo(Optional.ofNullable(taskLoggerThreadLocal.get()), infoLog);
}
} catch (Throwable e) {
LoggerUtils.logError(Optional.ofNullable(logger), e);
LoggerUtils.logError(Optional.ofNullable(taskLoggerThreadLocal.get()), e);
}
OSUtils.taskLoggerThreadLocal.remove();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EtachGu commented on issue #5709: [Bug][Worker] Sqoop task create file permission denied
Posted by GitBox <gi...@apache.org>.
EtachGu commented on issue #5709:
URL: https://github.com/apache/dolphinscheduler/issues/5709#issuecomment-871040908
Finally， I solved this problem，we must change the `supervisor.ini` configuration of Worker server， add umask=002
```ini
[program:worker]
command=%(ENV_DOLPHINSCHEDULER_BIN)s/dolphinscheduler-daemon.sh start worker-server
directory=%(ENV_DOLPHINSCHEDULER_HOME)s
priority=999
autostart=%(ENV_WORKER_START_ENABLED)s
autorestart=true
startsecs=10
stopwaitsecs=3
exitcodes=0
stopasgroup=true
killasgroup=true
redirect_stderr=true
stdout_logfile=/dev/fd/1
stdout_logfile_maxbytes=0
umask=002
```
and it works, GOOD
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #5709: [Bug][Worker] Sqoop task create file permission denied
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #5709:
URL: https://github.com/apache/dolphinscheduler/issues/5709#issuecomment-869641707
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/zh-cn/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EtachGu commented on issue #5709: [Bug][Worker] Sqoop task create file permission denied
Posted by GitBox <gi...@apache.org>.
EtachGu commented on issue #5709:
URL: https://github.com/apache/dolphinscheduler/issues/5709#issuecomment-870470495
有没有人看一下，要不用中文说一遍，Sqoop任务出错，提示无权限修改创建文件或修改文件。
定位原因：
`FileUtils.java`
```java
/**
* create directory and user
* @param execLocalPath execute local path
* @param userName user name
* @throws IOException errors
*/
public static void createWorkDirAndUserIfAbsent(String execLocalPath, String userName) throws IOException {
//if work dir exists, first delete
Set<PosixFilePermission> perms = PosixFilePermissions.fromString("rwxrwxrwx");
FileAttribute<Set<PosixFilePermission>> attr = PosixFilePermissions.asFileAttribute(perms);
File execLocalPathFile = new File(execLocalPath, attr);
if (execLocalPathFile.exists()) {
org.apache.commons.io.FileUtils.forceDelete(execLocalPathFile);
}
//create work dir
org.apache.commons.io.FileUtils.forceMkdir(execLocalPathFile); // 这个地方创建的文件夹默认权限 drwxr-xr-x
String mkdirLog = "create dir success " + execLocalPath;
LoggerUtils.logInfo(Optional.ofNullable(logger), mkdirLog);
LoggerUtils.logInfo(Optional.ofNullable(taskLoggerThreadLocal.get()), mkdirLog);
//if not exists this user,then create
OSUtils.taskLoggerThreadLocal.set(taskLoggerThreadLocal.get());
try {
if (!OSUtils.getUserList().contains(userName)) {
boolean isSuccessCreateUser = OSUtils.createUser(userName);
String infoLog;
if (isSuccessCreateUser) {
infoLog = String.format("create user name success %s", userName);
} else {
infoLog = String.format("create user name fail %s", userName);
}
LoggerUtils.logInfo(Optional.ofNullable(logger), infoLog);
LoggerUtils.logInfo(Optional.ofNullable(taskLoggerThreadLocal.get()), infoLog);
}
} catch (Throwable e) {
LoggerUtils.logError(Optional.ofNullable(logger), e);
LoggerUtils.logError(Optional.ofNullable(taskLoggerThreadLocal.get()), e);
}
OSUtils.taskLoggerThreadLocal.remove();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EtachGu commented on issue #5709: [Bug][Worker] Sqoop task create file permission denied
Posted by GitBox <gi...@apache.org>.
EtachGu commented on issue #5709:
URL: https://github.com/apache/dolphinscheduler/issues/5709#issuecomment-870470495
有没有人看一下，要不用中文说一遍，Sqoop任务出错，提示无权限修改创建文件或修改文件。
定位原因：
`FileUtils.java`
```java
/**
* create directory and user
* @param execLocalPath execute local path
* @param userName user name
* @throws IOException errors
*/
public static void createWorkDirAndUserIfAbsent(String execLocalPath, String userName) throws IOException {
//if work dir exists, first delete
Set<PosixFilePermission> perms = PosixFilePermissions.fromString("rwxrwxrwx");
FileAttribute<Set<PosixFilePermission>> attr = PosixFilePermissions.asFileAttribute(perms);
File execLocalPathFile = new File(execLocalPath, attr);
if (execLocalPathFile.exists()) {
org.apache.commons.io.FileUtils.forceDelete(execLocalPathFile);
}
//create work dir
org.apache.commons.io.FileUtils.forceMkdir(execLocalPathFile); // 这个地方创建的文件夹默认权限 drwxr-xr-x
String mkdirLog = "create dir success " + execLocalPath;
LoggerUtils.logInfo(Optional.ofNullable(logger), mkdirLog);
LoggerUtils.logInfo(Optional.ofNullable(taskLoggerThreadLocal.get()), mkdirLog);
//if not exists this user,then create
OSUtils.taskLoggerThreadLocal.set(taskLoggerThreadLocal.get());
try {
if (!OSUtils.getUserList().contains(userName)) {
boolean isSuccessCreateUser = OSUtils.createUser(userName);
String infoLog;
if (isSuccessCreateUser) {
infoLog = String.format("create user name success %s", userName);
} else {
infoLog = String.format("create user name fail %s", userName);
}
LoggerUtils.logInfo(Optional.ofNullable(logger), infoLog);
LoggerUtils.logInfo(Optional.ofNullable(taskLoggerThreadLocal.get()), infoLog);
}
} catch (Throwable e) {
LoggerUtils.logError(Optional.ofNullable(logger), e);
LoggerUtils.logError(Optional.ofNullable(taskLoggerThreadLocal.get()), e);
}
OSUtils.taskLoggerThreadLocal.remove();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] dailidong closed issue #5709: [Bug][Worker] Sqoop task create file permission denied
Posted by GitBox <gi...@apache.org>.
dailidong closed issue #5709:
URL: https://github.com/apache/dolphinscheduler/issues/5709
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EtachGu edited a comment on issue #5709: [Bug][Worker] Sqoop task create file permission denied
Posted by GitBox <gi...@apache.org>.
EtachGu edited a comment on issue #5709:
URL: https://github.com/apache/dolphinscheduler/issues/5709#issuecomment-870470495
有没有人看一下，要不用中文说一遍，Sqoop任务出错，提示无权限修改创建文件或修改文件。
定位原因：
`FileUtils.java`
```java
/**
* create directory and user
* @param execLocalPath execute local path
* @param userName user name
* @throws IOException errors
*/
public static void createWorkDirAndUserIfAbsent(String execLocalPath, String userName) throws IOException {
//if work dir exists, first delete
File execLocalPathFile = new File(execLocalPath);
if (execLocalPathFile.exists()) {
org.apache.commons.io.FileUtils.forceDelete(execLocalPathFile);
}
//create work dir
org.apache.commons.io.FileUtils.forceMkdir(execLocalPathFile); // 这个地方创建的文件夹默认权限 drwxr-xr-x
String mkdirLog = "create dir success " + execLocalPath;
LoggerUtils.logInfo(Optional.ofNullable(logger), mkdirLog);
LoggerUtils.logInfo(Optional.ofNullable(taskLoggerThreadLocal.get()), mkdirLog);
//if not exists this user,then create
OSUtils.taskLoggerThreadLocal.set(taskLoggerThreadLocal.get());
try {
if (!OSUtils.getUserList().contains(userName)) {
boolean isSuccessCreateUser = OSUtils.createUser(userName);
String infoLog;
if (isSuccessCreateUser) {
infoLog = String.format("create user name success %s", userName);
} else {
infoLog = String.format("create user name fail %s", userName);
}
LoggerUtils.logInfo(Optional.ofNullable(logger), infoLog);
LoggerUtils.logInfo(Optional.ofNullable(taskLoggerThreadLocal.get()), infoLog);
}
} catch (Throwable e) {
LoggerUtils.logError(Optional.ofNullable(logger), e);
LoggerUtils.logError(Optional.ofNullable(taskLoggerThreadLocal.get()), e);
}
OSUtils.taskLoggerThreadLocal.remove();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org