You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by "Ivan Veselovsky (JIRA)" <ji...@apache.org> on 2015/03/06 19:42:38 UTC
[jira] [Commented] (IGNITE-418) "terasort" example does not work on
secondary HDFS with DUAL_SYNC, DUAL_ASYNC mode
[ https://issues.apache.org/jira/browse/IGNITE-418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350687#comment-14350687 ]
Ivan Veselovsky commented on IGNITE-418:
----------------------------------------
The cause of the bug is that listing of the directory in DUAL_XXX mode always returns files, never directories.
Simple way to reproduce the bug:
listing the directory content we see that "d2" is a file, while when listing "d2" itself, we see that it is a directory (what is really the case):
ivan@ivan-N56JN:~/_hadoop-apache-test/hadoop-2.6.0/bin$ ./hadoop fs -mkdir -p /tmp/d1/d2/
ivan@ivan-N56JN:~/_hadoop-apache-test/hadoop-2.6.0/bin$ ./hadoop fs -ls /tmp/d1
Found 1 items
-rwxr-xr-x 3 ivan supergroup 0 2015-03-06 20:26 /tmp/d1/d2
ivan@ivan-N56JN:~/_hadoop-apache-test/hadoop-2.6.0/bin$ ./hadoop fs -ls -d /tmp/d1/d2
drwxr-xr-x - ivan supergroup 0 2015-03-06 20:26 /tmp/d1/d2
> "terasort" example does not work on secondary HDFS with DUAL_SYNC, DUAL_ASYNC mode
> ----------------------------------------------------------------------------------
>
> Key: IGNITE-418
> URL: https://issues.apache.org/jira/browse/IGNITE-418
> Project: Ignite
> Issue Type: Bug
> Components: hadoop
> Affects Versions: sprint-1
> Reporter: Ivan Veselovsky
> Assignee: Ivan Veselovsky
> Priority: Critical
> Fix For: sprint-2
>
>
> On attempt to run "terasort" Hadoop example on top of IGFS backed by HDFS in DUAL_SYNC, DUAL_ASYNC mode we have failure: hadoop job finished Ok, but its fesult files are not found:
> + ./hadoop jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar teragen 1000000 /tmp/teragen
> 15/03/05 19:07:26 INFO terasort.TeraSort: Generating 1000000 using 2
> 15/03/05 19:07:26 INFO mapreduce.JobSubmitter: number of splits:2
> 15/03/05 19:07:26 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_774fb810-21a7-48d9-81d1-ce1ac727841f_0001
> 15/03/05 19:07:27 INFO mapreduce.Job: The url to track the job: N/A
> 15/03/05 19:07:27 INFO mapreduce.Job: Running job: job_774fb810-21a7-48d9-81d1-ce1ac727841f_0001
> 15/03/05 19:07:29 INFO mapreduce.Job: Job job_774fb810-21a7-48d9-81d1-ce1ac727841f_0001 running in uber mode : false
> 15/03/05 19:07:29 INFO mapreduce.Job: map 0% reduce 0%
> 15/03/05 19:07:31 INFO mapreduce.Job: map 50% reduce 0%
> 15/03/05 19:07:31 INFO mapreduce.Job: map 100% reduce 100%
> 15/03/05 19:07:31 INFO mapreduce.Job: Job job_774fb810-21a7-48d9-81d1-ce1ac727841f_0001 completed successfully
> 15/03/05 19:07:31 INFO mapreduce.Job: Counters: 1
> org.apache.hadoop.examples.terasort.TeraGen$Counters
> CHECKSUM=2148987642402270
> + ./hadoop fs -ls /tmp/teragen
> ls: `/tmp/teragen': No such file or directory
> At the same time the following exception is on the Ignite node:
> [19:07:16] Topology snapshot [ver=1, nodes=1, CPUs=8, heap=1.0GB]
> [19:07:31,288][ERROR][Hadoop-task-774fb810-21a7-48d9-81d1-ce1ac727841f_1-COMMIT-0-0-#110%null%][IgfsMetaManager] Path rename in DUAL mode failed [source=/tmp/teragen/_temporary/0/task_774fb810-21a7-48d9-81d1-ce1ac727841f_0001_m_000000, destination=/tmp/teragen]
> class org.apache.ignite.IgniteCheckedException: Failed to rename (source path not found): /tmp/teragen/_temporary/0/task_774fb810-21a7-48d9-81d1-ce1ac727841f_0001_m_000000
> at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.fsException(IgfsMetaManager.java:2668)
> at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.access$1100(IgfsMetaManager.java:53)
> at org.apache.ignite.internal.processors.igfs.IgfsMetaManager$7.onSuccess(IgfsMetaManager.java:2052)
> at org.apache.ignite.internal.processors.igfs.IgfsMetaManager$7.onSuccess(IgfsMetaManager.java:2043)
> at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.synchronizeAndExecute(IgfsMetaManager.java:2527)
> at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.synchronizeAndExecute(IgfsMetaManager.java:2337)
> at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.renameDual(IgfsMetaManager.java:2103)
> at org.apache.ignite.internal.processors.igfs.IgfsImpl.rename(IgfsImpl.java:673)
> at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc.rename(HadoopIgfsInProc.java:128)
> at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$5.apply(HadoopIgfsWrapper.java:133)
> at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$5.apply(HadoopIgfsWrapper.java:130)
> at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.withReconnectHandling(HadoopIgfsWrapper.java:289)
> at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.rename(HadoopIgfsWrapper.java:130)
> at org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem.rename(IgniteHadoopFileSystem.java:698)
> at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:345)
> at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310)
> at org.apache.hadoop.mapred.FileOutputCommitter.commitJob(FileOutputCommitter.java:136)
> at org.apache.ignite.internal.processors.hadoop.v1.HadoopV1CleanupTask.run(HadoopV1CleanupTask.java:56)
> at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:193)
> at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:176)
> at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:120)
> at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:36)
> at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:183)
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:145)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: class org.apache.ignite.igfs.IgfsFileNotFoundException: Failed to rename (source path not found): /tmp/teragen/_temporary/0/task_774fb810-21a7-48d9-81d1-ce1ac727841f_0001_m_000000
> ... 23 more
> [19:07:31,294][ERROR][Hadoop-task-774fb810-21a7-48d9-81d1-ce1ac727841f_1-COMMIT-0-0-#110%null%][HadoopRunnableTask] Task execution failed.
> class org.apache.ignite.IgniteCheckedException: class org.apache.ignite.IgniteCheckedException: Failed to rename FileStatus [path=igfs://igfs@localhost/tmp/teragen/_temporary/0/task_774fb810-21a7-48d9-81d1-ce1ac727841f_0001_m_000000, isDir=false, len=0, mtime=1425571651255, atime=1425571651255] to igfs://igfs@localhost/tmp/teragen
> at org.apache.ignite.internal.processors.hadoop.v1.HadoopV1CleanupTask.run(HadoopV1CleanupTask.java:59)
> at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:193)
> at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:176)
> at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:120)
> at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:36)
> at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:183)
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:145)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Failed to rename FileStatus [path=igfs://igfs@localhost/tmp/teragen/_temporary/0/task_774fb810-21a7-48d9-81d1-ce1ac727841f_0001_m_000000, isDir=false, len=0, mtime=1425571651255, atime=1425571651255] to igfs://igfs@localhost/tmp/teragen
> at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:346)
> at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310)
> at org.apache.hadoop.mapred.FileOutputCommitter.commitJob(FileOutputCommitter.java:136)
> at org.apache.ignite.internal.processors.hadoop.v1.HadoopV1CleanupTask.run(HadoopV1CleanupTask.java:56)
> ... 7 more
> at org.apache.ignite.internal.processors.hadoop.HadoopUtils.transformException(HadoopUtils.java:273)
> at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:196)
> at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:176)
> at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:120)
> at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:36)
> at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:183)
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:145)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)