You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by "Ivan Veselovsky (JIRA)" <ji...@apache.org> on 2015/03/05 17:15:38 UTC
[jira] [Created] (IGNITE-418) "terasort" example does not work on
secondary HDFS with DUAL_SYNC, DUAL_ASYNC mode
Ivan Veselovsky created IGNITE-418:
--------------------------------------
Summary: "terasort" example does not work on secondary HDFS with DUAL_SYNC, DUAL_ASYNC mode
Key: IGNITE-418
URL: https://issues.apache.org/jira/browse/IGNITE-418
Project: Ignite
Issue Type: Bug
Reporter: Ivan Veselovsky
Assignee: Ivan Veselovsky
Priority: Critical
On attempt to run "terasort" Hadoop example on top of IGFS backed by HDFS in DUAL_SYNC, DUAL_ASYNC mode we have failure: hadoop job finished Ok, but its fesult files are not found:
+ ./hadoop jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar teragen 1000000 /tmp/teragen
15/03/05 19:07:26 INFO terasort.TeraSort: Generating 1000000 using 2
15/03/05 19:07:26 INFO mapreduce.JobSubmitter: number of splits:2
15/03/05 19:07:26 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_774fb810-21a7-48d9-81d1-ce1ac727841f_0001
15/03/05 19:07:27 INFO mapreduce.Job: The url to track the job: N/A
15/03/05 19:07:27 INFO mapreduce.Job: Running job: job_774fb810-21a7-48d9-81d1-ce1ac727841f_0001
15/03/05 19:07:29 INFO mapreduce.Job: Job job_774fb810-21a7-48d9-81d1-ce1ac727841f_0001 running in uber mode : false
15/03/05 19:07:29 INFO mapreduce.Job: map 0% reduce 0%
15/03/05 19:07:31 INFO mapreduce.Job: map 50% reduce 0%
15/03/05 19:07:31 INFO mapreduce.Job: map 100% reduce 100%
15/03/05 19:07:31 INFO mapreduce.Job: Job job_774fb810-21a7-48d9-81d1-ce1ac727841f_0001 completed successfully
15/03/05 19:07:31 INFO mapreduce.Job: Counters: 1
org.apache.hadoop.examples.terasort.TeraGen$Counters
CHECKSUM=2148987642402270
+ ./hadoop fs -ls /tmp/teragen
ls: `/tmp/teragen': No such file or directory
At the same time the following exception is on the Ignite node:
[19:07:16] Topology snapshot [ver=1, nodes=1, CPUs=8, heap=1.0GB]
[19:07:31,288][ERROR][Hadoop-task-774fb810-21a7-48d9-81d1-ce1ac727841f_1-COMMIT-0-0-#110%null%][IgfsMetaManager] Path rename in DUAL mode failed [source=/tmp/teragen/_temporary/0/task_774fb810-21a7-48d9-81d1-ce1ac727841f_0001_m_000000, destination=/tmp/teragen]
class org.apache.ignite.IgniteCheckedException: Failed to rename (source path not found): /tmp/teragen/_temporary/0/task_774fb810-21a7-48d9-81d1-ce1ac727841f_0001_m_000000
at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.fsException(IgfsMetaManager.java:2668)
at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.access$1100(IgfsMetaManager.java:53)
at org.apache.ignite.internal.processors.igfs.IgfsMetaManager$7.onSuccess(IgfsMetaManager.java:2052)
at org.apache.ignite.internal.processors.igfs.IgfsMetaManager$7.onSuccess(IgfsMetaManager.java:2043)
at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.synchronizeAndExecute(IgfsMetaManager.java:2527)
at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.synchronizeAndExecute(IgfsMetaManager.java:2337)
at org.apache.ignite.internal.processors.igfs.IgfsMetaManager.renameDual(IgfsMetaManager.java:2103)
at org.apache.ignite.internal.processors.igfs.IgfsImpl.rename(IgfsImpl.java:673)
at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc.rename(HadoopIgfsInProc.java:128)
at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$5.apply(HadoopIgfsWrapper.java:133)
at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$5.apply(HadoopIgfsWrapper.java:130)
at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.withReconnectHandling(HadoopIgfsWrapper.java:289)
at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.rename(HadoopIgfsWrapper.java:130)
at org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem.rename(IgniteHadoopFileSystem.java:698)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:345)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310)
at org.apache.hadoop.mapred.FileOutputCommitter.commitJob(FileOutputCommitter.java:136)
at org.apache.ignite.internal.processors.hadoop.v1.HadoopV1CleanupTask.run(HadoopV1CleanupTask.java:56)
at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:193)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:176)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:120)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:36)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:183)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:145)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.igfs.IgfsFileNotFoundException: Failed to rename (source path not found): /tmp/teragen/_temporary/0/task_774fb810-21a7-48d9-81d1-ce1ac727841f_0001_m_000000
... 23 more
[19:07:31,294][ERROR][Hadoop-task-774fb810-21a7-48d9-81d1-ce1ac727841f_1-COMMIT-0-0-#110%null%][HadoopRunnableTask] Task execution failed.
class org.apache.ignite.IgniteCheckedException: class org.apache.ignite.IgniteCheckedException: Failed to rename FileStatus [path=igfs://igfs@localhost/tmp/teragen/_temporary/0/task_774fb810-21a7-48d9-81d1-ce1ac727841f_0001_m_000000, isDir=false, len=0, mtime=1425571651255, atime=1425571651255] to igfs://igfs@localhost/tmp/teragen
at org.apache.ignite.internal.processors.hadoop.v1.HadoopV1CleanupTask.run(HadoopV1CleanupTask.java:59)
at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:193)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:176)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:120)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:36)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:183)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:145)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to rename FileStatus [path=igfs://igfs@localhost/tmp/teragen/_temporary/0/task_774fb810-21a7-48d9-81d1-ce1ac727841f_0001_m_000000, isDir=false, len=0, mtime=1425571651255, atime=1425571651255] to igfs://igfs@localhost/tmp/teragen
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:346)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310)
at org.apache.hadoop.mapred.FileOutputCommitter.commitJob(FileOutputCommitter.java:136)
at org.apache.ignite.internal.processors.hadoop.v1.HadoopV1CleanupTask.run(HadoopV1CleanupTask.java:56)
... 7 more
at org.apache.ignite.internal.processors.hadoop.HadoopUtils.transformException(HadoopUtils.java:273)
at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:196)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:176)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:120)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:36)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:183)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:145)
at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)