You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Walter Su (JIRA)" <ji...@apache.org> on 2015/07/03 03:27:04 UTC
[jira] [Assigned] (HDFS-8714) Folder ModificationTime in Millis
Changed When NameNode is restarted
[ https://issues.apache.org/jira/browse/HDFS-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Walter Su reassigned HDFS-8714:
-------------------------------
Assignee: Walter Su
> Folder ModificationTime in Millis Changed When NameNode is restarted
> --------------------------------------------------------------------
>
> Key: HDFS-8714
> URL: https://issues.apache.org/jira/browse/HDFS-8714
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Chandan Biswas
> Assignee: Walter Su
>
> *Steps to Produce*
> # Steps need to do in program
> ** Create a folder into HDFS
> ** Print folder modificationTime in millis
> ** Upload a file or copy a file to this newly created folder
> ** Print file and folder modificationTime in millis
> ** Restart the name node
> ** Print file and folder modificationTime in millis
> # Expected Result
> ** folder modification time should be the file modification time before name node restart
> ** folder modification time should not change after name node restart
> # Actual result
> ** folder modification time is not same with file modification time
> ** folder modification time is changed after name node restart and it's changed to file modification time
> *Impact of this behavior:* Before task is launched, distributed cache files/folders are checked for any modification. The checks are done by comparing file/folder modicationTime in millis. So any job that uses distributed cache has a potential chance of failure if
> # name node restarts and running tasks are resubmitted or
> # for e.g among 100 tasks 50 are in queue for run. Now name node restarts
> Here is the sample code I used for testing-
> {code}
> // file creating in hdfs
> final Path pathToFiles = new Path("/user/vagrant/chandan/test/");
> fileSystem.mkdirs(pathToFiles);
> System.out.println("HDFS Folder Modification Time in long Before file copy:"
> + fileSystem.getFileStatus(pathToFiles).getModificationTime());
> FileUtil.copy(fileSystem, new Path("/user/cloudera/test"), fileSystem, pathToFiles, false, configuration);
> System.out.println("HDFS File Modification Time in long:"
> + fileSystem.getFileStatus(new Path("/user/vagrant/chandan/test/test")).getModificationTime());
> System.out.println("HDFS Folder Modification Time in long After file copy:"
> + fileSystem.getFileStatus(pathToFiles).getModificationTime());
> for (int i = 0; i < 100; i++) {
> System.out.println("Normal HDFS Folder Modification Time in long:"
> + fileSystem.getFileStatus(pathToFiles).getModificationTime());
> System.out.println("Normal HDFS File Modification Time in long:"
> + fileSystem.getFileStatus(new Path("/user/vagrant/chandan/test/test")).getModificationTime());
> Thread.sleep(60000 * 2);
> }
> {code}
> Here is the output -
> {code}
> HDFS Folder Modification Time in long Before file copy:1435868217309
> HDFS File Modification Time in long:1435868217368
> HDFS Folder Modification Time in long After file copy:1435868217353
> Normal HDFS Folder Modification Time in long:1435868217353
> Normal HDFS File Modification Time in long:1435868217368
> Normal HDFS Folder Modification Time in long:1435868217353
> Normal HDFS File Modification Time in long:1435868217368
> Normal HDFS Folder Modification Time in long:1435868217368
> Normal HDFS File Modification Time in long:1435868217368
> {code}
> The last two lines are printed after name node restart.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)