You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/03/11 10:24:39 UTC
[jira] [Commented] (APEXMALHAR-2004) TimeBasedDirectoryScanner keep
reading same file
[ https://issues.apache.org/jira/browse/APEXMALHAR-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190697#comment-15190697 ]
ASF GitHub Bot commented on APEXMALHAR-2004:
--------------------------------------------
Github user tushargosavi commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/203#discussion_r55806762
--- Diff: library/src/main/java/com/datatorrent/lib/io/fs/FileSplitterInput.java ---
@@ -439,12 +439,12 @@ protected ScannedFileInfo createScannedFileInfo(Path parentPath, FileStatus pare
ScannedFileInfo info;
if (rootPath == null) {
info = parentStatus.isDirectory() ?
- new ScannedFileInfo(parentPath.toUri().getPath(), childPath.getName(), parentStatus.getModificationTime()) :
- new ScannedFileInfo(null, childPath.toUri().getPath(), parentStatus.getModificationTime());
+ new ScannedFileInfo(parentPath.toUri().getPath(), childPath.getName(), childStatus.getModificationTime()) :
+ new ScannedFileInfo(null, childPath.toUri().getPath(), childStatus.getModificationTime());
--- End diff --
done
> TimeBasedDirectoryScanner keep reading same file
> ------------------------------------------------
>
> Key: APEXMALHAR-2004
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2004
> Project: Apache Apex Malhar
> Issue Type: Bug
> Affects Versions: 3.3.0
> Reporter: Tushar Gosavi
> Assignee: Tushar Gosavi
>
> The recorded last modification time is of the parent directory, If modification time > last recorded modification time then we will read the file again.
> In case where directory modification time < file modification time, we will keep on reading the file again and again.
> Fix is to record file modification time instead of parent modification time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)