[jira] [Commented] (HDDS-3545) MR Jobhistory cannot work well with o3fs hadoop compatible file system

["maobaolong (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDDS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17141784#comment-17141784 ] 

maobaolong commented on HDDS-3545:
----------------------------------

Thanks [~elek] and [~arp] for the discussion in the apache slack channel for this tickets. Until now, we all think HDDS-2939 can make it possible, But not sure when HDDS-2939 would be available in master branch. 

The following is the reply from [~elek]
{quote}
HDDS-2939 is the umbrella of the new approach It should be possible to add some support with the current approach. IMHO the only tricky question is that how to make it optional. This is an additional update for each file update (update modification time). It might not be required for all the use cases.
With current approach you can:
updating /a/b/c/d
create /a/b/c/ (directory entry) of doesn't exist
update the modification time in the same batch
It makes everything slower (if it's required all the time) but can be possible. One way to make it optional to make it part of CreateFile and not CommitKey. CommitKey can be the pure ObjectStore method but CreateFile should provide the file system sematics
{quote}

So, i think until now, we cannot support MR jobhistory well, because MR jobhistory can display the new jobs when the modification time of the parent directory change, but now we cannot support the modification time of directory.


> MR Jobhistory cannot work well with o3fs hadoop compatible file system
> ----------------------------------------------------------------------
>
>                 Key: HDDS-3545
>                 URL: https://issues.apache.org/jira/browse/HDDS-3545
>             Project: Hadoop Distributed Data Store
>          Issue Type: Bug
>          Components: Ozone Filesystem
>            Reporter: maobaolong
>            Priority: Major
>              Labels: TriagePending
>
> After take a look at the code of JobHistory, i see  jobhistory use `fs.getModificationTime()` to get the directory modification time, and use it as a condition of starting scan the job directories. 
> But, for ozone, wile insert a child to a `directory`, the modification time of the `directory` don't update now.
> So we should update the modification time of `directory`, otherwise, MR Jobhistory and some other component which use the modification time of `directory` cannot work as expected.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org