[jira] Issue Comment Edited: (HADOOP-5188) Modifications to enable multiple types of logging

["Luca Telloli (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-5188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711213#action_12711213 ] 

Luca Telloli edited comment on HADOOP-5188 at 5/20/09 9:38 AM:
---------------------------------------------------------------

This patch adds the following features:
- supports for URI for property dfs.name.edits.dir (HADOOP-5832)
- abstraction of logging systems 

The abstraction might be still incomplete, but I think it's in a good shape. 

To achieve the first point (URI in dfs.name.edits.dir), the patch introduces a new abstraction called LoggingDevice. 
The trunk implementation currently assumes that every attribute inside the property dfs.name.edits.dir is a StorageDirectory, which is false if other types of logging are specified. LoggingDevice should help with it; additionally, a Logging device would incorporate all needed parameters for its instantiation by reading them from the initialization URI. 

Ideally, a new logging device should: 
- extend class LoggingDevice 
- implement interfaces EditLogInput/OutputStream

Additionally, in the current implementation there is one data structure that keeps the set of edit streams (FSEditLog.editStreams) and one structure that keeps the list of StorageDirectories. The two structures are maintained in sync whenever a edit stream fails (the corresponding storage directory is removed) or a storage directory fails (the corresponding edit stream is removed). 

My hope is that, by using logging devices, we could keep only one data structure. 


Finally: 
- the patch passes all NameNode unit tests
- the patch applies against current trunk (as of today)
- I'm starting the submit patch process to get more feedback from users

      was (Author: lucat):
    This patch adds the following features:
- supports for URI for property dfs.name.edits.dir (HADOOP-5832)
- abstraction of logging systems 

The abstraction might be still incomplete, but I think it's in a good shape. 

To achieve the first point (URI in dfs.name.edits.dir), the patch introduces a new abstraction called LoggingDevice. 
The trunk implementation currently assumes that every attribute inside the property dfs.name.edits.dir is a StorageDirectory, which is false if other types of logging are specified. LoggingDevice should help with it; additionally, a Logging device would incorporate all needed parameters for its instantiation by reading them from the initialization URI. 

Ideally, a new logging device should: 
- extend class LoggingDevice 
- implement interfaces EditLogInput/OutputStream

Additionally, in the current implementation there is one data structure that keeps the set of edit streams (FSEditLog.editStreams) and one structure that keeps the list of StorageDirectories. The two structures are maintained in sync whenever a edit stream fails (the corresponding storage directory is removed) or a storage directory fails (the corresponding edit stream is removed). 

My hope is that, by using logging devices, we could keep only one data structure. 


Finally: 
- the patch passes all NameNode unit tests
- I'm starting the submit patch process to get more feedback from users
  
> Modifications to enable multiple types of logging 
> --------------------------------------------------
>
>                 Key: HADOOP-5188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5188
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.20.0
>            Reporter: Luca Telloli
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5188.patch, HADOOP-5188.patch, HADOOP-5188.patch, HADOOP-5188.patch, HADOOP-5188.pdf
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.