You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Konstantin Shvachko (JIRA)" <ji...@apache.org> on 2008/08/06 03:50:44 UTC
[jira] Created: (HADOOP-3905) Factor out a common interface for
edits log.
Factor out a common interface for edits log.
--------------------------------------------
Key: HADOOP-3905
URL: https://issues.apache.org/jira/browse/HADOOP-3905
Project: Hadoop Core
Issue Type: Improvement
Components: dfs
Affects Versions: 0.19.0
Reporter: Konstantin Shvachko
Assignee: Konstantin Shvachko
Fix For: 0.19.0
Create a generic interface that would cover all edit log operations and be useful for implementing journaling functionality on different storage sources.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3905) Create a generic interface for
edits log.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624734#action_12624734 ]
Hudson commented on HADOOP-3905:
--------------------------------
Integrated in Hadoop-trunk #581 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/])
> Create a generic interface for edits log.
> -----------------------------------------
>
> Key: HADOOP-3905
> URL: https://issues.apache.org/jira/browse/HADOOP-3905
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.19.0
>
> Attachments: EditsLogStreams.patch, EditsLogStreams.patch
>
>
> Create a generic interface that would cover all edit log operations and be useful for implementing journaling functionality on different storage sources.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3905) Create a generic interface for edits
log.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-3905:
----------------------------------------
Hadoop Flags: [Reviewed]
Status: Patch Available (was: Open)
> Create a generic interface for edits log.
> -----------------------------------------
>
> Key: HADOOP-3905
> URL: https://issues.apache.org/jira/browse/HADOOP-3905
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.19.0
>
> Attachments: EditsLogStreams.patch, EditsLogStreams.patch
>
>
> Create a generic interface that would cover all edit log operations and be useful for implementing journaling functionality on different storage sources.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3905) Create a generic interface for
edits log.
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622977#action_12622977 ]
Lohit Vijayarenu commented on HADOOP-3905:
------------------------------------------
+1 this looks good to me.
> Create a generic interface for edits log.
> -----------------------------------------
>
> Key: HADOOP-3905
> URL: https://issues.apache.org/jira/browse/HADOOP-3905
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.19.0
>
> Attachments: EditsLogStreams.patch
>
>
> Create a generic interface that would cover all edit log operations and be useful for implementing journaling functionality on different storage sources.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3905) Create a generic interface for
edits log.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12623032#action_12623032 ]
Hadoop QA commented on HADOOP-3905:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12388330/EditsLogStreams.patch
against trunk revision 686181.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3060/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3060/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3060/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3060/console
This message is automatically generated.
> Create a generic interface for edits log.
> -----------------------------------------
>
> Key: HADOOP-3905
> URL: https://issues.apache.org/jira/browse/HADOOP-3905
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.19.0
>
> Attachments: EditsLogStreams.patch, EditsLogStreams.patch
>
>
> Create a generic interface that would cover all edit log operations and be useful for implementing journaling functionality on different storage sources.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3905) Create a generic interface for edits
log.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-3905:
----------------------------------------
Attachment: EditsLogStreams.patch
I fixed one JavaDoc warning.
> Create a generic interface for edits log.
> -----------------------------------------
>
> Key: HADOOP-3905
> URL: https://issues.apache.org/jira/browse/HADOOP-3905
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.19.0
>
> Attachments: EditsLogStreams.patch, EditsLogStreams.patch
>
>
> Create a generic interface that would cover all edit log operations and be useful for implementing journaling functionality on different storage sources.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3905) Factor out a common interface for
edits log.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620116#action_12620116 ]
Konstantin Shvachko commented on HADOOP-3905:
---------------------------------------------
HADOOP-3860 concluded particularly that edits log currently presents one of the main bottlenecks for the name-node performance.
Currently we can save edits records only into a file and are bound by hard drive performance.
We may try to stream edits e.g. into a database or another high performance system optimized for writes.
My primary focus though is to make it possible to stream records into a secondary name-node so that the latter could
always keep an up todate namespace state with a perspective of building an HA solution based on that.
> Factor out a common interface for edits log.
> --------------------------------------------
>
> Key: HADOOP-3905
> URL: https://issues.apache.org/jira/browse/HADOOP-3905
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.19.0
>
>
> Create a generic interface that would cover all edit log operations and be useful for implementing journaling functionality on different storage sources.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3905) Create a generic interface for edits
log.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-3905:
----------------------------------------
Attachment: EditsLogStreams.patch
In the patch.
# I substantially simplified what used to be called {{EditLogOutputStream}} by using existing hadoop class
{{org.apache.hadoop.io.DataOutputBuffer}} instead of a pair of classes import {{DataOutputStream}} and
{{ByteArrayOutputStream}}
# {{EditLogOutputStream}} is now an abstract class, and {{EditLogFileOutputStream}} is its implementation
for storing edits in a file.
# Introduced {{EditLogInputStream}} and its implementation called {{EditLogFileInputStream}} for reading
edits from a file.
# The rest of the {{FSEditLog}} remained unchanged.
The idea here is that one should write implementations of {{EditLogOutputStream}} and {{EditLogInputStream}}
for a different type of persistent storage and the rest of the logic for the edits log should remain unchanged.
Since I don't have other implementations ready yet it is hard to predict what else should be changed or abstracted.
FSImage and FSEditLog heavily depend on the storage directory file names so may be that should be changed somehow in the future.
> Create a generic interface for edits log.
> -----------------------------------------
>
> Key: HADOOP-3905
> URL: https://issues.apache.org/jira/browse/HADOOP-3905
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.19.0
>
> Attachments: EditsLogStreams.patch
>
>
> Create a generic interface that would cover all edit log operations and be useful for implementing journaling functionality on different storage sources.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3905) Create a generic interface for edits
log.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-3905:
----------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> Create a generic interface for edits log.
> -----------------------------------------
>
> Key: HADOOP-3905
> URL: https://issues.apache.org/jira/browse/HADOOP-3905
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.19.0
>
> Attachments: EditsLogStreams.patch, EditsLogStreams.patch
>
>
> Create a generic interface that would cover all edit log operations and be useful for implementing journaling functionality on different storage sources.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3905) Create a generic interface for edits
log.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-3905:
----------------------------------------
Summary: Create a generic interface for edits log. (was: Factor out a common interface for edits log.)
> Create a generic interface for edits log.
> -----------------------------------------
>
> Key: HADOOP-3905
> URL: https://issues.apache.org/jira/browse/HADOOP-3905
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.19.0
>
>
> Create a generic interface that would cover all edit log operations and be useful for implementing journaling functionality on different storage sources.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.