You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Ivan Kelly (JIRA)" <ji...@apache.org> on 2011/07/14 17:40:59 UTC
[jira] [Created] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Move EditLogOp serialisation and deserialation into one place
-------------------------------------------------------------
Key: HDFS-2149
URL: https://issues.apache.org/jira/browse/HDFS-2149
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Ivan Kelly
Assignee: Ivan Kelly
Fix For: 0.23.0
On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated HDFS-2149:
-----------------------------
Attachment: HDFS-2149.diff
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065368#comment-13065368 ]
Ivan Kelly commented on HDFS-2149:
----------------------------------
It's been mentioned in the past and it should be done. However that code confuses me at the moment. It should be possible to get rid of all the hardcoded duplicated op stuff though and replace it with reflection.
I'd also like to run a benchmark on the new code to ensure it doesn't add any slowdown. Whats the best way to do this?
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067949#comment-13067949 ]
Todd Lipcon commented on HDFS-2149:
-----------------------------------
Oh, I see. Let me review against trunk then, sorry for confusion.
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2149:
------------------------------
Status: Patch Available (was: Open)
marking patch available again to get hudson results
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068177#comment-13068177 ]
Todd Lipcon commented on HDFS-2149:
-----------------------------------
Looking over the patch, it seems like this is really two distinct parts:
1) cleans up the writing of edits:
-- rather than writing logEdit(opcode, Writable...) in FSEditLog, the API is logEdit(op record)
-- this API change filters down into EditLogOutputStream as well
-- writeRaw is introduced to be used from the BN (like in 1073 branch)
-- EditLogBackupOutputStream, which used to buffer Writables in JournalRecord, now directly buffers the op objects
2) Changes the API around *reading* ops to be an iterator-style interface rather than a byte-style interface, by pushing down the FSEditLogOp.Reader into EditLogInputStream.
I'm convinced that (1) above is a good idea, and I think the patch is mostly ready for commit modulo a couple things below.
I'm not yet convinced that (2) above is quite right yet. We might need another iteration or two to discuss the APIs.
Would you mind using this JIRA to get part (1) committed, and then work out the kinks in (2) next? Or is there some reason that part (1) depends on part (2) that I'm missing?
Issues:
*EditLogBackupOutputStream bug*:
Buffering the Op objects in EditLogBackupOutputStream is likely to cause a bug since the objects are reused. If a single thread logs twice before syncing (unlikely but possible for some operations) the edits will get modified in place and the BN will diverge. This might be a bug in the current BN as well, but the Writables weren't always reused, whereas these op objects are.
To fix this, I think we should make EditLogBackupOutputStream and EditLogFileOutputStream act the same - ie serialize the ops immediately into a buffer. Perhaps they could share this code in the abstract base class? Or, introduce a new classs like JournalDoubleBuffer?
*getInstance for AddCloseOp*:
Rather than have two separate ThreadLocals for AddCloseOp, I think it would be cleaner to add a "setOp()" function there on a single instance. My only reasoning is that then all of the classes are nicely symmetric.
*javadoc on writeRaw*:
I find this javadoc confusing. What do you mean by "should only be used in new code"? Also, the punctuation is incomplete which makes the comment hard to read. Perhaps just copy the javadoc from the 1073 branch s it doesn't need to be resolved at merge time?
*For brevity*:
Here's a thought: rather than giving each op its own ThreadLocal and getInstance method, why not make the {{opInstances}} EnumMap that we already have into a ThreadLocal? Something like:
{code}
// in FSEditLogOp.java:
ThreadLocal<EnumMap<FSEditLogOpCodes, FSEditLogOp>> opInstances =
new ThreadLocal<EnumMap<FSEditLogOpCodes, FSEditLogOp>>() {
@Override
protected EnumMap<FSEditLogOpCodes, FSEditLogOp> initialValue() {
// ...
}
};
<T extends FSEditLogOp> T getThreadLocalOp(
FSEditLogOpCodes opCode, Class<T> clazz) {
return clazz.cast(opInstances.get().get(opCode));
}
// in FSEditLog.java:
public void logCloseFile(String path, INodeFile newNode) {
AddCloseOp op = FSEditLogOp.getThreadLocalOp(OP_CLOSE, AddCloseOp.class)
.setPath(path)
...;
}
{code}
I think this would reduce the amount of boilerplate and shouldn't be any discernible perf difference.
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2149) Move EditLogOp serialization formats
into FsEditLogOp implementations
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082049#comment-13082049 ]
Hudson commented on HDFS-2149:
------------------------------
Integrated in Hadoop-Hdfs-1073-branch #23 (See [https://builds.apache.org/job/Hadoop-Hdfs-1073-branch/23/])
Merge trunk into HDFS-1073.
Resolved several conflicts due to merge of HDFS-2149 and HDFS-2212.
Changes during resolution were:
- move the writing of the transaction ID out of EditLogOutputStream to
FSEditLogOp.Writer to match trunk's organization
- remove JSPOOL related FsEditLogOp subclasses, add LogSegmentOp subclasses
- modify TestEditLogJournalFailures to not keep trying to use streams after
the simulated halt, since newer stricter assertions caused these writes to
fail
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152128
Files :
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileAppend2.java
* /hadoop/common/branches/HDFS-1073/hdfs/CHANGES.txt
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileStatus.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/docs/src/documentation/content/xdocs/hdfsproxy.xml
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestComputeInvalidateWork.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/contrib/build.xml
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestDatanodeDeath.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileCreationDelete.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileAppend3.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/DFSClient.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestEditLogJournalFailures.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileConcurrentReader.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/docs/src/documentation/content/xdocs/site.xml
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/unit/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithMultipleNameNodes.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestDecommission.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestEditsDoubleBuffer.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestLeaseRecovery2.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/common/Storage.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/webapps/datanode
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestMultiThreadedHflush.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileAppend4.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/UpgradeUtilities.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/contrib/hdfsproxy
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileCreation.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/aop/org/apache/hadoop/hdfs/TestFiPipelines.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileCorruption.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestComputeInvalidateWork.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestPipelines.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/webapps/hdfs
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestReadWhileWriting.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/webapps/secondary
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestRenameWhileOpen.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileCreationEmpty.java
* /hadoop/common/branches/HDFS-1073/hdfs
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/datanode/TestBlockReport.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/c++/libhdfs
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestStorageRestore.java
* /hadoop/common/branches/HDFS-1073/hdfs/build.xml
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileCreationClient.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingReplicationBlocks.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditsDoubleBuffer.java
* /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> Move EditLogOp serialization formats into FsEditLogOp implementations
> ---------------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071095#comment-13071095 ]
Ivan Kelly commented on HDFS-2149:
----------------------------------
I've addressed all these comments. For writeFields I've created a new FSEditLogOp.Writer which is symmetrical to FSEditLogOp.Reader. There is also an asymmetry in how the checksumming is done. I have solved this in HDFS-2187 though.
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068256#comment-13068256 ]
Ivan Kelly commented on HDFS-2149:
----------------------------------
I've removed (2) for the current patch but I absolutely think that it's the way to go with input. To turn it around, is there ever any situation where you would want to read anything other than an FSEditLogOp from an edit log stream? The only situation previously was to read the version number at the start. This is metadata though. In BK I won't store this in the stream at all. In file and bk cases I think it's better if this is accessed with a getVersion() method rather than the reader having to know about the stored format.
*EditLogBackupOutputStream bug*
Well spotted. I've changed this again to use a byte[] to store the ops once it's written to the stream. It's now very similar to what it was originally.
*getInstance for AddCloseOp*
I've changed this so that AddCloseOp is abstract and you call AddOp.getInstance() and CloseOp.getInstance() to get the actual instances.
*javadoc on writeRaw*
That should have read "Should not be used on new code". In any case, i've just pulled the comment from 1073 instead.
*brevity changes*
I've removed most of the boilerplate to replace with a opInstances ThreadLocal. However, i've retained the getInstance() calls on the ops themselves. FSEditLogOp.getThreadLocalOp(OP_CLOSE, AddCloseOp.class) looks clunky to me. You're passing in the op code and the op class. You can't get away with just the op code because the generics framework need to know what to return. You can't get away with just the class because the enummap is code->instance, not visa versa. In any case, the getInstance methods are quite small. It was the ThreadLocal stuff that looked very heavy, IMO.
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated HDFS-2149:
-----------------------------
Attachment: HDFS-2149.diff
Code ready for review/submission provides the benchmarks don't show anything nasty. I'm running those now.
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065802#comment-13065802 ]
Ivan Kelly commented on HDFS-2149:
----------------------------------
REPOST from HDFS-1073:
HDFS-2149 will probably remove LogHeader completely. I plan to add a getVersion() call to InputStreams and each stream will handle it's own metadata internally. So EditLogFileInputStream will read it's version on creation, or first call to read etc. The input and output stream will be packet based, so an input stream is basically an iterator over FSEditLogOp objects and output stream is a sink for FSEditLogOp objects. I think the way I've implemented the FSEditLogOp objects should avoid all extra copies and object creation. Whats more, there's plenty to room to improve this by removing the creation of ArrayWritables and DeprecatedUTF8 objects and just write strings and arrays directly.
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065429#comment-13065429 ]
Aaron T. Myers commented on HDFS-2149:
--------------------------------------
I also find the OEV code confusing. I think its implementation of the visitor pattern is slightly different from what is usually done.
As for a benchmark, I'd recommend either using {{NNThroughputBenchmark}} or writing your own custom micro-benchmark in the same vain as HDFS-1846.
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070873#comment-13070873 ]
Todd Lipcon commented on HDFS-2149:
-----------------------------------
- In all of the getInstance methods, you're doing FooOp.class.cast(...). Why not use the normal casting syntax (FooOp)...?
- in logOpen and logCloseFile, you should use the subclass types rather than AddCloseOp. The fact that they both inherit from AddCloseOp should be an internal detail of FSEditLogOp
- Indentation error:
{code}
EnumMap<FSEditLogOpCodes, FSEditLogOp> instances
= new EnumMap<FSEditLogOpCodes, FSEditLogOp>(FSEditLogOpCodes.class);
{code}
- I don't like the assymetry that Op.writeFields() writes the opcode, whereas readFields() doesn't (since it's read in readOp()). I think this could be fixed by pulling it up into the EditLogOutputStream {{write}} functions. Or, if you want it to all be in FsEditLogOp, add a new static function there called {{writeOp(FSEditLogOp, DataOutputStream)}} or something?
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated HDFS-2149:
-----------------------------
Attachment: HDFS-2149.diff
Fixed findbugs error
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated HDFS-2149:
-----------------------------
Attachment: HDFS-2149.diff
Initial drop of the changes. I have yet to figure out what to do with BackupNode which is using the editlog data streams as control streams also. I've commented out two lines related to this until I figure out what to do. Surprisingly, TestBackupNode passes without issue.
FSEditLog is using EditLogOutputStreams as OutputStreams now, and I've added code to add checksumming for all streams. Previously it was only happening for File streams, not backup streams.
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2149) Move EditLogOp serialization formats
into FsEditLogOp implementations
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2149:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk, thanks Ivan!
> Move EditLogOp serialization formats into FsEditLogOp implementations
> ---------------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Jitendra Nath Pandey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jitendra Nath Pandey updated HDFS-2149:
---------------------------------------
Issue Type: Sub-task (was: Improvement)
Parent: HDFS-1580
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2149) Move EditLogOp serialization formats
into FsEditLogOp implementations
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2149:
------------------------------
Component/s: name-node
Affects Version/s: 0.23.0
Hadoop Flags: [Reviewed]
Summary: Move EditLogOp serialization formats into FsEditLogOp implementations (was: Move EditLogOp serialisation and deserialation into one place)
> Move EditLogOp serialization formats into FsEditLogOp implementations
> ---------------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067612#comment-13067612 ]
Ivan Kelly commented on HDFS-2149:
----------------------------------
I ran these the NNThroughputBenchmark on these changes as follows:
The bench was ran on a quad core xeon, with 48G of Ram. A dedicated disk was used for the namenode.
Parameters: -op all -threads 100 -files 100000
The bench was run 5 times both for trunk and HDFS-2149 and the average and standard deviation recorded as below. In short this patch actually gives a slight improvement in performance, but nothing significant.
Numbers in the table are ops per second, bigger numbers are better.
|| op || Trunk Avg || Trunk stddev || HDFS-2149 Avg || HDFS-2149 stddev || Gain || %Gain ||
| create | 5174.918613 | 37.82819514 | 5180.145729 | 44.19491851 | +5.227116624 | +0.001010087 |
| open | 29913.30775 | 2063.514703 | 30322.53589 | 1413.393394 | +409.228147 | +0.013680471 |
| delete | 9135.00613 | 193.0711833 | 9329.309289 | 209.2509118 | +194.3031585 | +0.021270173 |
| fileStatus | 134478.4952 | 32823.61628 | 140988.636 | 37665.42285 | +6510.140784 | +0.048410274 |
| rename | 7199.803766 | 84.61922167 | 7084.922625 | 82.04274592 | -114.8811417 | -0.015956149 |
| blockReport | 693.4065934 | 111.5533471 | 719.8717949 | 69.17435973 | +26.46520147 | +0.038166931 |
| replication | 700 | 273.8612788 | 900 | 223.6067977 | +200 | +0.285714286 |
| clean | 600 | 223.6067977 | 750 | 353.5533906 | +150 | +0.25 |
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated HDFS-2149:
-----------------------------
Attachment: HDFS-2149.diff
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067917#comment-13067917 ]
Todd Lipcon commented on HDFS-2149:
-----------------------------------
hm, is this patch supposed to be against the tip of 1073 branch? Looks like it was made against a checkout prior to the BackupNode work
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071347#comment-13071347 ]
Todd Lipcon commented on HDFS-2149:
-----------------------------------
+1. Hudson QA bot timed out on the last test run, but seems unrelated to the patch. Since the two most recent patches only differ by the introduction of a 'static' keyword, I will commit based on the previous hudson +1 results. Thanks, Ivan!
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2149) Move EditLogOp serialization formats
into FsEditLogOp implementations
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071360#comment-13071360 ]
Hudson commented on HDFS-2149:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #806 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/806/])
HDFS-2149. Move EditLogOp serialization formats into FsEditLogOp implementations. Contributed by Ivan Kelly.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151238
Files :
* /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
* /hadoop/common/trunk/hdfs/CHANGES.txt
* /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
* /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java
* /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java
> Move EditLogOp serialization formats into FsEditLogOp implementations
> ---------------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2149) Move EditLogOp serialization formats
into FsEditLogOp implementations
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071703#comment-13071703 ]
Hudson commented on HDFS-2149:
------------------------------
Integrated in Hadoop-Hdfs-trunk #737 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/737/])
HDFS-2149. Move EditLogOp serialization formats into FsEditLogOp implementations. Contributed by Ivan Kelly.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151238
Files :
* /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
* /hadoop/common/trunk/hdfs/CHANGES.txt
* /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
* /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java
* /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java
> Move EditLogOp serialization formats into FsEditLogOp implementations
> ---------------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067940#comment-13067940 ]
Ivan Kelly commented on HDFS-2149:
----------------------------------
This is against trunk. There's nothing that would be specific to 1073 in it.
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065348#comment-13065348 ]
Aaron T. Myers commented on HDFS-2149:
--------------------------------------
This is a great cleanup, Ivan. Have you considered also refactoring the OEV to use this code as well? Presently, all of the edit log loading code is effectively duplicated in the OEV.
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2149) Move EditLogOp serialisation and
deserialation into one place
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071129#comment-13071129 ]
Hadoop QA commented on HDFS-2149:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12487829/HDFS-2149.diff
against trunk revision 1150960.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1018//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1018//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1018//console
This message is automatically generated.
> Move EditLogOp serialisation and deserialation into one place
> -------------------------------------------------------------
>
> Key: HDFS-2149
> URL: https://issues.apache.org/jira/browse/HDFS-2149
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.23.0
>
> Attachments: HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff, HDFS-2149.diff
>
>
> On trunk serialisation of editlog ops is in FSEditLog#log* and deserialisation is in FSEditLogOp.*Op . This improvement is to move the serialisation code into one place, i.e under FSEditLogOp.*Op.
> This is part of HDFS-1580.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira