You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Li Junjun (JIRA)" <ji...@apache.org> on 2012/05/21 05:28:41 UTC
[jira] [Created] (HDFS-3448) CLONE - Why open method in class
DFSClient would compare old LocatedBlocks and new LocatedBlocks?
Li Junjun created HDFS-3448:
-------------------------------
Summary: CLONE - Why open method in class DFSClient would compare old LocatedBlocks and new LocatedBlocks?
Key: HDFS-3448
URL: https://issues.apache.org/jira/browse/HDFS-3448
Project: Hadoop HDFS
Issue Type: Wish
Reporter: Li Junjun
Assignee: Todd Lipcon
This is in the package of org.apache.hadoop.hdfs, DFSClient.openInfo():
if (locatedBlocks != null) {
Iterator<LocatedBlock> oldIter = locatedBlocks.getLocatedBlocks().iterator();
Iterator<LocatedBlock> newIter = newInfo.getLocatedBlocks().iterator();
while (oldIter.hasNext() && newIter.hasNext()) {
if (! oldIter.next().getBlock().equals(newIter.next().getBlock())) {
throw new IOException("Blocklist for " + src + " has changed!");
}
}
}
Why we need compare old LocatedBlocks and new LocatedBlocks, and in what case it happen?
Why not "this.locatedBlocks = newInfo" directly?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3448) CLONE - Why open method in class
DFSClient would compare old LocatedBlocks and new LocatedBlocks?
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279950#comment-13279950 ]
Todd Lipcon commented on HDFS-3448:
-----------------------------------
Hi Li. {{Block.equals()}} already does what you're suggesting - it compares based on block ID alone - if the genstamp changes, the block is still considered equal.
> CLONE - Why open method in class DFSClient would compare old LocatedBlocks and new LocatedBlocks?
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-3448
> URL: https://issues.apache.org/jira/browse/HDFS-3448
> Project: Hadoop HDFS
> Issue Type: Wish
> Reporter: Li Junjun
> Assignee: Todd Lipcon
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I think there are two situations.
> 1,the file has been swapped with another file,we check the blockId,that's correct we throw Exceptions!
> 2,but if the file has not been swapped but has been appended ,we should just check the blockId ,and should not care about the block's stamp , because in fact we got the right and updated block list , cause file in hdfs can't be truncate .
> so how about we do it like this ?
> if ( oldIter.next().getBlock().getBlockId() != newIter.next().getBlock().getBlockId() ) { throw new IOException("Blocklist for " + src + " has changed!"); }
> after all , between two calls to openInfo() the file can be swapped and then appending,so we should not ignore the under construction file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3448) CLONE - Why open method in class
DFSClient would compare old LocatedBlocks and new LocatedBlocks?
Posted by "Li Junjun (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Junjun updated HDFS-3448:
----------------------------
Component/s: hdfs client
Affects Version/s: 1.0.1
> CLONE - Why open method in class DFSClient would compare old LocatedBlocks and new LocatedBlocks?
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-3448
> URL: https://issues.apache.org/jira/browse/HDFS-3448
> Project: Hadoop HDFS
> Issue Type: Wish
> Components: hdfs client
> Affects Versions: 1.0.1
> Reporter: Li Junjun
> Assignee: Todd Lipcon
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I think there are two situations.
> 1,the file has been swapped with another file,we check the blockId,that's correct we throw Exceptions!
> 2,but if the file has not been swapped but has been appended ,we should just check the blockId ,and should not care about the block's stamp , because in fact we got the right and updated block list , cause file in hdfs can't be truncate .
> so how about we do it like this ?
> if ( oldIter.next().getBlock().getBlockId() != newIter.next().getBlock().getBlockId() ) { throw new IOException("Blocklist for " + src + " has changed!"); }
> after all , between two calls to openInfo() the file can be swapped and then appending,so we should not ignore the under construction file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3448) CLONE - Why open method in class
DFSClient would compare old LocatedBlocks and new LocatedBlocks?
Posted by "Li Junjun (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Junjun updated HDFS-3448:
----------------------------
Description:
I think there are two situations.
1,the file has been swapped with another file,we check the blockId,that's correct we throw Exceptions!
2,but if the file has not been swapped but has been appended ,we should just check the blockId ,and should not care about the block's stamp , because in fact we got the right and updated block list , cause file in hdfs can't be truncate .
so how about we do it like this ?
if ( oldIter.next().getBlock().getBlockId() != newIter.next().getBlock().getBlockId() ) { throw new IOException("Blocklist for " + src + " has changed!"); }
after all , between two calls to openInfo() the file can be swapped and then appending,so we should not ignore the under construction file.
was:
This is in the package of org.apache.hadoop.hdfs, DFSClient.openInfo():
if (locatedBlocks != null) {
Iterator<LocatedBlock> oldIter = locatedBlocks.getLocatedBlocks().iterator();
Iterator<LocatedBlock> newIter = newInfo.getLocatedBlocks().iterator();
while (oldIter.hasNext() && newIter.hasNext()) {
if (! oldIter.next().getBlock().equals(newIter.next().getBlock())) {
throw new IOException("Blocklist for " + src + " has changed!");
}
}
}
Why we need compare old LocatedBlocks and new LocatedBlocks, and in what case it happen?
Why not "this.locatedBlocks = newInfo" directly?
> CLONE - Why open method in class DFSClient would compare old LocatedBlocks and new LocatedBlocks?
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-3448
> URL: https://issues.apache.org/jira/browse/HDFS-3448
> Project: Hadoop HDFS
> Issue Type: Wish
> Reporter: Li Junjun
> Assignee: Todd Lipcon
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I think there are two situations.
> 1,the file has been swapped with another file,we check the blockId,that's correct we throw Exceptions!
> 2,but if the file has not been swapped but has been appended ,we should just check the blockId ,and should not care about the block's stamp , because in fact we got the right and updated block list , cause file in hdfs can't be truncate .
> so how about we do it like this ?
> if ( oldIter.next().getBlock().getBlockId() != newIter.next().getBlock().getBlockId() ) { throw new IOException("Blocklist for " + src + " has changed!"); }
> after all , between two calls to openInfo() the file can be swapped and then appending,so we should not ignore the under construction file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3448) CLONE - Why open method in class
DFSClient would compare old LocatedBlocks and new LocatedBlocks?
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280267#comment-13280267 ]
Todd Lipcon commented on HDFS-3448:
-----------------------------------
Good catch. Want to submit a patch file against branch-1? A unit test would also be appreciated. Since append() is not supported in branch-1, I think a good unit test would be an addition to TestFileConcurrentReader which causes the writer trigger pipeline recovery after opening the file. Maybe something like:
- writer opens file for write
- writer writes some bytes and hflushes
- reader opens file but does not read any bytes (this will cause getBlockLocations() to be fetched but not actually talk to DNs)
- shut down one of three DNs
- writer writes some more data and hflushes again: this should trigger the genstamp to increase
- reader tries to read.
I think this should trigger the bug you found. Thanks!
> CLONE - Why open method in class DFSClient would compare old LocatedBlocks and new LocatedBlocks?
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-3448
> URL: https://issues.apache.org/jira/browse/HDFS-3448
> Project: Hadoop HDFS
> Issue Type: Wish
> Components: hdfs client
> Affects Versions: 1.0.1
> Reporter: Li Junjun
> Assignee: Todd Lipcon
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I think there are two situations.
> 1,the file has been swapped with another file,we check the blockId,that's correct we throw Exceptions!
> 2,but if the file has not been swapped but has been appended ,we should just check the blockId ,and should not care about the block's stamp , because in fact we got the right and updated block list , cause file in hdfs can't be truncate .
> so how about we do it like this ?
> if ( oldIter.next().getBlock().getBlockId() != newIter.next().getBlock().getBlockId() ) { throw new IOException("Blocklist for " + src + " has changed!"); }
> after all , between two calls to openInfo() the file can be swapped and then appending,so we should not ignore the under construction file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3448) CLONE - Why open method in class
DFSClient would compare old LocatedBlocks and new LocatedBlocks?
Posted by "Li Junjun (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279985#comment-13279985 ]
Li Junjun commented on HDFS-3448:
---------------------------------
I'm looking at hadoop 1.0.1
the file is org/apache/hadoop/hdfs/protocol/Block.java
public boolean equals(Object o) {
if (!(o instanceof Block)) {
return false;
}
final Block that = (Block)o;
//Wildcard generationStamp is ALLOWED here
return this.blockId == that.blockId
&& GenerationStamp.equalsWithWildcard(
this.generationStamp, that.generationStamp);
}
------------
after check the verison 0.23.1, I see the Block.equals() is like what you say.
I'm sorry,i am not familiar with jira.
> CLONE - Why open method in class DFSClient would compare old LocatedBlocks and new LocatedBlocks?
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-3448
> URL: https://issues.apache.org/jira/browse/HDFS-3448
> Project: Hadoop HDFS
> Issue Type: Wish
> Reporter: Li Junjun
> Assignee: Todd Lipcon
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I think there are two situations.
> 1,the file has been swapped with another file,we check the blockId,that's correct we throw Exceptions!
> 2,but if the file has not been swapped but has been appended ,we should just check the blockId ,and should not care about the block's stamp , because in fact we got the right and updated block list , cause file in hdfs can't be truncate .
> so how about we do it like this ?
> if ( oldIter.next().getBlock().getBlockId() != newIter.next().getBlock().getBlockId() ) { throw new IOException("Blocklist for " + src + " has changed!"); }
> after all , between two calls to openInfo() the file can be swapped and then appending,so we should not ignore the under construction file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira