You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Yoram Arnon (JIRA)" <ji...@apache.org> on 2006/04/19 00:07:17 UTC
[jira] Created: (HADOOP-146) potential conflict in block id's,
leading to data corruption
potential conflict in block id's, leading to data corruption
------------------------------------------------------------
Key: HADOOP-146
URL: http://issues.apache.org/jira/browse/HADOOP-146
Project: Hadoop
Type: Bug
Components: dfs
Versions: 0.1.0, 0.1.1
Reporter: Yoram Arnon
Assigned to: Konstantin Shvachko
Fix For: 0.3
currently, block id's are generated randomly, and are not tested for collisions with existing id's.
while ids are 64 bits, given enough time and a large enough FS, collisions are expected.
when a collision occurs, a random subset of blocks with that id will be removed as extra replicas, and the contents of that portion of the containing file are one random version of the block.
to solve this one could check for id collision when creating a new block, getting a new id in case of conflict. This approach requires the name node to keep track of all existing block id's (rather than just the ones who have reported in), and to identify old versions of a block id as in valid (in case a data node dies, a file is deleted, then a block id is reused for a new file).
Alternatively, one could simply use sequential block id's. Here the downsides are:
1. migration from an existing file system is hard, requiring compaction of the entire FS
2. once you cycle through 64 bits of id's (quite a few years at full blast), you're in trouble again (or run occasional/background compaction)
3. you must never lose the high watermark block id.
synchronized Block allocateBlock(UTF8 src) {
Block b = new Block();
FileUnderConstruction v = (FileUnderConstruction) pendingCreates.get(src);
v.add(b);
pendingCreateBlocks.add(b);
return b;
}
static Random r = new Random();
/**
*/
public Block() {
this.blkid = r.nextLong();
this.len = 0;
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-146) potential conflict in block id's,
leading to data corruption
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-146?page=comments#action_12383305 ]
Doug Cutting commented on HADOOP-146:
-------------------------------------
Oops. I meant, "Thanks Milind! " Sorry. I wasn't paying attention.
> potential conflict in block id's, leading to data corruption
> ------------------------------------------------------------
>
> Key: HADOOP-146
> URL: http://issues.apache.org/jira/browse/HADOOP-146
> Project: Hadoop
> Type: Bug
> Components: dfs
> Versions: 0.1.0, 0.1.1
> Reporter: Yoram Arnon
> Assignee: Konstantin Shvachko
> Fix For: 0.3
> Attachments: hadoop-146-random.patch
>
> currently, block id's are generated randomly, and are not tested for collisions with existing id's.
> while ids are 64 bits, given enough time and a large enough FS, collisions are expected.
> when a collision occurs, a random subset of blocks with that id will be removed as extra replicas, and the contents of that portion of the containing file are one random version of the block.
> to solve this one could check for id collision when creating a new block, getting a new id in case of conflict. This approach requires the name node to keep track of all existing block id's (rather than just the ones who have reported in), and to identify old versions of a block id as in valid (in case a data node dies, a file is deleted, then a block id is reused for a new file).
> Alternatively, one could simply use sequential block id's. Here the downsides are:
> 1. migration from an existing file system is hard, requiring compaction of the entire FS
> 2. once you cycle through 64 bits of id's (quite a few years at full blast), you're in trouble again (or run occasional/background compaction)
> 3. you must never lose the high watermark block id.
> synchronized Block allocateBlock(UTF8 src) {
> Block b = new Block();
> FileUnderConstruction v = (FileUnderConstruction) pendingCreates.get(src);
> v.add(b);
> pendingCreateBlocks.add(b);
> return b;
> }
> static Random r = new Random();
> /**
> */
> public Block() {
> this.blkid = r.nextLong();
> this.len = 0;
> }
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (HADOOP-146) potential conflict in block id's, leading to data corruption
Posted by Eric Baldeschwieler <er...@yahoo-inc.com>.
we might want to do something generic to support future migrations,
where you boot a new version of the FS next to the old one and then
just move files across. If we can manage to use hardlinks to migrate
data block files, it should go fast...
On Apr 18, 2006, at 4:39 PM, Doug Cutting (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/HADOOP-146?
> page=comments#action_12375011 ]
>
> Doug Cutting commented on HADOOP-146:
> -------------------------------------
>
> I'd vote for sequential allocation. It will take a *really* long
> time to cycle through all ids. Migration should not be expensive,
> since it just requires renaming block files, not copying them. The
> high-watermark block id can be logged with the block->name table.
>
> Here's one way to migrate: initially the high-water-mark id is
> zero. So all blocks in the name table are out-of-range, and hence
> need renaming. Renaming can be handled like other blockwork: the
> namenode can give datanodes rename commands. While a block is
> being renamed it must be kept in side tables, so that, e.g.,
> requests to read files whose blocks are partially renamed can still
> be handled.
>
>
>> potential conflict in block id's, leading to data corruption
>> ------------------------------------------------------------
>>
>> Key: HADOOP-146
>> URL: http://issues.apache.org/jira/browse/HADOOP-146
>> Project: Hadoop
>> Type: Bug
>
>> Components: dfs
>> Versions: 0.1.0, 0.1.1
>> Reporter: Yoram Arnon
>> Assignee: Konstantin Shvachko
>> Fix For: 0.3
>
>>
>> currently, block id's are generated randomly, and are not tested
>> for collisions with existing id's.
>> while ids are 64 bits, given enough time and a large enough FS,
>> collisions are expected.
>> when a collision occurs, a random subset of blocks with that id
>> will be removed as extra replicas, and the contents of that
>> portion of the containing file are one random version of the block.
>> to solve this one could check for id collision when creating a new
>> block, getting a new id in case of conflict. This approach
>> requires the name node to keep track of all existing block id's
>> (rather than just the ones who have reported in), and to identify
>> old versions of a block id as in valid (in case a data node dies,
>> a file is deleted, then a block id is reused for a new file).
>> Alternatively, one could simply use sequential block id's. Here
>> the downsides are:
>> 1. migration from an existing file system is hard, requiring
>> compaction of the entire FS
>> 2. once you cycle through 64 bits of id's (quite a few years at
>> full blast), you're in trouble again (or run occasional/background
>> compaction)
>> 3. you must never lose the high watermark block id.
>> synchronized Block allocateBlock(UTF8 src) {
>> Block b = new Block();
>> FileUnderConstruction v = (FileUnderConstruction)
>> pendingCreates.get(src);
>> v.add(b);
>> pendingCreateBlocks.add(b);
>> return b;
>> }
>> static Random r = new Random();
>> /**
>> */
>> public Block() {
>> this.blkid = r.nextLong();
>> this.len = 0;
>> }
>
> --
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the
> administrators:
> http://issues.apache.org/jira/secure/Administrators.jspa
> -
> For more information on JIRA, see:
> http://www.atlassian.com/software/jira
>
[jira] Commented: (HADOOP-146) potential conflict in block id's,
leading to data corruption
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-146?page=comments#action_12375011 ]
Doug Cutting commented on HADOOP-146:
-------------------------------------
I'd vote for sequential allocation. It will take a *really* long time to cycle through all ids. Migration should not be expensive, since it just requires renaming block files, not copying them. The high-watermark block id can be logged with the block->name table.
Here's one way to migrate: initially the high-water-mark id is zero. So all blocks in the name table are out-of-range, and hence need renaming. Renaming can be handled like other blockwork: the namenode can give datanodes rename commands. While a block is being renamed it must be kept in side tables, so that, e.g., requests to read files whose blocks are partially renamed can still be handled.
> potential conflict in block id's, leading to data corruption
> ------------------------------------------------------------
>
> Key: HADOOP-146
> URL: http://issues.apache.org/jira/browse/HADOOP-146
> Project: Hadoop
> Type: Bug
> Components: dfs
> Versions: 0.1.0, 0.1.1
> Reporter: Yoram Arnon
> Assignee: Konstantin Shvachko
> Fix For: 0.3
>
> currently, block id's are generated randomly, and are not tested for collisions with existing id's.
> while ids are 64 bits, given enough time and a large enough FS, collisions are expected.
> when a collision occurs, a random subset of blocks with that id will be removed as extra replicas, and the contents of that portion of the containing file are one random version of the block.
> to solve this one could check for id collision when creating a new block, getting a new id in case of conflict. This approach requires the name node to keep track of all existing block id's (rather than just the ones who have reported in), and to identify old versions of a block id as in valid (in case a data node dies, a file is deleted, then a block id is reused for a new file).
> Alternatively, one could simply use sequential block id's. Here the downsides are:
> 1. migration from an existing file system is hard, requiring compaction of the entire FS
> 2. once you cycle through 64 bits of id's (quite a few years at full blast), you're in trouble again (or run occasional/background compaction)
> 3. you must never lose the high watermark block id.
> synchronized Block allocateBlock(UTF8 src) {
> Block b = new Block();
> FileUnderConstruction v = (FileUnderConstruction) pendingCreates.get(src);
> v.add(b);
> pendingCreateBlocks.add(b);
> return b;
> }
> static Random r = new Random();
> /**
> */
> public Block() {
> this.blkid = r.nextLong();
> this.len = 0;
> }
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Resolved: (HADOOP-146) potential conflict in block id's,
leading to data corruption
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-146?page=all ]
Doug Cutting resolved HADOOP-146:
---------------------------------
Resolution: Fixed
I just committed this. Thanks, Konstantin!
> potential conflict in block id's, leading to data corruption
> ------------------------------------------------------------
>
> Key: HADOOP-146
> URL: http://issues.apache.org/jira/browse/HADOOP-146
> Project: Hadoop
> Type: Bug
> Components: dfs
> Versions: 0.1.0, 0.1.1
> Reporter: Yoram Arnon
> Assignee: Konstantin Shvachko
> Fix For: 0.3
> Attachments: hadoop-146-random.patch
>
> currently, block id's are generated randomly, and are not tested for collisions with existing id's.
> while ids are 64 bits, given enough time and a large enough FS, collisions are expected.
> when a collision occurs, a random subset of blocks with that id will be removed as extra replicas, and the contents of that portion of the containing file are one random version of the block.
> to solve this one could check for id collision when creating a new block, getting a new id in case of conflict. This approach requires the name node to keep track of all existing block id's (rather than just the ones who have reported in), and to identify old versions of a block id as in valid (in case a data node dies, a file is deleted, then a block id is reused for a new file).
> Alternatively, one could simply use sequential block id's. Here the downsides are:
> 1. migration from an existing file system is hard, requiring compaction of the entire FS
> 2. once you cycle through 64 bits of id's (quite a few years at full blast), you're in trouble again (or run occasional/background compaction)
> 3. you must never lose the high watermark block id.
> synchronized Block allocateBlock(UTF8 src) {
> Block b = new Block();
> FileUnderConstruction v = (FileUnderConstruction) pendingCreates.get(src);
> v.add(b);
> pendingCreateBlocks.add(b);
> return b;
> }
> static Random r = new Random();
> /**
> */
> public Block() {
> this.blkid = r.nextLong();
> this.len = 0;
> }
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-146) potential conflict in block id's,
leading to data corruption
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-146?page=all ]
Milind Bhandarkar updated HADOOP-146:
-------------------------------------
Attachment: hadoop-146-random.patch
I have attached a patch that fixes block-id conflicts for randomly generated block-ids. Now it checks after generating a block ID whether it already exists in the file system.
> potential conflict in block id's, leading to data corruption
> ------------------------------------------------------------
>
> Key: HADOOP-146
> URL: http://issues.apache.org/jira/browse/HADOOP-146
> Project: Hadoop
> Type: Bug
> Components: dfs
> Versions: 0.1.0, 0.1.1
> Reporter: Yoram Arnon
> Assignee: Konstantin Shvachko
> Fix For: 0.3
> Attachments: hadoop-146-random.patch
>
> currently, block id's are generated randomly, and are not tested for collisions with existing id's.
> while ids are 64 bits, given enough time and a large enough FS, collisions are expected.
> when a collision occurs, a random subset of blocks with that id will be removed as extra replicas, and the contents of that portion of the containing file are one random version of the block.
> to solve this one could check for id collision when creating a new block, getting a new id in case of conflict. This approach requires the name node to keep track of all existing block id's (rather than just the ones who have reported in), and to identify old versions of a block id as in valid (in case a data node dies, a file is deleted, then a block id is reused for a new file).
> Alternatively, one could simply use sequential block id's. Here the downsides are:
> 1. migration from an existing file system is hard, requiring compaction of the entire FS
> 2. once you cycle through 64 bits of id's (quite a few years at full blast), you're in trouble again (or run occasional/background compaction)
> 3. you must never lose the high watermark block id.
> synchronized Block allocateBlock(UTF8 src) {
> Block b = new Block();
> FileUnderConstruction v = (FileUnderConstruction) pendingCreates.get(src);
> v.add(b);
> pendingCreateBlocks.add(b);
> return b;
> }
> static Random r = new Random();
> /**
> */
> public Block() {
> this.blkid = r.nextLong();
> this.len = 0;
> }
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Closed: (HADOOP-146) potential conflict in block id's,
leading to data corruption
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-146?page=all ]
Doug Cutting closed HADOOP-146:
-------------------------------
> potential conflict in block id's, leading to data corruption
> ------------------------------------------------------------
>
> Key: HADOOP-146
> URL: http://issues.apache.org/jira/browse/HADOOP-146
> Project: Hadoop
> Type: Bug
> Components: dfs
> Versions: 0.1.0, 0.1.1
> Reporter: Yoram Arnon
> Assignee: Konstantin Shvachko
> Fix For: 0.3.0
> Attachments: hadoop-146-random.patch
>
> currently, block id's are generated randomly, and are not tested for collisions with existing id's.
> while ids are 64 bits, given enough time and a large enough FS, collisions are expected.
> when a collision occurs, a random subset of blocks with that id will be removed as extra replicas, and the contents of that portion of the containing file are one random version of the block.
> to solve this one could check for id collision when creating a new block, getting a new id in case of conflict. This approach requires the name node to keep track of all existing block id's (rather than just the ones who have reported in), and to identify old versions of a block id as in valid (in case a data node dies, a file is deleted, then a block id is reused for a new file).
> Alternatively, one could simply use sequential block id's. Here the downsides are:
> 1. migration from an existing file system is hard, requiring compaction of the entire FS
> 2. once you cycle through 64 bits of id's (quite a few years at full blast), you're in trouble again (or run occasional/background compaction)
> 3. you must never lose the high watermark block id.
> synchronized Block allocateBlock(UTF8 src) {
> Block b = new Block();
> FileUnderConstruction v = (FileUnderConstruction) pendingCreates.get(src);
> v.add(b);
> pendingCreateBlocks.add(b);
> return b;
> }
> static Random r = new Random();
> /**
> */
> public Block() {
> this.blkid = r.nextLong();
> this.len = 0;
> }
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira