You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2010/12/03 08:14:11 UTC
[jira] Created: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Dfs client name for a map/reduce task should have some randomness
-----------------------------------------------------------------
Key: HDFS-1526
URL: https://issues.apache.org/jira/browse/HDFS-1526
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Reporter: Hairong Kuang
Assignee: Hairong Kuang
Fix For: 0.23.0
Fsck shows one of the files in our dfs cluster is corrupt.
# /bin/hadoop fsck aFile -files -blocks -locations
aFile: 4633 bytes, 2 block(s):
aFile: CORRUPT block blk_-4597378336099313975
OK
0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
[T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated HDFS-1526:
--------------------------------
Status: Patch Available (was: Open)
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch, randClientId1.patch, randClientId2.patch, randClientId3.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (HDFS-1526) Dfs client name for a map/reduce
task should have some randomness
Posted by "Harsh J (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171865#comment-13171865 ]
Harsh J commented on HDFS-1526:
-------------------------------
Noticed this today while poking around with 0.23.0:
If its not mapreduce, labelling it as 'NONMAPREDUCE' only makes it harder to grep, cause there's still some 'MAPREDUCE' in it? Its a nitpick (cause IDs don't carry that string), but perhaps you may consider switching to something more 'REGULAR'?
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch, randClientId1.patch, randClientId2.patch, randClientId3.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12970587#action_12970587 ]
dhruba borthakur commented on HDFS-1526:
----------------------------------------
+1. I do not think we need any unit tests for this one.
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch, randClientId1.patch, randClientId2.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-1526:
--------------------------------
Description:
Fsck shows one of the files in our dfs cluster is corrupt.
/bin/hadoop fsck aFile -files -blocks -locations
aFile: 4633 bytes, 2 block(s):
aFile: CORRUPT block blk_-4597378336099313975
OK
0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
[T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
was:
Fsck shows one of the files in our dfs cluster is corrupt.
# /bin/hadoop fsck aFile -files -blocks -locations
aFile: 4633 bytes, 2 block(s):
aFile: CORRUPT block blk_-4597378336099313975
OK
0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
[T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch, randClientId1.patch, randClientId2.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12971083#action_12971083 ]
Mahadev konar commented on HDFS-1526:
-------------------------------------
+1 the patch looks good.
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch, randClientId1.patch, randClientId2.patch, randClientId3.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-1526:
--------------------------------
Attachment: randClientId1.patch
This patch makes random number generator to be static and concatenates a random number and the thread id to the client name.
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch, randClientId1.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> # /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12971113#action_12971113 ]
Hairong Kuang commented on HDFS-1526:
-------------------------------------
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appe
[exec] ar to include any new or modified tests.
[exec] Please justify why no new tests are needed for this patch.
[exec] Also please list what manual steps were performed to verify this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
[exec] +1 system test framework. The patch passed system test framework compile
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch, randClientId1.patch, randClientId2.patch, randClientId3.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "Allen Wittenauer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966668#action_12966668 ]
Allen Wittenauer commented on HDFS-1526:
----------------------------------------
-1 on random without some sort of check for uniqueness
+1 on thread id if it is guaranteed to be unique
... and this sounds like it should be a blocker for 0.22 and 0.21.1.
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> # /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966589#action_12966589 ]
Hairong Kuang commented on HDFS-1526:
-------------------------------------
Koji, your comment makes sense if all threads happen to have the same seed. Another option for this problem is to concatenate the thread id, which might be a safer.
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> # /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12970936#action_12970936 ]
Mahadev konar commented on HDFS-1526:
-------------------------------------
was just thinking abt this, would it be better for log parsing if both the DFSClient names looked the same? Meaning with and without mapreduce?
something like:
DFSClient_applicationid_randomint_threadid (where applicationid = mapred.task.id or else = "null" or some other constant).
I dont know much abt the dfsclient code so dont know if this would be useful or not.
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch, randClientId1.patch, randClientId2.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12970362#action_12970362 ]
dhruba borthakur commented on HDFS-1526:
----------------------------------------
I think you attached the wrong patch to this JIRA
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch, randClientId1.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> # /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-1526:
--------------------------------
Attachment: randClientId3.patch
Here is the patch that uses "NONMAPREDUCE" as app name for dfs clients that are not mapreduce tasks. (MAHADEV sounds good to me too. :)
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch, randClientId1.patch, randClientId2.patch, randClientId3.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HDFS-1526:
-----------------------------------
Status: Patch Available (was: Open)
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch, randClientId1.patch, randClientId2.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12971187#action_12971187 ]
Hairong Kuang commented on HDFS-1526:
-------------------------------------
All unit tests passed except for known failed ones.
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch, randClientId1.patch, randClientId2.patch, randClientId3.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12971062#action_12971062 ]
Hairong Kuang commented on HDFS-1526:
-------------------------------------
Which constant string we should choose? null seems not user friendly. How about an empty string or "NONMAPREDUCE"?
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch, randClientId1.patch, randClientId2.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12971066#action_12971066 ]
Mahadev konar commented on HDFS-1526:
-------------------------------------
agreed. NONMAPREDUCE seems fine (or you can use MAHADEV, just kidding!! :) )
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch, randClientId1.patch, randClientId2.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated HDFS-1526:
--------------------------------
Status: Open (was: Patch Available)
resubmitting for hudson.
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch, randClientId1.patch, randClientId2.patch, randClientId3.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966579#action_12966579 ]
Koji Noguchi commented on HDFS-1526:
------------------------------------
Hairong, using a random number (with time as a seed) for getting a unique filename reminds me of an old old data corruption bug we looked at. When dfsclient was writing to temporary file before Dhruba's HADOOP-1707, we had cases of multiple tasks using the same seed/time resulting in data corruption. Wouldn't your patch have the same problem?
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> # /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12970322#action_12970322 ]
dhruba borthakur commented on HDFS-1526:
----------------------------------------
To address Allen's comments, we should make the Random object be a static object so that the generated random numbers do not repeat within the same DistributedFileSystem object.
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> # /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-1526:
--------------------------------
Attachment: clientName.patch
Here comes the patch.
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> # /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12970970#action_12970970 ]
dhruba borthakur commented on HDFS-1526:
----------------------------------------
I agree with Mahadev, we could follow his suggestion.
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch, randClientId1.patch, randClientId2.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-1526:
--------------------------------
Resolution: Fixed
Release Note: Make a client name has this format: DFSClient_applicationid_randomint_threadid, where applicationid = mapred.task.id or else = "NONMAPREDUCE".
Hadoop Flags: [Incompatible change, Reviewed]
Status: Resolved (was: Patch Available)
I just committed this!
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch, randClientId1.patch, randClientId2.patch, randClientId3.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (HDFS-1526) Dfs client name for a map/reduce
task should have some randomness
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022464#comment-13022464 ]
Hudson commented on HDFS-1526:
------------------------------
Integrated in Hadoop-Hdfs-trunk #643 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/])
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch, randClientId1.patch, randClientId2.patch, randClientId3.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1526) Dfs client name for a map/reduce task
should have some randomness
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-1526:
--------------------------------
Attachment: randClientId2.patch
Oops, here is the right one.
> Dfs client name for a map/reduce task should have some randomness
> -----------------------------------------------------------------
>
> Key: HDFS-1526
> URL: https://issues.apache.org/jira/browse/HDFS-1526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.23.0
>
> Attachments: clientName.patch, randClientId1.patch, randClientId2.patch
>
>
> Fsck shows one of the files in our dfs cluster is corrupt.
> # /bin/hadoop fsck aFile -files -blocks -locations
> aFile: 4633 bytes, 2 block(s):
> aFile: CORRUPT block blk_-4597378336099313975
> OK
> 0. blk_-4597378336099313975_2284630101 len=0 repl=3 [...]
> 1. blk_5024052590403223424_2284630107 len=4633 repl=3 [...]Status: CORRUPT
> On disk, these two blocks are of the same size and the same content. It turns out the writer of the file is from a multiple threaded map task. Each thread may write to the same file. One possible interaction of two threads might make this to happen:
> [T1: create aFile] [T2: delete aFile] [T2: create aFile][T1: addBlock 0 to aFile][T2: addBlock1 to aFile]...
> Because T1 and T2 have the same client name, which is the map task id, the above interactions could be done without any lease exception, thus eventually leading to a corrupt file. To solve the problem, a mapreduce task's client name could be formed by its task id followed by a random number.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.