You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2007/09/06 18:56:32 UTC
[jira] Created: (HADOOP-1845) Datanodes get error message "is
valid, and cannot be written to"
Datanodes get error message "is valid, and cannot be written to"
-----------------------------------------------------------------
Key: HADOOP-1845
URL: https://issues.apache.org/jira/browse/HADOOP-1845
Project: Hadoop
Issue Type: Bug
Components: dfs
Affects Versions: 0.14.1
Reporter: Hairong Kuang
Fix For: 0.15.0
>> Copy from dev list:
Our cluster has 4 nodes and i set the mapred.subimt.replication parameter to 2 on all nodes and the master. Everything has been restarted.
Unfortuantely, we still have the same exception :
2007-09-05 17:01:59,623 ERROR org.apache.hadoop.dfs.DataNode:
DataXceiver: java.io.IOException: Block blk_-5969983648201186681 is valid, and cannot be written to.
at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:515)
at
org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:822)
at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:727)
at java.lang.Thread.run(Thread.java:595)
>> end of copy
The message shows that the namenode schedules to replicate a block to a datanode that already holds the block. The namenode block placement algorithm makes sure that it does not schedule a block to a datanode that is confirmed to hold a replica of the block. But it is not aware of any in-transit block placements (i.e. the scheduled but not confirmed block placements), so occasionally we may still see "is valid, and cannot be written to" errors.
A fix to the problem is to keep track of all in-transit block placements, and the block placement algorithm considers these to-be-confirmed replicas as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1845) Datanodes get error message "is
valid, and cannot be written to"
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525548 ]
Raghu Angadi commented on HADOOP-1845:
--------------------------------------
This is from Datanode log. As far as datanode is concerned, it is an error. Agreed, it is confusing to the user. We should fix this issue.
> Datanodes get error message "is valid, and cannot be written to"
> -----------------------------------------------------------------
>
> Key: HADOOP-1845
> URL: https://issues.apache.org/jira/browse/HADOOP-1845
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Fix For: 0.15.0
>
>
> >> Copy from dev list:
> Our cluster has 4 nodes and i set the mapred.subimt.replication parameter to 2 on all nodes and the master. Everything has been restarted.
> Unfortuantely, we still have the same exception :
> 2007-09-05 17:01:59,623 ERROR org.apache.hadoop.dfs.DataNode:
> DataXceiver: java.io.IOException: Block blk_-5969983648201186681 is valid, and cannot be written to.
> at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:515)
> at
> org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:822)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:727)
> at java.lang.Thread.run(Thread.java:595)
> >> end of copy
> The message shows that the namenode schedules to replicate a block to a datanode that already holds the block. The namenode block placement algorithm makes sure that it does not schedule a block to a datanode that is confirmed to hold a replica of the block. But it is not aware of any in-transit block placements (i.e. the scheduled but not confirmed block placements), so occasionally we may still see "is valid, and cannot be written to" errors.
> A fix to the problem is to keep track of all in-transit block placements, and the block placement algorithm considers these to-be-confirmed replicas as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1845) Datanodes get error message "is
valid, and cannot be written to"
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12526229 ]
Konstantin Shvachko commented on HADOOP-1845:
---------------------------------------------
This looks like a recent addition. Could you guys post with which versions of hadoop you can/started to see that behavior
so that we could track what could have caused it.
> Datanodes get error message "is valid, and cannot be written to"
> -----------------------------------------------------------------
>
> Key: HADOOP-1845
> URL: https://issues.apache.org/jira/browse/HADOOP-1845
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Fix For: 0.15.0
>
>
> >> Copy from dev list:
> Our cluster has 4 nodes and i set the mapred.subimt.replication parameter to 2 on all nodes and the master. Everything has been restarted.
> Unfortuantely, we still have the same exception :
> 2007-09-05 17:01:59,623 ERROR org.apache.hadoop.dfs.DataNode:
> DataXceiver: java.io.IOException: Block blk_-5969983648201186681 is valid, and cannot be written to.
> at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:515)
> at
> org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:822)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:727)
> at java.lang.Thread.run(Thread.java:595)
> >> end of copy
> The message shows that the namenode schedules to replicate a block to a datanode that already holds the block. The namenode block placement algorithm makes sure that it does not schedule a block to a datanode that is confirmed to hold a replica of the block. But it is not aware of any in-transit block placements (i.e. the scheduled but not confirmed block placements), so occasionally we may still see "is valid, and cannot be written to" errors.
> A fix to the problem is to keep track of all in-transit block placements, and the block placement algorithm considers these to-be-confirmed replicas as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1845) Datanodes get error message "is
valid, and cannot be written to"
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525766 ]
Raghu Angadi commented on HADOOP-1845:
--------------------------------------
> We can try to gather a (new) log but if I remember even the unit test had the exception.
The unit test failure was for a different reason. If there are many of this kind, we can fix that too. The proposed fix in this jira should fix the most commonly observed cause of these exceptions.
> Datanodes get error message "is valid, and cannot be written to"
> -----------------------------------------------------------------
>
> Key: HADOOP-1845
> URL: https://issues.apache.org/jira/browse/HADOOP-1845
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Fix For: 0.15.0
>
>
> >> Copy from dev list:
> Our cluster has 4 nodes and i set the mapred.subimt.replication parameter to 2 on all nodes and the master. Everything has been restarted.
> Unfortuantely, we still have the same exception :
> 2007-09-05 17:01:59,623 ERROR org.apache.hadoop.dfs.DataNode:
> DataXceiver: java.io.IOException: Block blk_-5969983648201186681 is valid, and cannot be written to.
> at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:515)
> at
> org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:822)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:727)
> at java.lang.Thread.run(Thread.java:595)
> >> end of copy
> The message shows that the namenode schedules to replicate a block to a datanode that already holds the block. The namenode block placement algorithm makes sure that it does not schedule a block to a datanode that is confirmed to hold a replica of the block. But it is not aware of any in-transit block placements (i.e. the scheduled but not confirmed block placements), so occasionally we may still see "is valid, and cannot be written to" errors.
> A fix to the problem is to keep track of all in-transit block placements, and the block placement algorithm considers these to-be-confirmed replicas as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1845) Datanodes get error message "is
valid, and cannot be written to"
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525544 ]
Doug Cutting commented on HADOOP-1845:
--------------------------------------
Does the current behavior actually cause any problems? Or is the message just disconcerting?
> Datanodes get error message "is valid, and cannot be written to"
> -----------------------------------------------------------------
>
> Key: HADOOP-1845
> URL: https://issues.apache.org/jira/browse/HADOOP-1845
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Fix For: 0.15.0
>
>
> >> Copy from dev list:
> Our cluster has 4 nodes and i set the mapred.subimt.replication parameter to 2 on all nodes and the master. Everything has been restarted.
> Unfortuantely, we still have the same exception :
> 2007-09-05 17:01:59,623 ERROR org.apache.hadoop.dfs.DataNode:
> DataXceiver: java.io.IOException: Block blk_-5969983648201186681 is valid, and cannot be written to.
> at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:515)
> at
> org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:822)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:727)
> at java.lang.Thread.run(Thread.java:595)
> >> end of copy
> The message shows that the namenode schedules to replicate a block to a datanode that already holds the block. The namenode block placement algorithm makes sure that it does not schedule a block to a datanode that is confirmed to hold a replica of the block. But it is not aware of any in-transit block placements (i.e. the scheduled but not confirmed block placements), so occasionally we may still see "is valid, and cannot be written to" errors.
> A fix to the problem is to keep track of all in-transit block placements, and the block placement algorithm considers these to-be-confirmed replicas as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1845) Datanodes get error message "is
valid, and cannot be written to"
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525546 ]
Raghu Angadi commented on HADOOP-1845:
--------------------------------------
Assuming default replication is 3, setting mapred.submit.replication to 2 should not result in this exception. Relevant log from namenode from original poster would be helpful.
The problem that Hairong described hasn't cause real failures yet as far as I know. It only slows down increasing replication.. not sure by how much.
> Datanodes get error message "is valid, and cannot be written to"
> -----------------------------------------------------------------
>
> Key: HADOOP-1845
> URL: https://issues.apache.org/jira/browse/HADOOP-1845
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Fix For: 0.15.0
>
>
> >> Copy from dev list:
> Our cluster has 4 nodes and i set the mapred.subimt.replication parameter to 2 on all nodes and the master. Everything has been restarted.
> Unfortuantely, we still have the same exception :
> 2007-09-05 17:01:59,623 ERROR org.apache.hadoop.dfs.DataNode:
> DataXceiver: java.io.IOException: Block blk_-5969983648201186681 is valid, and cannot be written to.
> at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:515)
> at
> org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:822)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:727)
> at java.lang.Thread.run(Thread.java:595)
> >> end of copy
> The message shows that the namenode schedules to replicate a block to a datanode that already holds the block. The namenode block placement algorithm makes sure that it does not schedule a block to a datanode that is confirmed to hold a replica of the block. But it is not aware of any in-transit block placements (i.e. the scheduled but not confirmed block placements), so occasionally we may still see "is valid, and cannot be written to" errors.
> A fix to the problem is to keep track of all in-transit block placements, and the block placement algorithm considers these to-be-confirmed replicas as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1845) Datanodes get error message "is
valid, and cannot be written to"
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528287 ]
dhruba borthakur commented on HADOOP-1845:
------------------------------------------
Hi Torsten, how many datanodes do you have in your cluster? The reason I asm asking this question is because the probability of these exception occuring increases if the cluster size is very small (or if only a very small number of nodes have avilable disk space)
> Datanodes get error message "is valid, and cannot be written to"
> -----------------------------------------------------------------
>
> Key: HADOOP-1845
> URL: https://issues.apache.org/jira/browse/HADOOP-1845
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Fix For: 0.15.0
>
>
> >> Copy from dev list:
> Our cluster has 4 nodes and i set the mapred.subimt.replication parameter to 2 on all nodes and the master. Everything has been restarted.
> Unfortuantely, we still have the same exception :
> 2007-09-05 17:01:59,623 ERROR org.apache.hadoop.dfs.DataNode:
> DataXceiver: java.io.IOException: Block blk_-5969983648201186681 is valid, and cannot be written to.
> at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:515)
> at
> org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:822)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:727)
> at java.lang.Thread.run(Thread.java:595)
> >> end of copy
> The message shows that the namenode schedules to replicate a block to a datanode that already holds the block. The namenode block placement algorithm makes sure that it does not schedule a block to a datanode that is confirmed to hold a replica of the block. But it is not aware of any in-transit block placements (i.e. the scheduled but not confirmed block placements), so occasionally we may still see "is valid, and cannot be written to" errors.
> A fix to the problem is to keep track of all in-transit block placements, and the block placement algorithm considers these to-be-confirmed replicas as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1845) Datanodes get error message "is
valid, and cannot be written to"
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530345 ]
Hairong Kuang commented on HADOOP-1845:
---------------------------------------
I feel that fundamentally namenode does not handle blockReceived well. HADOOP-1946 simply "amplied" the problem. BlockReceived should not trigger re-replication. We should put block placement decisions at file creation time in the PendingReplicationBlocks queue as well.
> Datanodes get error message "is valid, and cannot be written to"
> -----------------------------------------------------------------
>
> Key: HADOOP-1845
> URL: https://issues.apache.org/jira/browse/HADOOP-1845
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Fix For: 0.15.0
>
>
> >> Copy from dev list:
> Our cluster has 4 nodes and i set the mapred.subimt.replication parameter to 2 on all nodes and the master. Everything has been restarted.
> Unfortuantely, we still have the same exception :
> 2007-09-05 17:01:59,623 ERROR org.apache.hadoop.dfs.DataNode:
> DataXceiver: java.io.IOException: Block blk_-5969983648201186681 is valid, and cannot be written to.
> at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:515)
> at
> org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:822)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:727)
> at java.lang.Thread.run(Thread.java:595)
> >> end of copy
> The message shows that the namenode schedules to replicate a block to a datanode that already holds the block. The namenode block placement algorithm makes sure that it does not schedule a block to a datanode that is confirmed to hold a replica of the block. But it is not aware of any in-transit block placements (i.e. the scheduled but not confirmed block placements), so occasionally we may still see "is valid, and cannot be written to" errors.
> A fix to the problem is to keep track of all in-transit block placements, and the block placement algorithm considers these to-be-confirmed replicas as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1845) Datanodes get error message "is
valid, and cannot be written to"
Posted by "Torsten Curdt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528282 ]
Torsten Curdt commented on HADOOP-1845:
---------------------------------------
We have been seen this in 0.10 and are still seeing it in 0.14
> Datanodes get error message "is valid, and cannot be written to"
> -----------------------------------------------------------------
>
> Key: HADOOP-1845
> URL: https://issues.apache.org/jira/browse/HADOOP-1845
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Fix For: 0.15.0
>
>
> >> Copy from dev list:
> Our cluster has 4 nodes and i set the mapred.subimt.replication parameter to 2 on all nodes and the master. Everything has been restarted.
> Unfortuantely, we still have the same exception :
> 2007-09-05 17:01:59,623 ERROR org.apache.hadoop.dfs.DataNode:
> DataXceiver: java.io.IOException: Block blk_-5969983648201186681 is valid, and cannot be written to.
> at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:515)
> at
> org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:822)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:727)
> at java.lang.Thread.run(Thread.java:595)
> >> end of copy
> The message shows that the namenode schedules to replicate a block to a datanode that already holds the block. The namenode block placement algorithm makes sure that it does not schedule a block to a datanode that is confirmed to hold a replica of the block. But it is not aware of any in-transit block placements (i.e. the scheduled but not confirmed block placements), so occasionally we may still see "is valid, and cannot be written to" errors.
> A fix to the problem is to keep track of all in-transit block placements, and the block placement algorithm considers these to-be-confirmed replicas as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1845) Datanodes get error message "is
valid, and cannot be written to"
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530323 ]
Konstantin Shvachko commented on HADOOP-1845:
---------------------------------------------
HADOOP-1946 is the main reason for that behavior.
The data-nodes were very slow so that blockReceived() were in fact coming late after the file had been closed.
Do we still want to remove the exception mentioned in the initial description?
> Datanodes get error message "is valid, and cannot be written to"
> -----------------------------------------------------------------
>
> Key: HADOOP-1845
> URL: https://issues.apache.org/jira/browse/HADOOP-1845
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Fix For: 0.15.0
>
>
> >> Copy from dev list:
> Our cluster has 4 nodes and i set the mapred.subimt.replication parameter to 2 on all nodes and the master. Everything has been restarted.
> Unfortuantely, we still have the same exception :
> 2007-09-05 17:01:59,623 ERROR org.apache.hadoop.dfs.DataNode:
> DataXceiver: java.io.IOException: Block blk_-5969983648201186681 is valid, and cannot be written to.
> at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:515)
> at
> org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:822)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:727)
> at java.lang.Thread.run(Thread.java:595)
> >> end of copy
> The message shows that the namenode schedules to replicate a block to a datanode that already holds the block. The namenode block placement algorithm makes sure that it does not schedule a block to a datanode that is confirmed to hold a replica of the block. But it is not aware of any in-transit block placements (i.e. the scheduled but not confirmed block placements), so occasionally we may still see "is valid, and cannot be written to" errors.
> A fix to the problem is to keep track of all in-transit block placements, and the block placement algorithm considers these to-be-confirmed replicas as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1845) Datanodes get error message "is
valid, and cannot be written to"
Posted by "Torsten Curdt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525633 ]
Torsten Curdt commented on HADOOP-1845:
---------------------------------------
We can try to gather a (new) log but if I remember even the unit test had the exception. (see the thread on dev) With my still limited knowledge of the codebase and comparing to what is going on the cluster it seems it does not really cause huge problems. So it was mainly disconcerting. But - at some stage we have had so many of these exception in the logs that they were essentially spamming the logs. (More than hundred exceptions per minute!) ...which made it a bit more than just disconcerting. Maybe the word "panic" was more suited ;) At least OPS stopped believing this is not a problem and doesn't add to a good reputation of hadoop. It also make you lose the eye for real problems as it just drowns them in a see of information.
So yeah ...we should really fix this.
> Datanodes get error message "is valid, and cannot be written to"
> -----------------------------------------------------------------
>
> Key: HADOOP-1845
> URL: https://issues.apache.org/jira/browse/HADOOP-1845
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Fix For: 0.15.0
>
>
> >> Copy from dev list:
> Our cluster has 4 nodes and i set the mapred.subimt.replication parameter to 2 on all nodes and the master. Everything has been restarted.
> Unfortuantely, we still have the same exception :
> 2007-09-05 17:01:59,623 ERROR org.apache.hadoop.dfs.DataNode:
> DataXceiver: java.io.IOException: Block blk_-5969983648201186681 is valid, and cannot be written to.
> at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:515)
> at
> org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:822)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:727)
> at java.lang.Thread.run(Thread.java:595)
> >> end of copy
> The message shows that the namenode schedules to replicate a block to a datanode that already holds the block. The namenode block placement algorithm makes sure that it does not schedule a block to a datanode that is confirmed to hold a replica of the block. But it is not aware of any in-transit block placements (i.e. the scheduled but not confirmed block placements), so occasionally we may still see "is valid, and cannot be written to" errors.
> A fix to the problem is to keep track of all in-transit block placements, and the block placement algorithm considers these to-be-confirmed replicas as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1845) Datanodes get error message "is
valid, and cannot be written to"
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525547 ]
Jim Kellerman commented on HADOOP-1845:
---------------------------------------
Until I saw the explanation above, it was disconcerting as it looked like there was something wrong with the DFS, especially since the log message is an ERROR. If it was an INFO, I would have assumed it was mostly harmless.
> Datanodes get error message "is valid, and cannot be written to"
> -----------------------------------------------------------------
>
> Key: HADOOP-1845
> URL: https://issues.apache.org/jira/browse/HADOOP-1845
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Fix For: 0.15.0
>
>
> >> Copy from dev list:
> Our cluster has 4 nodes and i set the mapred.subimt.replication parameter to 2 on all nodes and the master. Everything has been restarted.
> Unfortuantely, we still have the same exception :
> 2007-09-05 17:01:59,623 ERROR org.apache.hadoop.dfs.DataNode:
> DataXceiver: java.io.IOException: Block blk_-5969983648201186681 is valid, and cannot be written to.
> at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:515)
> at
> org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:822)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:727)
> at java.lang.Thread.run(Thread.java:595)
> >> end of copy
> The message shows that the namenode schedules to replicate a block to a datanode that already holds the block. The namenode block placement algorithm makes sure that it does not schedule a block to a datanode that is confirmed to hold a replica of the block. But it is not aware of any in-transit block placements (i.e. the scheduled but not confirmed block placements), so occasionally we may still see "is valid, and cannot be written to" errors.
> A fix to the problem is to keep track of all in-transit block placements, and the block placement algorithm considers these to-be-confirmed replicas as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1845) Datanodes get error message "is
valid, and cannot be written to"
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12528609 ]
Konstantin Shvachko commented on HADOOP-1845:
---------------------------------------------
I am running a 2-node cluster and see this exception all the time.
Here is the log from the name-node that explains the behavior:
{code}
07/09/18 12:08:05 INFO dfs.StateChange: BLOCK* NameSystem.allocateBlock: /Dir0/file20. blk_-4934058791921230875 is created and added to pendingCreates and pendingCreateBlocks
07/09/18 12:08:06 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: node.x.x.111:50077 is added to blk_-4934058791921230875
07/09/18 12:08:08 INFO dfs.StateChange: BLOCK* NameSystem.pendingTransfer: ask node.x.x.111:50077 to replicate blk_-4934058791921230875 to datanode(s) node.x.x.222:50017
07/09/18 12:08:09 INFO dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: node.x.x.222:50017 is added to blk_-4934058791921230875
{code}
What we see here is a race condition between blockReceived() and sendHeartbeat().
The client writes 2 block replicas to 2 data-nodes. When finished writing replicas to their disks the data-nodes send blockReceived() to the name-node.
The following sequence of events causes the exception:
# node1.blockReceived(): the block is added to the list of under-replicated blocks, since it is supposed to have replication 2;
# node1.sendHeartbeat(): node1 starts replicating block to node2.
In respond node2 throws "block is valid, and cannot be written to" exception.
# node2.blockReceived(): everything goes back to normal.
On a 2-node cluster there is always one choice where the other replica can be placed. That is why the exception is inevitable if replication is requested.
On a bigger clusters the exception is rather rare, because replication most probably will be scheduled to a data-node that does not contain the block.
Which makes it even worse, because the actual transfer happens, the block becomes over-replicated so that now one of the replicas needs to be removed.
I think this decreases the overall performance of the cluster - unnecessary transfers of large blocks can be costly.
> Datanodes get error message "is valid, and cannot be written to"
> -----------------------------------------------------------------
>
> Key: HADOOP-1845
> URL: https://issues.apache.org/jira/browse/HADOOP-1845
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Fix For: 0.15.0
>
>
> >> Copy from dev list:
> Our cluster has 4 nodes and i set the mapred.subimt.replication parameter to 2 on all nodes and the master. Everything has been restarted.
> Unfortuantely, we still have the same exception :
> 2007-09-05 17:01:59,623 ERROR org.apache.hadoop.dfs.DataNode:
> DataXceiver: java.io.IOException: Block blk_-5969983648201186681 is valid, and cannot be written to.
> at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:515)
> at
> org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:822)
> at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:727)
> at java.lang.Thread.run(Thread.java:595)
> >> end of copy
> The message shows that the namenode schedules to replicate a block to a datanode that already holds the block. The namenode block placement algorithm makes sure that it does not schedule a block to a datanode that is confirmed to hold a replica of the block. But it is not aware of any in-transit block placements (i.e. the scheduled but not confirmed block placements), so occasionally we may still see "is valid, and cannot be written to" errors.
> A fix to the problem is to keep track of all in-transit block placements, and the block placement algorithm considers these to-be-confirmed replicas as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.