You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Yoram Arnon (JIRA)" <ji...@apache.org> on 2006/03/31 02:08:36 UTC
[jira] Assigned: (HADOOP-96) name server should log decisions that
affect data: block creation, removal, replication
[ http://issues.apache.org/jira/browse/HADOOP-96?page=all ]
Yoram Arnon reassigned HADOOP-96:
---------------------------------
Assign To: Hairong Kuang
> name server should log decisions that affect data: block creation, removal, replication
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-96
> URL: http://issues.apache.org/jira/browse/HADOOP-96
> Project: Hadoop
> Type: Improvement
> Components: dfs
> Versions: 0.1
> Reporter: Yoram Arnon
> Assignee: Hairong Kuang
> Priority: Critical
>
> currently, there's no way to analyze and debug DFS errors where blocks disapear.
> name server should log its decisions that affect data, including block creation, removal, replication:
> - block <b> created, assigned to datanodes A, B, ...
> - datanode A dead, block <b> underreplicated(1), replicating to datanode C
> - datanode B dead, block <b> underreplicated(2), replicating to datanode D
> - datanode A alive, block <b> overreplicated, removing from datanode D
> - block <removed> from datanodes C, D, ...
> that will enable me to track down, two weeks later, a block that's missing from a file, and to debug the name server.
> extra credit:
> - rotate log file, as it might grow large
> - make this behaviour optional/configurable
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Assigned: (HADOOP-96) name server should log decisions that affect data: block creation, removal, replication
Posted by Eric Baldeschwieler <er...@yahoo-inc.com>.
do we have an easy way of loading the some of the name nodes (and
data nodes) local directories into HDFs space? This is an easy trick
that would make it trivial to apply map-reduce to distributed log
grepping. Should we add that? Had good success with this sort of
thing with previous systems.
EG.
hdfs://blah:6666/SYSTEM/namenodes/<host:port>/logs/
just points allows one to HDFS browse the logs directory of a name
node. Can then use the plan-9 trick of putting all kinds of API info
into "files" on the data nodes (be they real files or CGI equivalents).
Should I file a bug on this?
On Mar 30, 2006, at 4:08 PM, Yoram Arnon (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/HADOOP-96?page=all ]
>
> Yoram Arnon reassigned HADOOP-96:
> ---------------------------------
>
> Assign To: Hairong Kuang
>
>> name server should log decisions that affect data: block creation,
>> removal, replication
>> ---------------------------------------------------------------------
>> ------------------
>>
>> Key: HADOOP-96
>> URL: http://issues.apache.org/jira/browse/HADOOP-96
>> Project: Hadoop
>> Type: Improvement
>> Components: dfs
>> Versions: 0.1
>> Reporter: Yoram Arnon
>> Assignee: Hairong Kuang
>> Priority: Critical
>
>>
>> currently, there's no way to analyze and debug DFS errors where
>> blocks disapear.
>> name server should log its decisions that affect data, including
>> block creation, removal, replication:
>> - block <b> created, assigned to datanodes A, B, ...
>> - datanode A dead, block <b> underreplicated(1), replicating to
>> datanode C
>> - datanode B dead, block <b> underreplicated(2), replicating to
>> datanode D
>> - datanode A alive, block <b> overreplicated, removing from
>> datanode D
>> - block <removed> from datanodes C, D, ...
>> that will enable me to track down, two weeks later, a block that's
>> missing from a file, and to debug the name server.
>> extra credit:
>> - rotate log file, as it might grow large
>> - make this behaviour optional/configurable
>
> --
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the
> administrators:
> http://issues.apache.org/jira/secure/Administrators.jspa
> -
> For more information on JIRA, see:
> http://www.atlassian.com/software/jira
>