You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Manoj Govindassamy (JIRA)" <ji...@apache.org> on 2017/06/09 00:45:18 UTC

[jira] [Updated] (HDFS-10999) Introduce separate stats for Replicated and Erasure Coded Blocks apart from the current Aggregated stats

     [ https://issues.apache.org/jira/browse/HDFS-10999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Manoj Govindassamy updated HDFS-10999:
--------------------------------------
    Attachment: HDFS-10999.04.patch

Thanks [~tasanuma0829] and [~andrew.wang] for the detailed review comments. Attaching patch v04 to address the following. Can you please take a look?

Takanobu:
bq. For unit tests: I think it would be good if we add more unit tests for these changes in this jira or follow-on jiras. Add more validations for new metrics in TestComputeInvalidateWork, TestCorruptReplicaInfo and TestLowRedundancyBlockQueues.
Done. Added new unit tests to verify stats added in {{CorruptRepicasMap}}, {{InvalidateBlocks}}, {{LowRedundancyBlocks}}

bq. TestUnderReplicatedBlocks covers only replicated files. If we use DFSTestUtil#verifyClientStats in TestReconstructStripedBlocks, we may be able to cover the EC case.
Done.

Andrew:
bq. Regarding "CorruptReplicaOne" and similar, the full name is "corrupt replication factor one". Abbreviating it as "CorruptReplicaOne" is a bit ambiguous since "replica" has its own meaning. I'd prefer we name all these as "CorruptReplOne" or "CorruptReplicationOne" instead.
Done. 

bq. BlocksStats#toString and ECBlockGroupsStats, optional, but should the names be expanded to match the field names? In terms of code formatting, it'd also be clearer if the key and value were on the same line.
Done.

bq. Need javadoc on methods in ECBlockGroupsStatsMBean and ReplicatedBlocksStatsMBean. Javadoc on aggregate methods in FSNamesystemMBean should make it clear that they are aggregates too.
Done.

bq. Looks like some of my testing comments from last time were not addressed. Do you prefer to handle this in a follow-on JIRA? If so, appreciate if you could file that and link it here. Still a bit concerned about the TODO in TestNameNodeMetrics.
Fixed the tests in TestNameNodeMetrics and the TODOs are removed now. Also, added bunch of new unit tests under other Tests to cover all the newly added stats. If at all more need to be added, I can take it up in new jiras. 


> Introduce separate stats for Replicated and Erasure Coded Blocks apart from the current Aggregated stats
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-10999
>                 URL: https://issues.apache.org/jira/browse/HDFS-10999
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: erasure-coding
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Wei-Chiu Chuang
>            Assignee: Manoj Govindassamy
>              Labels: hdfs-ec-3.0-nice-to-have, supportability
>         Attachments: HDFS-10999.01.patch, HDFS-10999.02.patch, HDFS-10999.03.patch, HDFS-10999.04.patch
>
>
> Per HDFS-9857, it seems in the Hadoop 3 world, people prefer the more generic term "low redundancy" to the old-fashioned "under replicated". But this term is still being used in messages in several places, such as web ui, dfsadmin and fsck. We should probably change them to avoid confusion.
> File this jira to discuss it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org