[jira] [Commented] (HDFS-12345) Scale testing HDFS NameNode with real metadata and workloads

["Wei-Chiu Chuang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576465#comment-16576465 ] 

Wei-Chiu Chuang commented on HDFS-12345:
----------------------------------------

Guys, this is an awesome work.

Here at Cloudera's HDFS team, we've been working to port it to CDH5/CDH6. From a development point of view, there are some minor issues dealing with private/test API compatibility when I ported it to different versions. Having this tool in Hadoop code base would make sure the community takes care of the API compat.

 

I would vote +1 to incorporate Dynamometer into Hadoop codebase (pending LinkedIn's legal team review).

 

Moving forward, there are several pending tasks in this tool from our perspective:
 # more unit tests
 # more use scenarios: we're interested in HA, snapshots, authentication (Kerberos), encryption (KMS), authorization (Sentry or Ranger)

 

Thanks, and look forward to more communication collaboration!

> Scale testing HDFS NameNode with real metadata and workloads
> ------------------------------------------------------------
>
>                 Key: HDFS-12345
>                 URL: https://issues.apache.org/jira/browse/HDFS-12345
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: namenode, test
>            Reporter: Zhe Zhang
>            Assignee: Erik Krogen
>            Priority: Major
>
> Dynamometer has now been open sourced on our [GitHub page|https://github.com/linkedin/dynamometer]. Read more at our [recent blog post|https://engineering.linkedin.com/blog/2018/02/dynamometer--scale-testing-hdfs-on-minimal-hardware-with-maximum].
> To encourage getting the tool into the open for others to use as quickly as possible, we went through our standard open sourcing process of releasing on GitHub. However we are interested in the possibility of donating this to Apache as part of Hadoop itself and would appreciate feedback on whether or not this is something that would be supported by the community.
> Also of note, previous [discussions on the dev mail lists|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201707.mbox/%3c98FCEFFA-FAFF-4CF1-A14D-4FAAB6567F31@gmail.com%3e]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org