Meeting notes from today's Hadoop storage community sync

[Wei-Chiu Chuang <we...@apache.org>]

Here's today's notes for future reference:
https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit?usp=sharing
10/16/2019

Attendee:

Weichiu, Cynthia, Craig, Stephen, Akira, David

Stephen introduced upgrade domain, which was developed at Twitter. Cloudera
is going to support this feature in the next release. The feature was
developed a few years back and quite complete, so Cloudera is just adding
UI and verification/guardrails to support this feature.

Akira is interested in decommission and maintenance mode. Decomm is slow at
Y! Japan. Akira’s interested in maintenance mode too, but they are on 2.6.x
so can’t try yet.

Stephen introduced the decommissioning improvement project. Decommissioning
in practice has a few weird behavior and tend to be slow.

HDFS-14814  a new decommissioning monitor. It reduces NameNode lock holding
time, and spread replication load across DataNodes. It also gives priority
to dead nodes than decommissioning nodes. But it’s hard to simulate its
performance. It will have to run on a real large cluster to prove it works.
Looking for community members to pick it up and introduce it in some large
clusters to try out.

HDFS-14861 instead of letting the block to go to the end of replication
queue, iterator is reset periodically.

EC is not considered yet.


Next week we will have the Hadoop storage community sync for the APAC time
(PDT 10pm Wednesday, CST 1pm Thursday). Looking for topics.

Best,
Weichiu