Re: [EXTERNAL] Mechanism that allow datanode to leave

[Adrian Schuepbach <DR...@zurich.ibm.com>]

Dear all

After a while it is finally getting more concrete. We would
like to add "Mechanism 1" to Apache Crail. I have
created the following "New Feature" in JIRA to track the
progress: https://issues.apache.org/jira/browse/CRAIL-111

We plan to borrow most of the mechanism we have in Pocket,
but will need to adapt it to the current Apache Crail version,
which is much newer and uses slightly different data structures.

This implementation step is only about the mechanism. As already
described, it won't be automatically invoked so that performance
and behaviour is not being affected in any way, when running
Apache Crail the usual way.
What we would like to add in addition is a trigger mechanism
to invoke the mechanism for testing purposes only.

When running Apache Crail as an elastic storage service,
a separate policy engine should decide when and how to scale-out
or scale-in the running Crail instance. It will call into
this mechanism to actually apply the decision.

We plan to present a design idea for the policy engine later
for a discussion among the Apache Crail community.


Please feel free to comment on this first step.

Thanks a lot
Adrian





From:   "Adrian Schüpbach" <ad...@gribex.net>
To:     dev@crail.apache.org
Date:   05/27/2020 11:32 AM
Subject:        [EXTERNAL] Mechanism that allow datanode to leave



Dear all

Crail supports dynamically adding new datanodes, while the Crail cluster
is running.

To build an elastic storage service, it would be nice to have
mechanisms and protocol extensions, which allow a datanode
to gracefully leave the cluster.

Especially in serverless environments, it would be great
if the Crail cluster could also dynamically grow and shrink
according to current storage capacity needs.

Since a while I am experimenting with an older version,
which is being used in Pocket. I changed and extended it
and gained some more experience, what I believe would be
good to have. I also believe that adding such functionality
natively to Apache Crail (instead of "around" Crail as in Pocket)
would help making Crail a storage service choice in serverless
environments. Furthermore, having this mechanisms natively
in Crail does not harm running Crail the classical way.

More concretely, I suggest to add the following:
- Add mechanisms to gracefully leave datanodes
   (with the namenode's help)
- Mechanism 1: Datanode leaves when no more blocks are allocated
   (as in Pocket)
- Mechanism 2: Namenode helps to move blocks from the leaving
   datanode to a remaining datanode. "helps" does not mean,
   that the namenode has to perform the actual data copying,
   but only to find new blocks and update the file block lists.
- Allow datanodes to express the wish to leave
   (ask namenode to initiate the process), by sending a message to
   the namenode.

I would also volunteer to add this functionality.



I would like to emphasize that I would like to add
the mechanisms in a way that they do not get invoked
automatically and that performance and current functionality
will no be affected in any way, when Crail is used the usual way.
Instead, adding the mechanisms allows building a dynamically
scaling system based on policy code, which can also run outside
of the namenode and datanodes.

Thanks
Adrian