You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crail.apache.org by Adrian Schüpbach <ad...@gribex.net> on 2020/05/27 09:31:54 UTC
Mechanism that allow datanode to leave
Dear all
Crail supports dynamically adding new datanodes, while the Crail cluster
is running.
To build an elastic storage service, it would be nice to have
mechanisms and protocol extensions, which allow a datanode
to gracefully leave the cluster.
Especially in serverless environments, it would be great
if the Crail cluster could also dynamically grow and shrink
according to current storage capacity needs.
Since a while I am experimenting with an older version,
which is being used in Pocket. I changed and extended it
and gained some more experience, what I believe would be
good to have. I also believe that adding such functionality
natively to Apache Crail (instead of "around" Crail as in Pocket)
would help making Crail a storage service choice in serverless
environments. Furthermore, having this mechanisms natively
in Crail does not harm running Crail the classical way.
More concretely, I suggest to add the following:
- Add mechanisms to gracefully leave datanodes
(with the namenode's help)
- Mechanism 1: Datanode leaves when no more blocks are allocated
(as in Pocket)
- Mechanism 2: Namenode helps to move blocks from the leaving
datanode to a remaining datanode. "helps" does not mean,
that the namenode has to perform the actual data copying,
but only to find new blocks and update the file block lists.
- Allow datanodes to express the wish to leave
(ask namenode to initiate the process), by sending a message to
the namenode.
I would also volunteer to add this functionality.
I would like to emphasize that I would like to add
the mechanisms in a way that they do not get invoked
automatically and that performance and current functionality
will no be affected in any way, when Crail is used the usual way.
Instead, adding the mechanisms allows building a dynamically
scaling system based on policy code, which can also run outside
of the namenode and datanodes.
Thanks
Adrian
Re: Mechanism that allow datanode to leave
Posted by Ana Klimovic <an...@gmail.com>.
Thanks Adrian, this sounds great!
Best,
Ana
On Thu, May 28, 2020 at 7:40 AM Adrian Schüpbach <
adrian.schuepbach@gribex.net> wrote:
> Thanks, Patrick! I will post updates to the list
> and hopefully have a first part of code ready for
> discussion/reviewing soon.
>
> Best regards
> Adrian
>
> On 27.05.20 18:00, Patrick Stuedi wrote:
> > Adrian, thanks for providing this info. Looking forward to the
> > implementation.
> >
> > Best regards,
> > Patrick
> >
> > On Wed, May 27, 2020 at 2:32 AM Adrian Schüpbach <
> > adrian.schuepbach@gribex.net> wrote:
> >
> >> Dear all
> >>
> >> Crail supports dynamically adding new datanodes, while the Crail cluster
> >> is running.
> >>
> >> To build an elastic storage service, it would be nice to have
> >> mechanisms and protocol extensions, which allow a datanode
> >> to gracefully leave the cluster.
> >>
> >> Especially in serverless environments, it would be great
> >> if the Crail cluster could also dynamically grow and shrink
> >> according to current storage capacity needs.
> >>
> >> Since a while I am experimenting with an older version,
> >> which is being used in Pocket. I changed and extended it
> >> and gained some more experience, what I believe would be
> >> good to have. I also believe that adding such functionality
> >> natively to Apache Crail (instead of "around" Crail as in Pocket)
> >> would help making Crail a storage service choice in serverless
> >> environments. Furthermore, having this mechanisms natively
> >> in Crail does not harm running Crail the classical way.
> >>
> >> More concretely, I suggest to add the following:
> >> - Add mechanisms to gracefully leave datanodes
> >> (with the namenode's help)
> >> - Mechanism 1: Datanode leaves when no more blocks are allocated
> >> (as in Pocket)
> >> - Mechanism 2: Namenode helps to move blocks from the leaving
> >> datanode to a remaining datanode. "helps" does not mean,
> >> that the namenode has to perform the actual data copying,
> >> but only to find new blocks and update the file block lists.
> >> - Allow datanodes to express the wish to leave
> >> (ask namenode to initiate the process), by sending a message to
> >> the namenode.
> >>
> >> I would also volunteer to add this functionality.
> >>
> >>
> >>
> >> I would like to emphasize that I would like to add
> >> the mechanisms in a way that they do not get invoked
> >> automatically and that performance and current functionality
> >> will no be affected in any way, when Crail is used the usual way.
> >> Instead, adding the mechanisms allows building a dynamically
> >> scaling system based on policy code, which can also run outside
> >> of the namenode and datanodes.
> >>
> >> Thanks
> >> Adrian
> >>
> >>
> >
>
Re: Mechanism that allow datanode to leave
Posted by Adrian Schüpbach <ad...@gribex.net>.
Thanks, Patrick! I will post updates to the list
and hopefully have a first part of code ready for
discussion/reviewing soon.
Best regards
Adrian
On 27.05.20 18:00, Patrick Stuedi wrote:
> Adrian, thanks for providing this info. Looking forward to the
> implementation.
>
> Best regards,
> Patrick
>
> On Wed, May 27, 2020 at 2:32 AM Adrian Schüpbach <
> adrian.schuepbach@gribex.net> wrote:
>
>> Dear all
>>
>> Crail supports dynamically adding new datanodes, while the Crail cluster
>> is running.
>>
>> To build an elastic storage service, it would be nice to have
>> mechanisms and protocol extensions, which allow a datanode
>> to gracefully leave the cluster.
>>
>> Especially in serverless environments, it would be great
>> if the Crail cluster could also dynamically grow and shrink
>> according to current storage capacity needs.
>>
>> Since a while I am experimenting with an older version,
>> which is being used in Pocket. I changed and extended it
>> and gained some more experience, what I believe would be
>> good to have. I also believe that adding such functionality
>> natively to Apache Crail (instead of "around" Crail as in Pocket)
>> would help making Crail a storage service choice in serverless
>> environments. Furthermore, having this mechanisms natively
>> in Crail does not harm running Crail the classical way.
>>
>> More concretely, I suggest to add the following:
>> - Add mechanisms to gracefully leave datanodes
>> (with the namenode's help)
>> - Mechanism 1: Datanode leaves when no more blocks are allocated
>> (as in Pocket)
>> - Mechanism 2: Namenode helps to move blocks from the leaving
>> datanode to a remaining datanode. "helps" does not mean,
>> that the namenode has to perform the actual data copying,
>> but only to find new blocks and update the file block lists.
>> - Allow datanodes to express the wish to leave
>> (ask namenode to initiate the process), by sending a message to
>> the namenode.
>>
>> I would also volunteer to add this functionality.
>>
>>
>>
>> I would like to emphasize that I would like to add
>> the mechanisms in a way that they do not get invoked
>> automatically and that performance and current functionality
>> will no be affected in any way, when Crail is used the usual way.
>> Instead, adding the mechanisms allows building a dynamically
>> scaling system based on policy code, which can also run outside
>> of the namenode and datanodes.
>>
>> Thanks
>> Adrian
>>
>>
>
Re: Mechanism that allow datanode to leave
Posted by Patrick Stuedi <ps...@gmail.com>.
Adrian, thanks for providing this info. Looking forward to the
implementation.
Best regards,
Patrick
On Wed, May 27, 2020 at 2:32 AM Adrian Schüpbach <
adrian.schuepbach@gribex.net> wrote:
> Dear all
>
> Crail supports dynamically adding new datanodes, while the Crail cluster
> is running.
>
> To build an elastic storage service, it would be nice to have
> mechanisms and protocol extensions, which allow a datanode
> to gracefully leave the cluster.
>
> Especially in serverless environments, it would be great
> if the Crail cluster could also dynamically grow and shrink
> according to current storage capacity needs.
>
> Since a while I am experimenting with an older version,
> which is being used in Pocket. I changed and extended it
> and gained some more experience, what I believe would be
> good to have. I also believe that adding such functionality
> natively to Apache Crail (instead of "around" Crail as in Pocket)
> would help making Crail a storage service choice in serverless
> environments. Furthermore, having this mechanisms natively
> in Crail does not harm running Crail the classical way.
>
> More concretely, I suggest to add the following:
> - Add mechanisms to gracefully leave datanodes
> (with the namenode's help)
> - Mechanism 1: Datanode leaves when no more blocks are allocated
> (as in Pocket)
> - Mechanism 2: Namenode helps to move blocks from the leaving
> datanode to a remaining datanode. "helps" does not mean,
> that the namenode has to perform the actual data copying,
> but only to find new blocks and update the file block lists.
> - Allow datanodes to express the wish to leave
> (ask namenode to initiate the process), by sending a message to
> the namenode.
>
> I would also volunteer to add this functionality.
>
>
>
> I would like to emphasize that I would like to add
> the mechanisms in a way that they do not get invoked
> automatically and that performance and current functionality
> will no be affected in any way, when Crail is used the usual way.
> Instead, adding the mechanisms allows building a dynamically
> scaling system based on policy code, which can also run outside
> of the namenode and datanodes.
>
> Thanks
> Adrian
>
>
Re: Mechanism that allow datanode to leave
Posted by Adrian Schüpbach <ad...@gribex.net>.
Hi Bernard, all
Thanks for the comments. Few answers inlined below.
>> Since a while I am experimenting with an older version,
>> which is being used in Pocket. I changed and extended it
>> and gained some more experience, what I believe would be
>> good to have. I also believe that adding such functionality
>> natively to Apache Crail (instead of "around" Crail as in Pocket)
>> would help making Crail a storage service choice in serverless
>> environments. Furthermore, having this mechanisms natively
>> in Crail does not harm running Crail the classical way.
>>
>> More concretely, I suggest to add the following:
>> - Add mechanisms to gracefully leave datanodes
>> (with the namenode's help)
>> - Mechanism 1: Datanode leaves when no more blocks are allocated
>> (as in Pocket)
>> - Mechanism 2: Namenode helps to move blocks from the leaving
>> datanode to a remaining datanode. "helps" does not mean,
>> that the namenode has to perform the actual data copying,
>> but only to find new blocks and update the file block lists.
> ...but that data copying would have to be done somehow.
> So the data node shall execute that? That would add
> client code to the data node..?
Yes, this is at least one possible approach I had in mind.
I don't think that the full client code is necessary,
if the namenode gives a list of replacement block per
allocated block of the datanode that requested to leave,
the datanode would only need to perform the copying of
its block to the new one (for all blocks). So I believe it
would be a rather simple mechanism.
Since the datanode is leaving anyways, I believe it is ok
to do some additional work. I know, the design goal is
that datanodes do not perform any active action, however
in this special case it is probably best, if the leaving node
does perform the action, instead of giving this additional
work to the namenode or the new datanode.
But we can discuss that.
>
> Would probably good if in a first step the simple
> mechanism (empty data node leaves) becomes available?
>
Yes, definitely this will be the first step.
>> - Allow datanodes to express the wish to leave
>> (ask namenode to initiate the process), by sending a message to
>> the namenode.
> Another way around would be to let the namenode tell
> the data node to go off. I think you have good reasons
> to do it the way you are proposing.
>
Yes, definitely it should be possible to tell the datanode to go off.
The other way would be in addition, a datanode should be able to
ask the namenode to leave.
There is a reason for this proposal: Thinking towards serverless,
one interesting way would be to run Crail as a Knative
service and let it scale by the Knative autoscaler. This way,
if it scales in, Knative would tell the POD using a hook that it will be
deleted soon. It gets a certain time to perform some cleanup actions,
before finally being removed from the cluster.
By implementing this hook, the datanode has a chance to tell
the namenode that it is about to leave and to copy the
remaining blocks to still running datanodes.
This additional mechanism is basically an "enabler",
but does not need to be used, especially not, if Crail
is used the regular way. So I don't think that just
having the possibility built-in would negatively
affect performance.
Thanks
Adrian
Re: Mechanism that allow datanode to leave
Posted by bernard metzler <bm...@gmx.ch>.
On 5/27/2020 11:31 AM, Adrian Schüpbach wrote:
> Dear all
>
> Crail supports dynamically adding new datanodes, while the Crail cluster
> is running.
>
> To build an elastic storage service, it would be nice to have
> mechanisms and protocol extensions, which allow a datanode
> to gracefully leave the cluster.
>
> Especially in serverless environments, it would be great
> if the Crail cluster could also dynamically grow and shrink
> according to current storage capacity needs.
>
I agree.
> Since a while I am experimenting with an older version,
> which is being used in Pocket. I changed and extended it
> and gained some more experience, what I believe would be
> good to have. I also believe that adding such functionality
> natively to Apache Crail (instead of "around" Crail as in Pocket)
> would help making Crail a storage service choice in serverless
> environments. Furthermore, having this mechanisms natively
> in Crail does not harm running Crail the classical way.
>
> More concretely, I suggest to add the following:
> - Add mechanisms to gracefully leave datanodes
> (with the namenode's help)
> - Mechanism 1: Datanode leaves when no more blocks are allocated
> (as in Pocket)
> - Mechanism 2: Namenode helps to move blocks from the leaving
> datanode to a remaining datanode. "helps" does not mean,
> that the namenode has to perform the actual data copying,
> but only to find new blocks and update the file block lists.
...but that data copying would have to be done somehow.
So the data node shall execute that? That would add
client code to the data node..?
Would probably good if in a first step the simple
mechanism (empty data node leaves) becomes available?
> - Allow datanodes to express the wish to leave
> (ask namenode to initiate the process), by sending a message to
> the namenode.
Another way around would be to let the namenode tell
the data node to go off. I think you have good reasons
to do it the way you are proposing.
>
> I would also volunteer to add this functionality.
>
great!
>
>
> I would like to emphasize that I would like to add
> the mechanisms in a way that they do not get invoked
> automatically and that performance and current functionality
> will no be affected in any way, when Crail is used the usual way.
> Instead, adding the mechanisms allows building a dynamically
> scaling system based on policy code, which can also run outside
> of the namenode and datanodes.
>
> Thanks
> Adrian
>
Re: [EXTERNAL] Mechanism that allow datanode to leave
Posted by Adrian Schuepbach <DR...@zurich.ibm.com>.
Dear all
After a while it is finally getting more concrete. We would
like to add "Mechanism 1" to Apache Crail. I have
created the following "New Feature" in JIRA to track the
progress: https://issues.apache.org/jira/browse/CRAIL-111
We plan to borrow most of the mechanism we have in Pocket,
but will need to adapt it to the current Apache Crail version,
which is much newer and uses slightly different data structures.
This implementation step is only about the mechanism. As already
described, it won't be automatically invoked so that performance
and behaviour is not being affected in any way, when running
Apache Crail the usual way.
What we would like to add in addition is a trigger mechanism
to invoke the mechanism for testing purposes only.
When running Apache Crail as an elastic storage service,
a separate policy engine should decide when and how to scale-out
or scale-in the running Crail instance. It will call into
this mechanism to actually apply the decision.
We plan to present a design idea for the policy engine later
for a discussion among the Apache Crail community.
Please feel free to comment on this first step.
Thanks a lot
Adrian
From: "Adrian Schüpbach" <ad...@gribex.net>
To: dev@crail.apache.org
Date: 05/27/2020 11:32 AM
Subject: [EXTERNAL] Mechanism that allow datanode to leave
Dear all
Crail supports dynamically adding new datanodes, while the Crail cluster
is running.
To build an elastic storage service, it would be nice to have
mechanisms and protocol extensions, which allow a datanode
to gracefully leave the cluster.
Especially in serverless environments, it would be great
if the Crail cluster could also dynamically grow and shrink
according to current storage capacity needs.
Since a while I am experimenting with an older version,
which is being used in Pocket. I changed and extended it
and gained some more experience, what I believe would be
good to have. I also believe that adding such functionality
natively to Apache Crail (instead of "around" Crail as in Pocket)
would help making Crail a storage service choice in serverless
environments. Furthermore, having this mechanisms natively
in Crail does not harm running Crail the classical way.
More concretely, I suggest to add the following:
- Add mechanisms to gracefully leave datanodes
(with the namenode's help)
- Mechanism 1: Datanode leaves when no more blocks are allocated
(as in Pocket)
- Mechanism 2: Namenode helps to move blocks from the leaving
datanode to a remaining datanode. "helps" does not mean,
that the namenode has to perform the actual data copying,
but only to find new blocks and update the file block lists.
- Allow datanodes to express the wish to leave
(ask namenode to initiate the process), by sending a message to
the namenode.
I would also volunteer to add this functionality.
I would like to emphasize that I would like to add
the mechanisms in a way that they do not get invoked
automatically and that performance and current functionality
will no be affected in any way, when Crail is used the usual way.
Instead, adding the mechanisms allows building a dynamically
scaling system based on policy code, which can also run outside
of the namenode and datanodes.
Thanks
Adrian