You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Stephan Hoermann <sr...@gmail.com> on 2016/06/16 02:29:24 UTC
Multi node maintenance for HDFS?
Hi,
How do people do multi node maintenance for HDFS without data loss?
We want to apply the ideas of immutable infrastructure to how we manage our
machines. We prebuild an OS image with the configuration and roll it out to
our nodes. When we have a patch we build a new image and roll that out
again. It takes us about 10 to 15 minutes to do that.
For our data nodes we want to keep the data on a separate partition/disks
so that when we rebuild we rejoin HDFS with the data don't start a
replication storm.
Now in order to scale this and quickly roll out upgrades we can't really do
a one node at a time upgrade so we need to be able to take out a percentage
of the nodes at a time. Ideally we would like to do this while keeping the
replication count of each block at 2 (so we can still handle failure while
we are doing an upgrade) and without starting a replication strategy.
Right now it doesn't look like that is really supported. Is anyone else
doing multi node upgrades and how do you solve these problems?
We are considering changing the replication strategy so that we divide all
our nodes into 3 evenly sized buckets and at maintenance remove a subset
from one bucket at a time. Does anyone have experience with doing something
similar?
Regards,
Stephan
Re: Multi node maintenance for HDFS?
Posted by Anu Engineer <ae...@hortonworks.com>.
We are still working on 2.8 release and this feature is targeted for 2.9. So we don’t have a time line now.
Thanks
Anu
From: Stephan Hoermann <sr...@gmail.com>
Date: Thursday, June 16, 2016 at 6:55 PM
To: Anu Engineer <ae...@hortonworks.com>, "user@hadoop.apache.org" <us...@hadoop.apache.org>
Subject: Re: Multi node maintenance for HDFS?
Hi Anu,
That was exactly what I was looking for! Do you have any idea when this will be released?
Thank you,
Stephan
On Fri, 17 Jun 2016 at 02:12 Anu Engineer <ae...@hortonworks.com>> wrote:
Hi Stephan,
AFAIK, this is not a solved problem in the version of Hadoop that you are using. There is a JIRA that you might be interested in.
Please look at the attached design documents of https://issues.apache.org/jira/browse/HDFS-7541 and see
if it addresses your problem.
Disclaimer: I have never used the feature proposed/implemented in this JIRA,
however most of the developers are from twitter and hopefully this is being used internally.
Thanks
Anu
From: Stephan Hoermann <sr...@gmail.com>>
Date: Wednesday, June 15, 2016 at 7:29 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Multi node maintenance for HDFS?
Hi,
How do people do multi node maintenance for HDFS without data loss?
We want to apply the ideas of immutable infrastructure to how we manage our machines. We prebuild an OS image with the configuration and roll it out to our nodes. When we have a patch we build a new image and roll that out again. It takes us about 10 to 15 minutes to do that.
For our data nodes we want to keep the data on a separate partition/disks so that when we rebuild we rejoin HDFS with the data don't start a replication storm.
Now in order to scale this and quickly roll out upgrades we can't really do a one node at a time upgrade so we need to be able to take out a percentage of the nodes at a time. Ideally we would like to do this while keeping the replication count of each block at 2 (so we can still handle failure while we are doing an upgrade) and without starting a replication strategy.
Right now it doesn't look like that is really supported. Is anyone else doing multi node upgrades and how do you solve these problems?
We are considering changing the replication strategy so that we divide all our nodes into 3 evenly sized buckets and at maintenance remove a subset from one bucket at a time. Does anyone have experience with doing something similar?
Regards,
Stephan
Re: Multi node maintenance for HDFS?
Posted by Stephan Hoermann <sr...@gmail.com>.
Hi Anu,
That was exactly what I was looking for! Do you have any idea when this
will be released?
Thank you,
Stephan
On Fri, 17 Jun 2016 at 02:12 Anu Engineer <ae...@hortonworks.com> wrote:
> Hi Stephan,
>
>
>
> AFAIK, this is not a solved problem in the version of Hadoop that you are
> using. There is a JIRA that you might be interested in.
>
> Please look at the attached design documents of
> https://issues.apache.org/jira/browse/HDFS-7541 and see
>
> if it addresses your problem.
>
>
>
> Disclaimer: I have never used the feature proposed/implemented in this
> JIRA,
>
> however most of the developers are from twitter and hopefully this is
> being used internally.
>
>
>
> Thanks
>
> Anu
>
>
>
>
>
>
>
> *From: *Stephan Hoermann <sr...@gmail.com>
> *Date: *Wednesday, June 15, 2016 at 7:29 PM
> *To: *"user@hadoop.apache.org" <us...@hadoop.apache.org>
> *Subject: *Multi node maintenance for HDFS?
>
>
>
> Hi,
>
>
>
> How do people do multi node maintenance for HDFS without data loss?
>
>
>
> We want to apply the ideas of immutable infrastructure to how we manage
> our machines. We prebuild an OS image with the configuration and roll it
> out to our nodes. When we have a patch we build a new image and roll that
> out again. It takes us about 10 to 15 minutes to do that.
>
>
>
> For our data nodes we want to keep the data on a separate partition/disks
> so that when we rebuild we rejoin HDFS with the data don't start a
> replication storm.
>
>
>
> Now in order to scale this and quickly roll out upgrades we can't really
> do a one node at a time upgrade so we need to be able to take out a
> percentage of the nodes at a time. Ideally we would like to do this while
> keeping the replication count of each block at 2 (so we can still handle
> failure while we are doing an upgrade) and without starting a replication
> strategy.
>
>
>
> Right now it doesn't look like that is really supported. Is anyone else
> doing multi node upgrades and how do you solve these problems?
>
>
>
> We are considering changing the replication strategy so that we divide all
> our nodes into 3 evenly sized buckets and at maintenance remove a subset
> from one bucket at a time. Does anyone have experience with doing something
> similar?
>
>
>
> Regards,
>
>
>
> Stephan
>
Re: Multi node maintenance for HDFS?
Posted by Anu Engineer <ae...@hortonworks.com>.
Hi Stephan,
AFAIK, this is not a solved problem in the version of Hadoop that you are using. There is a JIRA that you might be interested in.
Please look at the attached design documents of https://issues.apache.org/jira/browse/HDFS-7541 and see
if it addresses your problem.
Disclaimer: I have never used the feature proposed/implemented in this JIRA,
however most of the developers are from twitter and hopefully this is being used internally.
Thanks
Anu
From: Stephan Hoermann <sr...@gmail.com>
Date: Wednesday, June 15, 2016 at 7:29 PM
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
Subject: Multi node maintenance for HDFS?
Hi,
How do people do multi node maintenance for HDFS without data loss?
We want to apply the ideas of immutable infrastructure to how we manage our machines. We prebuild an OS image with the configuration and roll it out to our nodes. When we have a patch we build a new image and roll that out again. It takes us about 10 to 15 minutes to do that.
For our data nodes we want to keep the data on a separate partition/disks so that when we rebuild we rejoin HDFS with the data don't start a replication storm.
Now in order to scale this and quickly roll out upgrades we can't really do a one node at a time upgrade so we need to be able to take out a percentage of the nodes at a time. Ideally we would like to do this while keeping the replication count of each block at 2 (so we can still handle failure while we are doing an upgrade) and without starting a replication strategy.
Right now it doesn't look like that is really supported. Is anyone else doing multi node upgrades and how do you solve these problems?
We are considering changing the replication strategy so that we divide all our nodes into 3 evenly sized buckets and at maintenance remove a subset from one bucket at a time. Does anyone have experience with doing something similar?
Regards,
Stephan
Re: Multi node maintenance for HDFS?
Posted by Olivier Renault <or...@hortonworks.com>.
Sorry, it's not suitable, as I take the cluster down.
Thanks,
Olivier
On Fri, Jun 17, 2016 at 3:00 AM +0100, "Stephan Hoermann" <sr...@gmail.com>> wrote:
Hi Olivier,
Do your instructions apply for an online cluster? I couldn't see where it dealt with keeping the data online/available while the upgrade is happening.
I'm sorry I wasn't very precise. We don't just want to avoid data loss but also do the upgrade while the cluster is running and avoid unavailability of data. We only want 1 copy of each block offline while we are upgrading.
The ticket Anu linked to address seems to address our use case.
Thank you,
Stephan
On Fri, 17 Jun 2016 at 03:21 Olivier Renault <or...@hortonworks.com>> wrote:
Hi Stephan,
It happens that I've been working on this during the last two days. It was much easier that I was expecting.
https://.hortonworks.com/articles/40126/hdp-upgrade-using-reinstallation.html<https://community.hortonworks.com/articles/40126/hdp-upgrade-using-reinstallation.html>
Let me know if you've got any questions
Kind regards,
Olivier
From: Stephan Hoermann <sr...@gmail.com>>
Date: Thursday, 16 June 2016 at 04:29
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Multi node maintenance for HDFS?
Hi,
How do people do multi node maintenance for HDFS without data loss?
We want to apply the ideas of immutable infrastructure to how we manage our machines. We prebuild an OS image with the configuration and roll it out to our nodes. When we have a patch we build a new image and roll that out again. It takes us about 10 to 15 minutes to do that.
For our data nodes we want to keep the data on a separate partition/disks so that when we rebuild we rejoin HDFS with the data don't start a replication storm.
Now in order to scale this and quickly roll out upgrades we can't really do a one node at a time upgrade so we need to be able to take out a percentage of the nodes at a time. Ideally we would like to do this while keeping the replication count of each block at 2 (so we can still handle failure while we are doing an upgrade) and without starting a replication strategy.
Right now it doesn't look like that is really supported. Is anyone else doing multi node upgrades and how do you solve these problems?
We are considering changing the replication strategy so that we divide all our nodes into 3 evenly sized buckets and at maintenance remove a subset from one bucket at a time. Does anyone have experience with doing something similar?
Regards,
Stephan
Re: Multi node maintenance for HDFS?
Posted by Stephan Hoermann <sr...@gmail.com>.
Hi Olivier,
Do your instructions apply for an online cluster? I couldn't see where it
dealt with keeping the data online/available while the upgrade is happening.
I'm sorry I wasn't very precise. We don't just want to avoid data loss but
also do the upgrade while the cluster is running and avoid unavailability
of data. We only want 1 copy of each block offline while we are upgrading.
The ticket Anu linked to address seems to address our use case.
Thank you,
Stephan
On Fri, 17 Jun 2016 at 03:21 Olivier Renault <or...@hortonworks.com>
wrote:
> Hi Stephan,
>
>
>
> It happens that I’ve been working on this during the last two days. It was
> much easier that I was expecting.
>
>
>
https://.hortonworks.com/articles/40126/hdp-upgrade-using-reinstallation.html
> <https://community.hortonworks.com/articles/40126/hdp-upgrade-using-reinstallation.html>
>
>
> Let me know if you’ve got any questions
>
>
>
> Kind regards,
>
> Olivier
>
>
>
> *From: *Stephan Hoermann <sr...@gmail.com>
> *Date: *Thursday, 16 June 2016 at 04:29
>
> *To: *"user@hadoop.apache.org" <us...@hadoop.apache.org>
> *Subject: *Multi node maintenance for HDFS?
>
>
>
> Hi,
>
>
>
> How do people do multi node maintenance for HDFS without data loss?
>
>
>
> We want to apply the ideas of immutable infrastructure to how we manage
> our machines. We prebuild an OS image with the configuration and roll it
> out to our nodes. When we have a patch we build a new image and roll that
> out again. It takes us about 10 to 15 minutes to do that.
>
>
>
> For our data nodes we want to keep the data on a separate partition/disks
> so that when we rebuild we rejoin HDFS with the data don't start a
> replication storm.
>
>
>
> Now in order to scale this and quickly roll out upgrades we can't really
> do a one node at a time upgrade so we need to be able to take out a
> percentage of the nodes at a time. Ideally we would like to do this while
> keeping the replication count of each block at 2 (so we can still handle
> failure while we are doing an upgrade) and without starting a replication
> strategy.
>
>
>
> Right now it doesn't look like that is really supported. Is anyone else
> doing multi node upgrades and how do you solve these problems?
>
>
>
> We are considering changing the replication strategy so that we divide all
> our nodes into 3 evenly sized buckets and at maintenance remove a subset
> from one bucket at a time. Does anyone have experience with doing something
> similar?
>
>
>
> Regards,
>
>
>
> Stephan
>
Re: Multi node maintenance for HDFS?
Posted by Olivier Renault <or...@hortonworks.com>.
Hi Stephan,
It happens that I’ve been working on this during the last two days. It was much easier that I was expecting.
https://community.hortonworks.com/articles/40126/hdp-upgrade-using-reinstallation.html
Let me know if you’ve got any questions
Kind regards,
Olivier
From: Stephan Hoermann <sr...@gmail.com>
Date: Thursday, 16 June 2016 at 04:29
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
Subject: Multi node maintenance for HDFS?
Hi,
How do people do multi node maintenance for HDFS without data loss?
We want to apply the ideas of immutable infrastructure to how we manage our machines. We prebuild an OS image with the configuration and roll it out to our nodes. When we have a patch we build a new image and roll that out again. It takes us about 10 to 15 minutes to do that.
For our data nodes we want to keep the data on a separate partition/disks so that when we rebuild we rejoin HDFS with the data don't start a replication storm.
Now in order to scale this and quickly roll out upgrades we can't really do a one node at a time upgrade so we need to be able to take out a percentage of the nodes at a time. Ideally we would like to do this while keeping the replication count of each block at 2 (so we can still handle failure while we are doing an upgrade) and without starting a replication strategy.
Right now it doesn't look like that is really supported. Is anyone else doing multi node upgrades and how do you solve these problems?
We are considering changing the replication strategy so that we divide all our nodes into 3 evenly sized buckets and at maintenance remove a subset from one bucket at a time. Does anyone have experience with doing something similar?
Regards,
Stephan