You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Henry JunYoung KIM <he...@gmail.com> on 2013/04/01 07:26:50 UTC
are we able to decommission multi nodes at one time?
hi, hadoop users.
to decommission a single node, there is necessary to wait to remove a node from a cluster for awhile. (in my case, 20 ~ 30 minutes)
for the safety, I am decommissioning a node at a time.
for the performance, am I able to remove multi nodes at same time?
Re: are we able to decommission multi nodes at one time?
Posted by Yanbo Liang <ya...@gmail.com>.
It's alowable to decommission multi nodes at the same time.
Just write the all the hostnames which will be decommissioned to the
exclude file and run "bin/hadoop dfsadmin -refreshNodes".
However you need to ensure the decommissioned DataNodes are minority of all
the DataNodes in the cluster and the block replica can be guaranteed after
decommission.
For example, default replication level mapred.submit.replication=10.
So if you have less than 10 DataNodes after decommissioned, the decommision
process will hang.
2013/4/1 varun kumar <va...@gmail.com>
> How many nodes do you have and replication factor for it.
>
Re: are we able to decommission multi nodes at one time?
Posted by Yanbo Liang <ya...@gmail.com>.
It's alowable to decommission multi nodes at the same time.
Just write the all the hostnames which will be decommissioned to the
exclude file and run "bin/hadoop dfsadmin -refreshNodes".
However you need to ensure the decommissioned DataNodes are minority of all
the DataNodes in the cluster and the block replica can be guaranteed after
decommission.
For example, default replication level mapred.submit.replication=10.
So if you have less than 10 DataNodes after decommissioned, the decommision
process will hang.
2013/4/1 varun kumar <va...@gmail.com>
> How many nodes do you have and replication factor for it.
>
Re: are we able to decommission multi nodes at one time?
Posted by Henry Junyoung Kim <he...@gmail.com>.
thanks for all.
my strategies for removing from 15 DN to 8 DN.
1. kill two DNs at same time. : NN will detect nodes' down and he will try to retain replication factors of lost blocks.
2. check your NN web UI. there is an info to let you know counts of under-replicated blocks.
3. if it is returned to 0 again, you can do remove other 2 DNs again. let's start from first again.
this scenario is what I did. my cluster is working well. ;)
thanks for all.
2013. 4. 3., 오후 5:18, Azuryy Yu <az...@gmail.com> 작성:
> not at all. so don't worry about that.
>
>
> On Wed, Apr 3, 2013 at 2:04 PM, Yanbo Liang <ya...@gmail.com> wrote:
> It means that may be some replicas will be stay in under replica state?
>
>
> 2013/4/3 Azuryy Yu <az...@gmail.com>
> bq. then namenode start to copy block replicates on DN-2 to another DN, supposed DN-2.
>
> sorry for typo.
>
> Correct for it:
> then namenode start to copy block replicates on DN-1 to another DN, supposed DN-2.
>
>
> On Wed, Apr 3, 2013 at 9:51 AM, Azuryy Yu <az...@gmail.com> wrote:
> It's different.
> If you just want to stop DN-1 a short time, just kill the DataNode process on DN-1. then do what you want. during this time, Namenode cannot receive the heart beat from DN-1, then namenode start to copy block replicates on DN-2 to another DN, supposed DN-2.
>
> But when you start DN-1 again, Namenode receive the DN-1 registration, then namenode stop to copy the DN-1's block replicates even if NN doesn't finish coping.
>
> Am I explain clearly?
>
>
>
> On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <he...@gmail.com> wrote:
> @Harsh
>
> What's the reasons to make big gaps for removing nodes between decommission and just down nodes?
> In my understanding, both are necessary to copy un-replicated blocks to another alive nodes.
> If main costs of them are this one, total elapsed time couldn't be big different.
>
> Could you share some articles or documents to understand about decommissioning procedures?
> - explaining is always thanks ;)
>
>
> 2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
>
> > Yes, you can do the downtime work in steps of 2 DNs at a time,
> > especially since you mentioned the total work would be only ~30mins at
> > most.
> >
> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
> > <he...@gmail.com> wrote:
> >> the rest of nodes to be alive has enough size to store.
> >>
> >> for this one that you've mentioned.
> >>> its easier to do so in a rolling manner without need of a
> >>> decommission.
> >>
> >> to check my understanding, just shutting down 2 of them and then 2 more and then 2 more without decommissions.
> >>
> >> is this correct?
> >>
> >>
> >> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
> >>
> >>> Note though that its only possible to decommission 7 nodes at the same
> >>> time and expect it to finish iff the remaining 8 nodes have adequate
> >>> free space for the excess replicas.
> >>>
> >>> If you're just going to take them down for a short while (few mins
> >>> each), its easier to do so in a rolling manner without need of a
> >>> decommission. You can take upto two down at a time on a replication
> >>> average of 3 or 3+, and put it back in later without too much data
> >>> movement impact.
> >>>
> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
> >>>> It's reasonable to decommission 7 nodes at the same time.
> >>>> But may be it also takes long time to finish it.
> >>>> Because all the replicas in these 7 nodes need to be copied to remaining 8
> >>>> nodes.
> >>>> The size of transfer from these nodes to the remaining nodes is equal.
> >>>>
> >>>>
> >>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
> >>>>>
> >>>>> :)
> >>>>>
> >>>>> currently, I have 15 data nodes.
> >>>>> for some tests, I am trying to decommission until 8 nodes.
> >>>>>
> >>>>> Now, the total dfs used size is 52 TB which is including all replicated
> >>>>> blocks.
> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
> >>>>>
> >>>>> someone mentioned that I don't need to decommission node by node.
> >>>>> for this case, is there no problems if I decommissioned 7 nodes at the
> >>>>> same time?
> >>>>>
> >>>>>
> >>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
> >>>>>
> >>>>> I can translate it to native English: how many nodes you want to
> >>>>> decommission?
> >>>>>
> >>>>>
> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
> >>>>>>
> >>>>>> You want to decommission how many nodes?
> >>>>>>
> >>>>>>
> >>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
> >>>>>>>
> >>>>>>> 15 for datanodes and 3 for replication factor.
> >>>>>>>
> >>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
> >>>>>>>
> >>>>>>>> How many nodes do you have and replication factor for it.
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Harsh J
> >>
> >
> >
> >
> > --
> > Harsh J
>
>
>
>
>
Re: are we able to decommission multi nodes at one time?
Posted by Henry Junyoung Kim <he...@gmail.com>.
thanks for all.
my strategies for removing from 15 DN to 8 DN.
1. kill two DNs at same time. : NN will detect nodes' down and he will try to retain replication factors of lost blocks.
2. check your NN web UI. there is an info to let you know counts of under-replicated blocks.
3. if it is returned to 0 again, you can do remove other 2 DNs again. let's start from first again.
this scenario is what I did. my cluster is working well. ;)
thanks for all.
2013. 4. 3., 오후 5:18, Azuryy Yu <az...@gmail.com> 작성:
> not at all. so don't worry about that.
>
>
> On Wed, Apr 3, 2013 at 2:04 PM, Yanbo Liang <ya...@gmail.com> wrote:
> It means that may be some replicas will be stay in under replica state?
>
>
> 2013/4/3 Azuryy Yu <az...@gmail.com>
> bq. then namenode start to copy block replicates on DN-2 to another DN, supposed DN-2.
>
> sorry for typo.
>
> Correct for it:
> then namenode start to copy block replicates on DN-1 to another DN, supposed DN-2.
>
>
> On Wed, Apr 3, 2013 at 9:51 AM, Azuryy Yu <az...@gmail.com> wrote:
> It's different.
> If you just want to stop DN-1 a short time, just kill the DataNode process on DN-1. then do what you want. during this time, Namenode cannot receive the heart beat from DN-1, then namenode start to copy block replicates on DN-2 to another DN, supposed DN-2.
>
> But when you start DN-1 again, Namenode receive the DN-1 registration, then namenode stop to copy the DN-1's block replicates even if NN doesn't finish coping.
>
> Am I explain clearly?
>
>
>
> On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <he...@gmail.com> wrote:
> @Harsh
>
> What's the reasons to make big gaps for removing nodes between decommission and just down nodes?
> In my understanding, both are necessary to copy un-replicated blocks to another alive nodes.
> If main costs of them are this one, total elapsed time couldn't be big different.
>
> Could you share some articles or documents to understand about decommissioning procedures?
> - explaining is always thanks ;)
>
>
> 2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
>
> > Yes, you can do the downtime work in steps of 2 DNs at a time,
> > especially since you mentioned the total work would be only ~30mins at
> > most.
> >
> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
> > <he...@gmail.com> wrote:
> >> the rest of nodes to be alive has enough size to store.
> >>
> >> for this one that you've mentioned.
> >>> its easier to do so in a rolling manner without need of a
> >>> decommission.
> >>
> >> to check my understanding, just shutting down 2 of them and then 2 more and then 2 more without decommissions.
> >>
> >> is this correct?
> >>
> >>
> >> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
> >>
> >>> Note though that its only possible to decommission 7 nodes at the same
> >>> time and expect it to finish iff the remaining 8 nodes have adequate
> >>> free space for the excess replicas.
> >>>
> >>> If you're just going to take them down for a short while (few mins
> >>> each), its easier to do so in a rolling manner without need of a
> >>> decommission. You can take upto two down at a time on a replication
> >>> average of 3 or 3+, and put it back in later without too much data
> >>> movement impact.
> >>>
> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
> >>>> It's reasonable to decommission 7 nodes at the same time.
> >>>> But may be it also takes long time to finish it.
> >>>> Because all the replicas in these 7 nodes need to be copied to remaining 8
> >>>> nodes.
> >>>> The size of transfer from these nodes to the remaining nodes is equal.
> >>>>
> >>>>
> >>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
> >>>>>
> >>>>> :)
> >>>>>
> >>>>> currently, I have 15 data nodes.
> >>>>> for some tests, I am trying to decommission until 8 nodes.
> >>>>>
> >>>>> Now, the total dfs used size is 52 TB which is including all replicated
> >>>>> blocks.
> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
> >>>>>
> >>>>> someone mentioned that I don't need to decommission node by node.
> >>>>> for this case, is there no problems if I decommissioned 7 nodes at the
> >>>>> same time?
> >>>>>
> >>>>>
> >>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
> >>>>>
> >>>>> I can translate it to native English: how many nodes you want to
> >>>>> decommission?
> >>>>>
> >>>>>
> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
> >>>>>>
> >>>>>> You want to decommission how many nodes?
> >>>>>>
> >>>>>>
> >>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
> >>>>>>>
> >>>>>>> 15 for datanodes and 3 for replication factor.
> >>>>>>>
> >>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
> >>>>>>>
> >>>>>>>> How many nodes do you have and replication factor for it.
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Harsh J
> >>
> >
> >
> >
> > --
> > Harsh J
>
>
>
>
>
Re: are we able to decommission multi nodes at one time?
Posted by Henry Junyoung Kim <he...@gmail.com>.
thanks for all.
my strategies for removing from 15 DN to 8 DN.
1. kill two DNs at same time. : NN will detect nodes' down and he will try to retain replication factors of lost blocks.
2. check your NN web UI. there is an info to let you know counts of under-replicated blocks.
3. if it is returned to 0 again, you can do remove other 2 DNs again. let's start from first again.
this scenario is what I did. my cluster is working well. ;)
thanks for all.
2013. 4. 3., 오후 5:18, Azuryy Yu <az...@gmail.com> 작성:
> not at all. so don't worry about that.
>
>
> On Wed, Apr 3, 2013 at 2:04 PM, Yanbo Liang <ya...@gmail.com> wrote:
> It means that may be some replicas will be stay in under replica state?
>
>
> 2013/4/3 Azuryy Yu <az...@gmail.com>
> bq. then namenode start to copy block replicates on DN-2 to another DN, supposed DN-2.
>
> sorry for typo.
>
> Correct for it:
> then namenode start to copy block replicates on DN-1 to another DN, supposed DN-2.
>
>
> On Wed, Apr 3, 2013 at 9:51 AM, Azuryy Yu <az...@gmail.com> wrote:
> It's different.
> If you just want to stop DN-1 a short time, just kill the DataNode process on DN-1. then do what you want. during this time, Namenode cannot receive the heart beat from DN-1, then namenode start to copy block replicates on DN-2 to another DN, supposed DN-2.
>
> But when you start DN-1 again, Namenode receive the DN-1 registration, then namenode stop to copy the DN-1's block replicates even if NN doesn't finish coping.
>
> Am I explain clearly?
>
>
>
> On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <he...@gmail.com> wrote:
> @Harsh
>
> What's the reasons to make big gaps for removing nodes between decommission and just down nodes?
> In my understanding, both are necessary to copy un-replicated blocks to another alive nodes.
> If main costs of them are this one, total elapsed time couldn't be big different.
>
> Could you share some articles or documents to understand about decommissioning procedures?
> - explaining is always thanks ;)
>
>
> 2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
>
> > Yes, you can do the downtime work in steps of 2 DNs at a time,
> > especially since you mentioned the total work would be only ~30mins at
> > most.
> >
> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
> > <he...@gmail.com> wrote:
> >> the rest of nodes to be alive has enough size to store.
> >>
> >> for this one that you've mentioned.
> >>> its easier to do so in a rolling manner without need of a
> >>> decommission.
> >>
> >> to check my understanding, just shutting down 2 of them and then 2 more and then 2 more without decommissions.
> >>
> >> is this correct?
> >>
> >>
> >> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
> >>
> >>> Note though that its only possible to decommission 7 nodes at the same
> >>> time and expect it to finish iff the remaining 8 nodes have adequate
> >>> free space for the excess replicas.
> >>>
> >>> If you're just going to take them down for a short while (few mins
> >>> each), its easier to do so in a rolling manner without need of a
> >>> decommission. You can take upto two down at a time on a replication
> >>> average of 3 or 3+, and put it back in later without too much data
> >>> movement impact.
> >>>
> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
> >>>> It's reasonable to decommission 7 nodes at the same time.
> >>>> But may be it also takes long time to finish it.
> >>>> Because all the replicas in these 7 nodes need to be copied to remaining 8
> >>>> nodes.
> >>>> The size of transfer from these nodes to the remaining nodes is equal.
> >>>>
> >>>>
> >>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
> >>>>>
> >>>>> :)
> >>>>>
> >>>>> currently, I have 15 data nodes.
> >>>>> for some tests, I am trying to decommission until 8 nodes.
> >>>>>
> >>>>> Now, the total dfs used size is 52 TB which is including all replicated
> >>>>> blocks.
> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
> >>>>>
> >>>>> someone mentioned that I don't need to decommission node by node.
> >>>>> for this case, is there no problems if I decommissioned 7 nodes at the
> >>>>> same time?
> >>>>>
> >>>>>
> >>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
> >>>>>
> >>>>> I can translate it to native English: how many nodes you want to
> >>>>> decommission?
> >>>>>
> >>>>>
> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
> >>>>>>
> >>>>>> You want to decommission how many nodes?
> >>>>>>
> >>>>>>
> >>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
> >>>>>>>
> >>>>>>> 15 for datanodes and 3 for replication factor.
> >>>>>>>
> >>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
> >>>>>>>
> >>>>>>>> How many nodes do you have and replication factor for it.
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Harsh J
> >>
> >
> >
> >
> > --
> > Harsh J
>
>
>
>
>
Re: are we able to decommission multi nodes at one time?
Posted by Henry Junyoung Kim <he...@gmail.com>.
thanks for all.
my strategies for removing from 15 DN to 8 DN.
1. kill two DNs at same time. : NN will detect nodes' down and he will try to retain replication factors of lost blocks.
2. check your NN web UI. there is an info to let you know counts of under-replicated blocks.
3. if it is returned to 0 again, you can do remove other 2 DNs again. let's start from first again.
this scenario is what I did. my cluster is working well. ;)
thanks for all.
2013. 4. 3., 오후 5:18, Azuryy Yu <az...@gmail.com> 작성:
> not at all. so don't worry about that.
>
>
> On Wed, Apr 3, 2013 at 2:04 PM, Yanbo Liang <ya...@gmail.com> wrote:
> It means that may be some replicas will be stay in under replica state?
>
>
> 2013/4/3 Azuryy Yu <az...@gmail.com>
> bq. then namenode start to copy block replicates on DN-2 to another DN, supposed DN-2.
>
> sorry for typo.
>
> Correct for it:
> then namenode start to copy block replicates on DN-1 to another DN, supposed DN-2.
>
>
> On Wed, Apr 3, 2013 at 9:51 AM, Azuryy Yu <az...@gmail.com> wrote:
> It's different.
> If you just want to stop DN-1 a short time, just kill the DataNode process on DN-1. then do what you want. during this time, Namenode cannot receive the heart beat from DN-1, then namenode start to copy block replicates on DN-2 to another DN, supposed DN-2.
>
> But when you start DN-1 again, Namenode receive the DN-1 registration, then namenode stop to copy the DN-1's block replicates even if NN doesn't finish coping.
>
> Am I explain clearly?
>
>
>
> On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <he...@gmail.com> wrote:
> @Harsh
>
> What's the reasons to make big gaps for removing nodes between decommission and just down nodes?
> In my understanding, both are necessary to copy un-replicated blocks to another alive nodes.
> If main costs of them are this one, total elapsed time couldn't be big different.
>
> Could you share some articles or documents to understand about decommissioning procedures?
> - explaining is always thanks ;)
>
>
> 2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
>
> > Yes, you can do the downtime work in steps of 2 DNs at a time,
> > especially since you mentioned the total work would be only ~30mins at
> > most.
> >
> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
> > <he...@gmail.com> wrote:
> >> the rest of nodes to be alive has enough size to store.
> >>
> >> for this one that you've mentioned.
> >>> its easier to do so in a rolling manner without need of a
> >>> decommission.
> >>
> >> to check my understanding, just shutting down 2 of them and then 2 more and then 2 more without decommissions.
> >>
> >> is this correct?
> >>
> >>
> >> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
> >>
> >>> Note though that its only possible to decommission 7 nodes at the same
> >>> time and expect it to finish iff the remaining 8 nodes have adequate
> >>> free space for the excess replicas.
> >>>
> >>> If you're just going to take them down for a short while (few mins
> >>> each), its easier to do so in a rolling manner without need of a
> >>> decommission. You can take upto two down at a time on a replication
> >>> average of 3 or 3+, and put it back in later without too much data
> >>> movement impact.
> >>>
> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
> >>>> It's reasonable to decommission 7 nodes at the same time.
> >>>> But may be it also takes long time to finish it.
> >>>> Because all the replicas in these 7 nodes need to be copied to remaining 8
> >>>> nodes.
> >>>> The size of transfer from these nodes to the remaining nodes is equal.
> >>>>
> >>>>
> >>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
> >>>>>
> >>>>> :)
> >>>>>
> >>>>> currently, I have 15 data nodes.
> >>>>> for some tests, I am trying to decommission until 8 nodes.
> >>>>>
> >>>>> Now, the total dfs used size is 52 TB which is including all replicated
> >>>>> blocks.
> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
> >>>>>
> >>>>> someone mentioned that I don't need to decommission node by node.
> >>>>> for this case, is there no problems if I decommissioned 7 nodes at the
> >>>>> same time?
> >>>>>
> >>>>>
> >>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
> >>>>>
> >>>>> I can translate it to native English: how many nodes you want to
> >>>>> decommission?
> >>>>>
> >>>>>
> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
> >>>>>>
> >>>>>> You want to decommission how many nodes?
> >>>>>>
> >>>>>>
> >>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
> >>>>>>>
> >>>>>>> 15 for datanodes and 3 for replication factor.
> >>>>>>>
> >>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
> >>>>>>>
> >>>>>>>> How many nodes do you have and replication factor for it.
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Harsh J
> >>
> >
> >
> >
> > --
> > Harsh J
>
>
>
>
>
Re: are we able to decommission multi nodes at one time?
Posted by Azuryy Yu <az...@gmail.com>.
not at all. so don't worry about that.
On Wed, Apr 3, 2013 at 2:04 PM, Yanbo Liang <ya...@gmail.com> wrote:
> It means that may be some replicas will be stay in under replica state?
>
>
> 2013/4/3 Azuryy Yu <az...@gmail.com>
>
>> bq. then namenode start to copy block replicates on DN-2 to another DN,
>> supposed DN-2.
>>
>> sorry for typo.
>>
>> Correct for it:
>> then namenode start to copy block replicates on DN-1 to another DN,
>> supposed DN-2.
>>
>>
>> On Wed, Apr 3, 2013 at 9:51 AM, Azuryy Yu <az...@gmail.com> wrote:
>>
>>> It's different.
>>> If you just want to stop DN-1 a short time, just kill the DataNode
>>> process on DN-1. then do what you want. during this time, Namenode cannot
>>> receive the heart beat from DN-1, then namenode start to copy block
>>> replicates on DN-2 to another DN, supposed DN-2.
>>>
>>> But when you start DN-1 again, Namenode receive the DN-1 registration,
>>> then namenode stop to copy the DN-1's block replicates even if NN doesn't
>>> finish coping.
>>>
>>> Am I explain clearly?
>>>
>>>
>>>
>>> On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <
>>> henry.jykim@gmail.com> wrote:
>>>
>>>> @Harsh
>>>>
>>>> What's the reasons to make big gaps for removing nodes between
>>>> decommission and just down nodes?
>>>> In my understanding, both are necessary to copy un-replicated blocks to
>>>> another alive nodes.
>>>> If main costs of them are this one, total elapsed time couldn't be big
>>>> different.
>>>>
>>>> Could you share some articles or documents to understand about
>>>> decommissioning procedures?
>>>> - explaining is always thanks ;)
>>>>
>>>>
>>>> 2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
>>>>
>>>> > Yes, you can do the downtime work in steps of 2 DNs at a time,
>>>> > especially since you mentioned the total work would be only ~30mins at
>>>> > most.
>>>> >
>>>> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
>>>> > <he...@gmail.com> wrote:
>>>> >> the rest of nodes to be alive has enough size to store.
>>>> >>
>>>> >> for this one that you've mentioned.
>>>> >>> its easier to do so in a rolling manner without need of a
>>>> >>> decommission.
>>>> >>
>>>> >> to check my understanding, just shutting down 2 of them and then 2
>>>> more and then 2 more without decommissions.
>>>> >>
>>>> >> is this correct?
>>>> >>
>>>> >>
>>>> >> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>>>> >>
>>>> >>> Note though that its only possible to decommission 7 nodes at the
>>>> same
>>>> >>> time and expect it to finish iff the remaining 8 nodes have adequate
>>>> >>> free space for the excess replicas.
>>>> >>>
>>>> >>> If you're just going to take them down for a short while (few mins
>>>> >>> each), its easier to do so in a rolling manner without need of a
>>>> >>> decommission. You can take upto two down at a time on a replication
>>>> >>> average of 3 or 3+, and put it back in later without too much data
>>>> >>> movement impact.
>>>> >>>
>>>> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com>
>>>> wrote:
>>>> >>>> It's reasonable to decommission 7 nodes at the same time.
>>>> >>>> But may be it also takes long time to finish it.
>>>> >>>> Because all the replicas in these 7 nodes need to be copied to
>>>> remaining 8
>>>> >>>> nodes.
>>>> >>>> The size of transfer from these nodes to the remaining nodes is
>>>> equal.
>>>> >>>>
>>>> >>>>
>>>> >>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>>> >>>>>
>>>> >>>>> :)
>>>> >>>>>
>>>> >>>>> currently, I have 15 data nodes.
>>>> >>>>> for some tests, I am trying to decommission until 8 nodes.
>>>> >>>>>
>>>> >>>>> Now, the total dfs used size is 52 TB which is including all
>>>> replicated
>>>> >>>>> blocks.
>>>> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>> >>>>>
>>>> >>>>> someone mentioned that I don't need to decommission node by node.
>>>> >>>>> for this case, is there no problems if I decommissioned 7 nodes
>>>> at the
>>>> >>>>> same time?
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>>> >>>>>
>>>> >>>>> I can translate it to native English: how many nodes you want to
>>>> >>>>> decommission?
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <
>>>> yanbohappy@gmail.com> wrote:
>>>> >>>>>>
>>>> >>>>>> You want to decommission how many nodes?
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>> >>>>>>>
>>>> >>>>>>> 15 for datanodes and 3 for replication factor.
>>>> >>>>>>>
>>>> >>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>> >>>>>>>
>>>> >>>>>>>> How many nodes do you have and replication factor for it.
>>>> >>>>>>>
>>>> >>>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Harsh J
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Harsh J
>>>>
>>>>
>>>
>>
>
Re: are we able to decommission multi nodes at one time?
Posted by Azuryy Yu <az...@gmail.com>.
not at all. so don't worry about that.
On Wed, Apr 3, 2013 at 2:04 PM, Yanbo Liang <ya...@gmail.com> wrote:
> It means that may be some replicas will be stay in under replica state?
>
>
> 2013/4/3 Azuryy Yu <az...@gmail.com>
>
>> bq. then namenode start to copy block replicates on DN-2 to another DN,
>> supposed DN-2.
>>
>> sorry for typo.
>>
>> Correct for it:
>> then namenode start to copy block replicates on DN-1 to another DN,
>> supposed DN-2.
>>
>>
>> On Wed, Apr 3, 2013 at 9:51 AM, Azuryy Yu <az...@gmail.com> wrote:
>>
>>> It's different.
>>> If you just want to stop DN-1 a short time, just kill the DataNode
>>> process on DN-1. then do what you want. during this time, Namenode cannot
>>> receive the heart beat from DN-1, then namenode start to copy block
>>> replicates on DN-2 to another DN, supposed DN-2.
>>>
>>> But when you start DN-1 again, Namenode receive the DN-1 registration,
>>> then namenode stop to copy the DN-1's block replicates even if NN doesn't
>>> finish coping.
>>>
>>> Am I explain clearly?
>>>
>>>
>>>
>>> On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <
>>> henry.jykim@gmail.com> wrote:
>>>
>>>> @Harsh
>>>>
>>>> What's the reasons to make big gaps for removing nodes between
>>>> decommission and just down nodes?
>>>> In my understanding, both are necessary to copy un-replicated blocks to
>>>> another alive nodes.
>>>> If main costs of them are this one, total elapsed time couldn't be big
>>>> different.
>>>>
>>>> Could you share some articles or documents to understand about
>>>> decommissioning procedures?
>>>> - explaining is always thanks ;)
>>>>
>>>>
>>>> 2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
>>>>
>>>> > Yes, you can do the downtime work in steps of 2 DNs at a time,
>>>> > especially since you mentioned the total work would be only ~30mins at
>>>> > most.
>>>> >
>>>> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
>>>> > <he...@gmail.com> wrote:
>>>> >> the rest of nodes to be alive has enough size to store.
>>>> >>
>>>> >> for this one that you've mentioned.
>>>> >>> its easier to do so in a rolling manner without need of a
>>>> >>> decommission.
>>>> >>
>>>> >> to check my understanding, just shutting down 2 of them and then 2
>>>> more and then 2 more without decommissions.
>>>> >>
>>>> >> is this correct?
>>>> >>
>>>> >>
>>>> >> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>>>> >>
>>>> >>> Note though that its only possible to decommission 7 nodes at the
>>>> same
>>>> >>> time and expect it to finish iff the remaining 8 nodes have adequate
>>>> >>> free space for the excess replicas.
>>>> >>>
>>>> >>> If you're just going to take them down for a short while (few mins
>>>> >>> each), its easier to do so in a rolling manner without need of a
>>>> >>> decommission. You can take upto two down at a time on a replication
>>>> >>> average of 3 or 3+, and put it back in later without too much data
>>>> >>> movement impact.
>>>> >>>
>>>> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com>
>>>> wrote:
>>>> >>>> It's reasonable to decommission 7 nodes at the same time.
>>>> >>>> But may be it also takes long time to finish it.
>>>> >>>> Because all the replicas in these 7 nodes need to be copied to
>>>> remaining 8
>>>> >>>> nodes.
>>>> >>>> The size of transfer from these nodes to the remaining nodes is
>>>> equal.
>>>> >>>>
>>>> >>>>
>>>> >>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>>> >>>>>
>>>> >>>>> :)
>>>> >>>>>
>>>> >>>>> currently, I have 15 data nodes.
>>>> >>>>> for some tests, I am trying to decommission until 8 nodes.
>>>> >>>>>
>>>> >>>>> Now, the total dfs used size is 52 TB which is including all
>>>> replicated
>>>> >>>>> blocks.
>>>> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>> >>>>>
>>>> >>>>> someone mentioned that I don't need to decommission node by node.
>>>> >>>>> for this case, is there no problems if I decommissioned 7 nodes
>>>> at the
>>>> >>>>> same time?
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>>> >>>>>
>>>> >>>>> I can translate it to native English: how many nodes you want to
>>>> >>>>> decommission?
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <
>>>> yanbohappy@gmail.com> wrote:
>>>> >>>>>>
>>>> >>>>>> You want to decommission how many nodes?
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>> >>>>>>>
>>>> >>>>>>> 15 for datanodes and 3 for replication factor.
>>>> >>>>>>>
>>>> >>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>> >>>>>>>
>>>> >>>>>>>> How many nodes do you have and replication factor for it.
>>>> >>>>>>>
>>>> >>>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Harsh J
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Harsh J
>>>>
>>>>
>>>
>>
>
Re: are we able to decommission multi nodes at one time?
Posted by Azuryy Yu <az...@gmail.com>.
not at all. so don't worry about that.
On Wed, Apr 3, 2013 at 2:04 PM, Yanbo Liang <ya...@gmail.com> wrote:
> It means that may be some replicas will be stay in under replica state?
>
>
> 2013/4/3 Azuryy Yu <az...@gmail.com>
>
>> bq. then namenode start to copy block replicates on DN-2 to another DN,
>> supposed DN-2.
>>
>> sorry for typo.
>>
>> Correct for it:
>> then namenode start to copy block replicates on DN-1 to another DN,
>> supposed DN-2.
>>
>>
>> On Wed, Apr 3, 2013 at 9:51 AM, Azuryy Yu <az...@gmail.com> wrote:
>>
>>> It's different.
>>> If you just want to stop DN-1 a short time, just kill the DataNode
>>> process on DN-1. then do what you want. during this time, Namenode cannot
>>> receive the heart beat from DN-1, then namenode start to copy block
>>> replicates on DN-2 to another DN, supposed DN-2.
>>>
>>> But when you start DN-1 again, Namenode receive the DN-1 registration,
>>> then namenode stop to copy the DN-1's block replicates even if NN doesn't
>>> finish coping.
>>>
>>> Am I explain clearly?
>>>
>>>
>>>
>>> On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <
>>> henry.jykim@gmail.com> wrote:
>>>
>>>> @Harsh
>>>>
>>>> What's the reasons to make big gaps for removing nodes between
>>>> decommission and just down nodes?
>>>> In my understanding, both are necessary to copy un-replicated blocks to
>>>> another alive nodes.
>>>> If main costs of them are this one, total elapsed time couldn't be big
>>>> different.
>>>>
>>>> Could you share some articles or documents to understand about
>>>> decommissioning procedures?
>>>> - explaining is always thanks ;)
>>>>
>>>>
>>>> 2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
>>>>
>>>> > Yes, you can do the downtime work in steps of 2 DNs at a time,
>>>> > especially since you mentioned the total work would be only ~30mins at
>>>> > most.
>>>> >
>>>> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
>>>> > <he...@gmail.com> wrote:
>>>> >> the rest of nodes to be alive has enough size to store.
>>>> >>
>>>> >> for this one that you've mentioned.
>>>> >>> its easier to do so in a rolling manner without need of a
>>>> >>> decommission.
>>>> >>
>>>> >> to check my understanding, just shutting down 2 of them and then 2
>>>> more and then 2 more without decommissions.
>>>> >>
>>>> >> is this correct?
>>>> >>
>>>> >>
>>>> >> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>>>> >>
>>>> >>> Note though that its only possible to decommission 7 nodes at the
>>>> same
>>>> >>> time and expect it to finish iff the remaining 8 nodes have adequate
>>>> >>> free space for the excess replicas.
>>>> >>>
>>>> >>> If you're just going to take them down for a short while (few mins
>>>> >>> each), its easier to do so in a rolling manner without need of a
>>>> >>> decommission. You can take upto two down at a time on a replication
>>>> >>> average of 3 or 3+, and put it back in later without too much data
>>>> >>> movement impact.
>>>> >>>
>>>> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com>
>>>> wrote:
>>>> >>>> It's reasonable to decommission 7 nodes at the same time.
>>>> >>>> But may be it also takes long time to finish it.
>>>> >>>> Because all the replicas in these 7 nodes need to be copied to
>>>> remaining 8
>>>> >>>> nodes.
>>>> >>>> The size of transfer from these nodes to the remaining nodes is
>>>> equal.
>>>> >>>>
>>>> >>>>
>>>> >>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>>> >>>>>
>>>> >>>>> :)
>>>> >>>>>
>>>> >>>>> currently, I have 15 data nodes.
>>>> >>>>> for some tests, I am trying to decommission until 8 nodes.
>>>> >>>>>
>>>> >>>>> Now, the total dfs used size is 52 TB which is including all
>>>> replicated
>>>> >>>>> blocks.
>>>> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>> >>>>>
>>>> >>>>> someone mentioned that I don't need to decommission node by node.
>>>> >>>>> for this case, is there no problems if I decommissioned 7 nodes
>>>> at the
>>>> >>>>> same time?
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>>> >>>>>
>>>> >>>>> I can translate it to native English: how many nodes you want to
>>>> >>>>> decommission?
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <
>>>> yanbohappy@gmail.com> wrote:
>>>> >>>>>>
>>>> >>>>>> You want to decommission how many nodes?
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>> >>>>>>>
>>>> >>>>>>> 15 for datanodes and 3 for replication factor.
>>>> >>>>>>>
>>>> >>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>> >>>>>>>
>>>> >>>>>>>> How many nodes do you have and replication factor for it.
>>>> >>>>>>>
>>>> >>>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Harsh J
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Harsh J
>>>>
>>>>
>>>
>>
>
Re: are we able to decommission multi nodes at one time?
Posted by Azuryy Yu <az...@gmail.com>.
not at all. so don't worry about that.
On Wed, Apr 3, 2013 at 2:04 PM, Yanbo Liang <ya...@gmail.com> wrote:
> It means that may be some replicas will be stay in under replica state?
>
>
> 2013/4/3 Azuryy Yu <az...@gmail.com>
>
>> bq. then namenode start to copy block replicates on DN-2 to another DN,
>> supposed DN-2.
>>
>> sorry for typo.
>>
>> Correct for it:
>> then namenode start to copy block replicates on DN-1 to another DN,
>> supposed DN-2.
>>
>>
>> On Wed, Apr 3, 2013 at 9:51 AM, Azuryy Yu <az...@gmail.com> wrote:
>>
>>> It's different.
>>> If you just want to stop DN-1 a short time, just kill the DataNode
>>> process on DN-1. then do what you want. during this time, Namenode cannot
>>> receive the heart beat from DN-1, then namenode start to copy block
>>> replicates on DN-2 to another DN, supposed DN-2.
>>>
>>> But when you start DN-1 again, Namenode receive the DN-1 registration,
>>> then namenode stop to copy the DN-1's block replicates even if NN doesn't
>>> finish coping.
>>>
>>> Am I explain clearly?
>>>
>>>
>>>
>>> On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <
>>> henry.jykim@gmail.com> wrote:
>>>
>>>> @Harsh
>>>>
>>>> What's the reasons to make big gaps for removing nodes between
>>>> decommission and just down nodes?
>>>> In my understanding, both are necessary to copy un-replicated blocks to
>>>> another alive nodes.
>>>> If main costs of them are this one, total elapsed time couldn't be big
>>>> different.
>>>>
>>>> Could you share some articles or documents to understand about
>>>> decommissioning procedures?
>>>> - explaining is always thanks ;)
>>>>
>>>>
>>>> 2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
>>>>
>>>> > Yes, you can do the downtime work in steps of 2 DNs at a time,
>>>> > especially since you mentioned the total work would be only ~30mins at
>>>> > most.
>>>> >
>>>> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
>>>> > <he...@gmail.com> wrote:
>>>> >> the rest of nodes to be alive has enough size to store.
>>>> >>
>>>> >> for this one that you've mentioned.
>>>> >>> its easier to do so in a rolling manner without need of a
>>>> >>> decommission.
>>>> >>
>>>> >> to check my understanding, just shutting down 2 of them and then 2
>>>> more and then 2 more without decommissions.
>>>> >>
>>>> >> is this correct?
>>>> >>
>>>> >>
>>>> >> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>>>> >>
>>>> >>> Note though that its only possible to decommission 7 nodes at the
>>>> same
>>>> >>> time and expect it to finish iff the remaining 8 nodes have adequate
>>>> >>> free space for the excess replicas.
>>>> >>>
>>>> >>> If you're just going to take them down for a short while (few mins
>>>> >>> each), its easier to do so in a rolling manner without need of a
>>>> >>> decommission. You can take upto two down at a time on a replication
>>>> >>> average of 3 or 3+, and put it back in later without too much data
>>>> >>> movement impact.
>>>> >>>
>>>> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com>
>>>> wrote:
>>>> >>>> It's reasonable to decommission 7 nodes at the same time.
>>>> >>>> But may be it also takes long time to finish it.
>>>> >>>> Because all the replicas in these 7 nodes need to be copied to
>>>> remaining 8
>>>> >>>> nodes.
>>>> >>>> The size of transfer from these nodes to the remaining nodes is
>>>> equal.
>>>> >>>>
>>>> >>>>
>>>> >>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>>> >>>>>
>>>> >>>>> :)
>>>> >>>>>
>>>> >>>>> currently, I have 15 data nodes.
>>>> >>>>> for some tests, I am trying to decommission until 8 nodes.
>>>> >>>>>
>>>> >>>>> Now, the total dfs used size is 52 TB which is including all
>>>> replicated
>>>> >>>>> blocks.
>>>> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>> >>>>>
>>>> >>>>> someone mentioned that I don't need to decommission node by node.
>>>> >>>>> for this case, is there no problems if I decommissioned 7 nodes
>>>> at the
>>>> >>>>> same time?
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>>> >>>>>
>>>> >>>>> I can translate it to native English: how many nodes you want to
>>>> >>>>> decommission?
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <
>>>> yanbohappy@gmail.com> wrote:
>>>> >>>>>>
>>>> >>>>>> You want to decommission how many nodes?
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>> >>>>>>>
>>>> >>>>>>> 15 for datanodes and 3 for replication factor.
>>>> >>>>>>>
>>>> >>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>> >>>>>>>
>>>> >>>>>>>> How many nodes do you have and replication factor for it.
>>>> >>>>>>>
>>>> >>>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Harsh J
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Harsh J
>>>>
>>>>
>>>
>>
>
Re: are we able to decommission multi nodes at one time?
Posted by Yanbo Liang <ya...@gmail.com>.
It means that may be some replicas will be stay in under replica state?
2013/4/3 Azuryy Yu <az...@gmail.com>
> bq. then namenode start to copy block replicates on DN-2 to another DN,
> supposed DN-2.
>
> sorry for typo.
>
> Correct for it:
> then namenode start to copy block replicates on DN-1 to another DN,
> supposed DN-2.
>
>
> On Wed, Apr 3, 2013 at 9:51 AM, Azuryy Yu <az...@gmail.com> wrote:
>
>> It's different.
>> If you just want to stop DN-1 a short time, just kill the DataNode
>> process on DN-1. then do what you want. during this time, Namenode cannot
>> receive the heart beat from DN-1, then namenode start to copy block
>> replicates on DN-2 to another DN, supposed DN-2.
>>
>> But when you start DN-1 again, Namenode receive the DN-1 registration,
>> then namenode stop to copy the DN-1's block replicates even if NN doesn't
>> finish coping.
>>
>> Am I explain clearly?
>>
>>
>>
>> On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <henry.jykim@gmail.com
>> > wrote:
>>
>>> @Harsh
>>>
>>> What's the reasons to make big gaps for removing nodes between
>>> decommission and just down nodes?
>>> In my understanding, both are necessary to copy un-replicated blocks to
>>> another alive nodes.
>>> If main costs of them are this one, total elapsed time couldn't be big
>>> different.
>>>
>>> Could you share some articles or documents to understand about
>>> decommissioning procedures?
>>> - explaining is always thanks ;)
>>>
>>>
>>> 2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
>>>
>>> > Yes, you can do the downtime work in steps of 2 DNs at a time,
>>> > especially since you mentioned the total work would be only ~30mins at
>>> > most.
>>> >
>>> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
>>> > <he...@gmail.com> wrote:
>>> >> the rest of nodes to be alive has enough size to store.
>>> >>
>>> >> for this one that you've mentioned.
>>> >>> its easier to do so in a rolling manner without need of a
>>> >>> decommission.
>>> >>
>>> >> to check my understanding, just shutting down 2 of them and then 2
>>> more and then 2 more without decommissions.
>>> >>
>>> >> is this correct?
>>> >>
>>> >>
>>> >> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>>> >>
>>> >>> Note though that its only possible to decommission 7 nodes at the
>>> same
>>> >>> time and expect it to finish iff the remaining 8 nodes have adequate
>>> >>> free space for the excess replicas.
>>> >>>
>>> >>> If you're just going to take them down for a short while (few mins
>>> >>> each), its easier to do so in a rolling manner without need of a
>>> >>> decommission. You can take upto two down at a time on a replication
>>> >>> average of 3 or 3+, and put it back in later without too much data
>>> >>> movement impact.
>>> >>>
>>> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com>
>>> wrote:
>>> >>>> It's reasonable to decommission 7 nodes at the same time.
>>> >>>> But may be it also takes long time to finish it.
>>> >>>> Because all the replicas in these 7 nodes need to be copied to
>>> remaining 8
>>> >>>> nodes.
>>> >>>> The size of transfer from these nodes to the remaining nodes is
>>> equal.
>>> >>>>
>>> >>>>
>>> >>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>> >>>>>
>>> >>>>> :)
>>> >>>>>
>>> >>>>> currently, I have 15 data nodes.
>>> >>>>> for some tests, I am trying to decommission until 8 nodes.
>>> >>>>>
>>> >>>>> Now, the total dfs used size is 52 TB which is including all
>>> replicated
>>> >>>>> blocks.
>>> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>> >>>>>
>>> >>>>> someone mentioned that I don't need to decommission node by node.
>>> >>>>> for this case, is there no problems if I decommissioned 7 nodes at
>>> the
>>> >>>>> same time?
>>> >>>>>
>>> >>>>>
>>> >>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>> >>>>>
>>> >>>>> I can translate it to native English: how many nodes you want to
>>> >>>>> decommission?
>>> >>>>>
>>> >>>>>
>>> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com>
>>> wrote:
>>> >>>>>>
>>> >>>>>> You want to decommission how many nodes?
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>> >>>>>>>
>>> >>>>>>> 15 for datanodes and 3 for replication factor.
>>> >>>>>>>
>>> >>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>> >>>>>>>
>>> >>>>>>>> How many nodes do you have and replication factor for it.
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Harsh J
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Harsh J
>>>
>>>
>>
>
Re: are we able to decommission multi nodes at one time?
Posted by Yanbo Liang <ya...@gmail.com>.
It means that may be some replicas will be stay in under replica state?
2013/4/3 Azuryy Yu <az...@gmail.com>
> bq. then namenode start to copy block replicates on DN-2 to another DN,
> supposed DN-2.
>
> sorry for typo.
>
> Correct for it:
> then namenode start to copy block replicates on DN-1 to another DN,
> supposed DN-2.
>
>
> On Wed, Apr 3, 2013 at 9:51 AM, Azuryy Yu <az...@gmail.com> wrote:
>
>> It's different.
>> If you just want to stop DN-1 a short time, just kill the DataNode
>> process on DN-1. then do what you want. during this time, Namenode cannot
>> receive the heart beat from DN-1, then namenode start to copy block
>> replicates on DN-2 to another DN, supposed DN-2.
>>
>> But when you start DN-1 again, Namenode receive the DN-1 registration,
>> then namenode stop to copy the DN-1's block replicates even if NN doesn't
>> finish coping.
>>
>> Am I explain clearly?
>>
>>
>>
>> On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <henry.jykim@gmail.com
>> > wrote:
>>
>>> @Harsh
>>>
>>> What's the reasons to make big gaps for removing nodes between
>>> decommission and just down nodes?
>>> In my understanding, both are necessary to copy un-replicated blocks to
>>> another alive nodes.
>>> If main costs of them are this one, total elapsed time couldn't be big
>>> different.
>>>
>>> Could you share some articles or documents to understand about
>>> decommissioning procedures?
>>> - explaining is always thanks ;)
>>>
>>>
>>> 2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
>>>
>>> > Yes, you can do the downtime work in steps of 2 DNs at a time,
>>> > especially since you mentioned the total work would be only ~30mins at
>>> > most.
>>> >
>>> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
>>> > <he...@gmail.com> wrote:
>>> >> the rest of nodes to be alive has enough size to store.
>>> >>
>>> >> for this one that you've mentioned.
>>> >>> its easier to do so in a rolling manner without need of a
>>> >>> decommission.
>>> >>
>>> >> to check my understanding, just shutting down 2 of them and then 2
>>> more and then 2 more without decommissions.
>>> >>
>>> >> is this correct?
>>> >>
>>> >>
>>> >> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>>> >>
>>> >>> Note though that its only possible to decommission 7 nodes at the
>>> same
>>> >>> time and expect it to finish iff the remaining 8 nodes have adequate
>>> >>> free space for the excess replicas.
>>> >>>
>>> >>> If you're just going to take them down for a short while (few mins
>>> >>> each), its easier to do so in a rolling manner without need of a
>>> >>> decommission. You can take upto two down at a time on a replication
>>> >>> average of 3 or 3+, and put it back in later without too much data
>>> >>> movement impact.
>>> >>>
>>> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com>
>>> wrote:
>>> >>>> It's reasonable to decommission 7 nodes at the same time.
>>> >>>> But may be it also takes long time to finish it.
>>> >>>> Because all the replicas in these 7 nodes need to be copied to
>>> remaining 8
>>> >>>> nodes.
>>> >>>> The size of transfer from these nodes to the remaining nodes is
>>> equal.
>>> >>>>
>>> >>>>
>>> >>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>> >>>>>
>>> >>>>> :)
>>> >>>>>
>>> >>>>> currently, I have 15 data nodes.
>>> >>>>> for some tests, I am trying to decommission until 8 nodes.
>>> >>>>>
>>> >>>>> Now, the total dfs used size is 52 TB which is including all
>>> replicated
>>> >>>>> blocks.
>>> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>> >>>>>
>>> >>>>> someone mentioned that I don't need to decommission node by node.
>>> >>>>> for this case, is there no problems if I decommissioned 7 nodes at
>>> the
>>> >>>>> same time?
>>> >>>>>
>>> >>>>>
>>> >>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>> >>>>>
>>> >>>>> I can translate it to native English: how many nodes you want to
>>> >>>>> decommission?
>>> >>>>>
>>> >>>>>
>>> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com>
>>> wrote:
>>> >>>>>>
>>> >>>>>> You want to decommission how many nodes?
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>> >>>>>>>
>>> >>>>>>> 15 for datanodes and 3 for replication factor.
>>> >>>>>>>
>>> >>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>> >>>>>>>
>>> >>>>>>>> How many nodes do you have and replication factor for it.
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Harsh J
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Harsh J
>>>
>>>
>>
>
Re: are we able to decommission multi nodes at one time?
Posted by Yanbo Liang <ya...@gmail.com>.
It means that may be some replicas will be stay in under replica state?
2013/4/3 Azuryy Yu <az...@gmail.com>
> bq. then namenode start to copy block replicates on DN-2 to another DN,
> supposed DN-2.
>
> sorry for typo.
>
> Correct for it:
> then namenode start to copy block replicates on DN-1 to another DN,
> supposed DN-2.
>
>
> On Wed, Apr 3, 2013 at 9:51 AM, Azuryy Yu <az...@gmail.com> wrote:
>
>> It's different.
>> If you just want to stop DN-1 a short time, just kill the DataNode
>> process on DN-1. then do what you want. during this time, Namenode cannot
>> receive the heart beat from DN-1, then namenode start to copy block
>> replicates on DN-2 to another DN, supposed DN-2.
>>
>> But when you start DN-1 again, Namenode receive the DN-1 registration,
>> then namenode stop to copy the DN-1's block replicates even if NN doesn't
>> finish coping.
>>
>> Am I explain clearly?
>>
>>
>>
>> On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <henry.jykim@gmail.com
>> > wrote:
>>
>>> @Harsh
>>>
>>> What's the reasons to make big gaps for removing nodes between
>>> decommission and just down nodes?
>>> In my understanding, both are necessary to copy un-replicated blocks to
>>> another alive nodes.
>>> If main costs of them are this one, total elapsed time couldn't be big
>>> different.
>>>
>>> Could you share some articles or documents to understand about
>>> decommissioning procedures?
>>> - explaining is always thanks ;)
>>>
>>>
>>> 2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
>>>
>>> > Yes, you can do the downtime work in steps of 2 DNs at a time,
>>> > especially since you mentioned the total work would be only ~30mins at
>>> > most.
>>> >
>>> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
>>> > <he...@gmail.com> wrote:
>>> >> the rest of nodes to be alive has enough size to store.
>>> >>
>>> >> for this one that you've mentioned.
>>> >>> its easier to do so in a rolling manner without need of a
>>> >>> decommission.
>>> >>
>>> >> to check my understanding, just shutting down 2 of them and then 2
>>> more and then 2 more without decommissions.
>>> >>
>>> >> is this correct?
>>> >>
>>> >>
>>> >> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>>> >>
>>> >>> Note though that its only possible to decommission 7 nodes at the
>>> same
>>> >>> time and expect it to finish iff the remaining 8 nodes have adequate
>>> >>> free space for the excess replicas.
>>> >>>
>>> >>> If you're just going to take them down for a short while (few mins
>>> >>> each), its easier to do so in a rolling manner without need of a
>>> >>> decommission. You can take upto two down at a time on a replication
>>> >>> average of 3 or 3+, and put it back in later without too much data
>>> >>> movement impact.
>>> >>>
>>> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com>
>>> wrote:
>>> >>>> It's reasonable to decommission 7 nodes at the same time.
>>> >>>> But may be it also takes long time to finish it.
>>> >>>> Because all the replicas in these 7 nodes need to be copied to
>>> remaining 8
>>> >>>> nodes.
>>> >>>> The size of transfer from these nodes to the remaining nodes is
>>> equal.
>>> >>>>
>>> >>>>
>>> >>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>> >>>>>
>>> >>>>> :)
>>> >>>>>
>>> >>>>> currently, I have 15 data nodes.
>>> >>>>> for some tests, I am trying to decommission until 8 nodes.
>>> >>>>>
>>> >>>>> Now, the total dfs used size is 52 TB which is including all
>>> replicated
>>> >>>>> blocks.
>>> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>> >>>>>
>>> >>>>> someone mentioned that I don't need to decommission node by node.
>>> >>>>> for this case, is there no problems if I decommissioned 7 nodes at
>>> the
>>> >>>>> same time?
>>> >>>>>
>>> >>>>>
>>> >>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>> >>>>>
>>> >>>>> I can translate it to native English: how many nodes you want to
>>> >>>>> decommission?
>>> >>>>>
>>> >>>>>
>>> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com>
>>> wrote:
>>> >>>>>>
>>> >>>>>> You want to decommission how many nodes?
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>> >>>>>>>
>>> >>>>>>> 15 for datanodes and 3 for replication factor.
>>> >>>>>>>
>>> >>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>> >>>>>>>
>>> >>>>>>>> How many nodes do you have and replication factor for it.
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Harsh J
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Harsh J
>>>
>>>
>>
>
Re: are we able to decommission multi nodes at one time?
Posted by Yanbo Liang <ya...@gmail.com>.
It means that may be some replicas will be stay in under replica state?
2013/4/3 Azuryy Yu <az...@gmail.com>
> bq. then namenode start to copy block replicates on DN-2 to another DN,
> supposed DN-2.
>
> sorry for typo.
>
> Correct for it:
> then namenode start to copy block replicates on DN-1 to another DN,
> supposed DN-2.
>
>
> On Wed, Apr 3, 2013 at 9:51 AM, Azuryy Yu <az...@gmail.com> wrote:
>
>> It's different.
>> If you just want to stop DN-1 a short time, just kill the DataNode
>> process on DN-1. then do what you want. during this time, Namenode cannot
>> receive the heart beat from DN-1, then namenode start to copy block
>> replicates on DN-2 to another DN, supposed DN-2.
>>
>> But when you start DN-1 again, Namenode receive the DN-1 registration,
>> then namenode stop to copy the DN-1's block replicates even if NN doesn't
>> finish coping.
>>
>> Am I explain clearly?
>>
>>
>>
>> On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <henry.jykim@gmail.com
>> > wrote:
>>
>>> @Harsh
>>>
>>> What's the reasons to make big gaps for removing nodes between
>>> decommission and just down nodes?
>>> In my understanding, both are necessary to copy un-replicated blocks to
>>> another alive nodes.
>>> If main costs of them are this one, total elapsed time couldn't be big
>>> different.
>>>
>>> Could you share some articles or documents to understand about
>>> decommissioning procedures?
>>> - explaining is always thanks ;)
>>>
>>>
>>> 2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
>>>
>>> > Yes, you can do the downtime work in steps of 2 DNs at a time,
>>> > especially since you mentioned the total work would be only ~30mins at
>>> > most.
>>> >
>>> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
>>> > <he...@gmail.com> wrote:
>>> >> the rest of nodes to be alive has enough size to store.
>>> >>
>>> >> for this one that you've mentioned.
>>> >>> its easier to do so in a rolling manner without need of a
>>> >>> decommission.
>>> >>
>>> >> to check my understanding, just shutting down 2 of them and then 2
>>> more and then 2 more without decommissions.
>>> >>
>>> >> is this correct?
>>> >>
>>> >>
>>> >> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>>> >>
>>> >>> Note though that its only possible to decommission 7 nodes at the
>>> same
>>> >>> time and expect it to finish iff the remaining 8 nodes have adequate
>>> >>> free space for the excess replicas.
>>> >>>
>>> >>> If you're just going to take them down for a short while (few mins
>>> >>> each), its easier to do so in a rolling manner without need of a
>>> >>> decommission. You can take upto two down at a time on a replication
>>> >>> average of 3 or 3+, and put it back in later without too much data
>>> >>> movement impact.
>>> >>>
>>> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com>
>>> wrote:
>>> >>>> It's reasonable to decommission 7 nodes at the same time.
>>> >>>> But may be it also takes long time to finish it.
>>> >>>> Because all the replicas in these 7 nodes need to be copied to
>>> remaining 8
>>> >>>> nodes.
>>> >>>> The size of transfer from these nodes to the remaining nodes is
>>> equal.
>>> >>>>
>>> >>>>
>>> >>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>> >>>>>
>>> >>>>> :)
>>> >>>>>
>>> >>>>> currently, I have 15 data nodes.
>>> >>>>> for some tests, I am trying to decommission until 8 nodes.
>>> >>>>>
>>> >>>>> Now, the total dfs used size is 52 TB which is including all
>>> replicated
>>> >>>>> blocks.
>>> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>> >>>>>
>>> >>>>> someone mentioned that I don't need to decommission node by node.
>>> >>>>> for this case, is there no problems if I decommissioned 7 nodes at
>>> the
>>> >>>>> same time?
>>> >>>>>
>>> >>>>>
>>> >>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>> >>>>>
>>> >>>>> I can translate it to native English: how many nodes you want to
>>> >>>>> decommission?
>>> >>>>>
>>> >>>>>
>>> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com>
>>> wrote:
>>> >>>>>>
>>> >>>>>> You want to decommission how many nodes?
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>> >>>>>>>
>>> >>>>>>> 15 for datanodes and 3 for replication factor.
>>> >>>>>>>
>>> >>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>> >>>>>>>
>>> >>>>>>>> How many nodes do you have and replication factor for it.
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Harsh J
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Harsh J
>>>
>>>
>>
>
Re: are we able to decommission multi nodes at one time?
Posted by Azuryy Yu <az...@gmail.com>.
bq. then namenode start to copy block replicates on DN-2 to another DN,
supposed DN-2.
sorry for typo.
Correct for it:
then namenode start to copy block replicates on DN-1 to another DN,
supposed DN-2.
On Wed, Apr 3, 2013 at 9:51 AM, Azuryy Yu <az...@gmail.com> wrote:
> It's different.
> If you just want to stop DN-1 a short time, just kill the DataNode process
> on DN-1. then do what you want. during this time, Namenode cannot receive
> the heart beat from DN-1, then namenode start to copy block replicates on
> DN-2 to another DN, supposed DN-2.
>
> But when you start DN-1 again, Namenode receive the DN-1 registration,
> then namenode stop to copy the DN-1's block replicates even if NN doesn't
> finish coping.
>
> Am I explain clearly?
>
>
>
> On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <he...@gmail.com>wrote:
>
>> @Harsh
>>
>> What's the reasons to make big gaps for removing nodes between
>> decommission and just down nodes?
>> In my understanding, both are necessary to copy un-replicated blocks to
>> another alive nodes.
>> If main costs of them are this one, total elapsed time couldn't be big
>> different.
>>
>> Could you share some articles or documents to understand about
>> decommissioning procedures?
>> - explaining is always thanks ;)
>>
>>
>> 2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
>>
>> > Yes, you can do the downtime work in steps of 2 DNs at a time,
>> > especially since you mentioned the total work would be only ~30mins at
>> > most.
>> >
>> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
>> > <he...@gmail.com> wrote:
>> >> the rest of nodes to be alive has enough size to store.
>> >>
>> >> for this one that you've mentioned.
>> >>> its easier to do so in a rolling manner without need of a
>> >>> decommission.
>> >>
>> >> to check my understanding, just shutting down 2 of them and then 2
>> more and then 2 more without decommissions.
>> >>
>> >> is this correct?
>> >>
>> >>
>> >> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>> >>
>> >>> Note though that its only possible to decommission 7 nodes at the same
>> >>> time and expect it to finish iff the remaining 8 nodes have adequate
>> >>> free space for the excess replicas.
>> >>>
>> >>> If you're just going to take them down for a short while (few mins
>> >>> each), its easier to do so in a rolling manner without need of a
>> >>> decommission. You can take upto two down at a time on a replication
>> >>> average of 3 or 3+, and put it back in later without too much data
>> >>> movement impact.
>> >>>
>> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com>
>> wrote:
>> >>>> It's reasonable to decommission 7 nodes at the same time.
>> >>>> But may be it also takes long time to finish it.
>> >>>> Because all the replicas in these 7 nodes need to be copied to
>> remaining 8
>> >>>> nodes.
>> >>>> The size of transfer from these nodes to the remaining nodes is
>> equal.
>> >>>>
>> >>>>
>> >>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>> >>>>>
>> >>>>> :)
>> >>>>>
>> >>>>> currently, I have 15 data nodes.
>> >>>>> for some tests, I am trying to decommission until 8 nodes.
>> >>>>>
>> >>>>> Now, the total dfs used size is 52 TB which is including all
>> replicated
>> >>>>> blocks.
>> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>> >>>>>
>> >>>>> someone mentioned that I don't need to decommission node by node.
>> >>>>> for this case, is there no problems if I decommissioned 7 nodes at
>> the
>> >>>>> same time?
>> >>>>>
>> >>>>>
>> >>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>> >>>>>
>> >>>>> I can translate it to native English: how many nodes you want to
>> >>>>> decommission?
>> >>>>>
>> >>>>>
>> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com>
>> wrote:
>> >>>>>>
>> >>>>>> You want to decommission how many nodes?
>> >>>>>>
>> >>>>>>
>> >>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>> >>>>>>>
>> >>>>>>> 15 for datanodes and 3 for replication factor.
>> >>>>>>>
>> >>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>> >>>>>>>
>> >>>>>>>> How many nodes do you have and replication factor for it.
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Harsh J
>> >>
>> >
>> >
>> >
>> > --
>> > Harsh J
>>
>>
>
Re: are we able to decommission multi nodes at one time?
Posted by Azuryy Yu <az...@gmail.com>.
bq. then namenode start to copy block replicates on DN-2 to another DN,
supposed DN-2.
sorry for typo.
Correct for it:
then namenode start to copy block replicates on DN-1 to another DN,
supposed DN-2.
On Wed, Apr 3, 2013 at 9:51 AM, Azuryy Yu <az...@gmail.com> wrote:
> It's different.
> If you just want to stop DN-1 a short time, just kill the DataNode process
> on DN-1. then do what you want. during this time, Namenode cannot receive
> the heart beat from DN-1, then namenode start to copy block replicates on
> DN-2 to another DN, supposed DN-2.
>
> But when you start DN-1 again, Namenode receive the DN-1 registration,
> then namenode stop to copy the DN-1's block replicates even if NN doesn't
> finish coping.
>
> Am I explain clearly?
>
>
>
> On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <he...@gmail.com>wrote:
>
>> @Harsh
>>
>> What's the reasons to make big gaps for removing nodes between
>> decommission and just down nodes?
>> In my understanding, both are necessary to copy un-replicated blocks to
>> another alive nodes.
>> If main costs of them are this one, total elapsed time couldn't be big
>> different.
>>
>> Could you share some articles or documents to understand about
>> decommissioning procedures?
>> - explaining is always thanks ;)
>>
>>
>> 2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
>>
>> > Yes, you can do the downtime work in steps of 2 DNs at a time,
>> > especially since you mentioned the total work would be only ~30mins at
>> > most.
>> >
>> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
>> > <he...@gmail.com> wrote:
>> >> the rest of nodes to be alive has enough size to store.
>> >>
>> >> for this one that you've mentioned.
>> >>> its easier to do so in a rolling manner without need of a
>> >>> decommission.
>> >>
>> >> to check my understanding, just shutting down 2 of them and then 2
>> more and then 2 more without decommissions.
>> >>
>> >> is this correct?
>> >>
>> >>
>> >> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>> >>
>> >>> Note though that its only possible to decommission 7 nodes at the same
>> >>> time and expect it to finish iff the remaining 8 nodes have adequate
>> >>> free space for the excess replicas.
>> >>>
>> >>> If you're just going to take them down for a short while (few mins
>> >>> each), its easier to do so in a rolling manner without need of a
>> >>> decommission. You can take upto two down at a time on a replication
>> >>> average of 3 or 3+, and put it back in later without too much data
>> >>> movement impact.
>> >>>
>> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com>
>> wrote:
>> >>>> It's reasonable to decommission 7 nodes at the same time.
>> >>>> But may be it also takes long time to finish it.
>> >>>> Because all the replicas in these 7 nodes need to be copied to
>> remaining 8
>> >>>> nodes.
>> >>>> The size of transfer from these nodes to the remaining nodes is
>> equal.
>> >>>>
>> >>>>
>> >>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>> >>>>>
>> >>>>> :)
>> >>>>>
>> >>>>> currently, I have 15 data nodes.
>> >>>>> for some tests, I am trying to decommission until 8 nodes.
>> >>>>>
>> >>>>> Now, the total dfs used size is 52 TB which is including all
>> replicated
>> >>>>> blocks.
>> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>> >>>>>
>> >>>>> someone mentioned that I don't need to decommission node by node.
>> >>>>> for this case, is there no problems if I decommissioned 7 nodes at
>> the
>> >>>>> same time?
>> >>>>>
>> >>>>>
>> >>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>> >>>>>
>> >>>>> I can translate it to native English: how many nodes you want to
>> >>>>> decommission?
>> >>>>>
>> >>>>>
>> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com>
>> wrote:
>> >>>>>>
>> >>>>>> You want to decommission how many nodes?
>> >>>>>>
>> >>>>>>
>> >>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>> >>>>>>>
>> >>>>>>> 15 for datanodes and 3 for replication factor.
>> >>>>>>>
>> >>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>> >>>>>>>
>> >>>>>>>> How many nodes do you have and replication factor for it.
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Harsh J
>> >>
>> >
>> >
>> >
>> > --
>> > Harsh J
>>
>>
>
Re: are we able to decommission multi nodes at one time?
Posted by Azuryy Yu <az...@gmail.com>.
bq. then namenode start to copy block replicates on DN-2 to another DN,
supposed DN-2.
sorry for typo.
Correct for it:
then namenode start to copy block replicates on DN-1 to another DN,
supposed DN-2.
On Wed, Apr 3, 2013 at 9:51 AM, Azuryy Yu <az...@gmail.com> wrote:
> It's different.
> If you just want to stop DN-1 a short time, just kill the DataNode process
> on DN-1. then do what you want. during this time, Namenode cannot receive
> the heart beat from DN-1, then namenode start to copy block replicates on
> DN-2 to another DN, supposed DN-2.
>
> But when you start DN-1 again, Namenode receive the DN-1 registration,
> then namenode stop to copy the DN-1's block replicates even if NN doesn't
> finish coping.
>
> Am I explain clearly?
>
>
>
> On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <he...@gmail.com>wrote:
>
>> @Harsh
>>
>> What's the reasons to make big gaps for removing nodes between
>> decommission and just down nodes?
>> In my understanding, both are necessary to copy un-replicated blocks to
>> another alive nodes.
>> If main costs of them are this one, total elapsed time couldn't be big
>> different.
>>
>> Could you share some articles or documents to understand about
>> decommissioning procedures?
>> - explaining is always thanks ;)
>>
>>
>> 2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
>>
>> > Yes, you can do the downtime work in steps of 2 DNs at a time,
>> > especially since you mentioned the total work would be only ~30mins at
>> > most.
>> >
>> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
>> > <he...@gmail.com> wrote:
>> >> the rest of nodes to be alive has enough size to store.
>> >>
>> >> for this one that you've mentioned.
>> >>> its easier to do so in a rolling manner without need of a
>> >>> decommission.
>> >>
>> >> to check my understanding, just shutting down 2 of them and then 2
>> more and then 2 more without decommissions.
>> >>
>> >> is this correct?
>> >>
>> >>
>> >> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>> >>
>> >>> Note though that its only possible to decommission 7 nodes at the same
>> >>> time and expect it to finish iff the remaining 8 nodes have adequate
>> >>> free space for the excess replicas.
>> >>>
>> >>> If you're just going to take them down for a short while (few mins
>> >>> each), its easier to do so in a rolling manner without need of a
>> >>> decommission. You can take upto two down at a time on a replication
>> >>> average of 3 or 3+, and put it back in later without too much data
>> >>> movement impact.
>> >>>
>> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com>
>> wrote:
>> >>>> It's reasonable to decommission 7 nodes at the same time.
>> >>>> But may be it also takes long time to finish it.
>> >>>> Because all the replicas in these 7 nodes need to be copied to
>> remaining 8
>> >>>> nodes.
>> >>>> The size of transfer from these nodes to the remaining nodes is
>> equal.
>> >>>>
>> >>>>
>> >>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>> >>>>>
>> >>>>> :)
>> >>>>>
>> >>>>> currently, I have 15 data nodes.
>> >>>>> for some tests, I am trying to decommission until 8 nodes.
>> >>>>>
>> >>>>> Now, the total dfs used size is 52 TB which is including all
>> replicated
>> >>>>> blocks.
>> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>> >>>>>
>> >>>>> someone mentioned that I don't need to decommission node by node.
>> >>>>> for this case, is there no problems if I decommissioned 7 nodes at
>> the
>> >>>>> same time?
>> >>>>>
>> >>>>>
>> >>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>> >>>>>
>> >>>>> I can translate it to native English: how many nodes you want to
>> >>>>> decommission?
>> >>>>>
>> >>>>>
>> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com>
>> wrote:
>> >>>>>>
>> >>>>>> You want to decommission how many nodes?
>> >>>>>>
>> >>>>>>
>> >>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>> >>>>>>>
>> >>>>>>> 15 for datanodes and 3 for replication factor.
>> >>>>>>>
>> >>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>> >>>>>>>
>> >>>>>>>> How many nodes do you have and replication factor for it.
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Harsh J
>> >>
>> >
>> >
>> >
>> > --
>> > Harsh J
>>
>>
>
Re: are we able to decommission multi nodes at one time?
Posted by Azuryy Yu <az...@gmail.com>.
bq. then namenode start to copy block replicates on DN-2 to another DN,
supposed DN-2.
sorry for typo.
Correct for it:
then namenode start to copy block replicates on DN-1 to another DN,
supposed DN-2.
On Wed, Apr 3, 2013 at 9:51 AM, Azuryy Yu <az...@gmail.com> wrote:
> It's different.
> If you just want to stop DN-1 a short time, just kill the DataNode process
> on DN-1. then do what you want. during this time, Namenode cannot receive
> the heart beat from DN-1, then namenode start to copy block replicates on
> DN-2 to another DN, supposed DN-2.
>
> But when you start DN-1 again, Namenode receive the DN-1 registration,
> then namenode stop to copy the DN-1's block replicates even if NN doesn't
> finish coping.
>
> Am I explain clearly?
>
>
>
> On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <he...@gmail.com>wrote:
>
>> @Harsh
>>
>> What's the reasons to make big gaps for removing nodes between
>> decommission and just down nodes?
>> In my understanding, both are necessary to copy un-replicated blocks to
>> another alive nodes.
>> If main costs of them are this one, total elapsed time couldn't be big
>> different.
>>
>> Could you share some articles or documents to understand about
>> decommissioning procedures?
>> - explaining is always thanks ;)
>>
>>
>> 2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
>>
>> > Yes, you can do the downtime work in steps of 2 DNs at a time,
>> > especially since you mentioned the total work would be only ~30mins at
>> > most.
>> >
>> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
>> > <he...@gmail.com> wrote:
>> >> the rest of nodes to be alive has enough size to store.
>> >>
>> >> for this one that you've mentioned.
>> >>> its easier to do so in a rolling manner without need of a
>> >>> decommission.
>> >>
>> >> to check my understanding, just shutting down 2 of them and then 2
>> more and then 2 more without decommissions.
>> >>
>> >> is this correct?
>> >>
>> >>
>> >> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>> >>
>> >>> Note though that its only possible to decommission 7 nodes at the same
>> >>> time and expect it to finish iff the remaining 8 nodes have adequate
>> >>> free space for the excess replicas.
>> >>>
>> >>> If you're just going to take them down for a short while (few mins
>> >>> each), its easier to do so in a rolling manner without need of a
>> >>> decommission. You can take upto two down at a time on a replication
>> >>> average of 3 or 3+, and put it back in later without too much data
>> >>> movement impact.
>> >>>
>> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com>
>> wrote:
>> >>>> It's reasonable to decommission 7 nodes at the same time.
>> >>>> But may be it also takes long time to finish it.
>> >>>> Because all the replicas in these 7 nodes need to be copied to
>> remaining 8
>> >>>> nodes.
>> >>>> The size of transfer from these nodes to the remaining nodes is
>> equal.
>> >>>>
>> >>>>
>> >>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>> >>>>>
>> >>>>> :)
>> >>>>>
>> >>>>> currently, I have 15 data nodes.
>> >>>>> for some tests, I am trying to decommission until 8 nodes.
>> >>>>>
>> >>>>> Now, the total dfs used size is 52 TB which is including all
>> replicated
>> >>>>> blocks.
>> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>> >>>>>
>> >>>>> someone mentioned that I don't need to decommission node by node.
>> >>>>> for this case, is there no problems if I decommissioned 7 nodes at
>> the
>> >>>>> same time?
>> >>>>>
>> >>>>>
>> >>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>> >>>>>
>> >>>>> I can translate it to native English: how many nodes you want to
>> >>>>> decommission?
>> >>>>>
>> >>>>>
>> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com>
>> wrote:
>> >>>>>>
>> >>>>>> You want to decommission how many nodes?
>> >>>>>>
>> >>>>>>
>> >>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>> >>>>>>>
>> >>>>>>> 15 for datanodes and 3 for replication factor.
>> >>>>>>>
>> >>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>> >>>>>>>
>> >>>>>>>> How many nodes do you have and replication factor for it.
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Harsh J
>> >>
>> >
>> >
>> >
>> > --
>> > Harsh J
>>
>>
>
Re: are we able to decommission multi nodes at one time?
Posted by Azuryy Yu <az...@gmail.com>.
It's different.
If you just want to stop DN-1 a short time, just kill the DataNode process
on DN-1. then do what you want. during this time, Namenode cannot receive
the heart beat from DN-1, then namenode start to copy block replicates on
DN-2 to another DN, supposed DN-2.
But when you start DN-1 again, Namenode receive the DN-1 registration, then
namenode stop to copy the DN-1's block replicates even if NN doesn't finish
coping.
Am I explain clearly?
On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <he...@gmail.com>wrote:
> @Harsh
>
> What's the reasons to make big gaps for removing nodes between
> decommission and just down nodes?
> In my understanding, both are necessary to copy un-replicated blocks to
> another alive nodes.
> If main costs of them are this one, total elapsed time couldn't be big
> different.
>
> Could you share some articles or documents to understand about
> decommissioning procedures?
> - explaining is always thanks ;)
>
>
> 2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
>
> > Yes, you can do the downtime work in steps of 2 DNs at a time,
> > especially since you mentioned the total work would be only ~30mins at
> > most.
> >
> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
> > <he...@gmail.com> wrote:
> >> the rest of nodes to be alive has enough size to store.
> >>
> >> for this one that you've mentioned.
> >>> its easier to do so in a rolling manner without need of a
> >>> decommission.
> >>
> >> to check my understanding, just shutting down 2 of them and then 2 more
> and then 2 more without decommissions.
> >>
> >> is this correct?
> >>
> >>
> >> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
> >>
> >>> Note though that its only possible to decommission 7 nodes at the same
> >>> time and expect it to finish iff the remaining 8 nodes have adequate
> >>> free space for the excess replicas.
> >>>
> >>> If you're just going to take them down for a short while (few mins
> >>> each), its easier to do so in a rolling manner without need of a
> >>> decommission. You can take upto two down at a time on a replication
> >>> average of 3 or 3+, and put it back in later without too much data
> >>> movement impact.
> >>>
> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com>
> wrote:
> >>>> It's reasonable to decommission 7 nodes at the same time.
> >>>> But may be it also takes long time to finish it.
> >>>> Because all the replicas in these 7 nodes need to be copied to
> remaining 8
> >>>> nodes.
> >>>> The size of transfer from these nodes to the remaining nodes is equal.
> >>>>
> >>>>
> >>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
> >>>>>
> >>>>> :)
> >>>>>
> >>>>> currently, I have 15 data nodes.
> >>>>> for some tests, I am trying to decommission until 8 nodes.
> >>>>>
> >>>>> Now, the total dfs used size is 52 TB which is including all
> replicated
> >>>>> blocks.
> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
> >>>>>
> >>>>> someone mentioned that I don't need to decommission node by node.
> >>>>> for this case, is there no problems if I decommissioned 7 nodes at
> the
> >>>>> same time?
> >>>>>
> >>>>>
> >>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
> >>>>>
> >>>>> I can translate it to native English: how many nodes you want to
> >>>>> decommission?
> >>>>>
> >>>>>
> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com>
> wrote:
> >>>>>>
> >>>>>> You want to decommission how many nodes?
> >>>>>>
> >>>>>>
> >>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
> >>>>>>>
> >>>>>>> 15 for datanodes and 3 for replication factor.
> >>>>>>>
> >>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
> >>>>>>>
> >>>>>>>> How many nodes do you have and replication factor for it.
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Harsh J
> >>
> >
> >
> >
> > --
> > Harsh J
>
>
Re: are we able to decommission multi nodes at one time?
Posted by Azuryy Yu <az...@gmail.com>.
It's different.
If you just want to stop DN-1 a short time, just kill the DataNode process
on DN-1. then do what you want. during this time, Namenode cannot receive
the heart beat from DN-1, then namenode start to copy block replicates on
DN-2 to another DN, supposed DN-2.
But when you start DN-1 again, Namenode receive the DN-1 registration, then
namenode stop to copy the DN-1's block replicates even if NN doesn't finish
coping.
Am I explain clearly?
On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <he...@gmail.com>wrote:
> @Harsh
>
> What's the reasons to make big gaps for removing nodes between
> decommission and just down nodes?
> In my understanding, both are necessary to copy un-replicated blocks to
> another alive nodes.
> If main costs of them are this one, total elapsed time couldn't be big
> different.
>
> Could you share some articles or documents to understand about
> decommissioning procedures?
> - explaining is always thanks ;)
>
>
> 2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
>
> > Yes, you can do the downtime work in steps of 2 DNs at a time,
> > especially since you mentioned the total work would be only ~30mins at
> > most.
> >
> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
> > <he...@gmail.com> wrote:
> >> the rest of nodes to be alive has enough size to store.
> >>
> >> for this one that you've mentioned.
> >>> its easier to do so in a rolling manner without need of a
> >>> decommission.
> >>
> >> to check my understanding, just shutting down 2 of them and then 2 more
> and then 2 more without decommissions.
> >>
> >> is this correct?
> >>
> >>
> >> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
> >>
> >>> Note though that its only possible to decommission 7 nodes at the same
> >>> time and expect it to finish iff the remaining 8 nodes have adequate
> >>> free space for the excess replicas.
> >>>
> >>> If you're just going to take them down for a short while (few mins
> >>> each), its easier to do so in a rolling manner without need of a
> >>> decommission. You can take upto two down at a time on a replication
> >>> average of 3 or 3+, and put it back in later without too much data
> >>> movement impact.
> >>>
> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com>
> wrote:
> >>>> It's reasonable to decommission 7 nodes at the same time.
> >>>> But may be it also takes long time to finish it.
> >>>> Because all the replicas in these 7 nodes need to be copied to
> remaining 8
> >>>> nodes.
> >>>> The size of transfer from these nodes to the remaining nodes is equal.
> >>>>
> >>>>
> >>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
> >>>>>
> >>>>> :)
> >>>>>
> >>>>> currently, I have 15 data nodes.
> >>>>> for some tests, I am trying to decommission until 8 nodes.
> >>>>>
> >>>>> Now, the total dfs used size is 52 TB which is including all
> replicated
> >>>>> blocks.
> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
> >>>>>
> >>>>> someone mentioned that I don't need to decommission node by node.
> >>>>> for this case, is there no problems if I decommissioned 7 nodes at
> the
> >>>>> same time?
> >>>>>
> >>>>>
> >>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
> >>>>>
> >>>>> I can translate it to native English: how many nodes you want to
> >>>>> decommission?
> >>>>>
> >>>>>
> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com>
> wrote:
> >>>>>>
> >>>>>> You want to decommission how many nodes?
> >>>>>>
> >>>>>>
> >>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
> >>>>>>>
> >>>>>>> 15 for datanodes and 3 for replication factor.
> >>>>>>>
> >>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
> >>>>>>>
> >>>>>>>> How many nodes do you have and replication factor for it.
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Harsh J
> >>
> >
> >
> >
> > --
> > Harsh J
>
>
Re: are we able to decommission multi nodes at one time?
Posted by Azuryy Yu <az...@gmail.com>.
It's different.
If you just want to stop DN-1 a short time, just kill the DataNode process
on DN-1. then do what you want. during this time, Namenode cannot receive
the heart beat from DN-1, then namenode start to copy block replicates on
DN-2 to another DN, supposed DN-2.
But when you start DN-1 again, Namenode receive the DN-1 registration, then
namenode stop to copy the DN-1's block replicates even if NN doesn't finish
coping.
Am I explain clearly?
On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <he...@gmail.com>wrote:
> @Harsh
>
> What's the reasons to make big gaps for removing nodes between
> decommission and just down nodes?
> In my understanding, both are necessary to copy un-replicated blocks to
> another alive nodes.
> If main costs of them are this one, total elapsed time couldn't be big
> different.
>
> Could you share some articles or documents to understand about
> decommissioning procedures?
> - explaining is always thanks ;)
>
>
> 2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
>
> > Yes, you can do the downtime work in steps of 2 DNs at a time,
> > especially since you mentioned the total work would be only ~30mins at
> > most.
> >
> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
> > <he...@gmail.com> wrote:
> >> the rest of nodes to be alive has enough size to store.
> >>
> >> for this one that you've mentioned.
> >>> its easier to do so in a rolling manner without need of a
> >>> decommission.
> >>
> >> to check my understanding, just shutting down 2 of them and then 2 more
> and then 2 more without decommissions.
> >>
> >> is this correct?
> >>
> >>
> >> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
> >>
> >>> Note though that its only possible to decommission 7 nodes at the same
> >>> time and expect it to finish iff the remaining 8 nodes have adequate
> >>> free space for the excess replicas.
> >>>
> >>> If you're just going to take them down for a short while (few mins
> >>> each), its easier to do so in a rolling manner without need of a
> >>> decommission. You can take upto two down at a time on a replication
> >>> average of 3 or 3+, and put it back in later without too much data
> >>> movement impact.
> >>>
> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com>
> wrote:
> >>>> It's reasonable to decommission 7 nodes at the same time.
> >>>> But may be it also takes long time to finish it.
> >>>> Because all the replicas in these 7 nodes need to be copied to
> remaining 8
> >>>> nodes.
> >>>> The size of transfer from these nodes to the remaining nodes is equal.
> >>>>
> >>>>
> >>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
> >>>>>
> >>>>> :)
> >>>>>
> >>>>> currently, I have 15 data nodes.
> >>>>> for some tests, I am trying to decommission until 8 nodes.
> >>>>>
> >>>>> Now, the total dfs used size is 52 TB which is including all
> replicated
> >>>>> blocks.
> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
> >>>>>
> >>>>> someone mentioned that I don't need to decommission node by node.
> >>>>> for this case, is there no problems if I decommissioned 7 nodes at
> the
> >>>>> same time?
> >>>>>
> >>>>>
> >>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
> >>>>>
> >>>>> I can translate it to native English: how many nodes you want to
> >>>>> decommission?
> >>>>>
> >>>>>
> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com>
> wrote:
> >>>>>>
> >>>>>> You want to decommission how many nodes?
> >>>>>>
> >>>>>>
> >>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
> >>>>>>>
> >>>>>>> 15 for datanodes and 3 for replication factor.
> >>>>>>>
> >>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
> >>>>>>>
> >>>>>>>> How many nodes do you have and replication factor for it.
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Harsh J
> >>
> >
> >
> >
> > --
> > Harsh J
>
>
Re: are we able to decommission multi nodes at one time?
Posted by Azuryy Yu <az...@gmail.com>.
It's different.
If you just want to stop DN-1 a short time, just kill the DataNode process
on DN-1. then do what you want. during this time, Namenode cannot receive
the heart beat from DN-1, then namenode start to copy block replicates on
DN-2 to another DN, supposed DN-2.
But when you start DN-1 again, Namenode receive the DN-1 registration, then
namenode stop to copy the DN-1's block replicates even if NN doesn't finish
coping.
Am I explain clearly?
On Wed, Apr 3, 2013 at 9:43 AM, Henry Junyoung Kim <he...@gmail.com>wrote:
> @Harsh
>
> What's the reasons to make big gaps for removing nodes between
> decommission and just down nodes?
> In my understanding, both are necessary to copy un-replicated blocks to
> another alive nodes.
> If main costs of them are this one, total elapsed time couldn't be big
> different.
>
> Could you share some articles or documents to understand about
> decommissioning procedures?
> - explaining is always thanks ;)
>
>
> 2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
>
> > Yes, you can do the downtime work in steps of 2 DNs at a time,
> > especially since you mentioned the total work would be only ~30mins at
> > most.
> >
> > On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
> > <he...@gmail.com> wrote:
> >> the rest of nodes to be alive has enough size to store.
> >>
> >> for this one that you've mentioned.
> >>> its easier to do so in a rolling manner without need of a
> >>> decommission.
> >>
> >> to check my understanding, just shutting down 2 of them and then 2 more
> and then 2 more without decommissions.
> >>
> >> is this correct?
> >>
> >>
> >> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
> >>
> >>> Note though that its only possible to decommission 7 nodes at the same
> >>> time and expect it to finish iff the remaining 8 nodes have adequate
> >>> free space for the excess replicas.
> >>>
> >>> If you're just going to take them down for a short while (few mins
> >>> each), its easier to do so in a rolling manner without need of a
> >>> decommission. You can take upto two down at a time on a replication
> >>> average of 3 or 3+, and put it back in later without too much data
> >>> movement impact.
> >>>
> >>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com>
> wrote:
> >>>> It's reasonable to decommission 7 nodes at the same time.
> >>>> But may be it also takes long time to finish it.
> >>>> Because all the replicas in these 7 nodes need to be copied to
> remaining 8
> >>>> nodes.
> >>>> The size of transfer from these nodes to the remaining nodes is equal.
> >>>>
> >>>>
> >>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
> >>>>>
> >>>>> :)
> >>>>>
> >>>>> currently, I have 15 data nodes.
> >>>>> for some tests, I am trying to decommission until 8 nodes.
> >>>>>
> >>>>> Now, the total dfs used size is 52 TB which is including all
> replicated
> >>>>> blocks.
> >>>>> from 15 to 8, total spent time is almost 4 days long. ;(
> >>>>>
> >>>>> someone mentioned that I don't need to decommission node by node.
> >>>>> for this case, is there no problems if I decommissioned 7 nodes at
> the
> >>>>> same time?
> >>>>>
> >>>>>
> >>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
> >>>>>
> >>>>> I can translate it to native English: how many nodes you want to
> >>>>> decommission?
> >>>>>
> >>>>>
> >>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com>
> wrote:
> >>>>>>
> >>>>>> You want to decommission how many nodes?
> >>>>>>
> >>>>>>
> >>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
> >>>>>>>
> >>>>>>> 15 for datanodes and 3 for replication factor.
> >>>>>>>
> >>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
> >>>>>>>
> >>>>>>>> How many nodes do you have and replication factor for it.
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Harsh J
> >>
> >
> >
> >
> > --
> > Harsh J
>
>
Re: are we able to decommission multi nodes at one time?
Posted by Henry Junyoung Kim <he...@gmail.com>.
@Harsh
What's the reasons to make big gaps for removing nodes between decommission and just down nodes?
In my understanding, both are necessary to copy un-replicated blocks to another alive nodes.
If main costs of them are this one, total elapsed time couldn't be big different.
Could you share some articles or documents to understand about decommissioning procedures?
- explaining is always thanks ;)
2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
> Yes, you can do the downtime work in steps of 2 DNs at a time,
> especially since you mentioned the total work would be only ~30mins at
> most.
>
> On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
> <he...@gmail.com> wrote:
>> the rest of nodes to be alive has enough size to store.
>>
>> for this one that you've mentioned.
>>> its easier to do so in a rolling manner without need of a
>>> decommission.
>>
>> to check my understanding, just shutting down 2 of them and then 2 more and then 2 more without decommissions.
>>
>> is this correct?
>>
>>
>> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>>
>>> Note though that its only possible to decommission 7 nodes at the same
>>> time and expect it to finish iff the remaining 8 nodes have adequate
>>> free space for the excess replicas.
>>>
>>> If you're just going to take them down for a short while (few mins
>>> each), its easier to do so in a rolling manner without need of a
>>> decommission. You can take upto two down at a time on a replication
>>> average of 3 or 3+, and put it back in later without too much data
>>> movement impact.
>>>
>>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
>>>> It's reasonable to decommission 7 nodes at the same time.
>>>> But may be it also takes long time to finish it.
>>>> Because all the replicas in these 7 nodes need to be copied to remaining 8
>>>> nodes.
>>>> The size of transfer from these nodes to the remaining nodes is equal.
>>>>
>>>>
>>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>>>>
>>>>> :)
>>>>>
>>>>> currently, I have 15 data nodes.
>>>>> for some tests, I am trying to decommission until 8 nodes.
>>>>>
>>>>> Now, the total dfs used size is 52 TB which is including all replicated
>>>>> blocks.
>>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>>>
>>>>> someone mentioned that I don't need to decommission node by node.
>>>>> for this case, is there no problems if I decommissioned 7 nodes at the
>>>>> same time?
>>>>>
>>>>>
>>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>>>>
>>>>> I can translate it to native English: how many nodes you want to
>>>>> decommission?
>>>>>
>>>>>
>>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>>>>>>
>>>>>> You want to decommission how many nodes?
>>>>>>
>>>>>>
>>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>>>>>
>>>>>>> 15 for datanodes and 3 for replication factor.
>>>>>>>
>>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>>>>>
>>>>>>>> How many nodes do you have and replication factor for it.
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>
>
>
> --
> Harsh J
Re: are we able to decommission multi nodes at one time?
Posted by Henry Junyoung Kim <he...@gmail.com>.
one more question,
currently, our cluster is under decommissioning.
Without any safe stop steps, could I do downtime work forcibly?
2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
> Yes, you can do the downtime work in steps of 2 DNs at a time,
> especially since you mentioned the total work would be only ~30mins at
> most.
>
> On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
> <he...@gmail.com> wrote:
>> the rest of nodes to be alive has enough size to store.
>>
>> for this one that you've mentioned.
>>> its easier to do so in a rolling manner without need of a
>>> decommission.
>>
>> to check my understanding, just shutting down 2 of them and then 2 more and then 2 more without decommissions.
>>
>> is this correct?
>>
>>
>> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>>
>>> Note though that its only possible to decommission 7 nodes at the same
>>> time and expect it to finish iff the remaining 8 nodes have adequate
>>> free space for the excess replicas.
>>>
>>> If you're just going to take them down for a short while (few mins
>>> each), its easier to do so in a rolling manner without need of a
>>> decommission. You can take upto two down at a time on a replication
>>> average of 3 or 3+, and put it back in later without too much data
>>> movement impact.
>>>
>>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
>>>> It's reasonable to decommission 7 nodes at the same time.
>>>> But may be it also takes long time to finish it.
>>>> Because all the replicas in these 7 nodes need to be copied to remaining 8
>>>> nodes.
>>>> The size of transfer from these nodes to the remaining nodes is equal.
>>>>
>>>>
>>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>>>>
>>>>> :)
>>>>>
>>>>> currently, I have 15 data nodes.
>>>>> for some tests, I am trying to decommission until 8 nodes.
>>>>>
>>>>> Now, the total dfs used size is 52 TB which is including all replicated
>>>>> blocks.
>>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>>>
>>>>> someone mentioned that I don't need to decommission node by node.
>>>>> for this case, is there no problems if I decommissioned 7 nodes at the
>>>>> same time?
>>>>>
>>>>>
>>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>>>>
>>>>> I can translate it to native English: how many nodes you want to
>>>>> decommission?
>>>>>
>>>>>
>>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>>>>>>
>>>>>> You want to decommission how many nodes?
>>>>>>
>>>>>>
>>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>>>>>
>>>>>>> 15 for datanodes and 3 for replication factor.
>>>>>>>
>>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>>>>>
>>>>>>>> How many nodes do you have and replication factor for it.
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>
>
>
> --
> Harsh J
Re: are we able to decommission multi nodes at one time?
Posted by Henry Junyoung Kim <he...@gmail.com>.
@Harsh
What's the reasons to make big gaps for removing nodes between decommission and just down nodes?
In my understanding, both are necessary to copy un-replicated blocks to another alive nodes.
If main costs of them are this one, total elapsed time couldn't be big different.
Could you share some articles or documents to understand about decommissioning procedures?
- explaining is always thanks ;)
2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
> Yes, you can do the downtime work in steps of 2 DNs at a time,
> especially since you mentioned the total work would be only ~30mins at
> most.
>
> On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
> <he...@gmail.com> wrote:
>> the rest of nodes to be alive has enough size to store.
>>
>> for this one that you've mentioned.
>>> its easier to do so in a rolling manner without need of a
>>> decommission.
>>
>> to check my understanding, just shutting down 2 of them and then 2 more and then 2 more without decommissions.
>>
>> is this correct?
>>
>>
>> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>>
>>> Note though that its only possible to decommission 7 nodes at the same
>>> time and expect it to finish iff the remaining 8 nodes have adequate
>>> free space for the excess replicas.
>>>
>>> If you're just going to take them down for a short while (few mins
>>> each), its easier to do so in a rolling manner without need of a
>>> decommission. You can take upto two down at a time on a replication
>>> average of 3 or 3+, and put it back in later without too much data
>>> movement impact.
>>>
>>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
>>>> It's reasonable to decommission 7 nodes at the same time.
>>>> But may be it also takes long time to finish it.
>>>> Because all the replicas in these 7 nodes need to be copied to remaining 8
>>>> nodes.
>>>> The size of transfer from these nodes to the remaining nodes is equal.
>>>>
>>>>
>>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>>>>
>>>>> :)
>>>>>
>>>>> currently, I have 15 data nodes.
>>>>> for some tests, I am trying to decommission until 8 nodes.
>>>>>
>>>>> Now, the total dfs used size is 52 TB which is including all replicated
>>>>> blocks.
>>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>>>
>>>>> someone mentioned that I don't need to decommission node by node.
>>>>> for this case, is there no problems if I decommissioned 7 nodes at the
>>>>> same time?
>>>>>
>>>>>
>>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>>>>
>>>>> I can translate it to native English: how many nodes you want to
>>>>> decommission?
>>>>>
>>>>>
>>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>>>>>>
>>>>>> You want to decommission how many nodes?
>>>>>>
>>>>>>
>>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>>>>>
>>>>>>> 15 for datanodes and 3 for replication factor.
>>>>>>>
>>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>>>>>
>>>>>>>> How many nodes do you have and replication factor for it.
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>
>
>
> --
> Harsh J
Re: are we able to decommission multi nodes at one time?
Posted by Henry Junyoung Kim <he...@gmail.com>.
one more question,
currently, our cluster is under decommissioning.
Without any safe stop steps, could I do downtime work forcibly?
2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
> Yes, you can do the downtime work in steps of 2 DNs at a time,
> especially since you mentioned the total work would be only ~30mins at
> most.
>
> On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
> <he...@gmail.com> wrote:
>> the rest of nodes to be alive has enough size to store.
>>
>> for this one that you've mentioned.
>>> its easier to do so in a rolling manner without need of a
>>> decommission.
>>
>> to check my understanding, just shutting down 2 of them and then 2 more and then 2 more without decommissions.
>>
>> is this correct?
>>
>>
>> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>>
>>> Note though that its only possible to decommission 7 nodes at the same
>>> time and expect it to finish iff the remaining 8 nodes have adequate
>>> free space for the excess replicas.
>>>
>>> If you're just going to take them down for a short while (few mins
>>> each), its easier to do so in a rolling manner without need of a
>>> decommission. You can take upto two down at a time on a replication
>>> average of 3 or 3+, and put it back in later without too much data
>>> movement impact.
>>>
>>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
>>>> It's reasonable to decommission 7 nodes at the same time.
>>>> But may be it also takes long time to finish it.
>>>> Because all the replicas in these 7 nodes need to be copied to remaining 8
>>>> nodes.
>>>> The size of transfer from these nodes to the remaining nodes is equal.
>>>>
>>>>
>>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>>>>
>>>>> :)
>>>>>
>>>>> currently, I have 15 data nodes.
>>>>> for some tests, I am trying to decommission until 8 nodes.
>>>>>
>>>>> Now, the total dfs used size is 52 TB which is including all replicated
>>>>> blocks.
>>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>>>
>>>>> someone mentioned that I don't need to decommission node by node.
>>>>> for this case, is there no problems if I decommissioned 7 nodes at the
>>>>> same time?
>>>>>
>>>>>
>>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>>>>
>>>>> I can translate it to native English: how many nodes you want to
>>>>> decommission?
>>>>>
>>>>>
>>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>>>>>>
>>>>>> You want to decommission how many nodes?
>>>>>>
>>>>>>
>>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>>>>>
>>>>>>> 15 for datanodes and 3 for replication factor.
>>>>>>>
>>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>>>>>
>>>>>>>> How many nodes do you have and replication factor for it.
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>
>
>
> --
> Harsh J
Re: are we able to decommission multi nodes at one time?
Posted by Henry Junyoung Kim <he...@gmail.com>.
one more question,
currently, our cluster is under decommissioning.
Without any safe stop steps, could I do downtime work forcibly?
2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
> Yes, you can do the downtime work in steps of 2 DNs at a time,
> especially since you mentioned the total work would be only ~30mins at
> most.
>
> On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
> <he...@gmail.com> wrote:
>> the rest of nodes to be alive has enough size to store.
>>
>> for this one that you've mentioned.
>>> its easier to do so in a rolling manner without need of a
>>> decommission.
>>
>> to check my understanding, just shutting down 2 of them and then 2 more and then 2 more without decommissions.
>>
>> is this correct?
>>
>>
>> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>>
>>> Note though that its only possible to decommission 7 nodes at the same
>>> time and expect it to finish iff the remaining 8 nodes have adequate
>>> free space for the excess replicas.
>>>
>>> If you're just going to take them down for a short while (few mins
>>> each), its easier to do so in a rolling manner without need of a
>>> decommission. You can take upto two down at a time on a replication
>>> average of 3 or 3+, and put it back in later without too much data
>>> movement impact.
>>>
>>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
>>>> It's reasonable to decommission 7 nodes at the same time.
>>>> But may be it also takes long time to finish it.
>>>> Because all the replicas in these 7 nodes need to be copied to remaining 8
>>>> nodes.
>>>> The size of transfer from these nodes to the remaining nodes is equal.
>>>>
>>>>
>>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>>>>
>>>>> :)
>>>>>
>>>>> currently, I have 15 data nodes.
>>>>> for some tests, I am trying to decommission until 8 nodes.
>>>>>
>>>>> Now, the total dfs used size is 52 TB which is including all replicated
>>>>> blocks.
>>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>>>
>>>>> someone mentioned that I don't need to decommission node by node.
>>>>> for this case, is there no problems if I decommissioned 7 nodes at the
>>>>> same time?
>>>>>
>>>>>
>>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>>>>
>>>>> I can translate it to native English: how many nodes you want to
>>>>> decommission?
>>>>>
>>>>>
>>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>>>>>>
>>>>>> You want to decommission how many nodes?
>>>>>>
>>>>>>
>>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>>>>>
>>>>>>> 15 for datanodes and 3 for replication factor.
>>>>>>>
>>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>>>>>
>>>>>>>> How many nodes do you have and replication factor for it.
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>
>
>
> --
> Harsh J
Re: are we able to decommission multi nodes at one time?
Posted by Henry Junyoung Kim <he...@gmail.com>.
@Harsh
What's the reasons to make big gaps for removing nodes between decommission and just down nodes?
In my understanding, both are necessary to copy un-replicated blocks to another alive nodes.
If main costs of them are this one, total elapsed time couldn't be big different.
Could you share some articles or documents to understand about decommissioning procedures?
- explaining is always thanks ;)
2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
> Yes, you can do the downtime work in steps of 2 DNs at a time,
> especially since you mentioned the total work would be only ~30mins at
> most.
>
> On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
> <he...@gmail.com> wrote:
>> the rest of nodes to be alive has enough size to store.
>>
>> for this one that you've mentioned.
>>> its easier to do so in a rolling manner without need of a
>>> decommission.
>>
>> to check my understanding, just shutting down 2 of them and then 2 more and then 2 more without decommissions.
>>
>> is this correct?
>>
>>
>> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>>
>>> Note though that its only possible to decommission 7 nodes at the same
>>> time and expect it to finish iff the remaining 8 nodes have adequate
>>> free space for the excess replicas.
>>>
>>> If you're just going to take them down for a short while (few mins
>>> each), its easier to do so in a rolling manner without need of a
>>> decommission. You can take upto two down at a time on a replication
>>> average of 3 or 3+, and put it back in later without too much data
>>> movement impact.
>>>
>>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
>>>> It's reasonable to decommission 7 nodes at the same time.
>>>> But may be it also takes long time to finish it.
>>>> Because all the replicas in these 7 nodes need to be copied to remaining 8
>>>> nodes.
>>>> The size of transfer from these nodes to the remaining nodes is equal.
>>>>
>>>>
>>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>>>>
>>>>> :)
>>>>>
>>>>> currently, I have 15 data nodes.
>>>>> for some tests, I am trying to decommission until 8 nodes.
>>>>>
>>>>> Now, the total dfs used size is 52 TB which is including all replicated
>>>>> blocks.
>>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>>>
>>>>> someone mentioned that I don't need to decommission node by node.
>>>>> for this case, is there no problems if I decommissioned 7 nodes at the
>>>>> same time?
>>>>>
>>>>>
>>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>>>>
>>>>> I can translate it to native English: how many nodes you want to
>>>>> decommission?
>>>>>
>>>>>
>>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>>>>>>
>>>>>> You want to decommission how many nodes?
>>>>>>
>>>>>>
>>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>>>>>
>>>>>>> 15 for datanodes and 3 for replication factor.
>>>>>>>
>>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>>>>>
>>>>>>>> How many nodes do you have and replication factor for it.
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>
>
>
> --
> Harsh J
Re: are we able to decommission multi nodes at one time?
Posted by Henry Junyoung Kim <he...@gmail.com>.
@Harsh
What's the reasons to make big gaps for removing nodes between decommission and just down nodes?
In my understanding, both are necessary to copy un-replicated blocks to another alive nodes.
If main costs of them are this one, total elapsed time couldn't be big different.
Could you share some articles or documents to understand about decommissioning procedures?
- explaining is always thanks ;)
2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
> Yes, you can do the downtime work in steps of 2 DNs at a time,
> especially since you mentioned the total work would be only ~30mins at
> most.
>
> On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
> <he...@gmail.com> wrote:
>> the rest of nodes to be alive has enough size to store.
>>
>> for this one that you've mentioned.
>>> its easier to do so in a rolling manner without need of a
>>> decommission.
>>
>> to check my understanding, just shutting down 2 of them and then 2 more and then 2 more without decommissions.
>>
>> is this correct?
>>
>>
>> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>>
>>> Note though that its only possible to decommission 7 nodes at the same
>>> time and expect it to finish iff the remaining 8 nodes have adequate
>>> free space for the excess replicas.
>>>
>>> If you're just going to take them down for a short while (few mins
>>> each), its easier to do so in a rolling manner without need of a
>>> decommission. You can take upto two down at a time on a replication
>>> average of 3 or 3+, and put it back in later without too much data
>>> movement impact.
>>>
>>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
>>>> It's reasonable to decommission 7 nodes at the same time.
>>>> But may be it also takes long time to finish it.
>>>> Because all the replicas in these 7 nodes need to be copied to remaining 8
>>>> nodes.
>>>> The size of transfer from these nodes to the remaining nodes is equal.
>>>>
>>>>
>>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>>>>
>>>>> :)
>>>>>
>>>>> currently, I have 15 data nodes.
>>>>> for some tests, I am trying to decommission until 8 nodes.
>>>>>
>>>>> Now, the total dfs used size is 52 TB which is including all replicated
>>>>> blocks.
>>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>>>
>>>>> someone mentioned that I don't need to decommission node by node.
>>>>> for this case, is there no problems if I decommissioned 7 nodes at the
>>>>> same time?
>>>>>
>>>>>
>>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>>>>
>>>>> I can translate it to native English: how many nodes you want to
>>>>> decommission?
>>>>>
>>>>>
>>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>>>>>>
>>>>>> You want to decommission how many nodes?
>>>>>>
>>>>>>
>>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>>>>>
>>>>>>> 15 for datanodes and 3 for replication factor.
>>>>>>>
>>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>>>>>
>>>>>>>> How many nodes do you have and replication factor for it.
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>
>
>
> --
> Harsh J
Re: are we able to decommission multi nodes at one time?
Posted by Henry Junyoung Kim <he...@gmail.com>.
one more question,
currently, our cluster is under decommissioning.
Without any safe stop steps, could I do downtime work forcibly?
2013. 4. 2., 오후 5:37, Harsh J <ha...@cloudera.com> 작성:
> Yes, you can do the downtime work in steps of 2 DNs at a time,
> especially since you mentioned the total work would be only ~30mins at
> most.
>
> On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
> <he...@gmail.com> wrote:
>> the rest of nodes to be alive has enough size to store.
>>
>> for this one that you've mentioned.
>>> its easier to do so in a rolling manner without need of a
>>> decommission.
>>
>> to check my understanding, just shutting down 2 of them and then 2 more and then 2 more without decommissions.
>>
>> is this correct?
>>
>>
>> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>>
>>> Note though that its only possible to decommission 7 nodes at the same
>>> time and expect it to finish iff the remaining 8 nodes have adequate
>>> free space for the excess replicas.
>>>
>>> If you're just going to take them down for a short while (few mins
>>> each), its easier to do so in a rolling manner without need of a
>>> decommission. You can take upto two down at a time on a replication
>>> average of 3 or 3+, and put it back in later without too much data
>>> movement impact.
>>>
>>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
>>>> It's reasonable to decommission 7 nodes at the same time.
>>>> But may be it also takes long time to finish it.
>>>> Because all the replicas in these 7 nodes need to be copied to remaining 8
>>>> nodes.
>>>> The size of transfer from these nodes to the remaining nodes is equal.
>>>>
>>>>
>>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>>>>
>>>>> :)
>>>>>
>>>>> currently, I have 15 data nodes.
>>>>> for some tests, I am trying to decommission until 8 nodes.
>>>>>
>>>>> Now, the total dfs used size is 52 TB which is including all replicated
>>>>> blocks.
>>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>>>
>>>>> someone mentioned that I don't need to decommission node by node.
>>>>> for this case, is there no problems if I decommissioned 7 nodes at the
>>>>> same time?
>>>>>
>>>>>
>>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>>>>
>>>>> I can translate it to native English: how many nodes you want to
>>>>> decommission?
>>>>>
>>>>>
>>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>>>>>>
>>>>>> You want to decommission how many nodes?
>>>>>>
>>>>>>
>>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>>>>>
>>>>>>> 15 for datanodes and 3 for replication factor.
>>>>>>>
>>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>>>>>
>>>>>>>> How many nodes do you have and replication factor for it.
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>
>
>
> --
> Harsh J
Re: are we able to decommission multi nodes at one time?
Posted by Harsh J <ha...@cloudera.com>.
Yes, you can do the downtime work in steps of 2 DNs at a time,
especially since you mentioned the total work would be only ~30mins at
most.
On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
<he...@gmail.com> wrote:
> the rest of nodes to be alive has enough size to store.
>
> for this one that you've mentioned.
>> its easier to do so in a rolling manner without need of a
>> decommission.
>
> to check my understanding, just shutting down 2 of them and then 2 more and then 2 more without decommissions.
>
> is this correct?
>
>
> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>
>> Note though that its only possible to decommission 7 nodes at the same
>> time and expect it to finish iff the remaining 8 nodes have adequate
>> free space for the excess replicas.
>>
>> If you're just going to take them down for a short while (few mins
>> each), its easier to do so in a rolling manner without need of a
>> decommission. You can take upto two down at a time on a replication
>> average of 3 or 3+, and put it back in later without too much data
>> movement impact.
>>
>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
>>> It's reasonable to decommission 7 nodes at the same time.
>>> But may be it also takes long time to finish it.
>>> Because all the replicas in these 7 nodes need to be copied to remaining 8
>>> nodes.
>>> The size of transfer from these nodes to the remaining nodes is equal.
>>>
>>>
>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>>>
>>>> :)
>>>>
>>>> currently, I have 15 data nodes.
>>>> for some tests, I am trying to decommission until 8 nodes.
>>>>
>>>> Now, the total dfs used size is 52 TB which is including all replicated
>>>> blocks.
>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>>
>>>> someone mentioned that I don't need to decommission node by node.
>>>> for this case, is there no problems if I decommissioned 7 nodes at the
>>>> same time?
>>>>
>>>>
>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>>>
>>>> I can translate it to native English: how many nodes you want to
>>>> decommission?
>>>>
>>>>
>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>>>>>
>>>>> You want to decommission how many nodes?
>>>>>
>>>>>
>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>>>>
>>>>>> 15 for datanodes and 3 for replication factor.
>>>>>>
>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>>>>
>>>>>>> How many nodes do you have and replication factor for it.
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Harsh J
>
--
Harsh J
Re: are we able to decommission multi nodes at one time?
Posted by Harsh J <ha...@cloudera.com>.
Yes, you can do the downtime work in steps of 2 DNs at a time,
especially since you mentioned the total work would be only ~30mins at
most.
On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
<he...@gmail.com> wrote:
> the rest of nodes to be alive has enough size to store.
>
> for this one that you've mentioned.
>> its easier to do so in a rolling manner without need of a
>> decommission.
>
> to check my understanding, just shutting down 2 of them and then 2 more and then 2 more without decommissions.
>
> is this correct?
>
>
> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>
>> Note though that its only possible to decommission 7 nodes at the same
>> time and expect it to finish iff the remaining 8 nodes have adequate
>> free space for the excess replicas.
>>
>> If you're just going to take them down for a short while (few mins
>> each), its easier to do so in a rolling manner without need of a
>> decommission. You can take upto two down at a time on a replication
>> average of 3 or 3+, and put it back in later without too much data
>> movement impact.
>>
>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
>>> It's reasonable to decommission 7 nodes at the same time.
>>> But may be it also takes long time to finish it.
>>> Because all the replicas in these 7 nodes need to be copied to remaining 8
>>> nodes.
>>> The size of transfer from these nodes to the remaining nodes is equal.
>>>
>>>
>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>>>
>>>> :)
>>>>
>>>> currently, I have 15 data nodes.
>>>> for some tests, I am trying to decommission until 8 nodes.
>>>>
>>>> Now, the total dfs used size is 52 TB which is including all replicated
>>>> blocks.
>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>>
>>>> someone mentioned that I don't need to decommission node by node.
>>>> for this case, is there no problems if I decommissioned 7 nodes at the
>>>> same time?
>>>>
>>>>
>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>>>
>>>> I can translate it to native English: how many nodes you want to
>>>> decommission?
>>>>
>>>>
>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>>>>>
>>>>> You want to decommission how many nodes?
>>>>>
>>>>>
>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>>>>
>>>>>> 15 for datanodes and 3 for replication factor.
>>>>>>
>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>>>>
>>>>>>> How many nodes do you have and replication factor for it.
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Harsh J
>
--
Harsh J
Re: are we able to decommission multi nodes at one time?
Posted by Harsh J <ha...@cloudera.com>.
Yes, you can do the downtime work in steps of 2 DNs at a time,
especially since you mentioned the total work would be only ~30mins at
most.
On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
<he...@gmail.com> wrote:
> the rest of nodes to be alive has enough size to store.
>
> for this one that you've mentioned.
>> its easier to do so in a rolling manner without need of a
>> decommission.
>
> to check my understanding, just shutting down 2 of them and then 2 more and then 2 more without decommissions.
>
> is this correct?
>
>
> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>
>> Note though that its only possible to decommission 7 nodes at the same
>> time and expect it to finish iff the remaining 8 nodes have adequate
>> free space for the excess replicas.
>>
>> If you're just going to take them down for a short while (few mins
>> each), its easier to do so in a rolling manner without need of a
>> decommission. You can take upto two down at a time on a replication
>> average of 3 or 3+, and put it back in later without too much data
>> movement impact.
>>
>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
>>> It's reasonable to decommission 7 nodes at the same time.
>>> But may be it also takes long time to finish it.
>>> Because all the replicas in these 7 nodes need to be copied to remaining 8
>>> nodes.
>>> The size of transfer from these nodes to the remaining nodes is equal.
>>>
>>>
>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>>>
>>>> :)
>>>>
>>>> currently, I have 15 data nodes.
>>>> for some tests, I am trying to decommission until 8 nodes.
>>>>
>>>> Now, the total dfs used size is 52 TB which is including all replicated
>>>> blocks.
>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>>
>>>> someone mentioned that I don't need to decommission node by node.
>>>> for this case, is there no problems if I decommissioned 7 nodes at the
>>>> same time?
>>>>
>>>>
>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>>>
>>>> I can translate it to native English: how many nodes you want to
>>>> decommission?
>>>>
>>>>
>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>>>>>
>>>>> You want to decommission how many nodes?
>>>>>
>>>>>
>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>>>>
>>>>>> 15 for datanodes and 3 for replication factor.
>>>>>>
>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>>>>
>>>>>>> How many nodes do you have and replication factor for it.
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Harsh J
>
--
Harsh J
Re: are we able to decommission multi nodes at one time?
Posted by Harsh J <ha...@cloudera.com>.
Yes, you can do the downtime work in steps of 2 DNs at a time,
especially since you mentioned the total work would be only ~30mins at
most.
On Tue, Apr 2, 2013 at 1:46 PM, Henry Junyoung Kim
<he...@gmail.com> wrote:
> the rest of nodes to be alive has enough size to store.
>
> for this one that you've mentioned.
>> its easier to do so in a rolling manner without need of a
>> decommission.
>
> to check my understanding, just shutting down 2 of them and then 2 more and then 2 more without decommissions.
>
> is this correct?
>
>
> 2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
>
>> Note though that its only possible to decommission 7 nodes at the same
>> time and expect it to finish iff the remaining 8 nodes have adequate
>> free space for the excess replicas.
>>
>> If you're just going to take them down for a short while (few mins
>> each), its easier to do so in a rolling manner without need of a
>> decommission. You can take upto two down at a time on a replication
>> average of 3 or 3+, and put it back in later without too much data
>> movement impact.
>>
>> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
>>> It's reasonable to decommission 7 nodes at the same time.
>>> But may be it also takes long time to finish it.
>>> Because all the replicas in these 7 nodes need to be copied to remaining 8
>>> nodes.
>>> The size of transfer from these nodes to the remaining nodes is equal.
>>>
>>>
>>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>>>
>>>> :)
>>>>
>>>> currently, I have 15 data nodes.
>>>> for some tests, I am trying to decommission until 8 nodes.
>>>>
>>>> Now, the total dfs used size is 52 TB which is including all replicated
>>>> blocks.
>>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>>
>>>> someone mentioned that I don't need to decommission node by node.
>>>> for this case, is there no problems if I decommissioned 7 nodes at the
>>>> same time?
>>>>
>>>>
>>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>>>
>>>> I can translate it to native English: how many nodes you want to
>>>> decommission?
>>>>
>>>>
>>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>>>>>
>>>>> You want to decommission how many nodes?
>>>>>
>>>>>
>>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>>>>
>>>>>> 15 for datanodes and 3 for replication factor.
>>>>>>
>>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>>>>
>>>>>>> How many nodes do you have and replication factor for it.
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Harsh J
>
--
Harsh J
Re: are we able to decommission multi nodes at one time?
Posted by Henry Junyoung Kim <he...@gmail.com>.
the rest of nodes to be alive has enough size to store.
for this one that you've mentioned.
> its easier to do so in a rolling manner without need of a
> decommission.
to check my understanding, just shutting down 2 of them and then 2 more and then 2 more without decommissions.
is this correct?
2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
> Note though that its only possible to decommission 7 nodes at the same
> time and expect it to finish iff the remaining 8 nodes have adequate
> free space for the excess replicas.
>
> If you're just going to take them down for a short while (few mins
> each), its easier to do so in a rolling manner without need of a
> decommission. You can take upto two down at a time on a replication
> average of 3 or 3+, and put it back in later without too much data
> movement impact.
>
> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
>> It's reasonable to decommission 7 nodes at the same time.
>> But may be it also takes long time to finish it.
>> Because all the replicas in these 7 nodes need to be copied to remaining 8
>> nodes.
>> The size of transfer from these nodes to the remaining nodes is equal.
>>
>>
>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>>
>>> :)
>>>
>>> currently, I have 15 data nodes.
>>> for some tests, I am trying to decommission until 8 nodes.
>>>
>>> Now, the total dfs used size is 52 TB which is including all replicated
>>> blocks.
>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>
>>> someone mentioned that I don't need to decommission node by node.
>>> for this case, is there no problems if I decommissioned 7 nodes at the
>>> same time?
>>>
>>>
>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>>
>>> I can translate it to native English: how many nodes you want to
>>> decommission?
>>>
>>>
>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>>>>
>>>> You want to decommission how many nodes?
>>>>
>>>>
>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>>>
>>>>> 15 for datanodes and 3 for replication factor.
>>>>>
>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>>>
>>>>>> How many nodes do you have and replication factor for it.
>>>>>
>>>>
>>>
>>>
>>
>
>
>
> --
> Harsh J
Re: are we able to decommission multi nodes at one time?
Posted by Henry Junyoung Kim <he...@gmail.com>.
the rest of nodes to be alive has enough size to store.
for this one that you've mentioned.
> its easier to do so in a rolling manner without need of a
> decommission.
to check my understanding, just shutting down 2 of them and then 2 more and then 2 more without decommissions.
is this correct?
2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
> Note though that its only possible to decommission 7 nodes at the same
> time and expect it to finish iff the remaining 8 nodes have adequate
> free space for the excess replicas.
>
> If you're just going to take them down for a short while (few mins
> each), its easier to do so in a rolling manner without need of a
> decommission. You can take upto two down at a time on a replication
> average of 3 or 3+, and put it back in later without too much data
> movement impact.
>
> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
>> It's reasonable to decommission 7 nodes at the same time.
>> But may be it also takes long time to finish it.
>> Because all the replicas in these 7 nodes need to be copied to remaining 8
>> nodes.
>> The size of transfer from these nodes to the remaining nodes is equal.
>>
>>
>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>>
>>> :)
>>>
>>> currently, I have 15 data nodes.
>>> for some tests, I am trying to decommission until 8 nodes.
>>>
>>> Now, the total dfs used size is 52 TB which is including all replicated
>>> blocks.
>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>
>>> someone mentioned that I don't need to decommission node by node.
>>> for this case, is there no problems if I decommissioned 7 nodes at the
>>> same time?
>>>
>>>
>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>>
>>> I can translate it to native English: how many nodes you want to
>>> decommission?
>>>
>>>
>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>>>>
>>>> You want to decommission how many nodes?
>>>>
>>>>
>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>>>
>>>>> 15 for datanodes and 3 for replication factor.
>>>>>
>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>>>
>>>>>> How many nodes do you have and replication factor for it.
>>>>>
>>>>
>>>
>>>
>>
>
>
>
> --
> Harsh J
Re: are we able to decommission multi nodes at one time?
Posted by Henry Junyoung Kim <he...@gmail.com>.
the rest of nodes to be alive has enough size to store.
for this one that you've mentioned.
> its easier to do so in a rolling manner without need of a
> decommission.
to check my understanding, just shutting down 2 of them and then 2 more and then 2 more without decommissions.
is this correct?
2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
> Note though that its only possible to decommission 7 nodes at the same
> time and expect it to finish iff the remaining 8 nodes have adequate
> free space for the excess replicas.
>
> If you're just going to take them down for a short while (few mins
> each), its easier to do so in a rolling manner without need of a
> decommission. You can take upto two down at a time on a replication
> average of 3 or 3+, and put it back in later without too much data
> movement impact.
>
> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
>> It's reasonable to decommission 7 nodes at the same time.
>> But may be it also takes long time to finish it.
>> Because all the replicas in these 7 nodes need to be copied to remaining 8
>> nodes.
>> The size of transfer from these nodes to the remaining nodes is equal.
>>
>>
>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>>
>>> :)
>>>
>>> currently, I have 15 data nodes.
>>> for some tests, I am trying to decommission until 8 nodes.
>>>
>>> Now, the total dfs used size is 52 TB which is including all replicated
>>> blocks.
>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>
>>> someone mentioned that I don't need to decommission node by node.
>>> for this case, is there no problems if I decommissioned 7 nodes at the
>>> same time?
>>>
>>>
>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>>
>>> I can translate it to native English: how many nodes you want to
>>> decommission?
>>>
>>>
>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>>>>
>>>> You want to decommission how many nodes?
>>>>
>>>>
>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>>>
>>>>> 15 for datanodes and 3 for replication factor.
>>>>>
>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>>>
>>>>>> How many nodes do you have and replication factor for it.
>>>>>
>>>>
>>>
>>>
>>
>
>
>
> --
> Harsh J
Re: are we able to decommission multi nodes at one time?
Posted by Henry Junyoung Kim <he...@gmail.com>.
the rest of nodes to be alive has enough size to store.
for this one that you've mentioned.
> its easier to do so in a rolling manner without need of a
> decommission.
to check my understanding, just shutting down 2 of them and then 2 more and then 2 more without decommissions.
is this correct?
2013. 4. 2., 오후 4:54, Harsh J <ha...@cloudera.com> 작성:
> Note though that its only possible to decommission 7 nodes at the same
> time and expect it to finish iff the remaining 8 nodes have adequate
> free space for the excess replicas.
>
> If you're just going to take them down for a short while (few mins
> each), its easier to do so in a rolling manner without need of a
> decommission. You can take upto two down at a time on a replication
> average of 3 or 3+, and put it back in later without too much data
> movement impact.
>
> On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
>> It's reasonable to decommission 7 nodes at the same time.
>> But may be it also takes long time to finish it.
>> Because all the replicas in these 7 nodes need to be copied to remaining 8
>> nodes.
>> The size of transfer from these nodes to the remaining nodes is equal.
>>
>>
>> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>>
>>> :)
>>>
>>> currently, I have 15 data nodes.
>>> for some tests, I am trying to decommission until 8 nodes.
>>>
>>> Now, the total dfs used size is 52 TB which is including all replicated
>>> blocks.
>>> from 15 to 8, total spent time is almost 4 days long. ;(
>>>
>>> someone mentioned that I don't need to decommission node by node.
>>> for this case, is there no problems if I decommissioned 7 nodes at the
>>> same time?
>>>
>>>
>>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>>
>>> I can translate it to native English: how many nodes you want to
>>> decommission?
>>>
>>>
>>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>>>>
>>>> You want to decommission how many nodes?
>>>>
>>>>
>>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>>>
>>>>> 15 for datanodes and 3 for replication factor.
>>>>>
>>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>>>
>>>>>> How many nodes do you have and replication factor for it.
>>>>>
>>>>
>>>
>>>
>>
>
>
>
> --
> Harsh J
Re: are we able to decommission multi nodes at one time?
Posted by Harsh J <ha...@cloudera.com>.
Note though that its only possible to decommission 7 nodes at the same
time and expect it to finish iff the remaining 8 nodes have adequate
free space for the excess replicas.
If you're just going to take them down for a short while (few mins
each), its easier to do so in a rolling manner without need of a
decommission. You can take upto two down at a time on a replication
average of 3 or 3+, and put it back in later without too much data
movement impact.
On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
> It's reasonable to decommission 7 nodes at the same time.
> But may be it also takes long time to finish it.
> Because all the replicas in these 7 nodes need to be copied to remaining 8
> nodes.
> The size of transfer from these nodes to the remaining nodes is equal.
>
>
> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>
>> :)
>>
>> currently, I have 15 data nodes.
>> for some tests, I am trying to decommission until 8 nodes.
>>
>> Now, the total dfs used size is 52 TB which is including all replicated
>> blocks.
>> from 15 to 8, total spent time is almost 4 days long. ;(
>>
>> someone mentioned that I don't need to decommission node by node.
>> for this case, is there no problems if I decommissioned 7 nodes at the
>> same time?
>>
>>
>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>
>> I can translate it to native English: how many nodes you want to
>> decommission?
>>
>>
>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>>>
>>> You want to decommission how many nodes?
>>>
>>>
>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>>
>>>> 15 for datanodes and 3 for replication factor.
>>>>
>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>>
>>>> > How many nodes do you have and replication factor for it.
>>>>
>>>
>>
>>
>
--
Harsh J
Re: are we able to decommission multi nodes at one time?
Posted by Harsh J <ha...@cloudera.com>.
Note though that its only possible to decommission 7 nodes at the same
time and expect it to finish iff the remaining 8 nodes have adequate
free space for the excess replicas.
If you're just going to take them down for a short while (few mins
each), its easier to do so in a rolling manner without need of a
decommission. You can take upto two down at a time on a replication
average of 3 or 3+, and put it back in later without too much data
movement impact.
On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
> It's reasonable to decommission 7 nodes at the same time.
> But may be it also takes long time to finish it.
> Because all the replicas in these 7 nodes need to be copied to remaining 8
> nodes.
> The size of transfer from these nodes to the remaining nodes is equal.
>
>
> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>
>> :)
>>
>> currently, I have 15 data nodes.
>> for some tests, I am trying to decommission until 8 nodes.
>>
>> Now, the total dfs used size is 52 TB which is including all replicated
>> blocks.
>> from 15 to 8, total spent time is almost 4 days long. ;(
>>
>> someone mentioned that I don't need to decommission node by node.
>> for this case, is there no problems if I decommissioned 7 nodes at the
>> same time?
>>
>>
>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>
>> I can translate it to native English: how many nodes you want to
>> decommission?
>>
>>
>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>>>
>>> You want to decommission how many nodes?
>>>
>>>
>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>>
>>>> 15 for datanodes and 3 for replication factor.
>>>>
>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>>
>>>> > How many nodes do you have and replication factor for it.
>>>>
>>>
>>
>>
>
--
Harsh J
Re: are we able to decommission multi nodes at one time?
Posted by Harsh J <ha...@cloudera.com>.
Note though that its only possible to decommission 7 nodes at the same
time and expect it to finish iff the remaining 8 nodes have adequate
free space for the excess replicas.
If you're just going to take them down for a short while (few mins
each), its easier to do so in a rolling manner without need of a
decommission. You can take upto two down at a time on a replication
average of 3 or 3+, and put it back in later without too much data
movement impact.
On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
> It's reasonable to decommission 7 nodes at the same time.
> But may be it also takes long time to finish it.
> Because all the replicas in these 7 nodes need to be copied to remaining 8
> nodes.
> The size of transfer from these nodes to the remaining nodes is equal.
>
>
> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>
>> :)
>>
>> currently, I have 15 data nodes.
>> for some tests, I am trying to decommission until 8 nodes.
>>
>> Now, the total dfs used size is 52 TB which is including all replicated
>> blocks.
>> from 15 to 8, total spent time is almost 4 days long. ;(
>>
>> someone mentioned that I don't need to decommission node by node.
>> for this case, is there no problems if I decommissioned 7 nodes at the
>> same time?
>>
>>
>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>
>> I can translate it to native English: how many nodes you want to
>> decommission?
>>
>>
>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>>>
>>> You want to decommission how many nodes?
>>>
>>>
>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>>
>>>> 15 for datanodes and 3 for replication factor.
>>>>
>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>>
>>>> > How many nodes do you have and replication factor for it.
>>>>
>>>
>>
>>
>
--
Harsh J
Re: are we able to decommission multi nodes at one time?
Posted by Harsh J <ha...@cloudera.com>.
Note though that its only possible to decommission 7 nodes at the same
time and expect it to finish iff the remaining 8 nodes have adequate
free space for the excess replicas.
If you're just going to take them down for a short while (few mins
each), its easier to do so in a rolling manner without need of a
decommission. You can take upto two down at a time on a replication
average of 3 or 3+, and put it back in later without too much data
movement impact.
On Tue, Apr 2, 2013 at 1:06 PM, Yanbo Liang <ya...@gmail.com> wrote:
> It's reasonable to decommission 7 nodes at the same time.
> But may be it also takes long time to finish it.
> Because all the replicas in these 7 nodes need to be copied to remaining 8
> nodes.
> The size of transfer from these nodes to the remaining nodes is equal.
>
>
> 2013/4/2 Henry Junyoung Kim <he...@gmail.com>
>>
>> :)
>>
>> currently, I have 15 data nodes.
>> for some tests, I am trying to decommission until 8 nodes.
>>
>> Now, the total dfs used size is 52 TB which is including all replicated
>> blocks.
>> from 15 to 8, total spent time is almost 4 days long. ;(
>>
>> someone mentioned that I don't need to decommission node by node.
>> for this case, is there no problems if I decommissioned 7 nodes at the
>> same time?
>>
>>
>> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>>
>> I can translate it to native English: how many nodes you want to
>> decommission?
>>
>>
>> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>>>
>>> You want to decommission how many nodes?
>>>
>>>
>>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>>>
>>>> 15 for datanodes and 3 for replication factor.
>>>>
>>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>>
>>>> > How many nodes do you have and replication factor for it.
>>>>
>>>
>>
>>
>
--
Harsh J
Re: are we able to decommission multi nodes at one time?
Posted by Yanbo Liang <ya...@gmail.com>.
It's reasonable to decommission 7 nodes at the same time.
But may be it also takes long time to finish it.
Because all the replicas in these 7 nodes need to be copied to remaining 8
nodes.
The size of transfer from these nodes to the remaining nodes is equal.
2013/4/2 Henry Junyoung Kim <he...@gmail.com>
> :)
>
> currently, I have 15 data nodes.
> for some tests, I am trying to decommission until 8 nodes.
>
> Now, the total dfs used size is 52 TB which is including all replicated
> blocks.
> from 15 to 8, total spent time is almost 4 days long. ;(
>
> someone mentioned that I don't need to decommission node by node.
> for this case, is there no problems if I decommissioned 7 nodes at the
> same time?
>
>
> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>
> I can translate it to native English: how many nodes you want to
> decommission?
>
>
> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>
>> You want to decommission how many nodes?
>>
>>
>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>
>>> 15 for datanodes and 3 for replication factor.
>>>
>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>
>>> > How many nodes do you have and replication factor for it.
>>>
>>>
>>
>
>
Re: are we able to decommission multi nodes at one time?
Posted by Yanbo Liang <ya...@gmail.com>.
It's reasonable to decommission 7 nodes at the same time.
But may be it also takes long time to finish it.
Because all the replicas in these 7 nodes need to be copied to remaining 8
nodes.
The size of transfer from these nodes to the remaining nodes is equal.
2013/4/2 Henry Junyoung Kim <he...@gmail.com>
> :)
>
> currently, I have 15 data nodes.
> for some tests, I am trying to decommission until 8 nodes.
>
> Now, the total dfs used size is 52 TB which is including all replicated
> blocks.
> from 15 to 8, total spent time is almost 4 days long. ;(
>
> someone mentioned that I don't need to decommission node by node.
> for this case, is there no problems if I decommissioned 7 nodes at the
> same time?
>
>
> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>
> I can translate it to native English: how many nodes you want to
> decommission?
>
>
> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>
>> You want to decommission how many nodes?
>>
>>
>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>
>>> 15 for datanodes and 3 for replication factor.
>>>
>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>
>>> > How many nodes do you have and replication factor for it.
>>>
>>>
>>
>
>
Re: are we able to decommission multi nodes at one time?
Posted by Yanbo Liang <ya...@gmail.com>.
It's reasonable to decommission 7 nodes at the same time.
But may be it also takes long time to finish it.
Because all the replicas in these 7 nodes need to be copied to remaining 8
nodes.
The size of transfer from these nodes to the remaining nodes is equal.
2013/4/2 Henry Junyoung Kim <he...@gmail.com>
> :)
>
> currently, I have 15 data nodes.
> for some tests, I am trying to decommission until 8 nodes.
>
> Now, the total dfs used size is 52 TB which is including all replicated
> blocks.
> from 15 to 8, total spent time is almost 4 days long. ;(
>
> someone mentioned that I don't need to decommission node by node.
> for this case, is there no problems if I decommissioned 7 nodes at the
> same time?
>
>
> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>
> I can translate it to native English: how many nodes you want to
> decommission?
>
>
> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>
>> You want to decommission how many nodes?
>>
>>
>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>
>>> 15 for datanodes and 3 for replication factor.
>>>
>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>
>>> > How many nodes do you have and replication factor for it.
>>>
>>>
>>
>
>
Re: are we able to decommission multi nodes at one time?
Posted by Yanbo Liang <ya...@gmail.com>.
It's reasonable to decommission 7 nodes at the same time.
But may be it also takes long time to finish it.
Because all the replicas in these 7 nodes need to be copied to remaining 8
nodes.
The size of transfer from these nodes to the remaining nodes is equal.
2013/4/2 Henry Junyoung Kim <he...@gmail.com>
> :)
>
> currently, I have 15 data nodes.
> for some tests, I am trying to decommission until 8 nodes.
>
> Now, the total dfs used size is 52 TB which is including all replicated
> blocks.
> from 15 to 8, total spent time is almost 4 days long. ;(
>
> someone mentioned that I don't need to decommission node by node.
> for this case, is there no problems if I decommissioned 7 nodes at the
> same time?
>
>
> 2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
>
> I can translate it to native English: how many nodes you want to
> decommission?
>
>
> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
>
>> You want to decommission how many nodes?
>>
>>
>> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>>
>>> 15 for datanodes and 3 for replication factor.
>>>
>>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>>
>>> > How many nodes do you have and replication factor for it.
>>>
>>>
>>
>
>
Re: are we able to decommission multi nodes at one time?
Posted by Henry Junyoung Kim <he...@gmail.com>.
:)
currently, I have 15 data nodes.
for some tests, I am trying to decommission until 8 nodes.
Now, the total dfs used size is 52 TB which is including all replicated blocks.
from 15 to 8, total spent time is almost 4 days long. ;(
someone mentioned that I don't need to decommission node by node.
for this case, is there no problems if I decommissioned 7 nodes at the same time?
2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
> I can translate it to native English: how many nodes you want to decommission?
>
>
> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
> You want to decommission how many nodes?
>
>
> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
> 15 for datanodes and 3 for replication factor.
>
> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>
> > How many nodes do you have and replication factor for it.
>
>
>
Re: are we able to decommission multi nodes at one time?
Posted by Henry Junyoung Kim <he...@gmail.com>.
:)
currently, I have 15 data nodes.
for some tests, I am trying to decommission until 8 nodes.
Now, the total dfs used size is 52 TB which is including all replicated blocks.
from 15 to 8, total spent time is almost 4 days long. ;(
someone mentioned that I don't need to decommission node by node.
for this case, is there no problems if I decommissioned 7 nodes at the same time?
2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
> I can translate it to native English: how many nodes you want to decommission?
>
>
> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
> You want to decommission how many nodes?
>
>
> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
> 15 for datanodes and 3 for replication factor.
>
> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>
> > How many nodes do you have and replication factor for it.
>
>
>
Re: are we able to decommission multi nodes at one time?
Posted by Henry Junyoung Kim <he...@gmail.com>.
:)
currently, I have 15 data nodes.
for some tests, I am trying to decommission until 8 nodes.
Now, the total dfs used size is 52 TB which is including all replicated blocks.
from 15 to 8, total spent time is almost 4 days long. ;(
someone mentioned that I don't need to decommission node by node.
for this case, is there no problems if I decommissioned 7 nodes at the same time?
2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
> I can translate it to native English: how many nodes you want to decommission?
>
>
> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
> You want to decommission how many nodes?
>
>
> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
> 15 for datanodes and 3 for replication factor.
>
> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>
> > How many nodes do you have and replication factor for it.
>
>
>
Re: are we able to decommission multi nodes at one time?
Posted by Henry Junyoung Kim <he...@gmail.com>.
:)
currently, I have 15 data nodes.
for some tests, I am trying to decommission until 8 nodes.
Now, the total dfs used size is 52 TB which is including all replicated blocks.
from 15 to 8, total spent time is almost 4 days long. ;(
someone mentioned that I don't need to decommission node by node.
for this case, is there no problems if I decommissioned 7 nodes at the same time?
2013. 4. 2., 오후 12:14, Azuryy Yu <az...@gmail.com> 작성:
> I can translate it to native English: how many nodes you want to decommission?
>
>
> On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
> You want to decommission how many nodes?
>
>
> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
> 15 for datanodes and 3 for replication factor.
>
> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>
> > How many nodes do you have and replication factor for it.
>
>
>
Re: are we able to decommission multi nodes at one time?
Posted by Azuryy Yu <az...@gmail.com>.
I can translate it to native English: how many nodes you want to
decommission?
On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
> You want to decommission how many nodes?
>
>
> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>
>> 15 for datanodes and 3 for replication factor.
>>
>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>
>> > How many nodes do you have and replication factor for it.
>>
>>
>
Re: are we able to decommission multi nodes at one time?
Posted by Azuryy Yu <az...@gmail.com>.
I can translate it to native English: how many nodes you want to
decommission?
On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
> You want to decommission how many nodes?
>
>
> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>
>> 15 for datanodes and 3 for replication factor.
>>
>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>
>> > How many nodes do you have and replication factor for it.
>>
>>
>
Re: are we able to decommission multi nodes at one time?
Posted by Azuryy Yu <az...@gmail.com>.
I can translate it to native English: how many nodes you want to
decommission?
On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
> You want to decommission how many nodes?
>
>
> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>
>> 15 for datanodes and 3 for replication factor.
>>
>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>
>> > How many nodes do you have and replication factor for it.
>>
>>
>
Re: are we able to decommission multi nodes at one time?
Posted by Azuryy Yu <az...@gmail.com>.
I can translate it to native English: how many nodes you want to
decommission?
On Tue, Apr 2, 2013 at 11:01 AM, Yanbo Liang <ya...@gmail.com> wrote:
> You want to decommission how many nodes?
>
>
> 2013/4/2 Henry JunYoung KIM <he...@gmail.com>
>
>> 15 for datanodes and 3 for replication factor.
>>
>> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>>
>> > How many nodes do you have and replication factor for it.
>>
>>
>
Re: are we able to decommission multi nodes at one time?
Posted by Yanbo Liang <ya...@gmail.com>.
You want to decommission how many nodes?
2013/4/2 Henry JunYoung KIM <he...@gmail.com>
> 15 for datanodes and 3 for replication factor.
>
> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>
> > How many nodes do you have and replication factor for it.
>
>
Re: are we able to decommission multi nodes at one time?
Posted by Yanbo Liang <ya...@gmail.com>.
You want to decommission how many nodes?
2013/4/2 Henry JunYoung KIM <he...@gmail.com>
> 15 for datanodes and 3 for replication factor.
>
> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>
> > How many nodes do you have and replication factor for it.
>
>
Re: are we able to decommission multi nodes at one time?
Posted by Yanbo Liang <ya...@gmail.com>.
You want to decommission how many nodes?
2013/4/2 Henry JunYoung KIM <he...@gmail.com>
> 15 for datanodes and 3 for replication factor.
>
> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>
> > How many nodes do you have and replication factor for it.
>
>
Re: are we able to decommission multi nodes at one time?
Posted by Yanbo Liang <ya...@gmail.com>.
You want to decommission how many nodes?
2013/4/2 Henry JunYoung KIM <he...@gmail.com>
> 15 for datanodes and 3 for replication factor.
>
> 2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
>
> > How many nodes do you have and replication factor for it.
>
>
Re: are we able to decommission multi nodes at one time?
Posted by Henry JunYoung KIM <he...@gmail.com>.
15 for datanodes and 3 for replication factor.
2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
> How many nodes do you have and replication factor for it.
Re: are we able to decommission multi nodes at one time?
Posted by Henry JunYoung KIM <he...@gmail.com>.
15 for datanodes and 3 for replication factor.
2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
> How many nodes do you have and replication factor for it.
Re: are we able to decommission multi nodes at one time?
Posted by Henry JunYoung KIM <he...@gmail.com>.
15 for datanodes and 3 for replication factor.
2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
> How many nodes do you have and replication factor for it.
Re: are we able to decommission multi nodes at one time?
Posted by Yanbo Liang <ya...@gmail.com>.
It's alowable to decommission multi nodes at the same time.
Just write the all the hostnames which will be decommissioned to the
exclude file and run "bin/hadoop dfsadmin -refreshNodes".
However you need to ensure the decommissioned DataNodes are minority of all
the DataNodes in the cluster and the block replica can be guaranteed after
decommission.
For example, default replication level mapred.submit.replication=10.
So if you have less than 10 DataNodes after decommissioned, the decommision
process will hang.
2013/4/1 varun kumar <va...@gmail.com>
> How many nodes do you have and replication factor for it.
>
Re: are we able to decommission multi nodes at one time?
Posted by Henry JunYoung KIM <he...@gmail.com>.
15 for datanodes and 3 for replication factor.
2013. 4. 1., 오후 3:23, varun kumar <va...@gmail.com> 작성:
> How many nodes do you have and replication factor for it.
Re: are we able to decommission multi nodes at one time?
Posted by Yanbo Liang <ya...@gmail.com>.
It's alowable to decommission multi nodes at the same time.
Just write the all the hostnames which will be decommissioned to the
exclude file and run "bin/hadoop dfsadmin -refreshNodes".
However you need to ensure the decommissioned DataNodes are minority of all
the DataNodes in the cluster and the block replica can be guaranteed after
decommission.
For example, default replication level mapred.submit.replication=10.
So if you have less than 10 DataNodes after decommissioned, the decommision
process will hang.
2013/4/1 varun kumar <va...@gmail.com>
> How many nodes do you have and replication factor for it.
>
Re: are we able to decommission multi nodes at one time?
Posted by varun kumar <va...@gmail.com>.
How many nodes do you have and replication factor for it.
Re: are we able to decommission multi nodes at one time?
Posted by varun kumar <va...@gmail.com>.
How many nodes do you have and replication factor for it.
Re: are we able to decommission multi nodes at one time?
Posted by varun kumar <va...@gmail.com>.
How many nodes do you have and replication factor for it.
Re: are we able to decommission multi nodes at one time?
Posted by varun kumar <va...@gmail.com>.
How many nodes do you have and replication factor for it.