Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

Hi,

I encountered an issue which hangs the decommission operatoin. Its steps:
1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, in
hdfs-site.xml, set the 'dfs.replication' to 2
2. Add node dn3 into the cluster as a new datanode, and did not change the '
dfs.replication' value in hdfs-site.xml and keep it as 2
note: step 2 passed
3. Decommission dn3 from the cluster

Expected result: dn3 could be decommissioned successfully

Actual result: decommission progress hangs and the status always be
'Waiting DataNode status: Decommissioned'

However, if the initial cluster includes >= 3 datanodes, this issue won't
be encountered when add/remove another datanode.

Also, after step 2, I noticed that some block's expected replicas is 3, but
the 'dfs.replication' value in hdfs-site.xml is always 2!

Could anyone pls help provide some triages?

Thanks in advance!

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

Yes, you are correct: using fsck tool I found some files in my cluster
expected more replications than the value defined in dfs.replication. If I
set the expected replication of this files to a proper number, the
decommissioning process will go smoothly and the datanode could be
decommissioned finally.

However, many users do not mention this and might confuse the situation
that cluster always stays on decommissioning phase, so I think we can make
some improvements to allow user more easily to do precheck before
decommisioning datanode here, and help them find out all filies which might
lack of replications after decommissioning a datanode. For example, we can
told user that the expected replication of file1 and file26 are 6, but
after decommissioning a datanode the datanodes of the cluster will be 5 and
won't satisify file1 and file26 any more. In this way, user can decide
whether to continue the decommissioning work or to reduce the expected
replications of those files. For the way to implementation, I think we
could add a decommission-precheck script or a parameter to fsck tool.

Any comments?


2013/8/1 Harsh J <ha...@cloudera.com>

> As I said before, it is a per-file property and the config can be
> bypassed by clients that do not read the configs, place a manual API
> override, etc..
>
> If you want to really define a hard maximum and catch such clients,
> try setting dfs.replication.max to 2 at your NameNode.
>
> On Thu, Aug 1, 2013 at 8:07 AM, sam liu <sa...@gmail.com> wrote:
> > But, please mention that the value of 'dfs.replication' of the cluster is
> > always 2, even when the datanode number is 3. And I am pretty sure I did
> not
> > manually create any files with rep=3. So, why were some files of hdfs
> > created with repl=3, but not repl=2?
> >
> >
> > 2013/8/1 Harsh J <ha...@cloudera.com>
> >>
> >> The step (a) points to your problem and solution both. You have files
> >> being created with repl=3 on a 2 DN cluster which will prevent
> >> decommission. This is not a bug.
> >>
> >> On Wed, Jul 31, 2013 at 12:09 PM, sam liu <sa...@gmail.com>
> wrote:
> >> > I opened a jira for tracking this issue:
> >> > https://issues.apache.org/jira/browse/HDFS-5046
> >> >
> >> >
> >> > 2013/7/2 sam liu <sa...@gmail.com>
> >> >>
> >> >> Yes, the default replication factor is 3. However, in my case, it's
> >> >> strange: during decommission hangs, I found some block's expected
> >> >> replicas
> >> >> is 3, but the 'dfs.replication' value in hdfs-site.xml of every
> cluster
> >> >> node
> >> >> is always 2 from the beginning of cluster setup. Below is my steps:
> >> >>
> >> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2.
> And,
> >> >> in
> >> >> hdfs-site.xml, set the 'dfs.replication' to 2
> >> >> 2. Add node dn3 into the cluster as a new datanode, and did not
> change
> >> >> the
> >> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> >> >> note: step 2 passed
> >> >> 3. Decommission dn3 from the cluster
> >> >> Expected result: dn3 could be decommissioned successfully
> >> >> Actual result:
> >> >> a). decommission progress hangs and the status always be 'Waiting
> >> >> DataNode
> >> >> status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2
> /',
> >> >> the
> >> >> decommission continues and will be completed finally.
> >> >> b). However, if the initial cluster includes >= 3 datanodes, this
> issue
> >> >> won't be encountered when add/remove another datanode. For example,
> if
> >> >> I
> >> >> setup a cluster with 3 datanodes, and then I can successfully add the
> >> >> 4th
> >> >> datanode into it, and then also can successfully remove the 4th
> >> >> datanode
> >> >> from the cluster.
> >> >>
> >> >> I doubt it's a bug and plan to open a jira to Hadoop HDFS for this.
> Any
> >> >> comments?
> >> >>
> >> >> Thanks!
> >> >>
> >> >>
> >> >> 2013/6/21 Harsh J <ha...@cloudera.com>
> >> >>>
> >> >>> The dfs.replication is a per-file parameter. If you have a client
> that
> >> >>> does not use the supplied configs, then its default replication is 3
> >> >>> and all files it will create (as part of the app or via a job
> config)
> >> >>> will be with replication factor 3.
> >> >>>
> >> >>> You can do an -lsr to find all files and filter which ones have been
> >> >>> created with a factor of 3 (versus expected config of 2).
> >> >>>
> >> >>> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com>
> >> >>> wrote:
> >> >>> > Hi George,
> >> >>> >
> >> >>> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2.
> >> >>> > But
> >> >>> > still
> >> >>> > encounter this issue.
> >> >>> >
> >> >>> > Thanks!
> >> >>> >
> >> >>> >
> >> >>> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
> >> >>> >>
> >> >>> >>
> >> >>> >> Hi,
> >> >>> >>
> >> >>> >> I think i have faced this before, the problem is that you have
> the
> >> >>> >> rep
> >> >>> >> factor=3 so it seems to hang because it needs 3 nodes to achieve
> >> >>> >> the
> >> >>> >> factor
> >> >>> >> (replicas are not created on the same node). If you set the
> >> >>> >> replication
> >> >>> >> factor=2 i think you will not have this issue. So in general you
> >> >>> >> must
> >> >>> >> make
> >> >>> >> sure that the rep factor is <= to the available datanodes.
> >> >>> >>
> >> >>> >> BR,
> >> >>> >> George
> >> >>> >>
> >> >>> >>
> >> >>> >> On 6/21/2013 12:29 PM, sam liu wrote:
> >> >>> >>
> >> >>> >> Hi,
> >> >>> >>
> >> >>> >> I encountered an issue which hangs the decommission operatoin.
> Its
> >> >>> >> steps:
> >> >>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2.
> >> >>> >> And,
> >> >>> >> in
> >> >>> >> hdfs-site.xml, set the 'dfs.replication' to 2
> >> >>> >> 2. Add node dn3 into the cluster as a new datanode, and did not
> >> >>> >> change
> >> >>> >> the
> >> >>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> >> >>> >> note: step 2 passed
> >> >>> >> 3. Decommission dn3 from the cluster
> >> >>> >>
> >> >>> >> Expected result: dn3 could be decommissioned successfully
> >> >>> >>
> >> >>> >> Actual result: decommission progress hangs and the status always
> be
> >> >>> >> 'Waiting DataNode status: Decommissioned'
> >> >>> >>
> >> >>> >> However, if the initial cluster includes >= 3 datanodes, this
> issue
> >> >>> >> won't
> >> >>> >> be encountered when add/remove another datanode.
> >> >>> >>
> >> >>> >> Also, after step 2, I noticed that some block's expected replicas
> >> >>> >> is
> >> >>> >> 3,
> >> >>> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
> >> >>> >>
> >> >>> >> Could anyone pls help provide some triages?
> >> >>> >>
> >> >>> >> Thanks in advance!
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> --
> >> >>> >> ---------------------------
> >> >>> >>
> >> >>> >> George Kousiouris, PhD
> >> >>> >> Electrical and Computer Engineer
> >> >>> >> Division of Communications,
> >> >>> >> Electronics and Information Engineering
> >> >>> >> School of Electrical and Computer Engineering
> >> >>> >> Tel: +30 210 772 2546
> >> >>> >> Mobile: +30 6939354121
> >> >>> >> Fax: +30 210 772 2569
> >> >>> >> Email: gkousiou@mail.ntua.gr
> >> >>> >> Site: http://users.ntua.gr/gkousiou/
> >> >>> >>
> >> >>> >> National Technical University of Athens
> >> >>> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
> >> >>> >
> >> >>> >
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Harsh J
> >> >>
> >> >>
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

Yes, you are correct: using fsck tool I found some files in my cluster
expected more replications than the value defined in dfs.replication. If I
set the expected replication of this files to a proper number, the
decommissioning process will go smoothly and the datanode could be
decommissioned finally.

However, many users do not mention this and might confuse the situation
that cluster always stays on decommissioning phase, so I think we can make
some improvements to allow user more easily to do precheck before
decommisioning datanode here, and help them find out all filies which might
lack of replications after decommissioning a datanode. For example, we can
told user that the expected replication of file1 and file26 are 6, but
after decommissioning a datanode the datanodes of the cluster will be 5 and
won't satisify file1 and file26 any more. In this way, user can decide
whether to continue the decommissioning work or to reduce the expected
replications of those files. For the way to implementation, I think we
could add a decommission-precheck script or a parameter to fsck tool.

Any comments?


2013/8/1 Harsh J <ha...@cloudera.com>

> As I said before, it is a per-file property and the config can be
> bypassed by clients that do not read the configs, place a manual API
> override, etc..
>
> If you want to really define a hard maximum and catch such clients,
> try setting dfs.replication.max to 2 at your NameNode.
>
> On Thu, Aug 1, 2013 at 8:07 AM, sam liu <sa...@gmail.com> wrote:
> > But, please mention that the value of 'dfs.replication' of the cluster is
> > always 2, even when the datanode number is 3. And I am pretty sure I did
> not
> > manually create any files with rep=3. So, why were some files of hdfs
> > created with repl=3, but not repl=2?
> >
> >
> > 2013/8/1 Harsh J <ha...@cloudera.com>
> >>
> >> The step (a) points to your problem and solution both. You have files
> >> being created with repl=3 on a 2 DN cluster which will prevent
> >> decommission. This is not a bug.
> >>
> >> On Wed, Jul 31, 2013 at 12:09 PM, sam liu <sa...@gmail.com>
> wrote:
> >> > I opened a jira for tracking this issue:
> >> > https://issues.apache.org/jira/browse/HDFS-5046
> >> >
> >> >
> >> > 2013/7/2 sam liu <sa...@gmail.com>
> >> >>
> >> >> Yes, the default replication factor is 3. However, in my case, it's
> >> >> strange: during decommission hangs, I found some block's expected
> >> >> replicas
> >> >> is 3, but the 'dfs.replication' value in hdfs-site.xml of every
> cluster
> >> >> node
> >> >> is always 2 from the beginning of cluster setup. Below is my steps:
> >> >>
> >> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2.
> And,
> >> >> in
> >> >> hdfs-site.xml, set the 'dfs.replication' to 2
> >> >> 2. Add node dn3 into the cluster as a new datanode, and did not
> change
> >> >> the
> >> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> >> >> note: step 2 passed
> >> >> 3. Decommission dn3 from the cluster
> >> >> Expected result: dn3 could be decommissioned successfully
> >> >> Actual result:
> >> >> a). decommission progress hangs and the status always be 'Waiting
> >> >> DataNode
> >> >> status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2
> /',
> >> >> the
> >> >> decommission continues and will be completed finally.
> >> >> b). However, if the initial cluster includes >= 3 datanodes, this
> issue
> >> >> won't be encountered when add/remove another datanode. For example,
> if
> >> >> I
> >> >> setup a cluster with 3 datanodes, and then I can successfully add the
> >> >> 4th
> >> >> datanode into it, and then also can successfully remove the 4th
> >> >> datanode
> >> >> from the cluster.
> >> >>
> >> >> I doubt it's a bug and plan to open a jira to Hadoop HDFS for this.
> Any
> >> >> comments?
> >> >>
> >> >> Thanks!
> >> >>
> >> >>
> >> >> 2013/6/21 Harsh J <ha...@cloudera.com>
> >> >>>
> >> >>> The dfs.replication is a per-file parameter. If you have a client
> that
> >> >>> does not use the supplied configs, then its default replication is 3
> >> >>> and all files it will create (as part of the app or via a job
> config)
> >> >>> will be with replication factor 3.
> >> >>>
> >> >>> You can do an -lsr to find all files and filter which ones have been
> >> >>> created with a factor of 3 (versus expected config of 2).
> >> >>>
> >> >>> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com>
> >> >>> wrote:
> >> >>> > Hi George,
> >> >>> >
> >> >>> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2.
> >> >>> > But
> >> >>> > still
> >> >>> > encounter this issue.
> >> >>> >
> >> >>> > Thanks!
> >> >>> >
> >> >>> >
> >> >>> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
> >> >>> >>
> >> >>> >>
> >> >>> >> Hi,
> >> >>> >>
> >> >>> >> I think i have faced this before, the problem is that you have
> the
> >> >>> >> rep
> >> >>> >> factor=3 so it seems to hang because it needs 3 nodes to achieve
> >> >>> >> the
> >> >>> >> factor
> >> >>> >> (replicas are not created on the same node). If you set the
> >> >>> >> replication
> >> >>> >> factor=2 i think you will not have this issue. So in general you
> >> >>> >> must
> >> >>> >> make
> >> >>> >> sure that the rep factor is <= to the available datanodes.
> >> >>> >>
> >> >>> >> BR,
> >> >>> >> George
> >> >>> >>
> >> >>> >>
> >> >>> >> On 6/21/2013 12:29 PM, sam liu wrote:
> >> >>> >>
> >> >>> >> Hi,
> >> >>> >>
> >> >>> >> I encountered an issue which hangs the decommission operatoin.
> Its
> >> >>> >> steps:
> >> >>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2.
> >> >>> >> And,
> >> >>> >> in
> >> >>> >> hdfs-site.xml, set the 'dfs.replication' to 2
> >> >>> >> 2. Add node dn3 into the cluster as a new datanode, and did not
> >> >>> >> change
> >> >>> >> the
> >> >>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> >> >>> >> note: step 2 passed
> >> >>> >> 3. Decommission dn3 from the cluster
> >> >>> >>
> >> >>> >> Expected result: dn3 could be decommissioned successfully
> >> >>> >>
> >> >>> >> Actual result: decommission progress hangs and the status always
> be
> >> >>> >> 'Waiting DataNode status: Decommissioned'
> >> >>> >>
> >> >>> >> However, if the initial cluster includes >= 3 datanodes, this
> issue
> >> >>> >> won't
> >> >>> >> be encountered when add/remove another datanode.
> >> >>> >>
> >> >>> >> Also, after step 2, I noticed that some block's expected replicas
> >> >>> >> is
> >> >>> >> 3,
> >> >>> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
> >> >>> >>
> >> >>> >> Could anyone pls help provide some triages?
> >> >>> >>
> >> >>> >> Thanks in advance!
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> --
> >> >>> >> ---------------------------
> >> >>> >>
> >> >>> >> George Kousiouris, PhD
> >> >>> >> Electrical and Computer Engineer
> >> >>> >> Division of Communications,
> >> >>> >> Electronics and Information Engineering
> >> >>> >> School of Electrical and Computer Engineering
> >> >>> >> Tel: +30 210 772 2546
> >> >>> >> Mobile: +30 6939354121
> >> >>> >> Fax: +30 210 772 2569
> >> >>> >> Email: gkousiou@mail.ntua.gr
> >> >>> >> Site: http://users.ntua.gr/gkousiou/
> >> >>> >>
> >> >>> >> National Technical University of Athens
> >> >>> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
> >> >>> >
> >> >>> >
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Harsh J
> >> >>
> >> >>
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

Yes, you are correct: using fsck tool I found some files in my cluster
expected more replications than the value defined in dfs.replication. If I
set the expected replication of this files to a proper number, the
decommissioning process will go smoothly and the datanode could be
decommissioned finally.

However, many users do not mention this and might confuse the situation
that cluster always stays on decommissioning phase, so I think we can make
some improvements to allow user more easily to do precheck before
decommisioning datanode here, and help them find out all filies which might
lack of replications after decommissioning a datanode. For example, we can
told user that the expected replication of file1 and file26 are 6, but
after decommissioning a datanode the datanodes of the cluster will be 5 and
won't satisify file1 and file26 any more. In this way, user can decide
whether to continue the decommissioning work or to reduce the expected
replications of those files. For the way to implementation, I think we
could add a decommission-precheck script or a parameter to fsck tool.

Any comments?


2013/8/1 Harsh J <ha...@cloudera.com>

> As I said before, it is a per-file property and the config can be
> bypassed by clients that do not read the configs, place a manual API
> override, etc..
>
> If you want to really define a hard maximum and catch such clients,
> try setting dfs.replication.max to 2 at your NameNode.
>
> On Thu, Aug 1, 2013 at 8:07 AM, sam liu <sa...@gmail.com> wrote:
> > But, please mention that the value of 'dfs.replication' of the cluster is
> > always 2, even when the datanode number is 3. And I am pretty sure I did
> not
> > manually create any files with rep=3. So, why were some files of hdfs
> > created with repl=3, but not repl=2?
> >
> >
> > 2013/8/1 Harsh J <ha...@cloudera.com>
> >>
> >> The step (a) points to your problem and solution both. You have files
> >> being created with repl=3 on a 2 DN cluster which will prevent
> >> decommission. This is not a bug.
> >>
> >> On Wed, Jul 31, 2013 at 12:09 PM, sam liu <sa...@gmail.com>
> wrote:
> >> > I opened a jira for tracking this issue:
> >> > https://issues.apache.org/jira/browse/HDFS-5046
> >> >
> >> >
> >> > 2013/7/2 sam liu <sa...@gmail.com>
> >> >>
> >> >> Yes, the default replication factor is 3. However, in my case, it's
> >> >> strange: during decommission hangs, I found some block's expected
> >> >> replicas
> >> >> is 3, but the 'dfs.replication' value in hdfs-site.xml of every
> cluster
> >> >> node
> >> >> is always 2 from the beginning of cluster setup. Below is my steps:
> >> >>
> >> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2.
> And,
> >> >> in
> >> >> hdfs-site.xml, set the 'dfs.replication' to 2
> >> >> 2. Add node dn3 into the cluster as a new datanode, and did not
> change
> >> >> the
> >> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> >> >> note: step 2 passed
> >> >> 3. Decommission dn3 from the cluster
> >> >> Expected result: dn3 could be decommissioned successfully
> >> >> Actual result:
> >> >> a). decommission progress hangs and the status always be 'Waiting
> >> >> DataNode
> >> >> status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2
> /',
> >> >> the
> >> >> decommission continues and will be completed finally.
> >> >> b). However, if the initial cluster includes >= 3 datanodes, this
> issue
> >> >> won't be encountered when add/remove another datanode. For example,
> if
> >> >> I
> >> >> setup a cluster with 3 datanodes, and then I can successfully add the
> >> >> 4th
> >> >> datanode into it, and then also can successfully remove the 4th
> >> >> datanode
> >> >> from the cluster.
> >> >>
> >> >> I doubt it's a bug and plan to open a jira to Hadoop HDFS for this.
> Any
> >> >> comments?
> >> >>
> >> >> Thanks!
> >> >>
> >> >>
> >> >> 2013/6/21 Harsh J <ha...@cloudera.com>
> >> >>>
> >> >>> The dfs.replication is a per-file parameter. If you have a client
> that
> >> >>> does not use the supplied configs, then its default replication is 3
> >> >>> and all files it will create (as part of the app or via a job
> config)
> >> >>> will be with replication factor 3.
> >> >>>
> >> >>> You can do an -lsr to find all files and filter which ones have been
> >> >>> created with a factor of 3 (versus expected config of 2).
> >> >>>
> >> >>> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com>
> >> >>> wrote:
> >> >>> > Hi George,
> >> >>> >
> >> >>> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2.
> >> >>> > But
> >> >>> > still
> >> >>> > encounter this issue.
> >> >>> >
> >> >>> > Thanks!
> >> >>> >
> >> >>> >
> >> >>> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
> >> >>> >>
> >> >>> >>
> >> >>> >> Hi,
> >> >>> >>
> >> >>> >> I think i have faced this before, the problem is that you have
> the
> >> >>> >> rep
> >> >>> >> factor=3 so it seems to hang because it needs 3 nodes to achieve
> >> >>> >> the
> >> >>> >> factor
> >> >>> >> (replicas are not created on the same node). If you set the
> >> >>> >> replication
> >> >>> >> factor=2 i think you will not have this issue. So in general you
> >> >>> >> must
> >> >>> >> make
> >> >>> >> sure that the rep factor is <= to the available datanodes.
> >> >>> >>
> >> >>> >> BR,
> >> >>> >> George
> >> >>> >>
> >> >>> >>
> >> >>> >> On 6/21/2013 12:29 PM, sam liu wrote:
> >> >>> >>
> >> >>> >> Hi,
> >> >>> >>
> >> >>> >> I encountered an issue which hangs the decommission operatoin.
> Its
> >> >>> >> steps:
> >> >>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2.
> >> >>> >> And,
> >> >>> >> in
> >> >>> >> hdfs-site.xml, set the 'dfs.replication' to 2
> >> >>> >> 2. Add node dn3 into the cluster as a new datanode, and did not
> >> >>> >> change
> >> >>> >> the
> >> >>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> >> >>> >> note: step 2 passed
> >> >>> >> 3. Decommission dn3 from the cluster
> >> >>> >>
> >> >>> >> Expected result: dn3 could be decommissioned successfully
> >> >>> >>
> >> >>> >> Actual result: decommission progress hangs and the status always
> be
> >> >>> >> 'Waiting DataNode status: Decommissioned'
> >> >>> >>
> >> >>> >> However, if the initial cluster includes >= 3 datanodes, this
> issue
> >> >>> >> won't
> >> >>> >> be encountered when add/remove another datanode.
> >> >>> >>
> >> >>> >> Also, after step 2, I noticed that some block's expected replicas
> >> >>> >> is
> >> >>> >> 3,
> >> >>> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
> >> >>> >>
> >> >>> >> Could anyone pls help provide some triages?
> >> >>> >>
> >> >>> >> Thanks in advance!
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> --
> >> >>> >> ---------------------------
> >> >>> >>
> >> >>> >> George Kousiouris, PhD
> >> >>> >> Electrical and Computer Engineer
> >> >>> >> Division of Communications,
> >> >>> >> Electronics and Information Engineering
> >> >>> >> School of Electrical and Computer Engineering
> >> >>> >> Tel: +30 210 772 2546
> >> >>> >> Mobile: +30 6939354121
> >> >>> >> Fax: +30 210 772 2569
> >> >>> >> Email: gkousiou@mail.ntua.gr
> >> >>> >> Site: http://users.ntua.gr/gkousiou/
> >> >>> >>
> >> >>> >> National Technical University of Athens
> >> >>> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
> >> >>> >
> >> >>> >
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Harsh J
> >> >>
> >> >>
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

Yes, you are correct: using fsck tool I found some files in my cluster
expected more replications than the value defined in dfs.replication. If I
set the expected replication of this files to a proper number, the
decommissioning process will go smoothly and the datanode could be
decommissioned finally.

However, many users do not mention this and might confuse the situation
that cluster always stays on decommissioning phase, so I think we can make
some improvements to allow user more easily to do precheck before
decommisioning datanode here, and help them find out all filies which might
lack of replications after decommissioning a datanode. For example, we can
told user that the expected replication of file1 and file26 are 6, but
after decommissioning a datanode the datanodes of the cluster will be 5 and
won't satisify file1 and file26 any more. In this way, user can decide
whether to continue the decommissioning work or to reduce the expected
replications of those files. For the way to implementation, I think we
could add a decommission-precheck script or a parameter to fsck tool.

Any comments?


2013/8/1 Harsh J <ha...@cloudera.com>

> As I said before, it is a per-file property and the config can be
> bypassed by clients that do not read the configs, place a manual API
> override, etc..
>
> If you want to really define a hard maximum and catch such clients,
> try setting dfs.replication.max to 2 at your NameNode.
>
> On Thu, Aug 1, 2013 at 8:07 AM, sam liu <sa...@gmail.com> wrote:
> > But, please mention that the value of 'dfs.replication' of the cluster is
> > always 2, even when the datanode number is 3. And I am pretty sure I did
> not
> > manually create any files with rep=3. So, why were some files of hdfs
> > created with repl=3, but not repl=2?
> >
> >
> > 2013/8/1 Harsh J <ha...@cloudera.com>
> >>
> >> The step (a) points to your problem and solution both. You have files
> >> being created with repl=3 on a 2 DN cluster which will prevent
> >> decommission. This is not a bug.
> >>
> >> On Wed, Jul 31, 2013 at 12:09 PM, sam liu <sa...@gmail.com>
> wrote:
> >> > I opened a jira for tracking this issue:
> >> > https://issues.apache.org/jira/browse/HDFS-5046
> >> >
> >> >
> >> > 2013/7/2 sam liu <sa...@gmail.com>
> >> >>
> >> >> Yes, the default replication factor is 3. However, in my case, it's
> >> >> strange: during decommission hangs, I found some block's expected
> >> >> replicas
> >> >> is 3, but the 'dfs.replication' value in hdfs-site.xml of every
> cluster
> >> >> node
> >> >> is always 2 from the beginning of cluster setup. Below is my steps:
> >> >>
> >> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2.
> And,
> >> >> in
> >> >> hdfs-site.xml, set the 'dfs.replication' to 2
> >> >> 2. Add node dn3 into the cluster as a new datanode, and did not
> change
> >> >> the
> >> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> >> >> note: step 2 passed
> >> >> 3. Decommission dn3 from the cluster
> >> >> Expected result: dn3 could be decommissioned successfully
> >> >> Actual result:
> >> >> a). decommission progress hangs and the status always be 'Waiting
> >> >> DataNode
> >> >> status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2
> /',
> >> >> the
> >> >> decommission continues and will be completed finally.
> >> >> b). However, if the initial cluster includes >= 3 datanodes, this
> issue
> >> >> won't be encountered when add/remove another datanode. For example,
> if
> >> >> I
> >> >> setup a cluster with 3 datanodes, and then I can successfully add the
> >> >> 4th
> >> >> datanode into it, and then also can successfully remove the 4th
> >> >> datanode
> >> >> from the cluster.
> >> >>
> >> >> I doubt it's a bug and plan to open a jira to Hadoop HDFS for this.
> Any
> >> >> comments?
> >> >>
> >> >> Thanks!
> >> >>
> >> >>
> >> >> 2013/6/21 Harsh J <ha...@cloudera.com>
> >> >>>
> >> >>> The dfs.replication is a per-file parameter. If you have a client
> that
> >> >>> does not use the supplied configs, then its default replication is 3
> >> >>> and all files it will create (as part of the app or via a job
> config)
> >> >>> will be with replication factor 3.
> >> >>>
> >> >>> You can do an -lsr to find all files and filter which ones have been
> >> >>> created with a factor of 3 (versus expected config of 2).
> >> >>>
> >> >>> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com>
> >> >>> wrote:
> >> >>> > Hi George,
> >> >>> >
> >> >>> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2.
> >> >>> > But
> >> >>> > still
> >> >>> > encounter this issue.
> >> >>> >
> >> >>> > Thanks!
> >> >>> >
> >> >>> >
> >> >>> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
> >> >>> >>
> >> >>> >>
> >> >>> >> Hi,
> >> >>> >>
> >> >>> >> I think i have faced this before, the problem is that you have
> the
> >> >>> >> rep
> >> >>> >> factor=3 so it seems to hang because it needs 3 nodes to achieve
> >> >>> >> the
> >> >>> >> factor
> >> >>> >> (replicas are not created on the same node). If you set the
> >> >>> >> replication
> >> >>> >> factor=2 i think you will not have this issue. So in general you
> >> >>> >> must
> >> >>> >> make
> >> >>> >> sure that the rep factor is <= to the available datanodes.
> >> >>> >>
> >> >>> >> BR,
> >> >>> >> George
> >> >>> >>
> >> >>> >>
> >> >>> >> On 6/21/2013 12:29 PM, sam liu wrote:
> >> >>> >>
> >> >>> >> Hi,
> >> >>> >>
> >> >>> >> I encountered an issue which hangs the decommission operatoin.
> Its
> >> >>> >> steps:
> >> >>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2.
> >> >>> >> And,
> >> >>> >> in
> >> >>> >> hdfs-site.xml, set the 'dfs.replication' to 2
> >> >>> >> 2. Add node dn3 into the cluster as a new datanode, and did not
> >> >>> >> change
> >> >>> >> the
> >> >>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> >> >>> >> note: step 2 passed
> >> >>> >> 3. Decommission dn3 from the cluster
> >> >>> >>
> >> >>> >> Expected result: dn3 could be decommissioned successfully
> >> >>> >>
> >> >>> >> Actual result: decommission progress hangs and the status always
> be
> >> >>> >> 'Waiting DataNode status: Decommissioned'
> >> >>> >>
> >> >>> >> However, if the initial cluster includes >= 3 datanodes, this
> issue
> >> >>> >> won't
> >> >>> >> be encountered when add/remove another datanode.
> >> >>> >>
> >> >>> >> Also, after step 2, I noticed that some block's expected replicas
> >> >>> >> is
> >> >>> >> 3,
> >> >>> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
> >> >>> >>
> >> >>> >> Could anyone pls help provide some triages?
> >> >>> >>
> >> >>> >> Thanks in advance!
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> --
> >> >>> >> ---------------------------
> >> >>> >>
> >> >>> >> George Kousiouris, PhD
> >> >>> >> Electrical and Computer Engineer
> >> >>> >> Division of Communications,
> >> >>> >> Electronics and Information Engineering
> >> >>> >> School of Electrical and Computer Engineering
> >> >>> >> Tel: +30 210 772 2546
> >> >>> >> Mobile: +30 6939354121
> >> >>> >> Fax: +30 210 772 2569
> >> >>> >> Email: gkousiou@mail.ntua.gr
> >> >>> >> Site: http://users.ntua.gr/gkousiou/
> >> >>> >>
> >> >>> >> National Technical University of Athens
> >> >>> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
> >> >>> >
> >> >>> >
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Harsh J
> >> >>
> >> >>
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[Harsh J <ha...@cloudera.com>]

As I said before, it is a per-file property and the config can be
bypassed by clients that do not read the configs, place a manual API
override, etc..

If you want to really define a hard maximum and catch such clients,
try setting dfs.replication.max to 2 at your NameNode.

On Thu, Aug 1, 2013 at 8:07 AM, sam liu <sa...@gmail.com> wrote:
> But, please mention that the value of 'dfs.replication' of the cluster is
> always 2, even when the datanode number is 3. And I am pretty sure I did not
> manually create any files with rep=3. So, why were some files of hdfs
> created with repl=3, but not repl=2?
>
>
> 2013/8/1 Harsh J <ha...@cloudera.com>
>>
>> The step (a) points to your problem and solution both. You have files
>> being created with repl=3 on a 2 DN cluster which will prevent
>> decommission. This is not a bug.
>>
>> On Wed, Jul 31, 2013 at 12:09 PM, sam liu <sa...@gmail.com> wrote:
>> > I opened a jira for tracking this issue:
>> > https://issues.apache.org/jira/browse/HDFS-5046
>> >
>> >
>> > 2013/7/2 sam liu <sa...@gmail.com>
>> >>
>> >> Yes, the default replication factor is 3. However, in my case, it's
>> >> strange: during decommission hangs, I found some block's expected
>> >> replicas
>> >> is 3, but the 'dfs.replication' value in hdfs-site.xml of every cluster
>> >> node
>> >> is always 2 from the beginning of cluster setup. Below is my steps:
>> >>
>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
>> >> in
>> >> hdfs-site.xml, set the 'dfs.replication' to 2
>> >> 2. Add node dn3 into the cluster as a new datanode, and did not change
>> >> the
>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>> >> note: step 2 passed
>> >> 3. Decommission dn3 from the cluster
>> >> Expected result: dn3 could be decommissioned successfully
>> >> Actual result:
>> >> a). decommission progress hangs and the status always be 'Waiting
>> >> DataNode
>> >> status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /',
>> >> the
>> >> decommission continues and will be completed finally.
>> >> b). However, if the initial cluster includes >= 3 datanodes, this issue
>> >> won't be encountered when add/remove another datanode. For example, if
>> >> I
>> >> setup a cluster with 3 datanodes, and then I can successfully add the
>> >> 4th
>> >> datanode into it, and then also can successfully remove the 4th
>> >> datanode
>> >> from the cluster.
>> >>
>> >> I doubt it's a bug and plan to open a jira to Hadoop HDFS for this. Any
>> >> comments?
>> >>
>> >> Thanks!
>> >>
>> >>
>> >> 2013/6/21 Harsh J <ha...@cloudera.com>
>> >>>
>> >>> The dfs.replication is a per-file parameter. If you have a client that
>> >>> does not use the supplied configs, then its default replication is 3
>> >>> and all files it will create (as part of the app or via a job config)
>> >>> will be with replication factor 3.
>> >>>
>> >>> You can do an -lsr to find all files and filter which ones have been
>> >>> created with a factor of 3 (versus expected config of 2).
>> >>>
>> >>> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com>
>> >>> wrote:
>> >>> > Hi George,
>> >>> >
>> >>> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2.
>> >>> > But
>> >>> > still
>> >>> > encounter this issue.
>> >>> >
>> >>> > Thanks!
>> >>> >
>> >>> >
>> >>> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
>> >>> >>
>> >>> >>
>> >>> >> Hi,
>> >>> >>
>> >>> >> I think i have faced this before, the problem is that you have the
>> >>> >> rep
>> >>> >> factor=3 so it seems to hang because it needs 3 nodes to achieve
>> >>> >> the
>> >>> >> factor
>> >>> >> (replicas are not created on the same node). If you set the
>> >>> >> replication
>> >>> >> factor=2 i think you will not have this issue. So in general you
>> >>> >> must
>> >>> >> make
>> >>> >> sure that the rep factor is <= to the available datanodes.
>> >>> >>
>> >>> >> BR,
>> >>> >> George
>> >>> >>
>> >>> >>
>> >>> >> On 6/21/2013 12:29 PM, sam liu wrote:
>> >>> >>
>> >>> >> Hi,
>> >>> >>
>> >>> >> I encountered an issue which hangs the decommission operatoin. Its
>> >>> >> steps:
>> >>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2.
>> >>> >> And,
>> >>> >> in
>> >>> >> hdfs-site.xml, set the 'dfs.replication' to 2
>> >>> >> 2. Add node dn3 into the cluster as a new datanode, and did not
>> >>> >> change
>> >>> >> the
>> >>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>> >>> >> note: step 2 passed
>> >>> >> 3. Decommission dn3 from the cluster
>> >>> >>
>> >>> >> Expected result: dn3 could be decommissioned successfully
>> >>> >>
>> >>> >> Actual result: decommission progress hangs and the status always be
>> >>> >> 'Waiting DataNode status: Decommissioned'
>> >>> >>
>> >>> >> However, if the initial cluster includes >= 3 datanodes, this issue
>> >>> >> won't
>> >>> >> be encountered when add/remove another datanode.
>> >>> >>
>> >>> >> Also, after step 2, I noticed that some block's expected replicas
>> >>> >> is
>> >>> >> 3,
>> >>> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
>> >>> >>
>> >>> >> Could anyone pls help provide some triages?
>> >>> >>
>> >>> >> Thanks in advance!
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> ---------------------------
>> >>> >>
>> >>> >> George Kousiouris, PhD
>> >>> >> Electrical and Computer Engineer
>> >>> >> Division of Communications,
>> >>> >> Electronics and Information Engineering
>> >>> >> School of Electrical and Computer Engineering
>> >>> >> Tel: +30 210 772 2546
>> >>> >> Mobile: +30 6939354121
>> >>> >> Fax: +30 210 772 2569
>> >>> >> Email: gkousiou@mail.ntua.gr
>> >>> >> Site: http://users.ntua.gr/gkousiou/
>> >>> >>
>> >>> >> National Technical University of Athens
>> >>> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
>> >>> >
>> >>> >
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Harsh J
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>
>



-- 
Harsh J

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[Harsh J <ha...@cloudera.com>]

As I said before, it is a per-file property and the config can be
bypassed by clients that do not read the configs, place a manual API
override, etc..

If you want to really define a hard maximum and catch such clients,
try setting dfs.replication.max to 2 at your NameNode.

On Thu, Aug 1, 2013 at 8:07 AM, sam liu <sa...@gmail.com> wrote:
> But, please mention that the value of 'dfs.replication' of the cluster is
> always 2, even when the datanode number is 3. And I am pretty sure I did not
> manually create any files with rep=3. So, why were some files of hdfs
> created with repl=3, but not repl=2?
>
>
> 2013/8/1 Harsh J <ha...@cloudera.com>
>>
>> The step (a) points to your problem and solution both. You have files
>> being created with repl=3 on a 2 DN cluster which will prevent
>> decommission. This is not a bug.
>>
>> On Wed, Jul 31, 2013 at 12:09 PM, sam liu <sa...@gmail.com> wrote:
>> > I opened a jira for tracking this issue:
>> > https://issues.apache.org/jira/browse/HDFS-5046
>> >
>> >
>> > 2013/7/2 sam liu <sa...@gmail.com>
>> >>
>> >> Yes, the default replication factor is 3. However, in my case, it's
>> >> strange: during decommission hangs, I found some block's expected
>> >> replicas
>> >> is 3, but the 'dfs.replication' value in hdfs-site.xml of every cluster
>> >> node
>> >> is always 2 from the beginning of cluster setup. Below is my steps:
>> >>
>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
>> >> in
>> >> hdfs-site.xml, set the 'dfs.replication' to 2
>> >> 2. Add node dn3 into the cluster as a new datanode, and did not change
>> >> the
>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>> >> note: step 2 passed
>> >> 3. Decommission dn3 from the cluster
>> >> Expected result: dn3 could be decommissioned successfully
>> >> Actual result:
>> >> a). decommission progress hangs and the status always be 'Waiting
>> >> DataNode
>> >> status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /',
>> >> the
>> >> decommission continues and will be completed finally.
>> >> b). However, if the initial cluster includes >= 3 datanodes, this issue
>> >> won't be encountered when add/remove another datanode. For example, if
>> >> I
>> >> setup a cluster with 3 datanodes, and then I can successfully add the
>> >> 4th
>> >> datanode into it, and then also can successfully remove the 4th
>> >> datanode
>> >> from the cluster.
>> >>
>> >> I doubt it's a bug and plan to open a jira to Hadoop HDFS for this. Any
>> >> comments?
>> >>
>> >> Thanks!
>> >>
>> >>
>> >> 2013/6/21 Harsh J <ha...@cloudera.com>
>> >>>
>> >>> The dfs.replication is a per-file parameter. If you have a client that
>> >>> does not use the supplied configs, then its default replication is 3
>> >>> and all files it will create (as part of the app or via a job config)
>> >>> will be with replication factor 3.
>> >>>
>> >>> You can do an -lsr to find all files and filter which ones have been
>> >>> created with a factor of 3 (versus expected config of 2).
>> >>>
>> >>> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com>
>> >>> wrote:
>> >>> > Hi George,
>> >>> >
>> >>> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2.
>> >>> > But
>> >>> > still
>> >>> > encounter this issue.
>> >>> >
>> >>> > Thanks!
>> >>> >
>> >>> >
>> >>> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
>> >>> >>
>> >>> >>
>> >>> >> Hi,
>> >>> >>
>> >>> >> I think i have faced this before, the problem is that you have the
>> >>> >> rep
>> >>> >> factor=3 so it seems to hang because it needs 3 nodes to achieve
>> >>> >> the
>> >>> >> factor
>> >>> >> (replicas are not created on the same node). If you set the
>> >>> >> replication
>> >>> >> factor=2 i think you will not have this issue. So in general you
>> >>> >> must
>> >>> >> make
>> >>> >> sure that the rep factor is <= to the available datanodes.
>> >>> >>
>> >>> >> BR,
>> >>> >> George
>> >>> >>
>> >>> >>
>> >>> >> On 6/21/2013 12:29 PM, sam liu wrote:
>> >>> >>
>> >>> >> Hi,
>> >>> >>
>> >>> >> I encountered an issue which hangs the decommission operatoin. Its
>> >>> >> steps:
>> >>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2.
>> >>> >> And,
>> >>> >> in
>> >>> >> hdfs-site.xml, set the 'dfs.replication' to 2
>> >>> >> 2. Add node dn3 into the cluster as a new datanode, and did not
>> >>> >> change
>> >>> >> the
>> >>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>> >>> >> note: step 2 passed
>> >>> >> 3. Decommission dn3 from the cluster
>> >>> >>
>> >>> >> Expected result: dn3 could be decommissioned successfully
>> >>> >>
>> >>> >> Actual result: decommission progress hangs and the status always be
>> >>> >> 'Waiting DataNode status: Decommissioned'
>> >>> >>
>> >>> >> However, if the initial cluster includes >= 3 datanodes, this issue
>> >>> >> won't
>> >>> >> be encountered when add/remove another datanode.
>> >>> >>
>> >>> >> Also, after step 2, I noticed that some block's expected replicas
>> >>> >> is
>> >>> >> 3,
>> >>> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
>> >>> >>
>> >>> >> Could anyone pls help provide some triages?
>> >>> >>
>> >>> >> Thanks in advance!
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> ---------------------------
>> >>> >>
>> >>> >> George Kousiouris, PhD
>> >>> >> Electrical and Computer Engineer
>> >>> >> Division of Communications,
>> >>> >> Electronics and Information Engineering
>> >>> >> School of Electrical and Computer Engineering
>> >>> >> Tel: +30 210 772 2546
>> >>> >> Mobile: +30 6939354121
>> >>> >> Fax: +30 210 772 2569
>> >>> >> Email: gkousiou@mail.ntua.gr
>> >>> >> Site: http://users.ntua.gr/gkousiou/
>> >>> >>
>> >>> >> National Technical University of Athens
>> >>> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
>> >>> >
>> >>> >
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Harsh J
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>
>



-- 
Harsh J

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[Harsh J <ha...@cloudera.com>]

As I said before, it is a per-file property and the config can be
bypassed by clients that do not read the configs, place a manual API
override, etc..

If you want to really define a hard maximum and catch such clients,
try setting dfs.replication.max to 2 at your NameNode.

On Thu, Aug 1, 2013 at 8:07 AM, sam liu <sa...@gmail.com> wrote:
> But, please mention that the value of 'dfs.replication' of the cluster is
> always 2, even when the datanode number is 3. And I am pretty sure I did not
> manually create any files with rep=3. So, why were some files of hdfs
> created with repl=3, but not repl=2?
>
>
> 2013/8/1 Harsh J <ha...@cloudera.com>
>>
>> The step (a) points to your problem and solution both. You have files
>> being created with repl=3 on a 2 DN cluster which will prevent
>> decommission. This is not a bug.
>>
>> On Wed, Jul 31, 2013 at 12:09 PM, sam liu <sa...@gmail.com> wrote:
>> > I opened a jira for tracking this issue:
>> > https://issues.apache.org/jira/browse/HDFS-5046
>> >
>> >
>> > 2013/7/2 sam liu <sa...@gmail.com>
>> >>
>> >> Yes, the default replication factor is 3. However, in my case, it's
>> >> strange: during decommission hangs, I found some block's expected
>> >> replicas
>> >> is 3, but the 'dfs.replication' value in hdfs-site.xml of every cluster
>> >> node
>> >> is always 2 from the beginning of cluster setup. Below is my steps:
>> >>
>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
>> >> in
>> >> hdfs-site.xml, set the 'dfs.replication' to 2
>> >> 2. Add node dn3 into the cluster as a new datanode, and did not change
>> >> the
>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>> >> note: step 2 passed
>> >> 3. Decommission dn3 from the cluster
>> >> Expected result: dn3 could be decommissioned successfully
>> >> Actual result:
>> >> a). decommission progress hangs and the status always be 'Waiting
>> >> DataNode
>> >> status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /',
>> >> the
>> >> decommission continues and will be completed finally.
>> >> b). However, if the initial cluster includes >= 3 datanodes, this issue
>> >> won't be encountered when add/remove another datanode. For example, if
>> >> I
>> >> setup a cluster with 3 datanodes, and then I can successfully add the
>> >> 4th
>> >> datanode into it, and then also can successfully remove the 4th
>> >> datanode
>> >> from the cluster.
>> >>
>> >> I doubt it's a bug and plan to open a jira to Hadoop HDFS for this. Any
>> >> comments?
>> >>
>> >> Thanks!
>> >>
>> >>
>> >> 2013/6/21 Harsh J <ha...@cloudera.com>
>> >>>
>> >>> The dfs.replication is a per-file parameter. If you have a client that
>> >>> does not use the supplied configs, then its default replication is 3
>> >>> and all files it will create (as part of the app or via a job config)
>> >>> will be with replication factor 3.
>> >>>
>> >>> You can do an -lsr to find all files and filter which ones have been
>> >>> created with a factor of 3 (versus expected config of 2).
>> >>>
>> >>> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com>
>> >>> wrote:
>> >>> > Hi George,
>> >>> >
>> >>> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2.
>> >>> > But
>> >>> > still
>> >>> > encounter this issue.
>> >>> >
>> >>> > Thanks!
>> >>> >
>> >>> >
>> >>> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
>> >>> >>
>> >>> >>
>> >>> >> Hi,
>> >>> >>
>> >>> >> I think i have faced this before, the problem is that you have the
>> >>> >> rep
>> >>> >> factor=3 so it seems to hang because it needs 3 nodes to achieve
>> >>> >> the
>> >>> >> factor
>> >>> >> (replicas are not created on the same node). If you set the
>> >>> >> replication
>> >>> >> factor=2 i think you will not have this issue. So in general you
>> >>> >> must
>> >>> >> make
>> >>> >> sure that the rep factor is <= to the available datanodes.
>> >>> >>
>> >>> >> BR,
>> >>> >> George
>> >>> >>
>> >>> >>
>> >>> >> On 6/21/2013 12:29 PM, sam liu wrote:
>> >>> >>
>> >>> >> Hi,
>> >>> >>
>> >>> >> I encountered an issue which hangs the decommission operatoin. Its
>> >>> >> steps:
>> >>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2.
>> >>> >> And,
>> >>> >> in
>> >>> >> hdfs-site.xml, set the 'dfs.replication' to 2
>> >>> >> 2. Add node dn3 into the cluster as a new datanode, and did not
>> >>> >> change
>> >>> >> the
>> >>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>> >>> >> note: step 2 passed
>> >>> >> 3. Decommission dn3 from the cluster
>> >>> >>
>> >>> >> Expected result: dn3 could be decommissioned successfully
>> >>> >>
>> >>> >> Actual result: decommission progress hangs and the status always be
>> >>> >> 'Waiting DataNode status: Decommissioned'
>> >>> >>
>> >>> >> However, if the initial cluster includes >= 3 datanodes, this issue
>> >>> >> won't
>> >>> >> be encountered when add/remove another datanode.
>> >>> >>
>> >>> >> Also, after step 2, I noticed that some block's expected replicas
>> >>> >> is
>> >>> >> 3,
>> >>> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
>> >>> >>
>> >>> >> Could anyone pls help provide some triages?
>> >>> >>
>> >>> >> Thanks in advance!
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> ---------------------------
>> >>> >>
>> >>> >> George Kousiouris, PhD
>> >>> >> Electrical and Computer Engineer
>> >>> >> Division of Communications,
>> >>> >> Electronics and Information Engineering
>> >>> >> School of Electrical and Computer Engineering
>> >>> >> Tel: +30 210 772 2546
>> >>> >> Mobile: +30 6939354121
>> >>> >> Fax: +30 210 772 2569
>> >>> >> Email: gkousiou@mail.ntua.gr
>> >>> >> Site: http://users.ntua.gr/gkousiou/
>> >>> >>
>> >>> >> National Technical University of Athens
>> >>> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
>> >>> >
>> >>> >
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Harsh J
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>
>



-- 
Harsh J

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[Harsh J <ha...@cloudera.com>]

As I said before, it is a per-file property and the config can be
bypassed by clients that do not read the configs, place a manual API
override, etc..

If you want to really define a hard maximum and catch such clients,
try setting dfs.replication.max to 2 at your NameNode.

On Thu, Aug 1, 2013 at 8:07 AM, sam liu <sa...@gmail.com> wrote:
> But, please mention that the value of 'dfs.replication' of the cluster is
> always 2, even when the datanode number is 3. And I am pretty sure I did not
> manually create any files with rep=3. So, why were some files of hdfs
> created with repl=3, but not repl=2?
>
>
> 2013/8/1 Harsh J <ha...@cloudera.com>
>>
>> The step (a) points to your problem and solution both. You have files
>> being created with repl=3 on a 2 DN cluster which will prevent
>> decommission. This is not a bug.
>>
>> On Wed, Jul 31, 2013 at 12:09 PM, sam liu <sa...@gmail.com> wrote:
>> > I opened a jira for tracking this issue:
>> > https://issues.apache.org/jira/browse/HDFS-5046
>> >
>> >
>> > 2013/7/2 sam liu <sa...@gmail.com>
>> >>
>> >> Yes, the default replication factor is 3. However, in my case, it's
>> >> strange: during decommission hangs, I found some block's expected
>> >> replicas
>> >> is 3, but the 'dfs.replication' value in hdfs-site.xml of every cluster
>> >> node
>> >> is always 2 from the beginning of cluster setup. Below is my steps:
>> >>
>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
>> >> in
>> >> hdfs-site.xml, set the 'dfs.replication' to 2
>> >> 2. Add node dn3 into the cluster as a new datanode, and did not change
>> >> the
>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>> >> note: step 2 passed
>> >> 3. Decommission dn3 from the cluster
>> >> Expected result: dn3 could be decommissioned successfully
>> >> Actual result:
>> >> a). decommission progress hangs and the status always be 'Waiting
>> >> DataNode
>> >> status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /',
>> >> the
>> >> decommission continues and will be completed finally.
>> >> b). However, if the initial cluster includes >= 3 datanodes, this issue
>> >> won't be encountered when add/remove another datanode. For example, if
>> >> I
>> >> setup a cluster with 3 datanodes, and then I can successfully add the
>> >> 4th
>> >> datanode into it, and then also can successfully remove the 4th
>> >> datanode
>> >> from the cluster.
>> >>
>> >> I doubt it's a bug and plan to open a jira to Hadoop HDFS for this. Any
>> >> comments?
>> >>
>> >> Thanks!
>> >>
>> >>
>> >> 2013/6/21 Harsh J <ha...@cloudera.com>
>> >>>
>> >>> The dfs.replication is a per-file parameter. If you have a client that
>> >>> does not use the supplied configs, then its default replication is 3
>> >>> and all files it will create (as part of the app or via a job config)
>> >>> will be with replication factor 3.
>> >>>
>> >>> You can do an -lsr to find all files and filter which ones have been
>> >>> created with a factor of 3 (versus expected config of 2).
>> >>>
>> >>> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com>
>> >>> wrote:
>> >>> > Hi George,
>> >>> >
>> >>> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2.
>> >>> > But
>> >>> > still
>> >>> > encounter this issue.
>> >>> >
>> >>> > Thanks!
>> >>> >
>> >>> >
>> >>> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
>> >>> >>
>> >>> >>
>> >>> >> Hi,
>> >>> >>
>> >>> >> I think i have faced this before, the problem is that you have the
>> >>> >> rep
>> >>> >> factor=3 so it seems to hang because it needs 3 nodes to achieve
>> >>> >> the
>> >>> >> factor
>> >>> >> (replicas are not created on the same node). If you set the
>> >>> >> replication
>> >>> >> factor=2 i think you will not have this issue. So in general you
>> >>> >> must
>> >>> >> make
>> >>> >> sure that the rep factor is <= to the available datanodes.
>> >>> >>
>> >>> >> BR,
>> >>> >> George
>> >>> >>
>> >>> >>
>> >>> >> On 6/21/2013 12:29 PM, sam liu wrote:
>> >>> >>
>> >>> >> Hi,
>> >>> >>
>> >>> >> I encountered an issue which hangs the decommission operatoin. Its
>> >>> >> steps:
>> >>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2.
>> >>> >> And,
>> >>> >> in
>> >>> >> hdfs-site.xml, set the 'dfs.replication' to 2
>> >>> >> 2. Add node dn3 into the cluster as a new datanode, and did not
>> >>> >> change
>> >>> >> the
>> >>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>> >>> >> note: step 2 passed
>> >>> >> 3. Decommission dn3 from the cluster
>> >>> >>
>> >>> >> Expected result: dn3 could be decommissioned successfully
>> >>> >>
>> >>> >> Actual result: decommission progress hangs and the status always be
>> >>> >> 'Waiting DataNode status: Decommissioned'
>> >>> >>
>> >>> >> However, if the initial cluster includes >= 3 datanodes, this issue
>> >>> >> won't
>> >>> >> be encountered when add/remove another datanode.
>> >>> >>
>> >>> >> Also, after step 2, I noticed that some block's expected replicas
>> >>> >> is
>> >>> >> 3,
>> >>> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
>> >>> >>
>> >>> >> Could anyone pls help provide some triages?
>> >>> >>
>> >>> >> Thanks in advance!
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> ---------------------------
>> >>> >>
>> >>> >> George Kousiouris, PhD
>> >>> >> Electrical and Computer Engineer
>> >>> >> Division of Communications,
>> >>> >> Electronics and Information Engineering
>> >>> >> School of Electrical and Computer Engineering
>> >>> >> Tel: +30 210 772 2546
>> >>> >> Mobile: +30 6939354121
>> >>> >> Fax: +30 210 772 2569
>> >>> >> Email: gkousiou@mail.ntua.gr
>> >>> >> Site: http://users.ntua.gr/gkousiou/
>> >>> >>
>> >>> >> National Technical University of Athens
>> >>> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
>> >>> >
>> >>> >
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Harsh J
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Harsh J
>
>



-- 
Harsh J

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

But, please mention that the value of 'dfs.replication' of the cluster is
always 2, even when the datanode number is 3. And I am pretty sure I did
not manually create any files with rep=3. So, why were some files of hdfs
created with repl=3, but not repl=2?


2013/8/1 Harsh J <ha...@cloudera.com>

> The step (a) points to your problem and solution both. You have files
> being created with repl=3 on a 2 DN cluster which will prevent
> decommission. This is not a bug.
>
> On Wed, Jul 31, 2013 at 12:09 PM, sam liu <sa...@gmail.com> wrote:
> > I opened a jira for tracking this issue:
> > https://issues.apache.org/jira/browse/HDFS-5046
> >
> >
> > 2013/7/2 sam liu <sa...@gmail.com>
> >>
> >> Yes, the default replication factor is 3. However, in my case, it's
> >> strange: during decommission hangs, I found some block's expected
> replicas
> >> is 3, but the 'dfs.replication' value in hdfs-site.xml of every cluster
> node
> >> is always 2 from the beginning of cluster setup. Below is my steps:
> >>
> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
> in
> >> hdfs-site.xml, set the 'dfs.replication' to 2
> >> 2. Add node dn3 into the cluster as a new datanode, and did not change
> the
> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> >> note: step 2 passed
> >> 3. Decommission dn3 from the cluster
> >> Expected result: dn3 could be decommissioned successfully
> >> Actual result:
> >> a). decommission progress hangs and the status always be 'Waiting
> DataNode
> >> status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /',
> the
> >> decommission continues and will be completed finally.
> >> b). However, if the initial cluster includes >= 3 datanodes, this issue
> >> won't be encountered when add/remove another datanode. For example, if I
> >> setup a cluster with 3 datanodes, and then I can successfully add the
> 4th
> >> datanode into it, and then also can successfully remove the 4th datanode
> >> from the cluster.
> >>
> >> I doubt it's a bug and plan to open a jira to Hadoop HDFS for this. Any
> >> comments?
> >>
> >> Thanks!
> >>
> >>
> >> 2013/6/21 Harsh J <ha...@cloudera.com>
> >>>
> >>> The dfs.replication is a per-file parameter. If you have a client that
> >>> does not use the supplied configs, then its default replication is 3
> >>> and all files it will create (as part of the app or via a job config)
> >>> will be with replication factor 3.
> >>>
> >>> You can do an -lsr to find all files and filter which ones have been
> >>> created with a factor of 3 (versus expected config of 2).
> >>>
> >>> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com>
> wrote:
> >>> > Hi George,
> >>> >
> >>> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2.
> But
> >>> > still
> >>> > encounter this issue.
> >>> >
> >>> > Thanks!
> >>> >
> >>> >
> >>> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
> >>> >>
> >>> >>
> >>> >> Hi,
> >>> >>
> >>> >> I think i have faced this before, the problem is that you have the
> rep
> >>> >> factor=3 so it seems to hang because it needs 3 nodes to achieve the
> >>> >> factor
> >>> >> (replicas are not created on the same node). If you set the
> >>> >> replication
> >>> >> factor=2 i think you will not have this issue. So in general you
> must
> >>> >> make
> >>> >> sure that the rep factor is <= to the available datanodes.
> >>> >>
> >>> >> BR,
> >>> >> George
> >>> >>
> >>> >>
> >>> >> On 6/21/2013 12:29 PM, sam liu wrote:
> >>> >>
> >>> >> Hi,
> >>> >>
> >>> >> I encountered an issue which hangs the decommission operatoin. Its
> >>> >> steps:
> >>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2.
> And,
> >>> >> in
> >>> >> hdfs-site.xml, set the 'dfs.replication' to 2
> >>> >> 2. Add node dn3 into the cluster as a new datanode, and did not
> change
> >>> >> the
> >>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> >>> >> note: step 2 passed
> >>> >> 3. Decommission dn3 from the cluster
> >>> >>
> >>> >> Expected result: dn3 could be decommissioned successfully
> >>> >>
> >>> >> Actual result: decommission progress hangs and the status always be
> >>> >> 'Waiting DataNode status: Decommissioned'
> >>> >>
> >>> >> However, if the initial cluster includes >= 3 datanodes, this issue
> >>> >> won't
> >>> >> be encountered when add/remove another datanode.
> >>> >>
> >>> >> Also, after step 2, I noticed that some block's expected replicas is
> >>> >> 3,
> >>> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
> >>> >>
> >>> >> Could anyone pls help provide some triages?
> >>> >>
> >>> >> Thanks in advance!
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> ---------------------------
> >>> >>
> >>> >> George Kousiouris, PhD
> >>> >> Electrical and Computer Engineer
> >>> >> Division of Communications,
> >>> >> Electronics and Information Engineering
> >>> >> School of Electrical and Computer Engineering
> >>> >> Tel: +30 210 772 2546
> >>> >> Mobile: +30 6939354121
> >>> >> Fax: +30 210 772 2569
> >>> >> Email: gkousiou@mail.ntua.gr
> >>> >> Site: http://users.ntua.gr/gkousiou/
> >>> >>
> >>> >> National Technical University of Athens
> >>> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Harsh J
> >>
> >>
> >
>
>
>
> --
> Harsh J
>

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

But, please mention that the value of 'dfs.replication' of the cluster is
always 2, even when the datanode number is 3. And I am pretty sure I did
not manually create any files with rep=3. So, why were some files of hdfs
created with repl=3, but not repl=2?


2013/8/1 Harsh J <ha...@cloudera.com>

> The step (a) points to your problem and solution both. You have files
> being created with repl=3 on a 2 DN cluster which will prevent
> decommission. This is not a bug.
>
> On Wed, Jul 31, 2013 at 12:09 PM, sam liu <sa...@gmail.com> wrote:
> > I opened a jira for tracking this issue:
> > https://issues.apache.org/jira/browse/HDFS-5046
> >
> >
> > 2013/7/2 sam liu <sa...@gmail.com>
> >>
> >> Yes, the default replication factor is 3. However, in my case, it's
> >> strange: during decommission hangs, I found some block's expected
> replicas
> >> is 3, but the 'dfs.replication' value in hdfs-site.xml of every cluster
> node
> >> is always 2 from the beginning of cluster setup. Below is my steps:
> >>
> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
> in
> >> hdfs-site.xml, set the 'dfs.replication' to 2
> >> 2. Add node dn3 into the cluster as a new datanode, and did not change
> the
> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> >> note: step 2 passed
> >> 3. Decommission dn3 from the cluster
> >> Expected result: dn3 could be decommissioned successfully
> >> Actual result:
> >> a). decommission progress hangs and the status always be 'Waiting
> DataNode
> >> status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /',
> the
> >> decommission continues and will be completed finally.
> >> b). However, if the initial cluster includes >= 3 datanodes, this issue
> >> won't be encountered when add/remove another datanode. For example, if I
> >> setup a cluster with 3 datanodes, and then I can successfully add the
> 4th
> >> datanode into it, and then also can successfully remove the 4th datanode
> >> from the cluster.
> >>
> >> I doubt it's a bug and plan to open a jira to Hadoop HDFS for this. Any
> >> comments?
> >>
> >> Thanks!
> >>
> >>
> >> 2013/6/21 Harsh J <ha...@cloudera.com>
> >>>
> >>> The dfs.replication is a per-file parameter. If you have a client that
> >>> does not use the supplied configs, then its default replication is 3
> >>> and all files it will create (as part of the app or via a job config)
> >>> will be with replication factor 3.
> >>>
> >>> You can do an -lsr to find all files and filter which ones have been
> >>> created with a factor of 3 (versus expected config of 2).
> >>>
> >>> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com>
> wrote:
> >>> > Hi George,
> >>> >
> >>> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2.
> But
> >>> > still
> >>> > encounter this issue.
> >>> >
> >>> > Thanks!
> >>> >
> >>> >
> >>> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
> >>> >>
> >>> >>
> >>> >> Hi,
> >>> >>
> >>> >> I think i have faced this before, the problem is that you have the
> rep
> >>> >> factor=3 so it seems to hang because it needs 3 nodes to achieve the
> >>> >> factor
> >>> >> (replicas are not created on the same node). If you set the
> >>> >> replication
> >>> >> factor=2 i think you will not have this issue. So in general you
> must
> >>> >> make
> >>> >> sure that the rep factor is <= to the available datanodes.
> >>> >>
> >>> >> BR,
> >>> >> George
> >>> >>
> >>> >>
> >>> >> On 6/21/2013 12:29 PM, sam liu wrote:
> >>> >>
> >>> >> Hi,
> >>> >>
> >>> >> I encountered an issue which hangs the decommission operatoin. Its
> >>> >> steps:
> >>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2.
> And,
> >>> >> in
> >>> >> hdfs-site.xml, set the 'dfs.replication' to 2
> >>> >> 2. Add node dn3 into the cluster as a new datanode, and did not
> change
> >>> >> the
> >>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> >>> >> note: step 2 passed
> >>> >> 3. Decommission dn3 from the cluster
> >>> >>
> >>> >> Expected result: dn3 could be decommissioned successfully
> >>> >>
> >>> >> Actual result: decommission progress hangs and the status always be
> >>> >> 'Waiting DataNode status: Decommissioned'
> >>> >>
> >>> >> However, if the initial cluster includes >= 3 datanodes, this issue
> >>> >> won't
> >>> >> be encountered when add/remove another datanode.
> >>> >>
> >>> >> Also, after step 2, I noticed that some block's expected replicas is
> >>> >> 3,
> >>> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
> >>> >>
> >>> >> Could anyone pls help provide some triages?
> >>> >>
> >>> >> Thanks in advance!
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> ---------------------------
> >>> >>
> >>> >> George Kousiouris, PhD
> >>> >> Electrical and Computer Engineer
> >>> >> Division of Communications,
> >>> >> Electronics and Information Engineering
> >>> >> School of Electrical and Computer Engineering
> >>> >> Tel: +30 210 772 2546
> >>> >> Mobile: +30 6939354121
> >>> >> Fax: +30 210 772 2569
> >>> >> Email: gkousiou@mail.ntua.gr
> >>> >> Site: http://users.ntua.gr/gkousiou/
> >>> >>
> >>> >> National Technical University of Athens
> >>> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Harsh J
> >>
> >>
> >
>
>
>
> --
> Harsh J
>

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

But, please mention that the value of 'dfs.replication' of the cluster is
always 2, even when the datanode number is 3. And I am pretty sure I did
not manually create any files with rep=3. So, why were some files of hdfs
created with repl=3, but not repl=2?


2013/8/1 Harsh J <ha...@cloudera.com>

> The step (a) points to your problem and solution both. You have files
> being created with repl=3 on a 2 DN cluster which will prevent
> decommission. This is not a bug.
>
> On Wed, Jul 31, 2013 at 12:09 PM, sam liu <sa...@gmail.com> wrote:
> > I opened a jira for tracking this issue:
> > https://issues.apache.org/jira/browse/HDFS-5046
> >
> >
> > 2013/7/2 sam liu <sa...@gmail.com>
> >>
> >> Yes, the default replication factor is 3. However, in my case, it's
> >> strange: during decommission hangs, I found some block's expected
> replicas
> >> is 3, but the 'dfs.replication' value in hdfs-site.xml of every cluster
> node
> >> is always 2 from the beginning of cluster setup. Below is my steps:
> >>
> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
> in
> >> hdfs-site.xml, set the 'dfs.replication' to 2
> >> 2. Add node dn3 into the cluster as a new datanode, and did not change
> the
> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> >> note: step 2 passed
> >> 3. Decommission dn3 from the cluster
> >> Expected result: dn3 could be decommissioned successfully
> >> Actual result:
> >> a). decommission progress hangs and the status always be 'Waiting
> DataNode
> >> status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /',
> the
> >> decommission continues and will be completed finally.
> >> b). However, if the initial cluster includes >= 3 datanodes, this issue
> >> won't be encountered when add/remove another datanode. For example, if I
> >> setup a cluster with 3 datanodes, and then I can successfully add the
> 4th
> >> datanode into it, and then also can successfully remove the 4th datanode
> >> from the cluster.
> >>
> >> I doubt it's a bug and plan to open a jira to Hadoop HDFS for this. Any
> >> comments?
> >>
> >> Thanks!
> >>
> >>
> >> 2013/6/21 Harsh J <ha...@cloudera.com>
> >>>
> >>> The dfs.replication is a per-file parameter. If you have a client that
> >>> does not use the supplied configs, then its default replication is 3
> >>> and all files it will create (as part of the app or via a job config)
> >>> will be with replication factor 3.
> >>>
> >>> You can do an -lsr to find all files and filter which ones have been
> >>> created with a factor of 3 (versus expected config of 2).
> >>>
> >>> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com>
> wrote:
> >>> > Hi George,
> >>> >
> >>> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2.
> But
> >>> > still
> >>> > encounter this issue.
> >>> >
> >>> > Thanks!
> >>> >
> >>> >
> >>> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
> >>> >>
> >>> >>
> >>> >> Hi,
> >>> >>
> >>> >> I think i have faced this before, the problem is that you have the
> rep
> >>> >> factor=3 so it seems to hang because it needs 3 nodes to achieve the
> >>> >> factor
> >>> >> (replicas are not created on the same node). If you set the
> >>> >> replication
> >>> >> factor=2 i think you will not have this issue. So in general you
> must
> >>> >> make
> >>> >> sure that the rep factor is <= to the available datanodes.
> >>> >>
> >>> >> BR,
> >>> >> George
> >>> >>
> >>> >>
> >>> >> On 6/21/2013 12:29 PM, sam liu wrote:
> >>> >>
> >>> >> Hi,
> >>> >>
> >>> >> I encountered an issue which hangs the decommission operatoin. Its
> >>> >> steps:
> >>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2.
> And,
> >>> >> in
> >>> >> hdfs-site.xml, set the 'dfs.replication' to 2
> >>> >> 2. Add node dn3 into the cluster as a new datanode, and did not
> change
> >>> >> the
> >>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> >>> >> note: step 2 passed
> >>> >> 3. Decommission dn3 from the cluster
> >>> >>
> >>> >> Expected result: dn3 could be decommissioned successfully
> >>> >>
> >>> >> Actual result: decommission progress hangs and the status always be
> >>> >> 'Waiting DataNode status: Decommissioned'
> >>> >>
> >>> >> However, if the initial cluster includes >= 3 datanodes, this issue
> >>> >> won't
> >>> >> be encountered when add/remove another datanode.
> >>> >>
> >>> >> Also, after step 2, I noticed that some block's expected replicas is
> >>> >> 3,
> >>> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
> >>> >>
> >>> >> Could anyone pls help provide some triages?
> >>> >>
> >>> >> Thanks in advance!
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> ---------------------------
> >>> >>
> >>> >> George Kousiouris, PhD
> >>> >> Electrical and Computer Engineer
> >>> >> Division of Communications,
> >>> >> Electronics and Information Engineering
> >>> >> School of Electrical and Computer Engineering
> >>> >> Tel: +30 210 772 2546
> >>> >> Mobile: +30 6939354121
> >>> >> Fax: +30 210 772 2569
> >>> >> Email: gkousiou@mail.ntua.gr
> >>> >> Site: http://users.ntua.gr/gkousiou/
> >>> >>
> >>> >> National Technical University of Athens
> >>> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Harsh J
> >>
> >>
> >
>
>
>
> --
> Harsh J
>

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

But, please mention that the value of 'dfs.replication' of the cluster is
always 2, even when the datanode number is 3. And I am pretty sure I did
not manually create any files with rep=3. So, why were some files of hdfs
created with repl=3, but not repl=2?


2013/8/1 Harsh J <ha...@cloudera.com>

> The step (a) points to your problem and solution both. You have files
> being created with repl=3 on a 2 DN cluster which will prevent
> decommission. This is not a bug.
>
> On Wed, Jul 31, 2013 at 12:09 PM, sam liu <sa...@gmail.com> wrote:
> > I opened a jira for tracking this issue:
> > https://issues.apache.org/jira/browse/HDFS-5046
> >
> >
> > 2013/7/2 sam liu <sa...@gmail.com>
> >>
> >> Yes, the default replication factor is 3. However, in my case, it's
> >> strange: during decommission hangs, I found some block's expected
> replicas
> >> is 3, but the 'dfs.replication' value in hdfs-site.xml of every cluster
> node
> >> is always 2 from the beginning of cluster setup. Below is my steps:
> >>
> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
> in
> >> hdfs-site.xml, set the 'dfs.replication' to 2
> >> 2. Add node dn3 into the cluster as a new datanode, and did not change
> the
> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> >> note: step 2 passed
> >> 3. Decommission dn3 from the cluster
> >> Expected result: dn3 could be decommissioned successfully
> >> Actual result:
> >> a). decommission progress hangs and the status always be 'Waiting
> DataNode
> >> status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /',
> the
> >> decommission continues and will be completed finally.
> >> b). However, if the initial cluster includes >= 3 datanodes, this issue
> >> won't be encountered when add/remove another datanode. For example, if I
> >> setup a cluster with 3 datanodes, and then I can successfully add the
> 4th
> >> datanode into it, and then also can successfully remove the 4th datanode
> >> from the cluster.
> >>
> >> I doubt it's a bug and plan to open a jira to Hadoop HDFS for this. Any
> >> comments?
> >>
> >> Thanks!
> >>
> >>
> >> 2013/6/21 Harsh J <ha...@cloudera.com>
> >>>
> >>> The dfs.replication is a per-file parameter. If you have a client that
> >>> does not use the supplied configs, then its default replication is 3
> >>> and all files it will create (as part of the app or via a job config)
> >>> will be with replication factor 3.
> >>>
> >>> You can do an -lsr to find all files and filter which ones have been
> >>> created with a factor of 3 (versus expected config of 2).
> >>>
> >>> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com>
> wrote:
> >>> > Hi George,
> >>> >
> >>> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2.
> But
> >>> > still
> >>> > encounter this issue.
> >>> >
> >>> > Thanks!
> >>> >
> >>> >
> >>> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
> >>> >>
> >>> >>
> >>> >> Hi,
> >>> >>
> >>> >> I think i have faced this before, the problem is that you have the
> rep
> >>> >> factor=3 so it seems to hang because it needs 3 nodes to achieve the
> >>> >> factor
> >>> >> (replicas are not created on the same node). If you set the
> >>> >> replication
> >>> >> factor=2 i think you will not have this issue. So in general you
> must
> >>> >> make
> >>> >> sure that the rep factor is <= to the available datanodes.
> >>> >>
> >>> >> BR,
> >>> >> George
> >>> >>
> >>> >>
> >>> >> On 6/21/2013 12:29 PM, sam liu wrote:
> >>> >>
> >>> >> Hi,
> >>> >>
> >>> >> I encountered an issue which hangs the decommission operatoin. Its
> >>> >> steps:
> >>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2.
> And,
> >>> >> in
> >>> >> hdfs-site.xml, set the 'dfs.replication' to 2
> >>> >> 2. Add node dn3 into the cluster as a new datanode, and did not
> change
> >>> >> the
> >>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> >>> >> note: step 2 passed
> >>> >> 3. Decommission dn3 from the cluster
> >>> >>
> >>> >> Expected result: dn3 could be decommissioned successfully
> >>> >>
> >>> >> Actual result: decommission progress hangs and the status always be
> >>> >> 'Waiting DataNode status: Decommissioned'
> >>> >>
> >>> >> However, if the initial cluster includes >= 3 datanodes, this issue
> >>> >> won't
> >>> >> be encountered when add/remove another datanode.
> >>> >>
> >>> >> Also, after step 2, I noticed that some block's expected replicas is
> >>> >> 3,
> >>> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
> >>> >>
> >>> >> Could anyone pls help provide some triages?
> >>> >>
> >>> >> Thanks in advance!
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> ---------------------------
> >>> >>
> >>> >> George Kousiouris, PhD
> >>> >> Electrical and Computer Engineer
> >>> >> Division of Communications,
> >>> >> Electronics and Information Engineering
> >>> >> School of Electrical and Computer Engineering
> >>> >> Tel: +30 210 772 2546
> >>> >> Mobile: +30 6939354121
> >>> >> Fax: +30 210 772 2569
> >>> >> Email: gkousiou@mail.ntua.gr
> >>> >> Site: http://users.ntua.gr/gkousiou/
> >>> >>
> >>> >> National Technical University of Athens
> >>> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Harsh J
> >>
> >>
> >
>
>
>
> --
> Harsh J
>

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[Harsh J <ha...@cloudera.com>]

The step (a) points to your problem and solution both. You have files
being created with repl=3 on a 2 DN cluster which will prevent
decommission. This is not a bug.

On Wed, Jul 31, 2013 at 12:09 PM, sam liu <sa...@gmail.com> wrote:
> I opened a jira for tracking this issue:
> https://issues.apache.org/jira/browse/HDFS-5046
>
>
> 2013/7/2 sam liu <sa...@gmail.com>
>>
>> Yes, the default replication factor is 3. However, in my case, it's
>> strange: during decommission hangs, I found some block's expected replicas
>> is 3, but the 'dfs.replication' value in hdfs-site.xml of every cluster node
>> is always 2 from the beginning of cluster setup. Below is my steps:
>>
>> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, in
>> hdfs-site.xml, set the 'dfs.replication' to 2
>> 2. Add node dn3 into the cluster as a new datanode, and did not change the
>> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>> note: step 2 passed
>> 3. Decommission dn3 from the cluster
>> Expected result: dn3 could be decommissioned successfully
>> Actual result:
>> a). decommission progress hangs and the status always be 'Waiting DataNode
>> status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /', the
>> decommission continues and will be completed finally.
>> b). However, if the initial cluster includes >= 3 datanodes, this issue
>> won't be encountered when add/remove another datanode. For example, if I
>> setup a cluster with 3 datanodes, and then I can successfully add the 4th
>> datanode into it, and then also can successfully remove the 4th datanode
>> from the cluster.
>>
>> I doubt it's a bug and plan to open a jira to Hadoop HDFS for this. Any
>> comments?
>>
>> Thanks!
>>
>>
>> 2013/6/21 Harsh J <ha...@cloudera.com>
>>>
>>> The dfs.replication is a per-file parameter. If you have a client that
>>> does not use the supplied configs, then its default replication is 3
>>> and all files it will create (as part of the app or via a job config)
>>> will be with replication factor 3.
>>>
>>> You can do an -lsr to find all files and filter which ones have been
>>> created with a factor of 3 (versus expected config of 2).
>>>
>>> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com> wrote:
>>> > Hi George,
>>> >
>>> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2. But
>>> > still
>>> > encounter this issue.
>>> >
>>> > Thanks!
>>> >
>>> >
>>> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
>>> >>
>>> >>
>>> >> Hi,
>>> >>
>>> >> I think i have faced this before, the problem is that you have the rep
>>> >> factor=3 so it seems to hang because it needs 3 nodes to achieve the
>>> >> factor
>>> >> (replicas are not created on the same node). If you set the
>>> >> replication
>>> >> factor=2 i think you will not have this issue. So in general you must
>>> >> make
>>> >> sure that the rep factor is <= to the available datanodes.
>>> >>
>>> >> BR,
>>> >> George
>>> >>
>>> >>
>>> >> On 6/21/2013 12:29 PM, sam liu wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> I encountered an issue which hangs the decommission operatoin. Its
>>> >> steps:
>>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
>>> >> in
>>> >> hdfs-site.xml, set the 'dfs.replication' to 2
>>> >> 2. Add node dn3 into the cluster as a new datanode, and did not change
>>> >> the
>>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>>> >> note: step 2 passed
>>> >> 3. Decommission dn3 from the cluster
>>> >>
>>> >> Expected result: dn3 could be decommissioned successfully
>>> >>
>>> >> Actual result: decommission progress hangs and the status always be
>>> >> 'Waiting DataNode status: Decommissioned'
>>> >>
>>> >> However, if the initial cluster includes >= 3 datanodes, this issue
>>> >> won't
>>> >> be encountered when add/remove another datanode.
>>> >>
>>> >> Also, after step 2, I noticed that some block's expected replicas is
>>> >> 3,
>>> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
>>> >>
>>> >> Could anyone pls help provide some triages?
>>> >>
>>> >> Thanks in advance!
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> ---------------------------
>>> >>
>>> >> George Kousiouris, PhD
>>> >> Electrical and Computer Engineer
>>> >> Division of Communications,
>>> >> Electronics and Information Engineering
>>> >> School of Electrical and Computer Engineering
>>> >> Tel: +30 210 772 2546
>>> >> Mobile: +30 6939354121
>>> >> Fax: +30 210 772 2569
>>> >> Email: gkousiou@mail.ntua.gr
>>> >> Site: http://users.ntua.gr/gkousiou/
>>> >>
>>> >> National Technical University of Athens
>>> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>



-- 
Harsh J

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[Harsh J <ha...@cloudera.com>]

The step (a) points to your problem and solution both. You have files
being created with repl=3 on a 2 DN cluster which will prevent
decommission. This is not a bug.

On Wed, Jul 31, 2013 at 12:09 PM, sam liu <sa...@gmail.com> wrote:
> I opened a jira for tracking this issue:
> https://issues.apache.org/jira/browse/HDFS-5046
>
>
> 2013/7/2 sam liu <sa...@gmail.com>
>>
>> Yes, the default replication factor is 3. However, in my case, it's
>> strange: during decommission hangs, I found some block's expected replicas
>> is 3, but the 'dfs.replication' value in hdfs-site.xml of every cluster node
>> is always 2 from the beginning of cluster setup. Below is my steps:
>>
>> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, in
>> hdfs-site.xml, set the 'dfs.replication' to 2
>> 2. Add node dn3 into the cluster as a new datanode, and did not change the
>> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>> note: step 2 passed
>> 3. Decommission dn3 from the cluster
>> Expected result: dn3 could be decommissioned successfully
>> Actual result:
>> a). decommission progress hangs and the status always be 'Waiting DataNode
>> status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /', the
>> decommission continues and will be completed finally.
>> b). However, if the initial cluster includes >= 3 datanodes, this issue
>> won't be encountered when add/remove another datanode. For example, if I
>> setup a cluster with 3 datanodes, and then I can successfully add the 4th
>> datanode into it, and then also can successfully remove the 4th datanode
>> from the cluster.
>>
>> I doubt it's a bug and plan to open a jira to Hadoop HDFS for this. Any
>> comments?
>>
>> Thanks!
>>
>>
>> 2013/6/21 Harsh J <ha...@cloudera.com>
>>>
>>> The dfs.replication is a per-file parameter. If you have a client that
>>> does not use the supplied configs, then its default replication is 3
>>> and all files it will create (as part of the app or via a job config)
>>> will be with replication factor 3.
>>>
>>> You can do an -lsr to find all files and filter which ones have been
>>> created with a factor of 3 (versus expected config of 2).
>>>
>>> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com> wrote:
>>> > Hi George,
>>> >
>>> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2. But
>>> > still
>>> > encounter this issue.
>>> >
>>> > Thanks!
>>> >
>>> >
>>> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
>>> >>
>>> >>
>>> >> Hi,
>>> >>
>>> >> I think i have faced this before, the problem is that you have the rep
>>> >> factor=3 so it seems to hang because it needs 3 nodes to achieve the
>>> >> factor
>>> >> (replicas are not created on the same node). If you set the
>>> >> replication
>>> >> factor=2 i think you will not have this issue. So in general you must
>>> >> make
>>> >> sure that the rep factor is <= to the available datanodes.
>>> >>
>>> >> BR,
>>> >> George
>>> >>
>>> >>
>>> >> On 6/21/2013 12:29 PM, sam liu wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> I encountered an issue which hangs the decommission operatoin. Its
>>> >> steps:
>>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
>>> >> in
>>> >> hdfs-site.xml, set the 'dfs.replication' to 2
>>> >> 2. Add node dn3 into the cluster as a new datanode, and did not change
>>> >> the
>>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>>> >> note: step 2 passed
>>> >> 3. Decommission dn3 from the cluster
>>> >>
>>> >> Expected result: dn3 could be decommissioned successfully
>>> >>
>>> >> Actual result: decommission progress hangs and the status always be
>>> >> 'Waiting DataNode status: Decommissioned'
>>> >>
>>> >> However, if the initial cluster includes >= 3 datanodes, this issue
>>> >> won't
>>> >> be encountered when add/remove another datanode.
>>> >>
>>> >> Also, after step 2, I noticed that some block's expected replicas is
>>> >> 3,
>>> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
>>> >>
>>> >> Could anyone pls help provide some triages?
>>> >>
>>> >> Thanks in advance!
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> ---------------------------
>>> >>
>>> >> George Kousiouris, PhD
>>> >> Electrical and Computer Engineer
>>> >> Division of Communications,
>>> >> Electronics and Information Engineering
>>> >> School of Electrical and Computer Engineering
>>> >> Tel: +30 210 772 2546
>>> >> Mobile: +30 6939354121
>>> >> Fax: +30 210 772 2569
>>> >> Email: gkousiou@mail.ntua.gr
>>> >> Site: http://users.ntua.gr/gkousiou/
>>> >>
>>> >> National Technical University of Athens
>>> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>



-- 
Harsh J

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[Harsh J <ha...@cloudera.com>]

The step (a) points to your problem and solution both. You have files
being created with repl=3 on a 2 DN cluster which will prevent
decommission. This is not a bug.

On Wed, Jul 31, 2013 at 12:09 PM, sam liu <sa...@gmail.com> wrote:
> I opened a jira for tracking this issue:
> https://issues.apache.org/jira/browse/HDFS-5046
>
>
> 2013/7/2 sam liu <sa...@gmail.com>
>>
>> Yes, the default replication factor is 3. However, in my case, it's
>> strange: during decommission hangs, I found some block's expected replicas
>> is 3, but the 'dfs.replication' value in hdfs-site.xml of every cluster node
>> is always 2 from the beginning of cluster setup. Below is my steps:
>>
>> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, in
>> hdfs-site.xml, set the 'dfs.replication' to 2
>> 2. Add node dn3 into the cluster as a new datanode, and did not change the
>> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>> note: step 2 passed
>> 3. Decommission dn3 from the cluster
>> Expected result: dn3 could be decommissioned successfully
>> Actual result:
>> a). decommission progress hangs and the status always be 'Waiting DataNode
>> status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /', the
>> decommission continues and will be completed finally.
>> b). However, if the initial cluster includes >= 3 datanodes, this issue
>> won't be encountered when add/remove another datanode. For example, if I
>> setup a cluster with 3 datanodes, and then I can successfully add the 4th
>> datanode into it, and then also can successfully remove the 4th datanode
>> from the cluster.
>>
>> I doubt it's a bug and plan to open a jira to Hadoop HDFS for this. Any
>> comments?
>>
>> Thanks!
>>
>>
>> 2013/6/21 Harsh J <ha...@cloudera.com>
>>>
>>> The dfs.replication is a per-file parameter. If you have a client that
>>> does not use the supplied configs, then its default replication is 3
>>> and all files it will create (as part of the app or via a job config)
>>> will be with replication factor 3.
>>>
>>> You can do an -lsr to find all files and filter which ones have been
>>> created with a factor of 3 (versus expected config of 2).
>>>
>>> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com> wrote:
>>> > Hi George,
>>> >
>>> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2. But
>>> > still
>>> > encounter this issue.
>>> >
>>> > Thanks!
>>> >
>>> >
>>> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
>>> >>
>>> >>
>>> >> Hi,
>>> >>
>>> >> I think i have faced this before, the problem is that you have the rep
>>> >> factor=3 so it seems to hang because it needs 3 nodes to achieve the
>>> >> factor
>>> >> (replicas are not created on the same node). If you set the
>>> >> replication
>>> >> factor=2 i think you will not have this issue. So in general you must
>>> >> make
>>> >> sure that the rep factor is <= to the available datanodes.
>>> >>
>>> >> BR,
>>> >> George
>>> >>
>>> >>
>>> >> On 6/21/2013 12:29 PM, sam liu wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> I encountered an issue which hangs the decommission operatoin. Its
>>> >> steps:
>>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
>>> >> in
>>> >> hdfs-site.xml, set the 'dfs.replication' to 2
>>> >> 2. Add node dn3 into the cluster as a new datanode, and did not change
>>> >> the
>>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>>> >> note: step 2 passed
>>> >> 3. Decommission dn3 from the cluster
>>> >>
>>> >> Expected result: dn3 could be decommissioned successfully
>>> >>
>>> >> Actual result: decommission progress hangs and the status always be
>>> >> 'Waiting DataNode status: Decommissioned'
>>> >>
>>> >> However, if the initial cluster includes >= 3 datanodes, this issue
>>> >> won't
>>> >> be encountered when add/remove another datanode.
>>> >>
>>> >> Also, after step 2, I noticed that some block's expected replicas is
>>> >> 3,
>>> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
>>> >>
>>> >> Could anyone pls help provide some triages?
>>> >>
>>> >> Thanks in advance!
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> ---------------------------
>>> >>
>>> >> George Kousiouris, PhD
>>> >> Electrical and Computer Engineer
>>> >> Division of Communications,
>>> >> Electronics and Information Engineering
>>> >> School of Electrical and Computer Engineering
>>> >> Tel: +30 210 772 2546
>>> >> Mobile: +30 6939354121
>>> >> Fax: +30 210 772 2569
>>> >> Email: gkousiou@mail.ntua.gr
>>> >> Site: http://users.ntua.gr/gkousiou/
>>> >>
>>> >> National Technical University of Athens
>>> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>



-- 
Harsh J

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[Harsh J <ha...@cloudera.com>]

The step (a) points to your problem and solution both. You have files
being created with repl=3 on a 2 DN cluster which will prevent
decommission. This is not a bug.

On Wed, Jul 31, 2013 at 12:09 PM, sam liu <sa...@gmail.com> wrote:
> I opened a jira for tracking this issue:
> https://issues.apache.org/jira/browse/HDFS-5046
>
>
> 2013/7/2 sam liu <sa...@gmail.com>
>>
>> Yes, the default replication factor is 3. However, in my case, it's
>> strange: during decommission hangs, I found some block's expected replicas
>> is 3, but the 'dfs.replication' value in hdfs-site.xml of every cluster node
>> is always 2 from the beginning of cluster setup. Below is my steps:
>>
>> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, in
>> hdfs-site.xml, set the 'dfs.replication' to 2
>> 2. Add node dn3 into the cluster as a new datanode, and did not change the
>> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>> note: step 2 passed
>> 3. Decommission dn3 from the cluster
>> Expected result: dn3 could be decommissioned successfully
>> Actual result:
>> a). decommission progress hangs and the status always be 'Waiting DataNode
>> status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /', the
>> decommission continues and will be completed finally.
>> b). However, if the initial cluster includes >= 3 datanodes, this issue
>> won't be encountered when add/remove another datanode. For example, if I
>> setup a cluster with 3 datanodes, and then I can successfully add the 4th
>> datanode into it, and then also can successfully remove the 4th datanode
>> from the cluster.
>>
>> I doubt it's a bug and plan to open a jira to Hadoop HDFS for this. Any
>> comments?
>>
>> Thanks!
>>
>>
>> 2013/6/21 Harsh J <ha...@cloudera.com>
>>>
>>> The dfs.replication is a per-file parameter. If you have a client that
>>> does not use the supplied configs, then its default replication is 3
>>> and all files it will create (as part of the app or via a job config)
>>> will be with replication factor 3.
>>>
>>> You can do an -lsr to find all files and filter which ones have been
>>> created with a factor of 3 (versus expected config of 2).
>>>
>>> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com> wrote:
>>> > Hi George,
>>> >
>>> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2. But
>>> > still
>>> > encounter this issue.
>>> >
>>> > Thanks!
>>> >
>>> >
>>> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
>>> >>
>>> >>
>>> >> Hi,
>>> >>
>>> >> I think i have faced this before, the problem is that you have the rep
>>> >> factor=3 so it seems to hang because it needs 3 nodes to achieve the
>>> >> factor
>>> >> (replicas are not created on the same node). If you set the
>>> >> replication
>>> >> factor=2 i think you will not have this issue. So in general you must
>>> >> make
>>> >> sure that the rep factor is <= to the available datanodes.
>>> >>
>>> >> BR,
>>> >> George
>>> >>
>>> >>
>>> >> On 6/21/2013 12:29 PM, sam liu wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> I encountered an issue which hangs the decommission operatoin. Its
>>> >> steps:
>>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
>>> >> in
>>> >> hdfs-site.xml, set the 'dfs.replication' to 2
>>> >> 2. Add node dn3 into the cluster as a new datanode, and did not change
>>> >> the
>>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>>> >> note: step 2 passed
>>> >> 3. Decommission dn3 from the cluster
>>> >>
>>> >> Expected result: dn3 could be decommissioned successfully
>>> >>
>>> >> Actual result: decommission progress hangs and the status always be
>>> >> 'Waiting DataNode status: Decommissioned'
>>> >>
>>> >> However, if the initial cluster includes >= 3 datanodes, this issue
>>> >> won't
>>> >> be encountered when add/remove another datanode.
>>> >>
>>> >> Also, after step 2, I noticed that some block's expected replicas is
>>> >> 3,
>>> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
>>> >>
>>> >> Could anyone pls help provide some triages?
>>> >>
>>> >> Thanks in advance!
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> ---------------------------
>>> >>
>>> >> George Kousiouris, PhD
>>> >> Electrical and Computer Engineer
>>> >> Division of Communications,
>>> >> Electronics and Information Engineering
>>> >> School of Electrical and Computer Engineering
>>> >> Tel: +30 210 772 2546
>>> >> Mobile: +30 6939354121
>>> >> Fax: +30 210 772 2569
>>> >> Email: gkousiou@mail.ntua.gr
>>> >> Site: http://users.ntua.gr/gkousiou/
>>> >>
>>> >> National Technical University of Athens
>>> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>



-- 
Harsh J

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

I opened a jira for tracking this issue:
https://issues.apache.org/jira/browse/HDFS-5046


2013/7/2 sam liu <sa...@gmail.com>

> Yes, the default replication factor is 3. However, in my case, it's
> strange: during decommission hangs, I found some block's expected replicas
> is 3, but the 'dfs.replication' value in hdfs-site.xml of every cluster
> node is always 2 from the beginning of cluster setup. Below is my steps:
>
> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, in
> hdfs-site.xml, set the 'dfs.replication' to 2
> 2. Add node dn3 into the cluster as a new datanode, and did not change the
> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> note: step 2 passed
>  3. Decommission dn3 from the cluster
> Expected result: dn3 could be decommissioned successfully
> Actual result:
> a). decommission progress hangs and the status always be 'Waiting DataNode
> status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /', the
> decommission continues and will be completed finally.
> b). However, if the initial cluster includes >= 3 datanodes, this issue
> won't be encountered when add/remove another datanode. For example, if I
> setup a cluster with 3 datanodes, and then I can successfully add the 4th
> datanode into it, and then also can successfully remove the 4th datanode
> from the cluster.
>
> I doubt it's a bug and plan to open a jira to Hadoop HDFS for this. Any
> comments?
>
> Thanks!
>
>
> 2013/6/21 Harsh J <ha...@cloudera.com>
>
>> The dfs.replication is a per-file parameter. If you have a client that
>> does not use the supplied configs, then its default replication is 3
>> and all files it will create (as part of the app or via a job config)
>> will be with replication factor 3.
>>
>> You can do an -lsr to find all files and filter which ones have been
>> created with a factor of 3 (versus expected config of 2).
>>
>> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com> wrote:
>> > Hi George,
>> >
>> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2. But
>> still
>> > encounter this issue.
>> >
>> > Thanks!
>> >
>> >
>> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
>> >>
>> >>
>> >> Hi,
>> >>
>> >> I think i have faced this before, the problem is that you have the rep
>> >> factor=3 so it seems to hang because it needs 3 nodes to achieve the
>> factor
>> >> (replicas are not created on the same node). If you set the replication
>> >> factor=2 i think you will not have this issue. So in general you must
>> make
>> >> sure that the rep factor is <= to the available datanodes.
>> >>
>> >> BR,
>> >> George
>> >>
>> >>
>> >> On 6/21/2013 12:29 PM, sam liu wrote:
>> >>
>> >> Hi,
>> >>
>> >> I encountered an issue which hangs the decommission operatoin. Its
>> steps:
>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
>> in
>> >> hdfs-site.xml, set the 'dfs.replication' to 2
>> >> 2. Add node dn3 into the cluster as a new datanode, and did not change
>> the
>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>> >> note: step 2 passed
>> >> 3. Decommission dn3 from the cluster
>> >>
>> >> Expected result: dn3 could be decommissioned successfully
>> >>
>> >> Actual result: decommission progress hangs and the status always be
>> >> 'Waiting DataNode status: Decommissioned'
>> >>
>> >> However, if the initial cluster includes >= 3 datanodes, this issue
>> won't
>> >> be encountered when add/remove another datanode.
>> >>
>> >> Also, after step 2, I noticed that some block's expected replicas is 3,
>> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
>> >>
>> >> Could anyone pls help provide some triages?
>> >>
>> >> Thanks in advance!
>> >>
>> >>
>> >>
>> >> --
>> >> ---------------------------
>> >>
>> >> George Kousiouris, PhD
>> >> Electrical and Computer Engineer
>> >> Division of Communications,
>> >> Electronics and Information Engineering
>> >> School of Electrical and Computer Engineering
>> >> Tel: +30 210 772 2546
>> >> Mobile: +30 6939354121
>> >> Fax: +30 210 772 2569
>> >> Email: gkousiou@mail.ntua.gr
>> >> Site: http://users.ntua.gr/gkousiou/
>> >>
>> >> National Technical University of Athens
>> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

I opened a jira for tracking this issue:
https://issues.apache.org/jira/browse/HDFS-5046


2013/7/2 sam liu <sa...@gmail.com>

> Yes, the default replication factor is 3. However, in my case, it's
> strange: during decommission hangs, I found some block's expected replicas
> is 3, but the 'dfs.replication' value in hdfs-site.xml of every cluster
> node is always 2 from the beginning of cluster setup. Below is my steps:
>
> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, in
> hdfs-site.xml, set the 'dfs.replication' to 2
> 2. Add node dn3 into the cluster as a new datanode, and did not change the
> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> note: step 2 passed
>  3. Decommission dn3 from the cluster
> Expected result: dn3 could be decommissioned successfully
> Actual result:
> a). decommission progress hangs and the status always be 'Waiting DataNode
> status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /', the
> decommission continues and will be completed finally.
> b). However, if the initial cluster includes >= 3 datanodes, this issue
> won't be encountered when add/remove another datanode. For example, if I
> setup a cluster with 3 datanodes, and then I can successfully add the 4th
> datanode into it, and then also can successfully remove the 4th datanode
> from the cluster.
>
> I doubt it's a bug and plan to open a jira to Hadoop HDFS for this. Any
> comments?
>
> Thanks!
>
>
> 2013/6/21 Harsh J <ha...@cloudera.com>
>
>> The dfs.replication is a per-file parameter. If you have a client that
>> does not use the supplied configs, then its default replication is 3
>> and all files it will create (as part of the app or via a job config)
>> will be with replication factor 3.
>>
>> You can do an -lsr to find all files and filter which ones have been
>> created with a factor of 3 (versus expected config of 2).
>>
>> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com> wrote:
>> > Hi George,
>> >
>> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2. But
>> still
>> > encounter this issue.
>> >
>> > Thanks!
>> >
>> >
>> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
>> >>
>> >>
>> >> Hi,
>> >>
>> >> I think i have faced this before, the problem is that you have the rep
>> >> factor=3 so it seems to hang because it needs 3 nodes to achieve the
>> factor
>> >> (replicas are not created on the same node). If you set the replication
>> >> factor=2 i think you will not have this issue. So in general you must
>> make
>> >> sure that the rep factor is <= to the available datanodes.
>> >>
>> >> BR,
>> >> George
>> >>
>> >>
>> >> On 6/21/2013 12:29 PM, sam liu wrote:
>> >>
>> >> Hi,
>> >>
>> >> I encountered an issue which hangs the decommission operatoin. Its
>> steps:
>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
>> in
>> >> hdfs-site.xml, set the 'dfs.replication' to 2
>> >> 2. Add node dn3 into the cluster as a new datanode, and did not change
>> the
>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>> >> note: step 2 passed
>> >> 3. Decommission dn3 from the cluster
>> >>
>> >> Expected result: dn3 could be decommissioned successfully
>> >>
>> >> Actual result: decommission progress hangs and the status always be
>> >> 'Waiting DataNode status: Decommissioned'
>> >>
>> >> However, if the initial cluster includes >= 3 datanodes, this issue
>> won't
>> >> be encountered when add/remove another datanode.
>> >>
>> >> Also, after step 2, I noticed that some block's expected replicas is 3,
>> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
>> >>
>> >> Could anyone pls help provide some triages?
>> >>
>> >> Thanks in advance!
>> >>
>> >>
>> >>
>> >> --
>> >> ---------------------------
>> >>
>> >> George Kousiouris, PhD
>> >> Electrical and Computer Engineer
>> >> Division of Communications,
>> >> Electronics and Information Engineering
>> >> School of Electrical and Computer Engineering
>> >> Tel: +30 210 772 2546
>> >> Mobile: +30 6939354121
>> >> Fax: +30 210 772 2569
>> >> Email: gkousiou@mail.ntua.gr
>> >> Site: http://users.ntua.gr/gkousiou/
>> >>
>> >> National Technical University of Athens
>> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

I opened a jira for tracking this issue:
https://issues.apache.org/jira/browse/HDFS-5046


2013/7/2 sam liu <sa...@gmail.com>

> Yes, the default replication factor is 3. However, in my case, it's
> strange: during decommission hangs, I found some block's expected replicas
> is 3, but the 'dfs.replication' value in hdfs-site.xml of every cluster
> node is always 2 from the beginning of cluster setup. Below is my steps:
>
> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, in
> hdfs-site.xml, set the 'dfs.replication' to 2
> 2. Add node dn3 into the cluster as a new datanode, and did not change the
> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> note: step 2 passed
>  3. Decommission dn3 from the cluster
> Expected result: dn3 could be decommissioned successfully
> Actual result:
> a). decommission progress hangs and the status always be 'Waiting DataNode
> status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /', the
> decommission continues and will be completed finally.
> b). However, if the initial cluster includes >= 3 datanodes, this issue
> won't be encountered when add/remove another datanode. For example, if I
> setup a cluster with 3 datanodes, and then I can successfully add the 4th
> datanode into it, and then also can successfully remove the 4th datanode
> from the cluster.
>
> I doubt it's a bug and plan to open a jira to Hadoop HDFS for this. Any
> comments?
>
> Thanks!
>
>
> 2013/6/21 Harsh J <ha...@cloudera.com>
>
>> The dfs.replication is a per-file parameter. If you have a client that
>> does not use the supplied configs, then its default replication is 3
>> and all files it will create (as part of the app or via a job config)
>> will be with replication factor 3.
>>
>> You can do an -lsr to find all files and filter which ones have been
>> created with a factor of 3 (versus expected config of 2).
>>
>> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com> wrote:
>> > Hi George,
>> >
>> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2. But
>> still
>> > encounter this issue.
>> >
>> > Thanks!
>> >
>> >
>> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
>> >>
>> >>
>> >> Hi,
>> >>
>> >> I think i have faced this before, the problem is that you have the rep
>> >> factor=3 so it seems to hang because it needs 3 nodes to achieve the
>> factor
>> >> (replicas are not created on the same node). If you set the replication
>> >> factor=2 i think you will not have this issue. So in general you must
>> make
>> >> sure that the rep factor is <= to the available datanodes.
>> >>
>> >> BR,
>> >> George
>> >>
>> >>
>> >> On 6/21/2013 12:29 PM, sam liu wrote:
>> >>
>> >> Hi,
>> >>
>> >> I encountered an issue which hangs the decommission operatoin. Its
>> steps:
>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
>> in
>> >> hdfs-site.xml, set the 'dfs.replication' to 2
>> >> 2. Add node dn3 into the cluster as a new datanode, and did not change
>> the
>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>> >> note: step 2 passed
>> >> 3. Decommission dn3 from the cluster
>> >>
>> >> Expected result: dn3 could be decommissioned successfully
>> >>
>> >> Actual result: decommission progress hangs and the status always be
>> >> 'Waiting DataNode status: Decommissioned'
>> >>
>> >> However, if the initial cluster includes >= 3 datanodes, this issue
>> won't
>> >> be encountered when add/remove another datanode.
>> >>
>> >> Also, after step 2, I noticed that some block's expected replicas is 3,
>> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
>> >>
>> >> Could anyone pls help provide some triages?
>> >>
>> >> Thanks in advance!
>> >>
>> >>
>> >>
>> >> --
>> >> ---------------------------
>> >>
>> >> George Kousiouris, PhD
>> >> Electrical and Computer Engineer
>> >> Division of Communications,
>> >> Electronics and Information Engineering
>> >> School of Electrical and Computer Engineering
>> >> Tel: +30 210 772 2546
>> >> Mobile: +30 6939354121
>> >> Fax: +30 210 772 2569
>> >> Email: gkousiou@mail.ntua.gr
>> >> Site: http://users.ntua.gr/gkousiou/
>> >>
>> >> National Technical University of Athens
>> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

I opened a jira for tracking this issue:
https://issues.apache.org/jira/browse/HDFS-5046


2013/7/2 sam liu <sa...@gmail.com>

> Yes, the default replication factor is 3. However, in my case, it's
> strange: during decommission hangs, I found some block's expected replicas
> is 3, but the 'dfs.replication' value in hdfs-site.xml of every cluster
> node is always 2 from the beginning of cluster setup. Below is my steps:
>
> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, in
> hdfs-site.xml, set the 'dfs.replication' to 2
> 2. Add node dn3 into the cluster as a new datanode, and did not change the
> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> note: step 2 passed
>  3. Decommission dn3 from the cluster
> Expected result: dn3 could be decommissioned successfully
> Actual result:
> a). decommission progress hangs and the status always be 'Waiting DataNode
> status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /', the
> decommission continues and will be completed finally.
> b). However, if the initial cluster includes >= 3 datanodes, this issue
> won't be encountered when add/remove another datanode. For example, if I
> setup a cluster with 3 datanodes, and then I can successfully add the 4th
> datanode into it, and then also can successfully remove the 4th datanode
> from the cluster.
>
> I doubt it's a bug and plan to open a jira to Hadoop HDFS for this. Any
> comments?
>
> Thanks!
>
>
> 2013/6/21 Harsh J <ha...@cloudera.com>
>
>> The dfs.replication is a per-file parameter. If you have a client that
>> does not use the supplied configs, then its default replication is 3
>> and all files it will create (as part of the app or via a job config)
>> will be with replication factor 3.
>>
>> You can do an -lsr to find all files and filter which ones have been
>> created with a factor of 3 (versus expected config of 2).
>>
>> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com> wrote:
>> > Hi George,
>> >
>> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2. But
>> still
>> > encounter this issue.
>> >
>> > Thanks!
>> >
>> >
>> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
>> >>
>> >>
>> >> Hi,
>> >>
>> >> I think i have faced this before, the problem is that you have the rep
>> >> factor=3 so it seems to hang because it needs 3 nodes to achieve the
>> factor
>> >> (replicas are not created on the same node). If you set the replication
>> >> factor=2 i think you will not have this issue. So in general you must
>> make
>> >> sure that the rep factor is <= to the available datanodes.
>> >>
>> >> BR,
>> >> George
>> >>
>> >>
>> >> On 6/21/2013 12:29 PM, sam liu wrote:
>> >>
>> >> Hi,
>> >>
>> >> I encountered an issue which hangs the decommission operatoin. Its
>> steps:
>> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
>> in
>> >> hdfs-site.xml, set the 'dfs.replication' to 2
>> >> 2. Add node dn3 into the cluster as a new datanode, and did not change
>> the
>> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>> >> note: step 2 passed
>> >> 3. Decommission dn3 from the cluster
>> >>
>> >> Expected result: dn3 could be decommissioned successfully
>> >>
>> >> Actual result: decommission progress hangs and the status always be
>> >> 'Waiting DataNode status: Decommissioned'
>> >>
>> >> However, if the initial cluster includes >= 3 datanodes, this issue
>> won't
>> >> be encountered when add/remove another datanode.
>> >>
>> >> Also, after step 2, I noticed that some block's expected replicas is 3,
>> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
>> >>
>> >> Could anyone pls help provide some triages?
>> >>
>> >> Thanks in advance!
>> >>
>> >>
>> >>
>> >> --
>> >> ---------------------------
>> >>
>> >> George Kousiouris, PhD
>> >> Electrical and Computer Engineer
>> >> Division of Communications,
>> >> Electronics and Information Engineering
>> >> School of Electrical and Computer Engineering
>> >> Tel: +30 210 772 2546
>> >> Mobile: +30 6939354121
>> >> Fax: +30 210 772 2569
>> >> Email: gkousiou@mail.ntua.gr
>> >> Site: http://users.ntua.gr/gkousiou/
>> >>
>> >> National Technical University of Athens
>> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

Yes, the default replication factor is 3. However, in my case, it's
strange: during decommission hangs, I found some block's expected replicas
is 3, but the 'dfs.replication' value in hdfs-site.xml of every cluster
node is always 2 from the beginning of cluster setup. Below is my steps:
1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, in
hdfs-site.xml, set the 'dfs.replication' to 2
2. Add node dn3 into the cluster as a new datanode, and did not change the '
dfs.replication' value in hdfs-site.xml and keep it as 2
note: step 2 passed
3. Decommission dn3 from the cluster
Expected result: dn3 could be decommissioned successfully
Actual result:
a). decommission progress hangs and the status always be 'Waiting DataNode
status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /', the
decommission continues and will be completed finally.
b). However, if the initial cluster includes >= 3 datanodes, this issue
won't be encountered when add/remove another datanode. For example, if I
setup a cluster with 3 datanodes, and then I can successfully add the 4th
datanode into it, and then also can successfully remove the 4th datanode
from the cluster.

I doubt it's a bug and plan to open a jira to Hadoop HDFS for this. Any
comments?

Thanks!


2013/6/21 Harsh J <ha...@cloudera.com>

> The dfs.replication is a per-file parameter. If you have a client that
> does not use the supplied configs, then its default replication is 3
> and all files it will create (as part of the app or via a job config)
> will be with replication factor 3.
>
> You can do an -lsr to find all files and filter which ones have been
> created with a factor of 3 (versus expected config of 2).
>
> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com> wrote:
> > Hi George,
> >
> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2. But
> still
> > encounter this issue.
> >
> > Thanks!
> >
> >
> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
> >>
> >>
> >> Hi,
> >>
> >> I think i have faced this before, the problem is that you have the rep
> >> factor=3 so it seems to hang because it needs 3 nodes to achieve the
> factor
> >> (replicas are not created on the same node). If you set the replication
> >> factor=2 i think you will not have this issue. So in general you must
> make
> >> sure that the rep factor is <= to the available datanodes.
> >>
> >> BR,
> >> George
> >>
> >>
> >> On 6/21/2013 12:29 PM, sam liu wrote:
> >>
> >> Hi,
> >>
> >> I encountered an issue which hangs the decommission operatoin. Its
> steps:
> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
> in
> >> hdfs-site.xml, set the 'dfs.replication' to 2
> >> 2. Add node dn3 into the cluster as a new datanode, and did not change
> the
> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> >> note: step 2 passed
> >> 3. Decommission dn3 from the cluster
> >>
> >> Expected result: dn3 could be decommissioned successfully
> >>
> >> Actual result: decommission progress hangs and the status always be
> >> 'Waiting DataNode status: Decommissioned'
> >>
> >> However, if the initial cluster includes >= 3 datanodes, this issue
> won't
> >> be encountered when add/remove another datanode.
> >>
> >> Also, after step 2, I noticed that some block's expected replicas is 3,
> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
> >>
> >> Could anyone pls help provide some triages?
> >>
> >> Thanks in advance!
> >>
> >>
> >>
> >> --
> >> ---------------------------
> >>
> >> George Kousiouris, PhD
> >> Electrical and Computer Engineer
> >> Division of Communications,
> >> Electronics and Information Engineering
> >> School of Electrical and Computer Engineering
> >> Tel: +30 210 772 2546
> >> Mobile: +30 6939354121
> >> Fax: +30 210 772 2569
> >> Email: gkousiou@mail.ntua.gr
> >> Site: http://users.ntua.gr/gkousiou/
> >>
> >> National Technical University of Athens
> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
> >
> >
>
>
>
> --
> Harsh J
>

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

Yes, the default replication factor is 3. However, in my case, it's
strange: during decommission hangs, I found some block's expected replicas
is 3, but the 'dfs.replication' value in hdfs-site.xml of every cluster
node is always 2 from the beginning of cluster setup. Below is my steps:
1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, in
hdfs-site.xml, set the 'dfs.replication' to 2
2. Add node dn3 into the cluster as a new datanode, and did not change the '
dfs.replication' value in hdfs-site.xml and keep it as 2
note: step 2 passed
3. Decommission dn3 from the cluster
Expected result: dn3 could be decommissioned successfully
Actual result:
a). decommission progress hangs and the status always be 'Waiting DataNode
status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /', the
decommission continues and will be completed finally.
b). However, if the initial cluster includes >= 3 datanodes, this issue
won't be encountered when add/remove another datanode. For example, if I
setup a cluster with 3 datanodes, and then I can successfully add the 4th
datanode into it, and then also can successfully remove the 4th datanode
from the cluster.

I doubt it's a bug and plan to open a jira to Hadoop HDFS for this. Any
comments?

Thanks!


2013/6/21 Harsh J <ha...@cloudera.com>

> The dfs.replication is a per-file parameter. If you have a client that
> does not use the supplied configs, then its default replication is 3
> and all files it will create (as part of the app or via a job config)
> will be with replication factor 3.
>
> You can do an -lsr to find all files and filter which ones have been
> created with a factor of 3 (versus expected config of 2).
>
> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com> wrote:
> > Hi George,
> >
> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2. But
> still
> > encounter this issue.
> >
> > Thanks!
> >
> >
> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
> >>
> >>
> >> Hi,
> >>
> >> I think i have faced this before, the problem is that you have the rep
> >> factor=3 so it seems to hang because it needs 3 nodes to achieve the
> factor
> >> (replicas are not created on the same node). If you set the replication
> >> factor=2 i think you will not have this issue. So in general you must
> make
> >> sure that the rep factor is <= to the available datanodes.
> >>
> >> BR,
> >> George
> >>
> >>
> >> On 6/21/2013 12:29 PM, sam liu wrote:
> >>
> >> Hi,
> >>
> >> I encountered an issue which hangs the decommission operatoin. Its
> steps:
> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
> in
> >> hdfs-site.xml, set the 'dfs.replication' to 2
> >> 2. Add node dn3 into the cluster as a new datanode, and did not change
> the
> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> >> note: step 2 passed
> >> 3. Decommission dn3 from the cluster
> >>
> >> Expected result: dn3 could be decommissioned successfully
> >>
> >> Actual result: decommission progress hangs and the status always be
> >> 'Waiting DataNode status: Decommissioned'
> >>
> >> However, if the initial cluster includes >= 3 datanodes, this issue
> won't
> >> be encountered when add/remove another datanode.
> >>
> >> Also, after step 2, I noticed that some block's expected replicas is 3,
> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
> >>
> >> Could anyone pls help provide some triages?
> >>
> >> Thanks in advance!
> >>
> >>
> >>
> >> --
> >> ---------------------------
> >>
> >> George Kousiouris, PhD
> >> Electrical and Computer Engineer
> >> Division of Communications,
> >> Electronics and Information Engineering
> >> School of Electrical and Computer Engineering
> >> Tel: +30 210 772 2546
> >> Mobile: +30 6939354121
> >> Fax: +30 210 772 2569
> >> Email: gkousiou@mail.ntua.gr
> >> Site: http://users.ntua.gr/gkousiou/
> >>
> >> National Technical University of Athens
> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
> >
> >
>
>
>
> --
> Harsh J
>

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

Yes, the default replication factor is 3. However, in my case, it's
strange: during decommission hangs, I found some block's expected replicas
is 3, but the 'dfs.replication' value in hdfs-site.xml of every cluster
node is always 2 from the beginning of cluster setup. Below is my steps:
1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, in
hdfs-site.xml, set the 'dfs.replication' to 2
2. Add node dn3 into the cluster as a new datanode, and did not change the '
dfs.replication' value in hdfs-site.xml and keep it as 2
note: step 2 passed
3. Decommission dn3 from the cluster
Expected result: dn3 could be decommissioned successfully
Actual result:
a). decommission progress hangs and the status always be 'Waiting DataNode
status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /', the
decommission continues and will be completed finally.
b). However, if the initial cluster includes >= 3 datanodes, this issue
won't be encountered when add/remove another datanode. For example, if I
setup a cluster with 3 datanodes, and then I can successfully add the 4th
datanode into it, and then also can successfully remove the 4th datanode
from the cluster.

I doubt it's a bug and plan to open a jira to Hadoop HDFS for this. Any
comments?

Thanks!


2013/6/21 Harsh J <ha...@cloudera.com>

> The dfs.replication is a per-file parameter. If you have a client that
> does not use the supplied configs, then its default replication is 3
> and all files it will create (as part of the app or via a job config)
> will be with replication factor 3.
>
> You can do an -lsr to find all files and filter which ones have been
> created with a factor of 3 (versus expected config of 2).
>
> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com> wrote:
> > Hi George,
> >
> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2. But
> still
> > encounter this issue.
> >
> > Thanks!
> >
> >
> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
> >>
> >>
> >> Hi,
> >>
> >> I think i have faced this before, the problem is that you have the rep
> >> factor=3 so it seems to hang because it needs 3 nodes to achieve the
> factor
> >> (replicas are not created on the same node). If you set the replication
> >> factor=2 i think you will not have this issue. So in general you must
> make
> >> sure that the rep factor is <= to the available datanodes.
> >>
> >> BR,
> >> George
> >>
> >>
> >> On 6/21/2013 12:29 PM, sam liu wrote:
> >>
> >> Hi,
> >>
> >> I encountered an issue which hangs the decommission operatoin. Its
> steps:
> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
> in
> >> hdfs-site.xml, set the 'dfs.replication' to 2
> >> 2. Add node dn3 into the cluster as a new datanode, and did not change
> the
> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> >> note: step 2 passed
> >> 3. Decommission dn3 from the cluster
> >>
> >> Expected result: dn3 could be decommissioned successfully
> >>
> >> Actual result: decommission progress hangs and the status always be
> >> 'Waiting DataNode status: Decommissioned'
> >>
> >> However, if the initial cluster includes >= 3 datanodes, this issue
> won't
> >> be encountered when add/remove another datanode.
> >>
> >> Also, after step 2, I noticed that some block's expected replicas is 3,
> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
> >>
> >> Could anyone pls help provide some triages?
> >>
> >> Thanks in advance!
> >>
> >>
> >>
> >> --
> >> ---------------------------
> >>
> >> George Kousiouris, PhD
> >> Electrical and Computer Engineer
> >> Division of Communications,
> >> Electronics and Information Engineering
> >> School of Electrical and Computer Engineering
> >> Tel: +30 210 772 2546
> >> Mobile: +30 6939354121
> >> Fax: +30 210 772 2569
> >> Email: gkousiou@mail.ntua.gr
> >> Site: http://users.ntua.gr/gkousiou/
> >>
> >> National Technical University of Athens
> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
> >
> >
>
>
>
> --
> Harsh J
>

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

Yes, the default replication factor is 3. However, in my case, it's
strange: during decommission hangs, I found some block's expected replicas
is 3, but the 'dfs.replication' value in hdfs-site.xml of every cluster
node is always 2 from the beginning of cluster setup. Below is my steps:
1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, in
hdfs-site.xml, set the 'dfs.replication' to 2
2. Add node dn3 into the cluster as a new datanode, and did not change the '
dfs.replication' value in hdfs-site.xml and keep it as 2
note: step 2 passed
3. Decommission dn3 from the cluster
Expected result: dn3 could be decommissioned successfully
Actual result:
a). decommission progress hangs and the status always be 'Waiting DataNode
status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /', the
decommission continues and will be completed finally.
b). However, if the initial cluster includes >= 3 datanodes, this issue
won't be encountered when add/remove another datanode. For example, if I
setup a cluster with 3 datanodes, and then I can successfully add the 4th
datanode into it, and then also can successfully remove the 4th datanode
from the cluster.

I doubt it's a bug and plan to open a jira to Hadoop HDFS for this. Any
comments?

Thanks!


2013/6/21 Harsh J <ha...@cloudera.com>

> The dfs.replication is a per-file parameter. If you have a client that
> does not use the supplied configs, then its default replication is 3
> and all files it will create (as part of the app or via a job config)
> will be with replication factor 3.
>
> You can do an -lsr to find all files and filter which ones have been
> created with a factor of 3 (versus expected config of 2).
>
> On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com> wrote:
> > Hi George,
> >
> > Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2. But
> still
> > encounter this issue.
> >
> > Thanks!
> >
> >
> > 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
> >>
> >>
> >> Hi,
> >>
> >> I think i have faced this before, the problem is that you have the rep
> >> factor=3 so it seems to hang because it needs 3 nodes to achieve the
> factor
> >> (replicas are not created on the same node). If you set the replication
> >> factor=2 i think you will not have this issue. So in general you must
> make
> >> sure that the rep factor is <= to the available datanodes.
> >>
> >> BR,
> >> George
> >>
> >>
> >> On 6/21/2013 12:29 PM, sam liu wrote:
> >>
> >> Hi,
> >>
> >> I encountered an issue which hangs the decommission operatoin. Its
> steps:
> >> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
> in
> >> hdfs-site.xml, set the 'dfs.replication' to 2
> >> 2. Add node dn3 into the cluster as a new datanode, and did not change
> the
> >> 'dfs.replication' value in hdfs-site.xml and keep it as 2
> >> note: step 2 passed
> >> 3. Decommission dn3 from the cluster
> >>
> >> Expected result: dn3 could be decommissioned successfully
> >>
> >> Actual result: decommission progress hangs and the status always be
> >> 'Waiting DataNode status: Decommissioned'
> >>
> >> However, if the initial cluster includes >= 3 datanodes, this issue
> won't
> >> be encountered when add/remove another datanode.
> >>
> >> Also, after step 2, I noticed that some block's expected replicas is 3,
> >> but the 'dfs.replication' value in hdfs-site.xml is always 2!
> >>
> >> Could anyone pls help provide some triages?
> >>
> >> Thanks in advance!
> >>
> >>
> >>
> >> --
> >> ---------------------------
> >>
> >> George Kousiouris, PhD
> >> Electrical and Computer Engineer
> >> Division of Communications,
> >> Electronics and Information Engineering
> >> School of Electrical and Computer Engineering
> >> Tel: +30 210 772 2546
> >> Mobile: +30 6939354121
> >> Fax: +30 210 772 2569
> >> Email: gkousiou@mail.ntua.gr
> >> Site: http://users.ntua.gr/gkousiou/
> >>
> >> National Technical University of Athens
> >> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
> >
> >
>
>
>
> --
> Harsh J
>

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[Harsh J <ha...@cloudera.com>]

The dfs.replication is a per-file parameter. If you have a client that
does not use the supplied configs, then its default replication is 3
and all files it will create (as part of the app or via a job config)
will be with replication factor 3.

You can do an -lsr to find all files and filter which ones have been
created with a factor of 3 (versus expected config of 2).

On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com> wrote:
> Hi George,
>
> Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2. But still
> encounter this issue.
>
> Thanks!
>
>
> 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
>>
>>
>> Hi,
>>
>> I think i have faced this before, the problem is that you have the rep
>> factor=3 so it seems to hang because it needs 3 nodes to achieve the factor
>> (replicas are not created on the same node). If you set the replication
>> factor=2 i think you will not have this issue. So in general you must make
>> sure that the rep factor is <= to the available datanodes.
>>
>> BR,
>> George
>>
>>
>> On 6/21/2013 12:29 PM, sam liu wrote:
>>
>> Hi,
>>
>> I encountered an issue which hangs the decommission operatoin. Its steps:
>> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, in
>> hdfs-site.xml, set the 'dfs.replication' to 2
>> 2. Add node dn3 into the cluster as a new datanode, and did not change the
>> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>> note: step 2 passed
>> 3. Decommission dn3 from the cluster
>>
>> Expected result: dn3 could be decommissioned successfully
>>
>> Actual result: decommission progress hangs and the status always be
>> 'Waiting DataNode status: Decommissioned'
>>
>> However, if the initial cluster includes >= 3 datanodes, this issue won't
>> be encountered when add/remove another datanode.
>>
>> Also, after step 2, I noticed that some block's expected replicas is 3,
>> but the 'dfs.replication' value in hdfs-site.xml is always 2!
>>
>> Could anyone pls help provide some triages?
>>
>> Thanks in advance!
>>
>>
>>
>> --
>> ---------------------------
>>
>> George Kousiouris, PhD
>> Electrical and Computer Engineer
>> Division of Communications,
>> Electronics and Information Engineering
>> School of Electrical and Computer Engineering
>> Tel: +30 210 772 2546
>> Mobile: +30 6939354121
>> Fax: +30 210 772 2569
>> Email: gkousiou@mail.ntua.gr
>> Site: http://users.ntua.gr/gkousiou/
>>
>> National Technical University of Athens
>> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
>
>



-- 
Harsh J

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[Harsh J <ha...@cloudera.com>]

The dfs.replication is a per-file parameter. If you have a client that
does not use the supplied configs, then its default replication is 3
and all files it will create (as part of the app or via a job config)
will be with replication factor 3.

You can do an -lsr to find all files and filter which ones have been
created with a factor of 3 (versus expected config of 2).

On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com> wrote:
> Hi George,
>
> Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2. But still
> encounter this issue.
>
> Thanks!
>
>
> 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
>>
>>
>> Hi,
>>
>> I think i have faced this before, the problem is that you have the rep
>> factor=3 so it seems to hang because it needs 3 nodes to achieve the factor
>> (replicas are not created on the same node). If you set the replication
>> factor=2 i think you will not have this issue. So in general you must make
>> sure that the rep factor is <= to the available datanodes.
>>
>> BR,
>> George
>>
>>
>> On 6/21/2013 12:29 PM, sam liu wrote:
>>
>> Hi,
>>
>> I encountered an issue which hangs the decommission operatoin. Its steps:
>> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, in
>> hdfs-site.xml, set the 'dfs.replication' to 2
>> 2. Add node dn3 into the cluster as a new datanode, and did not change the
>> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>> note: step 2 passed
>> 3. Decommission dn3 from the cluster
>>
>> Expected result: dn3 could be decommissioned successfully
>>
>> Actual result: decommission progress hangs and the status always be
>> 'Waiting DataNode status: Decommissioned'
>>
>> However, if the initial cluster includes >= 3 datanodes, this issue won't
>> be encountered when add/remove another datanode.
>>
>> Also, after step 2, I noticed that some block's expected replicas is 3,
>> but the 'dfs.replication' value in hdfs-site.xml is always 2!
>>
>> Could anyone pls help provide some triages?
>>
>> Thanks in advance!
>>
>>
>>
>> --
>> ---------------------------
>>
>> George Kousiouris, PhD
>> Electrical and Computer Engineer
>> Division of Communications,
>> Electronics and Information Engineering
>> School of Electrical and Computer Engineering
>> Tel: +30 210 772 2546
>> Mobile: +30 6939354121
>> Fax: +30 210 772 2569
>> Email: gkousiou@mail.ntua.gr
>> Site: http://users.ntua.gr/gkousiou/
>>
>> National Technical University of Athens
>> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
>
>



-- 
Harsh J

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[Harsh J <ha...@cloudera.com>]

The dfs.replication is a per-file parameter. If you have a client that
does not use the supplied configs, then its default replication is 3
and all files it will create (as part of the app or via a job config)
will be with replication factor 3.

You can do an -lsr to find all files and filter which ones have been
created with a factor of 3 (versus expected config of 2).

On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com> wrote:
> Hi George,
>
> Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2. But still
> encounter this issue.
>
> Thanks!
>
>
> 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
>>
>>
>> Hi,
>>
>> I think i have faced this before, the problem is that you have the rep
>> factor=3 so it seems to hang because it needs 3 nodes to achieve the factor
>> (replicas are not created on the same node). If you set the replication
>> factor=2 i think you will not have this issue. So in general you must make
>> sure that the rep factor is <= to the available datanodes.
>>
>> BR,
>> George
>>
>>
>> On 6/21/2013 12:29 PM, sam liu wrote:
>>
>> Hi,
>>
>> I encountered an issue which hangs the decommission operatoin. Its steps:
>> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, in
>> hdfs-site.xml, set the 'dfs.replication' to 2
>> 2. Add node dn3 into the cluster as a new datanode, and did not change the
>> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>> note: step 2 passed
>> 3. Decommission dn3 from the cluster
>>
>> Expected result: dn3 could be decommissioned successfully
>>
>> Actual result: decommission progress hangs and the status always be
>> 'Waiting DataNode status: Decommissioned'
>>
>> However, if the initial cluster includes >= 3 datanodes, this issue won't
>> be encountered when add/remove another datanode.
>>
>> Also, after step 2, I noticed that some block's expected replicas is 3,
>> but the 'dfs.replication' value in hdfs-site.xml is always 2!
>>
>> Could anyone pls help provide some triages?
>>
>> Thanks in advance!
>>
>>
>>
>> --
>> ---------------------------
>>
>> George Kousiouris, PhD
>> Electrical and Computer Engineer
>> Division of Communications,
>> Electronics and Information Engineering
>> School of Electrical and Computer Engineering
>> Tel: +30 210 772 2546
>> Mobile: +30 6939354121
>> Fax: +30 210 772 2569
>> Email: gkousiou@mail.ntua.gr
>> Site: http://users.ntua.gr/gkousiou/
>>
>> National Technical University of Athens
>> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
>
>



-- 
Harsh J

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[Harsh J <ha...@cloudera.com>]

The dfs.replication is a per-file parameter. If you have a client that
does not use the supplied configs, then its default replication is 3
and all files it will create (as part of the app or via a job config)
will be with replication factor 3.

You can do an -lsr to find all files and filter which ones have been
created with a factor of 3 (versus expected config of 2).

On Fri, Jun 21, 2013 at 3:13 PM, sam liu <sa...@gmail.com> wrote:
> Hi George,
>
> Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2. But still
> encounter this issue.
>
> Thanks!
>
>
> 2013/6/21 George Kousiouris <gk...@mail.ntua.gr>
>>
>>
>> Hi,
>>
>> I think i have faced this before, the problem is that you have the rep
>> factor=3 so it seems to hang because it needs 3 nodes to achieve the factor
>> (replicas are not created on the same node). If you set the replication
>> factor=2 i think you will not have this issue. So in general you must make
>> sure that the rep factor is <= to the available datanodes.
>>
>> BR,
>> George
>>
>>
>> On 6/21/2013 12:29 PM, sam liu wrote:
>>
>> Hi,
>>
>> I encountered an issue which hangs the decommission operatoin. Its steps:
>> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, in
>> hdfs-site.xml, set the 'dfs.replication' to 2
>> 2. Add node dn3 into the cluster as a new datanode, and did not change the
>> 'dfs.replication' value in hdfs-site.xml and keep it as 2
>> note: step 2 passed
>> 3. Decommission dn3 from the cluster
>>
>> Expected result: dn3 could be decommissioned successfully
>>
>> Actual result: decommission progress hangs and the status always be
>> 'Waiting DataNode status: Decommissioned'
>>
>> However, if the initial cluster includes >= 3 datanodes, this issue won't
>> be encountered when add/remove another datanode.
>>
>> Also, after step 2, I noticed that some block's expected replicas is 3,
>> but the 'dfs.replication' value in hdfs-site.xml is always 2!
>>
>> Could anyone pls help provide some triages?
>>
>> Thanks in advance!
>>
>>
>>
>> --
>> ---------------------------
>>
>> George Kousiouris, PhD
>> Electrical and Computer Engineer
>> Division of Communications,
>> Electronics and Information Engineering
>> School of Electrical and Computer Engineering
>> Tel: +30 210 772 2546
>> Mobile: +30 6939354121
>> Fax: +30 210 772 2569
>> Email: gkousiou@mail.ntua.gr
>> Site: http://users.ntua.gr/gkousiou/
>>
>> National Technical University of Athens
>> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
>
>



-- 
Harsh J

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

Hi George,

Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2. But
still encounter this issue.

Thanks!


2013/6/21 George Kousiouris <gk...@mail.ntua.gr>

>
> Hi,
>
> I think i have faced this before, the problem is that you have the rep
> factor=3 so it seems to hang because it needs 3 nodes to achieve the factor
> (replicas are not created on the same node). If you set the replication
> factor=2 i think you will not have this issue. So in general you must make
> sure that the rep factor is <= to the available datanodes.
>
> BR,
> George
>
>
> On 6/21/2013 12:29 PM, sam liu wrote:
>
>  Hi,
>
> I encountered an issue which hangs the decommission operatoin. Its steps:
>  1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
> in hdfs-site.xml, set the 'dfs.replication' to 2
>  2. Add node dn3 into the cluster as a new datanode, and did not change
> the 'dfs.replication' value in hdfs-site.xml and keep it as 2
>  note: step 2 passed
>  3. Decommission dn3 from the cluster
>
>  Expected result: dn3 could be decommissioned successfully
>
>  Actual result: decommission progress hangs and the status always be
> 'Waiting DataNode status: Decommissioned'
>
>  However, if the initial cluster includes >= 3 datanodes, this issue
> won't be encountered when add/remove another datanode.
>
>  Also, after step 2, I noticed that some block's expected replicas is 3,
> but the 'dfs.replication' value in hdfs-site.xml is always 2!
>
>  Could anyone pls help provide some triages?
>
>  Thanks in advance!
>
>
>
> --
> ---------------------------
>
> George Kousiouris, PhD
> Electrical and Computer Engineer
> Division of Communications,
> Electronics and Information Engineering
> School of Electrical and Computer Engineering
> Tel: +30 210 772 2546
> Mobile: +30 6939354121
> Fax: +30 210 772 2569
> Email: gkousiou@mail.ntua.gr
> Site: http://users.ntua.gr/gkousiou/
>
> National Technical University of Athens
> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
>
>

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

Hi George,

Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2. But
still encounter this issue.

Thanks!


2013/6/21 George Kousiouris <gk...@mail.ntua.gr>

>
> Hi,
>
> I think i have faced this before, the problem is that you have the rep
> factor=3 so it seems to hang because it needs 3 nodes to achieve the factor
> (replicas are not created on the same node). If you set the replication
> factor=2 i think you will not have this issue. So in general you must make
> sure that the rep factor is <= to the available datanodes.
>
> BR,
> George
>
>
> On 6/21/2013 12:29 PM, sam liu wrote:
>
>  Hi,
>
> I encountered an issue which hangs the decommission operatoin. Its steps:
>  1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
> in hdfs-site.xml, set the 'dfs.replication' to 2
>  2. Add node dn3 into the cluster as a new datanode, and did not change
> the 'dfs.replication' value in hdfs-site.xml and keep it as 2
>  note: step 2 passed
>  3. Decommission dn3 from the cluster
>
>  Expected result: dn3 could be decommissioned successfully
>
>  Actual result: decommission progress hangs and the status always be
> 'Waiting DataNode status: Decommissioned'
>
>  However, if the initial cluster includes >= 3 datanodes, this issue
> won't be encountered when add/remove another datanode.
>
>  Also, after step 2, I noticed that some block's expected replicas is 3,
> but the 'dfs.replication' value in hdfs-site.xml is always 2!
>
>  Could anyone pls help provide some triages?
>
>  Thanks in advance!
>
>
>
> --
> ---------------------------
>
> George Kousiouris, PhD
> Electrical and Computer Engineer
> Division of Communications,
> Electronics and Information Engineering
> School of Electrical and Computer Engineering
> Tel: +30 210 772 2546
> Mobile: +30 6939354121
> Fax: +30 210 772 2569
> Email: gkousiou@mail.ntua.gr
> Site: http://users.ntua.gr/gkousiou/
>
> National Technical University of Athens
> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
>
>

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

Hi George,

Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2. But
still encounter this issue.

Thanks!


2013/6/21 George Kousiouris <gk...@mail.ntua.gr>

>
> Hi,
>
> I think i have faced this before, the problem is that you have the rep
> factor=3 so it seems to hang because it needs 3 nodes to achieve the factor
> (replicas are not created on the same node). If you set the replication
> factor=2 i think you will not have this issue. So in general you must make
> sure that the rep factor is <= to the available datanodes.
>
> BR,
> George
>
>
> On 6/21/2013 12:29 PM, sam liu wrote:
>
>  Hi,
>
> I encountered an issue which hangs the decommission operatoin. Its steps:
>  1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
> in hdfs-site.xml, set the 'dfs.replication' to 2
>  2. Add node dn3 into the cluster as a new datanode, and did not change
> the 'dfs.replication' value in hdfs-site.xml and keep it as 2
>  note: step 2 passed
>  3. Decommission dn3 from the cluster
>
>  Expected result: dn3 could be decommissioned successfully
>
>  Actual result: decommission progress hangs and the status always be
> 'Waiting DataNode status: Decommissioned'
>
>  However, if the initial cluster includes >= 3 datanodes, this issue
> won't be encountered when add/remove another datanode.
>
>  Also, after step 2, I noticed that some block's expected replicas is 3,
> but the 'dfs.replication' value in hdfs-site.xml is always 2!
>
>  Could anyone pls help provide some triages?
>
>  Thanks in advance!
>
>
>
> --
> ---------------------------
>
> George Kousiouris, PhD
> Electrical and Computer Engineer
> Division of Communications,
> Electronics and Information Engineering
> School of Electrical and Computer Engineering
> Tel: +30 210 772 2546
> Mobile: +30 6939354121
> Fax: +30 210 772 2569
> Email: gkousiou@mail.ntua.gr
> Site: http://users.ntua.gr/gkousiou/
>
> National Technical University of Athens
> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
>
>

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[sam liu <sa...@gmail.com>]

Hi George,

Actually, in my hdfs-site.xml, I always set 'dfs.replication'to 2. But
still encounter this issue.

Thanks!


2013/6/21 George Kousiouris <gk...@mail.ntua.gr>

>
> Hi,
>
> I think i have faced this before, the problem is that you have the rep
> factor=3 so it seems to hang because it needs 3 nodes to achieve the factor
> (replicas are not created on the same node). If you set the replication
> factor=2 i think you will not have this issue. So in general you must make
> sure that the rep factor is <= to the available datanodes.
>
> BR,
> George
>
>
> On 6/21/2013 12:29 PM, sam liu wrote:
>
>  Hi,
>
> I encountered an issue which hangs the decommission operatoin. Its steps:
>  1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And,
> in hdfs-site.xml, set the 'dfs.replication' to 2
>  2. Add node dn3 into the cluster as a new datanode, and did not change
> the 'dfs.replication' value in hdfs-site.xml and keep it as 2
>  note: step 2 passed
>  3. Decommission dn3 from the cluster
>
>  Expected result: dn3 could be decommissioned successfully
>
>  Actual result: decommission progress hangs and the status always be
> 'Waiting DataNode status: Decommissioned'
>
>  However, if the initial cluster includes >= 3 datanodes, this issue
> won't be encountered when add/remove another datanode.
>
>  Also, after step 2, I noticed that some block's expected replicas is 3,
> but the 'dfs.replication' value in hdfs-site.xml is always 2!
>
>  Could anyone pls help provide some triages?
>
>  Thanks in advance!
>
>
>
> --
> ---------------------------
>
> George Kousiouris, PhD
> Electrical and Computer Engineer
> Division of Communications,
> Electronics and Information Engineering
> School of Electrical and Computer Engineering
> Tel: +30 210 772 2546
> Mobile: +30 6939354121
> Fax: +30 210 772 2569
> Email: gkousiou@mail.ntua.gr
> Site: http://users.ntua.gr/gkousiou/
>
> National Technical University of Athens
> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
>
>

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[George Kousiouris <gk...@mail.ntua.gr>]

Hi,

I think i have faced this before, the problem is that you have the rep 
factor=3 so it seems to hang because it needs 3 nodes to achieve the 
factor (replicas are not created on the same node). If you set the 
replication factor=2 i think you will not have this issue. So in general 
you must make sure that the rep factor is <= to the available datanodes.

BR,
George

On 6/21/2013 12:29 PM, sam liu wrote:
> Hi,
>
> I encountered an issue which hangs the decommission operatoin. Its steps:
> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, 
> in hdfs-site.xml, set the 'dfs.replication' to 2
> 2. Add node dn3 into the cluster as a new datanode, and did not change 
> the 'dfs.replication' value in hdfs-site.xml and keep it as 2
> note: step 2 passed
> 3. Decommission dn3 from the cluster
>
> Expected result: dn3 could be decommissioned successfully
>
> Actual result: decommission progress hangs and the status always be 
> 'Waiting DataNode status: Decommissioned'
>
> However, if the initial cluster includes >= 3 datanodes, this issue 
> won't be encountered when add/remove another datanode.
>
> Also, after step 2, I noticed that some block's expected replicas is 
> 3, but the 'dfs.replication' value in hdfs-site.xml is always 2!
>
> Could anyone pls help provide some triages?
>
> Thanks in advance!


-- 
---------------------------

George Kousiouris, PhD
Electrical and Computer Engineer
Division of Communications,
Electronics and Information Engineering
School of Electrical and Computer Engineering
Tel: +30 210 772 2546
Mobile: +30 6939354121
Fax: +30 210 772 2569
Email: gkousiou@mail.ntua.gr
Site: http://users.ntua.gr/gkousiou/

National Technical University of Athens
9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[George Kousiouris <gk...@mail.ntua.gr>]

Hi,

I think i have faced this before, the problem is that you have the rep 
factor=3 so it seems to hang because it needs 3 nodes to achieve the 
factor (replicas are not created on the same node). If you set the 
replication factor=2 i think you will not have this issue. So in general 
you must make sure that the rep factor is <= to the available datanodes.

BR,
George

On 6/21/2013 12:29 PM, sam liu wrote:
> Hi,
>
> I encountered an issue which hangs the decommission operatoin. Its steps:
> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, 
> in hdfs-site.xml, set the 'dfs.replication' to 2
> 2. Add node dn3 into the cluster as a new datanode, and did not change 
> the 'dfs.replication' value in hdfs-site.xml and keep it as 2
> note: step 2 passed
> 3. Decommission dn3 from the cluster
>
> Expected result: dn3 could be decommissioned successfully
>
> Actual result: decommission progress hangs and the status always be 
> 'Waiting DataNode status: Decommissioned'
>
> However, if the initial cluster includes >= 3 datanodes, this issue 
> won't be encountered when add/remove another datanode.
>
> Also, after step 2, I noticed that some block's expected replicas is 
> 3, but the 'dfs.replication' value in hdfs-site.xml is always 2!
>
> Could anyone pls help provide some triages?
>
> Thanks in advance!


-- 
---------------------------

George Kousiouris, PhD
Electrical and Computer Engineer
Division of Communications,
Electronics and Information Engineering
School of Electrical and Computer Engineering
Tel: +30 210 772 2546
Mobile: +30 6939354121
Fax: +30 210 772 2569
Email: gkousiou@mail.ntua.gr
Site: http://users.ntua.gr/gkousiou/

National Technical University of Athens
9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[George Kousiouris <gk...@mail.ntua.gr>]

Hi,

I think i have faced this before, the problem is that you have the rep 
factor=3 so it seems to hang because it needs 3 nodes to achieve the 
factor (replicas are not created on the same node). If you set the 
replication factor=2 i think you will not have this issue. So in general 
you must make sure that the rep factor is <= to the available datanodes.

BR,
George

On 6/21/2013 12:29 PM, sam liu wrote:
> Hi,
>
> I encountered an issue which hangs the decommission operatoin. Its steps:
> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, 
> in hdfs-site.xml, set the 'dfs.replication' to 2
> 2. Add node dn3 into the cluster as a new datanode, and did not change 
> the 'dfs.replication' value in hdfs-site.xml and keep it as 2
> note: step 2 passed
> 3. Decommission dn3 from the cluster
>
> Expected result: dn3 could be decommissioned successfully
>
> Actual result: decommission progress hangs and the status always be 
> 'Waiting DataNode status: Decommissioned'
>
> However, if the initial cluster includes >= 3 datanodes, this issue 
> won't be encountered when add/remove another datanode.
>
> Also, after step 2, I noticed that some block's expected replicas is 
> 3, but the 'dfs.replication' value in hdfs-site.xml is always 2!
>
> Could anyone pls help provide some triages?
>
> Thanks in advance!


-- 
---------------------------

George Kousiouris, PhD
Electrical and Computer Engineer
Division of Communications,
Electronics and Information Engineering
School of Electrical and Computer Engineering
Tel: +30 210 772 2546
Mobile: +30 6939354121
Fax: +30 210 772 2569
Email: gkousiou@mail.ntua.gr
Site: http://users.ntua.gr/gkousiou/

National Technical University of Athens
9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece

Re: Hang when add/remove a datanode into/from a 2 datanode cluster

[George Kousiouris <gk...@mail.ntua.gr>]

Hi,

I think i have faced this before, the problem is that you have the rep 
factor=3 so it seems to hang because it needs 3 nodes to achieve the 
factor (replicas are not created on the same node). If you set the 
replication factor=2 i think you will not have this issue. So in general 
you must make sure that the rep factor is <= to the available datanodes.

BR,
George

On 6/21/2013 12:29 PM, sam liu wrote:
> Hi,
>
> I encountered an issue which hangs the decommission operatoin. Its steps:
> 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, 
> in hdfs-site.xml, set the 'dfs.replication' to 2
> 2. Add node dn3 into the cluster as a new datanode, and did not change 
> the 'dfs.replication' value in hdfs-site.xml and keep it as 2
> note: step 2 passed
> 3. Decommission dn3 from the cluster
>
> Expected result: dn3 could be decommissioned successfully
>
> Actual result: decommission progress hangs and the status always be 
> 'Waiting DataNode status: Decommissioned'
>
> However, if the initial cluster includes >= 3 datanodes, this issue 
> won't be encountered when add/remove another datanode.
>
> Also, after step 2, I noticed that some block's expected replicas is 
> 3, but the 'dfs.replication' value in hdfs-site.xml is always 2!
>
> Could anyone pls help provide some triages?
>
> Thanks in advance!


-- 
---------------------------

George Kousiouris, PhD
Electrical and Computer Engineer
Division of Communications,
Electronics and Information Engineering
School of Electrical and Computer Engineering
Tel: +30 210 772 2546
Mobile: +30 6939354121
Fax: +30 210 772 2569
Email: gkousiou@mail.ntua.gr
Site: http://users.ntua.gr/gkousiou/

National Technical University of Athens
9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece