You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-user@hadoop.apache.org by Dhiraj Kamble <Dh...@sandisk.com> on 2014/10/14 07:08:01 UTC

Redirect Writes/Reads to a particular Node

Hi,

Is it possible to redirect writes to one particular node i.e. store the primary replica  always on the same node; and have reads served from this primary node. If the primary node goes down; then hadoop replication works as per its policy; but when this node comes up it should again become the primary node. I don't see any config parameter available for core-site.xml or hdfs-site.xml to serve this purpose.

Is there any way I can do this.


Regards,
Dhiraj


________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

RE: Redirect Writes/Reads to a particular Node

Posted by Rakesh R <ra...@huawei.com>.

Hi Dhiraj,

AFAIK there is a mechanism to pass the set of 'favoredNodes datanodes' that should be favored as targets while creating a file. But this is only considered as a hint, sometimes due to cluster state(for example: given dn doesn't have sufficient space, doesn't available etc.), namenode may not be able to place the blocks on these datanodes then the hadoop replication works as per its policy. Please go through the below api which can be used for achieving this behavior and hope this will help you to get some idea.

DistributedFileSystem.java

 /* Same as
   * {@link #create(Path, FsPermission, boolean, int, short, long,
   * Progressable)} with the addition of favoredNodes that is a hint to
   * where the namenode should place the file blocks.
   * The favored nodes hint is not persisted in HDFS. Hence it may be honored
   * at the creation time only. HDFS could move the blocks during balancing or
   * replication, to move the blocks from favored nodes. A value of null means
   * no favored nodes for this create
   */
  public HdfsDataOutputStream create(final Path f,
      final FsPermission permission, final boolean overwrite,
      final int bufferSize, final short replication, final long blockSize,
      final Progressable progress, final InetSocketAddress[] favoredNodes)

Regards,
Rakesh

From: Dhiraj Kamble [mailto:Dhiraj.Kamble@sandisk.com]
Sent: 14 October 2014 10:38
To: user@hadoop.apache.org
Subject: Redirect Writes/Reads to a particular Node

Hi,

Is it possible to redirect writes to one particular node i.e. store the primary replica  always on the same node; and have reads served from this primary node. If the primary node goes down; then hadoop replication works as per its policy; but when this node comes up it should again become the primary node. I don't see any config parameter available for core-site.xml or hdfs-site.xml to serve this purpose.

Is there any way I can do this.


Regards,
Dhiraj


________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

RE: Redirect Writes/Reads to a particular Node

Posted by Rakesh R <ra...@huawei.com>.

Hi Dhiraj,

AFAIK there is a mechanism to pass the set of 'favoredNodes datanodes' that should be favored as targets while creating a file. But this is only considered as a hint, sometimes due to cluster state(for example: given dn doesn't have sufficient space, doesn't available etc.), namenode may not be able to place the blocks on these datanodes then the hadoop replication works as per its policy. Please go through the below api which can be used for achieving this behavior and hope this will help you to get some idea.

DistributedFileSystem.java

 /* Same as
   * {@link #create(Path, FsPermission, boolean, int, short, long,
   * Progressable)} with the addition of favoredNodes that is a hint to
   * where the namenode should place the file blocks.
   * The favored nodes hint is not persisted in HDFS. Hence it may be honored
   * at the creation time only. HDFS could move the blocks during balancing or
   * replication, to move the blocks from favored nodes. A value of null means
   * no favored nodes for this create
   */
  public HdfsDataOutputStream create(final Path f,
      final FsPermission permission, final boolean overwrite,
      final int bufferSize, final short replication, final long blockSize,
      final Progressable progress, final InetSocketAddress[] favoredNodes)

Regards,
Rakesh

From: Dhiraj Kamble [mailto:Dhiraj.Kamble@sandisk.com]
Sent: 14 October 2014 10:38
To: user@hadoop.apache.org
Subject: Redirect Writes/Reads to a particular Node

Hi,

Is it possible to redirect writes to one particular node i.e. store the primary replica  always on the same node; and have reads served from this primary node. If the primary node goes down; then hadoop replication works as per its policy; but when this node comes up it should again become the primary node. I don't see any config parameter available for core-site.xml or hdfs-site.xml to serve this purpose.

Is there any way I can do this.


Regards,
Dhiraj


________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

RE: Redirect Writes/Reads to a particular Node

Posted by Rakesh R <ra...@huawei.com>.

Hi Dhiraj,

AFAIK there is a mechanism to pass the set of 'favoredNodes datanodes' that should be favored as targets while creating a file. But this is only considered as a hint, sometimes due to cluster state(for example: given dn doesn't have sufficient space, doesn't available etc.), namenode may not be able to place the blocks on these datanodes then the hadoop replication works as per its policy. Please go through the below api which can be used for achieving this behavior and hope this will help you to get some idea.

DistributedFileSystem.java

 /* Same as
   * {@link #create(Path, FsPermission, boolean, int, short, long,
   * Progressable)} with the addition of favoredNodes that is a hint to
   * where the namenode should place the file blocks.
   * The favored nodes hint is not persisted in HDFS. Hence it may be honored
   * at the creation time only. HDFS could move the blocks during balancing or
   * replication, to move the blocks from favored nodes. A value of null means
   * no favored nodes for this create
   */
  public HdfsDataOutputStream create(final Path f,
      final FsPermission permission, final boolean overwrite,
      final int bufferSize, final short replication, final long blockSize,
      final Progressable progress, final InetSocketAddress[] favoredNodes)

Regards,
Rakesh

From: Dhiraj Kamble [mailto:Dhiraj.Kamble@sandisk.com]
Sent: 14 October 2014 10:38
To: user@hadoop.apache.org
Subject: Redirect Writes/Reads to a particular Node

Hi,

Is it possible to redirect writes to one particular node i.e. store the primary replica  always on the same node; and have reads served from this primary node. If the primary node goes down; then hadoop replication works as per its policy; but when this node comes up it should again become the primary node. I don't see any config parameter available for core-site.xml or hdfs-site.xml to serve this purpose.

Is there any way I can do this.


Regards,
Dhiraj


________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

RE: Redirect Writes/Reads to a particular Node

Posted by Rakesh R <ra...@huawei.com>.

Hi Dhiraj,

AFAIK there is a mechanism to pass the set of 'favoredNodes datanodes' that should be favored as targets while creating a file. But this is only considered as a hint, sometimes due to cluster state(for example: given dn doesn't have sufficient space, doesn't available etc.), namenode may not be able to place the blocks on these datanodes then the hadoop replication works as per its policy. Please go through the below api which can be used for achieving this behavior and hope this will help you to get some idea.

DistributedFileSystem.java

 /* Same as
   * {@link #create(Path, FsPermission, boolean, int, short, long,
   * Progressable)} with the addition of favoredNodes that is a hint to
   * where the namenode should place the file blocks.
   * The favored nodes hint is not persisted in HDFS. Hence it may be honored
   * at the creation time only. HDFS could move the blocks during balancing or
   * replication, to move the blocks from favored nodes. A value of null means
   * no favored nodes for this create
   */
  public HdfsDataOutputStream create(final Path f,
      final FsPermission permission, final boolean overwrite,
      final int bufferSize, final short replication, final long blockSize,
      final Progressable progress, final InetSocketAddress[] favoredNodes)

Regards,
Rakesh

From: Dhiraj Kamble [mailto:Dhiraj.Kamble@sandisk.com]
Sent: 14 October 2014 10:38
To: user@hadoop.apache.org
Subject: Redirect Writes/Reads to a particular Node

Hi,

Is it possible to redirect writes to one particular node i.e. store the primary replica  always on the same node; and have reads served from this primary node. If the primary node goes down; then hadoop replication works as per its policy; but when this node comes up it should again become the primary node. I don't see any config parameter available for core-site.xml or hdfs-site.xml to serve this purpose.

Is there any way I can do this.


Regards,
Dhiraj


________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).