You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Vijayadarshan REDDY <vi...@dbs.com> on 2015/04/15 12:25:03 UTC
Restriction of disk space on HDFS
Hi Guys
Quick question - Using the fair scheduler we can restrict access to map tasks, reduce tasks and overall system resources for each queue. Using the same mechanism we don't see a any parameter to allocate disk usage by the queues.
Can you pls let us know if there is there a way to do this in CDH5?
Thanks and Regards,
Vijayadarshan Reddy
T&O-Core BI Technology
DBS Bank Ltd
Email : vijayadarshan@dbs.com
Mobile : +65 83157090
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the individual or entity named above and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this message in error, please immediately notify the sender and delete the mail. Thank you.
Re: Restriction of disk space on HDFS
Posted by Nitin Pawar <ni...@gmail.com>.
HDFS and job scheduling queues are entirely different systems.
HDFS disk quota are set at a directory level and then you can intern limit
the permissions of that directory to a group and then it indirectly means
this group has this much dist quota
On Wed, Apr 15, 2015 at 3:55 PM, Vijayadarshan REDDY <vi...@dbs.com>
wrote:
> Hi Guys
>
>
>
> Quick question - Using the fair scheduler we can restrict access to map
> tasks, reduce tasks and overall system resources for each queue. Using the
> same mechanism we don’t see a any parameter to allocate disk usage by the
> queues.
>
>
>
> Can you pls let us know if there is there a way to do this in CDH5?
>
>
>
>
>
> *Thanks and Regards,*
>
> *Vijayadarshan Reddy*
>
> *T&O-Core BI Technology*
>
> *DBS Bank Ltd*
>
> *Email : vijayadarshan@dbs.com <vi...@dbs.com>*
>
> *Mobile : +65 83157090 <%2B65%2083157090>*
>
>
> CONFIDENTIAL NOTE: The information contained in this email is intended
> only for the use of the individual or entity named above and may contain
> information that is privileged, confidential and exempt from disclosure
> under applicable law. If the reader of this message is not the intended
> recipient, you are hereby notified that any dissemination, distribution or
> copying of this communication is strictly prohibited. If you have received
> this message in error, please immediately notify the sender and delete the
> mail. Thank you.
>
--
Nitin Pawar
Re: Restriction of disk space on HDFS
Posted by Nitin Pawar <ni...@gmail.com>.
HDFS and job scheduling queues are entirely different systems.
HDFS disk quota are set at a directory level and then you can intern limit
the permissions of that directory to a group and then it indirectly means
this group has this much dist quota
On Wed, Apr 15, 2015 at 3:55 PM, Vijayadarshan REDDY <vi...@dbs.com>
wrote:
> Hi Guys
>
>
>
> Quick question - Using the fair scheduler we can restrict access to map
> tasks, reduce tasks and overall system resources for each queue. Using the
> same mechanism we don’t see a any parameter to allocate disk usage by the
> queues.
>
>
>
> Can you pls let us know if there is there a way to do this in CDH5?
>
>
>
>
>
> *Thanks and Regards,*
>
> *Vijayadarshan Reddy*
>
> *T&O-Core BI Technology*
>
> *DBS Bank Ltd*
>
> *Email : vijayadarshan@dbs.com <vi...@dbs.com>*
>
> *Mobile : +65 83157090 <%2B65%2083157090>*
>
>
> CONFIDENTIAL NOTE: The information contained in this email is intended
> only for the use of the individual or entity named above and may contain
> information that is privileged, confidential and exempt from disclosure
> under applicable law. If the reader of this message is not the intended
> recipient, you are hereby notified that any dissemination, distribution or
> copying of this communication is strictly prohibited. If you have received
> this message in error, please immediately notify the sender and delete the
> mail. Thank you.
>
--
Nitin Pawar
RE: Restriction of disk space on HDFS
Posted by Ravi Shankar <ra...@gmail.com>.
Please refer
https://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/YarnComma
nds.html#resourcemanager
Best regards,
Nair
From: Vijayadarshan REDDY [mailto:vijayadarshan@dbs.com]
Sent: Wednesday, April 15, 2015 6:25 AM
To: user@hadoop.apache.org
Subject: Restriction of disk space on HDFS
Hi Guys
Quick question - Using the fair scheduler we can restrict access to map
tasks, reduce tasks and overall system resources for each queue. Using the
same mechanism we don't see a any parameter to allocate disk usage by the
queues.
Can you pls let us know if there is there a way to do this in CDH5?
Thanks and Regards,
Vijayadarshan Reddy
T&O-Core BI Technology
DBS Bank Ltd
Email : vijayadarshan@dbs.com <ma...@dbs.com>
Mobile : +65 83157090
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the
individual or entity named above and may contain information that is
privileged, confidential and exempt from disclosure under applicable law. If
the reader of this message is not the intended recipient, you are hereby
notified that any dissemination, distribution or copying of this
communication is strictly prohibited. If you have received this message in
error, please immediately notify the sender and delete the mail. Thank you.
Re: Restriction of disk space on HDFS
Posted by Nitin Pawar <ni...@gmail.com>.
HDFS and job scheduling queues are entirely different systems.
HDFS disk quota are set at a directory level and then you can intern limit
the permissions of that directory to a group and then it indirectly means
this group has this much dist quota
On Wed, Apr 15, 2015 at 3:55 PM, Vijayadarshan REDDY <vi...@dbs.com>
wrote:
> Hi Guys
>
>
>
> Quick question - Using the fair scheduler we can restrict access to map
> tasks, reduce tasks and overall system resources for each queue. Using the
> same mechanism we don’t see a any parameter to allocate disk usage by the
> queues.
>
>
>
> Can you pls let us know if there is there a way to do this in CDH5?
>
>
>
>
>
> *Thanks and Regards,*
>
> *Vijayadarshan Reddy*
>
> *T&O-Core BI Technology*
>
> *DBS Bank Ltd*
>
> *Email : vijayadarshan@dbs.com <vi...@dbs.com>*
>
> *Mobile : +65 83157090 <%2B65%2083157090>*
>
>
> CONFIDENTIAL NOTE: The information contained in this email is intended
> only for the use of the individual or entity named above and may contain
> information that is privileged, confidential and exempt from disclosure
> under applicable law. If the reader of this message is not the intended
> recipient, you are hereby notified that any dissemination, distribution or
> copying of this communication is strictly prohibited. If you have received
> this message in error, please immediately notify the sender and delete the
> mail. Thank you.
>
--
Nitin Pawar
RE: Restriction of disk space on HDFS
Posted by Ravi Shankar <ra...@gmail.com>.
Please refer
https://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/YarnComma
nds.html#resourcemanager
Best regards,
Nair
From: Vijayadarshan REDDY [mailto:vijayadarshan@dbs.com]
Sent: Wednesday, April 15, 2015 6:25 AM
To: user@hadoop.apache.org
Subject: Restriction of disk space on HDFS
Hi Guys
Quick question - Using the fair scheduler we can restrict access to map
tasks, reduce tasks and overall system resources for each queue. Using the
same mechanism we don't see a any parameter to allocate disk usage by the
queues.
Can you pls let us know if there is there a way to do this in CDH5?
Thanks and Regards,
Vijayadarshan Reddy
T&O-Core BI Technology
DBS Bank Ltd
Email : vijayadarshan@dbs.com <ma...@dbs.com>
Mobile : +65 83157090
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the
individual or entity named above and may contain information that is
privileged, confidential and exempt from disclosure under applicable law. If
the reader of this message is not the intended recipient, you are hereby
notified that any dissemination, distribution or copying of this
communication is strictly prohibited. If you have received this message in
error, please immediately notify the sender and delete the mail. Thank you.
RE: Restriction of disk space on HDFS
Posted by Ravi Shankar <ra...@gmail.com>.
Please refer
https://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/YarnComma
nds.html#resourcemanager
Best regards,
Nair
From: Vijayadarshan REDDY [mailto:vijayadarshan@dbs.com]
Sent: Wednesday, April 15, 2015 6:25 AM
To: user@hadoop.apache.org
Subject: Restriction of disk space on HDFS
Hi Guys
Quick question - Using the fair scheduler we can restrict access to map
tasks, reduce tasks and overall system resources for each queue. Using the
same mechanism we don't see a any parameter to allocate disk usage by the
queues.
Can you pls let us know if there is there a way to do this in CDH5?
Thanks and Regards,
Vijayadarshan Reddy
T&O-Core BI Technology
DBS Bank Ltd
Email : vijayadarshan@dbs.com <ma...@dbs.com>
Mobile : +65 83157090
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the
individual or entity named above and may contain information that is
privileged, confidential and exempt from disclosure under applicable law. If
the reader of this message is not the intended recipient, you are hereby
notified that any dissemination, distribution or copying of this
communication is strictly prohibited. If you have received this message in
error, please immediately notify the sender and delete the mail. Thank you.
Re: Restriction of disk space on HDFS
Posted by Nitin Pawar <ni...@gmail.com>.
HDFS and job scheduling queues are entirely different systems.
HDFS disk quota are set at a directory level and then you can intern limit
the permissions of that directory to a group and then it indirectly means
this group has this much dist quota
On Wed, Apr 15, 2015 at 3:55 PM, Vijayadarshan REDDY <vi...@dbs.com>
wrote:
> Hi Guys
>
>
>
> Quick question - Using the fair scheduler we can restrict access to map
> tasks, reduce tasks and overall system resources for each queue. Using the
> same mechanism we don’t see a any parameter to allocate disk usage by the
> queues.
>
>
>
> Can you pls let us know if there is there a way to do this in CDH5?
>
>
>
>
>
> *Thanks and Regards,*
>
> *Vijayadarshan Reddy*
>
> *T&O-Core BI Technology*
>
> *DBS Bank Ltd*
>
> *Email : vijayadarshan@dbs.com <vi...@dbs.com>*
>
> *Mobile : +65 83157090 <%2B65%2083157090>*
>
>
> CONFIDENTIAL NOTE: The information contained in this email is intended
> only for the use of the individual or entity named above and may contain
> information that is privileged, confidential and exempt from disclosure
> under applicable law. If the reader of this message is not the intended
> recipient, you are hereby notified that any dissemination, distribution or
> copying of this communication is strictly prohibited. If you have received
> this message in error, please immediately notify the sender and delete the
> mail. Thank you.
>
--
Nitin Pawar
RE: Restriction of disk space on HDFS
Posted by Ravi Shankar <ra...@gmail.com>.
Please refer
https://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/YarnComma
nds.html#resourcemanager
Best regards,
Nair
From: Vijayadarshan REDDY [mailto:vijayadarshan@dbs.com]
Sent: Wednesday, April 15, 2015 6:25 AM
To: user@hadoop.apache.org
Subject: Restriction of disk space on HDFS
Hi Guys
Quick question - Using the fair scheduler we can restrict access to map
tasks, reduce tasks and overall system resources for each queue. Using the
same mechanism we don't see a any parameter to allocate disk usage by the
queues.
Can you pls let us know if there is there a way to do this in CDH5?
Thanks and Regards,
Vijayadarshan Reddy
T&O-Core BI Technology
DBS Bank Ltd
Email : vijayadarshan@dbs.com <ma...@dbs.com>
Mobile : +65 83157090
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the
individual or entity named above and may contain information that is
privileged, confidential and exempt from disclosure under applicable law. If
the reader of this message is not the intended recipient, you are hereby
notified that any dissemination, distribution or copying of this
communication is strictly prohibited. If you have received this message in
error, please immediately notify the sender and delete the mail. Thank you.