You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Rita <rm...@gmail.com> on 2011/10/27 02:24:09 UTC
correct way to reserve space
What is the correct way to reserve space for hdfs?
I currently have 2 filesystem, /fs1 and /fs2 and I would like to reserve
space for non-dfs operations. For example, for /fs1 i would like to reserve
30gb of space for non-dfs and 10gb of space for /fs2 ?
I fear HADOOP-2991 is still haunting us?
I am using CDH 3U1
--
--- Get your facts first, then you can distort them as you please.--
Re: correct way to reserve space
Posted by Rita <rm...@gmail.com>.
Thanks Harsh and everyone
On Thu, Oct 27, 2011 at 6:45 AM, Harsh J <ha...@cloudera.com> wrote:
> Rita/Alexander,
>
> You may be interested in continuing the discussion for this feature at
> https://issues.apache.org/jira/browse/HDFS-1564
>
> Sorry I could not post that link earlier, as I was on my phone.
>
> On Thu, Oct 27, 2011 at 1:15 PM, Alexander C.H. Lorenz
> <wg...@googlemail.com> wrote:
> > Hi Harsh,
> >
> > ah, nice to know, thanks ;)
> >
> > best,
> > Alex
> >
> > On Thu, Oct 27, 2011 at 9:27 AM, Harsh J <ha...@cloudera.com> wrote:
> >
> >> The percentage opt is not valid on 0.20 iirc.
> >>
> >> On Thursday, October 27, 2011, Alexander C.H. Lorenz <
> >> wget.null@googlemail.com> wrote:
> >> > Hi,
> >> >
> >> > in hdfs-site.xml:
> >> >
> >> > <property>
> >> > <name>dfs.datanode.du.reserved</name>
> >> > <value> VALUE_HERE (maybe 30000000000) </value>
> >> > </property>
> >> >
> >> > <property>
> >> > <name>dfs.datanode.du.pct</name>
> >> > <value>0.9f</value>
> >> > </property>
> >> >
> >> > But not available for each partition, its a general parameter. Correct
> me
> >> > please if I wrong.
> >> >
> >> > regards,
> >> > Alex
> >> >
> >> >
> >> >
> >> > On Thu, Oct 27, 2011 at 2:24 AM, Rita <rm...@gmail.com> wrote:
> >> >
> >> >> What is the correct way to reserve space for hdfs?
> >> >>
> >> >> I currently have 2 filesystem, /fs1 and /fs2 and I would like to
> reserve
> >> >> space for non-dfs operations. For example, for /fs1 i would like to
> >> reserve
> >> >> 30gb of space for non-dfs and 10gb of space for /fs2 ?
> >> >>
> >> >>
> >> >> I fear HADOOP-2991 is still haunting us?
> >> >>
> >> >> I am using CDH 3U1
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> --- Get your facts first, then you can distort them as you please.--
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Alexander Lorenz
> >> > http://mapredit.blogspot.com
> >> >
> >>
> >> --
> >> Harsh J
> >>
> >
> >
> >
> > --
> > Alexander Lorenz
> > http://mapredit.blogspot.com
> >
>
>
>
> --
> Harsh J
>
--
--- Get your facts first, then you can distort them as you please.--
Re: correct way to reserve space
Posted by Harsh J <ha...@cloudera.com>.
Rita/Alexander,
You may be interested in continuing the discussion for this feature at
https://issues.apache.org/jira/browse/HDFS-1564
Sorry I could not post that link earlier, as I was on my phone.
On Thu, Oct 27, 2011 at 1:15 PM, Alexander C.H. Lorenz
<wg...@googlemail.com> wrote:
> Hi Harsh,
>
> ah, nice to know, thanks ;)
>
> best,
> Alex
>
> On Thu, Oct 27, 2011 at 9:27 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> The percentage opt is not valid on 0.20 iirc.
>>
>> On Thursday, October 27, 2011, Alexander C.H. Lorenz <
>> wget.null@googlemail.com> wrote:
>> > Hi,
>> >
>> > in hdfs-site.xml:
>> >
>> > <property>
>> > <name>dfs.datanode.du.reserved</name>
>> > <value> VALUE_HERE (maybe 30000000000) </value>
>> > </property>
>> >
>> > <property>
>> > <name>dfs.datanode.du.pct</name>
>> > <value>0.9f</value>
>> > </property>
>> >
>> > But not available for each partition, its a general parameter. Correct me
>> > please if I wrong.
>> >
>> > regards,
>> > Alex
>> >
>> >
>> >
>> > On Thu, Oct 27, 2011 at 2:24 AM, Rita <rm...@gmail.com> wrote:
>> >
>> >> What is the correct way to reserve space for hdfs?
>> >>
>> >> I currently have 2 filesystem, /fs1 and /fs2 and I would like to reserve
>> >> space for non-dfs operations. For example, for /fs1 i would like to
>> reserve
>> >> 30gb of space for non-dfs and 10gb of space for /fs2 ?
>> >>
>> >>
>> >> I fear HADOOP-2991 is still haunting us?
>> >>
>> >> I am using CDH 3U1
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> --- Get your facts first, then you can distort them as you please.--
>> >>
>> >
>> >
>> >
>> > --
>> > Alexander Lorenz
>> > http://mapredit.blogspot.com
>> >
>>
>> --
>> Harsh J
>>
>
>
>
> --
> Alexander Lorenz
> http://mapredit.blogspot.com
>
--
Harsh J
Re: correct way to reserve space
Posted by "Alexander C.H. Lorenz" <wg...@googlemail.com>.
Hi Harsh,
ah, nice to know, thanks ;)
best,
Alex
On Thu, Oct 27, 2011 at 9:27 AM, Harsh J <ha...@cloudera.com> wrote:
> The percentage opt is not valid on 0.20 iirc.
>
> On Thursday, October 27, 2011, Alexander C.H. Lorenz <
> wget.null@googlemail.com> wrote:
> > Hi,
> >
> > in hdfs-site.xml:
> >
> > <property>
> > <name>dfs.datanode.du.reserved</name>
> > <value> VALUE_HERE (maybe 30000000000) </value>
> > </property>
> >
> > <property>
> > <name>dfs.datanode.du.pct</name>
> > <value>0.9f</value>
> > </property>
> >
> > But not available for each partition, its a general parameter. Correct me
> > please if I wrong.
> >
> > regards,
> > Alex
> >
> >
> >
> > On Thu, Oct 27, 2011 at 2:24 AM, Rita <rm...@gmail.com> wrote:
> >
> >> What is the correct way to reserve space for hdfs?
> >>
> >> I currently have 2 filesystem, /fs1 and /fs2 and I would like to reserve
> >> space for non-dfs operations. For example, for /fs1 i would like to
> reserve
> >> 30gb of space for non-dfs and 10gb of space for /fs2 ?
> >>
> >>
> >> I fear HADOOP-2991 is still haunting us?
> >>
> >> I am using CDH 3U1
> >>
> >>
> >>
> >>
> >> --
> >> --- Get your facts first, then you can distort them as you please.--
> >>
> >
> >
> >
> > --
> > Alexander Lorenz
> > http://mapredit.blogspot.com
> >
>
> --
> Harsh J
>
--
Alexander Lorenz
http://mapredit.blogspot.com
Re: correct way to reserve space
Posted by Harsh J <ha...@cloudera.com>.
The percentage opt is not valid on 0.20 iirc.
On Thursday, October 27, 2011, Alexander C.H. Lorenz <
wget.null@googlemail.com> wrote:
> Hi,
>
> in hdfs-site.xml:
>
> <property>
> <name>dfs.datanode.du.reserved</name>
> <value> VALUE_HERE (maybe 30000000000) </value>
> </property>
>
> <property>
> <name>dfs.datanode.du.pct</name>
> <value>0.9f</value>
> </property>
>
> But not available for each partition, its a general parameter. Correct me
> please if I wrong.
>
> regards,
> Alex
>
>
>
> On Thu, Oct 27, 2011 at 2:24 AM, Rita <rm...@gmail.com> wrote:
>
>> What is the correct way to reserve space for hdfs?
>>
>> I currently have 2 filesystem, /fs1 and /fs2 and I would like to reserve
>> space for non-dfs operations. For example, for /fs1 i would like to
reserve
>> 30gb of space for non-dfs and 10gb of space for /fs2 ?
>>
>>
>> I fear HADOOP-2991 is still haunting us?
>>
>> I am using CDH 3U1
>>
>>
>>
>>
>> --
>> --- Get your facts first, then you can distort them as you please.--
>>
>
>
>
> --
> Alexander Lorenz
> http://mapredit.blogspot.com
>
--
Harsh J
Re: correct way to reserve space
Posted by "Alexander C.H. Lorenz" <wg...@googlemail.com>.
Hi,
in hdfs-site.xml:
<property>
<name>dfs.datanode.du.reserved</name>
<value> VALUE_HERE (maybe 30000000000) </value>
</property>
<property>
<name>dfs.datanode.du.pct</name>
<value>0.9f</value>
</property>
But not available for each partition, its a general parameter. Correct me
please if I wrong.
regards,
Alex
On Thu, Oct 27, 2011 at 2:24 AM, Rita <rm...@gmail.com> wrote:
> What is the correct way to reserve space for hdfs?
>
> I currently have 2 filesystem, /fs1 and /fs2 and I would like to reserve
> space for non-dfs operations. For example, for /fs1 i would like to reserve
> 30gb of space for non-dfs and 10gb of space for /fs2 ?
>
>
> I fear HADOOP-2991 is still haunting us?
>
> I am using CDH 3U1
>
>
>
>
> --
> --- Get your facts first, then you can distort them as you please.--
>
--
Alexander Lorenz
http://mapredit.blogspot.com