You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Edward Capriolo <ed...@gmail.com> on 2009/05/02 16:13:19 UTC
Re: Getting free and used space
You can also pull these variables from the name node, datanode with
JMX. I am doing this to graph them with cacti. Both the JMX READ/WRITE
and READ user can access this variable.
On Tue, Apr 28, 2009 at 8:29 AM, Stas Oskin <st...@gmail.com> wrote:
> Hi.
>
> Any idea if the getDiskStatus() function requires superuser rights?
>
> Or it can work for any user?
>
> Thanks.
>
> 2009/4/9 Aaron Kimball <aa...@cloudera.com>
>
>> You can insert this propery into the jobconf, or specify it on the command
>> line e.g.: -D hadoop.job.ugi=username,group,group,group.....
>>
>> - Aaron
>>
>> On Wed, Apr 8, 2009 at 7:04 AM, Brian Bockelman <bbockelm@cse.unl.edu
>> >wrote:
>>
>> > Hey Stas,
>> >
>> > What we do locally is apply the latest patch for this issue:
>> > https://issues.apache.org/jira/browse/HADOOP-4368
>> >
>> > This makes getUsed (actually, it switches to FileSystem.getStatus) not a
>> > privileged action.
>> >
>> > As far as specifying the user ... gee, I can't think of it off the top of
>> > my head. It's a variable you can insert into the JobConf, but I'd have
>> to
>> > poke around google or the code to remember which one (I try to not
>> override
>> > it if possible).
>> >
>> > Brian
>> >
>> >
>> > On Apr 8, 2009, at 8:51 AM, Stas Oskin wrote:
>> >
>> > Hi.
>> >>
>> >> Thanks for the explanation.
>> >>
>> >> Now for the easier part - how do I specify the user when connecting? :)
>> >>
>> >> Is it a config file level, or run-time level setting?
>> >>
>> >> Regards.
>> >>
>> >> 2009/4/8 Brian Bockelman <bb...@cse.unl.edu>
>> >>
>> >> Hey Stas,
>> >>>
>> >>> Did you try this as a privileged user? There might be some permission
>> >>> errors... in most of the released versions, getUsed() is only available
>> >>> to
>> >>> the Hadoop superuser. It may be that the exception isn't propagating
>> >>> correctly.
>> >>>
>> >>> Brian
>> >>>
>> >>>
>> >>> On Apr 8, 2009, at 3:13 AM, Stas Oskin wrote:
>> >>>
>> >>> Hi.
>> >>>
>> >>>>
>> >>>> I'm trying to use the API to get the overall used and free spaces.
>> >>>>
>> >>>> I tried this function getUsed(), but it always returns 0.
>> >>>>
>> >>>> Any idea?
>> >>>>
>> >>>> Thanks.
>> >>>>
>> >>>>
>> >>>
>> >>>
>> >
>>
>
Re: Getting free and used space
Posted by Stas Oskin <st...@gmail.com>.
Thanks.
On Wed, Jun 23, 2010 at 7:01 PM, Ted Yu <yu...@gmail.com> wrote:
> Both DFSClient and DistributedFileSystem have this method:
> public DiskStatus getDiskStatus() throws IOException {
>
> You can obtain usage data from:
> public static class DiskStatus {
> which is defined in
> src/hdfs/org/apache/hadoop/hdfs/DistributedFileSystem.java
>
> On Wed, Jun 23, 2010 at 8:49 AM, Stas Oskin <st...@gmail.com> wrote:
>
> > Hi there.
> >
> > I noticed that function getUsed() is no longer present in Hadoop 0.20.
> >
> > What are the new functions of getting used and free space in DFS, and are
> > they working correctly, without any special admin permissions?
> >
> > Thanks!
> >
>
Re: Getting free and used space
Posted by Ted Yu <yu...@gmail.com>.
Both DFSClient and DistributedFileSystem have this method:
public DiskStatus getDiskStatus() throws IOException {
You can obtain usage data from:
public static class DiskStatus {
which is defined in
src/hdfs/org/apache/hadoop/hdfs/DistributedFileSystem.java
On Wed, Jun 23, 2010 at 8:49 AM, Stas Oskin <st...@gmail.com> wrote:
> Hi there.
>
> I noticed that function getUsed() is no longer present in Hadoop 0.20.
>
> What are the new functions of getting used and free space in DFS, and are
> they working correctly, without any special admin permissions?
>
> Thanks!
>
Re: Getting free and used space
Posted by Stas Oskin <st...@gmail.com>.
Hi there.
I noticed that function getUsed() is no longer present in Hadoop 0.20.
What are the new functions of getting used and free space in DFS, and are
they working correctly, without any special admin permissions?
Thanks!
Re: Getting free and used space
Posted by Stas Oskin <st...@gmail.com>.
Hi.
Actually, I'm trying to use the getDiskStatus() function, but it doesn't
seem to work so well in 0.18.3.
Can someone advice of a reliable way to get the HDFS overall free and used
space?
Same function that reports the space to the NameNode web panel?
Thanks.
2009/5/2 Edward Capriolo <ed...@gmail.com>
> You can also pull these variables from the name node, datanode with
> JMX. I am doing this to graph them with cacti. Both the JMX READ/WRITE
> and READ user can access this variable.
>
> On Tue, Apr 28, 2009 at 8:29 AM, Stas Oskin <st...@gmail.com> wrote:
> > Hi.
> >
> > Any idea if the getDiskStatus() function requires superuser rights?
> >
> > Or it can work for any user?
> >
> > Thanks.
> >
> > 2009/4/9 Aaron Kimball <aa...@cloudera.com>
> >
> >> You can insert this propery into the jobconf, or specify it on the
> command
> >> line e.g.: -D hadoop.job.ugi=username,group,group,group.....
> >>
> >> - Aaron
> >>
> >> On Wed, Apr 8, 2009 at 7:04 AM, Brian Bockelman <bbockelm@cse.unl.edu
> >> >wrote:
> >>
> >> > Hey Stas,
> >> >
> >> > What we do locally is apply the latest patch for this issue:
> >> > https://issues.apache.org/jira/browse/HADOOP-4368
> >> >
> >> > This makes getUsed (actually, it switches to FileSystem.getStatus) not
> a
> >> > privileged action.
> >> >
> >> > As far as specifying the user ... gee, I can't think of it off the top
> of
> >> > my head. It's a variable you can insert into the JobConf, but I'd
> have
> >> to
> >> > poke around google or the code to remember which one (I try to not
> >> override
> >> > it if possible).
> >> >
> >> > Brian
> >> >
> >> >
> >> > On Apr 8, 2009, at 8:51 AM, Stas Oskin wrote:
> >> >
> >> > Hi.
> >> >>
> >> >> Thanks for the explanation.
> >> >>
> >> >> Now for the easier part - how do I specify the user when connecting?
> :)
> >> >>
> >> >> Is it a config file level, or run-time level setting?
> >> >>
> >> >> Regards.
> >> >>
> >> >> 2009/4/8 Brian Bockelman <bb...@cse.unl.edu>
> >> >>
> >> >> Hey Stas,
> >> >>>
> >> >>> Did you try this as a privileged user? There might be some
> permission
> >> >>> errors... in most of the released versions, getUsed() is only
> available
> >> >>> to
> >> >>> the Hadoop superuser. It may be that the exception isn't
> propagating
> >> >>> correctly.
> >> >>>
> >> >>> Brian
> >> >>>
> >> >>>
> >> >>> On Apr 8, 2009, at 3:13 AM, Stas Oskin wrote:
> >> >>>
> >> >>> Hi.
> >> >>>
> >> >>>>
> >> >>>> I'm trying to use the API to get the overall used and free spaces.
> >> >>>>
> >> >>>> I tried this function getUsed(), but it always returns 0.
> >> >>>>
> >> >>>> Any idea?
> >> >>>>
> >> >>>> Thanks.
> >> >>>>
> >> >>>>
> >> >>>
> >> >>>
> >> >
> >>
> >
>