You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by ta...@nttdata.co.jp on 2010/01/25 10:01:14 UTC
HDFS Quota
Hi everyone,
I set Space Quotas for the amount of space on HDFS.
But I have some questions.
■Question
Q1：Why does diskspace become MByet when the file of KByte is input?(1024 calculations in HDFS)
Q2：Is there a person who has information for this problem that quota exceed exception,but it creates file of size 0?
Q3：Why did input file's capacity exceed the limit(space quota)?
■Details of question
★First time
Even when I put file which was smaller than the capacity set in Space Quota,
it became an error.
The input file's capacity which I put is 64000=64kbyte.
・input file
$ ls -ltr
$ -rw-r--r-- 1 root root 64000 Jan 21 04:42 xaa
$ du -h xaa
68K xaa
・hdfs-site.xml(replication)
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
・Space Quota
$ ./bin/hadoop dfsadmin -setSpaceQuota 192000 /user/root/input/
$ ./bin/hadoop fs -count -q /user/root/input/
none inf 192000 192000 1 0 0 hdfs://drbd-test-vm03/user/root/input
$ ./bin/hadoop dfs -put input/xaa /user/root/input/
10/01/21 19:35:58 WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.hdfs.protocol.QuotaExceededException: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of /user/root/input is exceeded: namespace quota=-1 file count=2, diskspace quota=192000 diskspace=134217728
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:96)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:58)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2755)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2046)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2232)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of /user/root/input is exceeded: namespace quota=-1 file count=2, diskspace quota=192000 diskspace=134217728
at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INodeDirectoryWithQuota.java:161)
at org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.updateNumItemsInTree(INodeDirectoryWithQuota.java:134)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:859)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:265)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem.java:1427)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1274)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2873)
... 3 more
10/01/21 19:35:58 WARN hdfs.DFSClient: Error Recovery for block null bad datanode[0] nodes == null
10/01/21 19:35:58 WARN hdfs.DFSClient: Could not get block locations. Source file "/user/root/input/xaa" - Aborting...
put: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of /user/root/input is exceeded: namespace quota=-1 file count=2, diskspace quota=192000 diskspace=134217728
When I saw the above error messages,it was described as diskspace=134217728.
Perhaps, I think that it is "134217728/1024^2=128MByte".
I understood that this is why The quota(192000=192Kbyte) of /user/root/input was exceeded.
Q1：Why does diskspace become MByet when the file of KByte is input?(1024 calculations in HDFS)
[root@drbd-test-vm03 current]# ./bin/hadoop dfs -lsr /user/root/input/xaa
-rw-r--r-- 2 root supergroup 0 2010-01-21 19:35 /user/root/input/xaa
Quota exceed exception,but it creates file of size 0.
It seems that this is the similar problem as the problem described in following URL.
http://issues.apache.org/jira/browse/HDFS-172
Has not this problem solved yet?
Q2：Is there a person who has information for this problem that quota exceed exception,but it creates file of size 0?
★Since the second times
The result was different the first time and the second times.
It doesn't make an error even if it exceeds capacity since the second times.
Imput file's capacity is 64000=64k.
Reprication figure is 2.
Calculated result
102400-(64000×2)=-25600<-- Why does it become a "-" mark without becoming an error?
I thought that the space quota is a limit on the number of bytes used by files at that directory.
However, the result shows the thing to which my assumption is wrong.
Q3：Why did input file's capacity exceed the limit(space quota)?
[root@drbd-test-vm03 current]# ./bin/hadoop dfsadmin -setSpaceQuota 100K /user/root/input/
[root@drbd-test-vm03 current]# ./bin/hadoop fs -count -q /user/root/input/
none inf 102400 102400 1 0 0 hdfs://drbd-test-vm03/user/root/input
[root@drbd-test-vm03 current]# ./bin/hadoop dfs -put xaa /user/root/input/
[root@drbd-test-vm03 current]# ./bin/hadoop fs -count -q /user/root/input/
none inf 102400 -25600 1 1 64000 hdfs://drbd-test-vm03/user/root/input
Best regards,
Tadashi.
RE: HDFS Quota
Posted by ta...@nttdata.co.jp.
Hi Boris,
Thank you for your investigation
in spite of a busy schedule.
What situation is it?
Best regards,
Tadashi.
> -----Original Message-----
> From: tatebet@nttdata.co.jp [mailto:tatebet@nttdata.co.jp]
> Sent: Tuesday, January 26, 2010 4:05 PM
> To: hdfs-user@hadoop.apache.org
> Subject: RE: HDFS Quota
>
> Hi Boris,
>
> Thank you for your reply.
>
> > From your example it looks like you have namespace quota (not diskspace),
> > i.e. quota for quantities of files (not size)
> I used following command.
>
> $ ./bin/hadoop dfsadmin -setSpaceQuota 192000 /user/root/input/
>
> I thought this was a limit on total size(192000) of all the files
> under the directory tree(/user/root/input/).
>
> Does quota for quantities of files (not size) mean Name Quotas?
>
> I found the following sentences.
> Name Quotas is a hard limit on the number of file and directory.
> Space Quotas is a hard limit on the number of bytes used by files.
>
> Sorry,I'm confused about the difference between namespace and diskspace.
>
> > Yes, I think this is by design. When HDFS creates a file, it doesn't know
> > how big the file will be. So diskspace quota is checked on writing to
> the
> > file (not on creation).
> I understood that.
> However, I think that the system recognize that the file exists even by
> 0 bytes.
> Therefore, should I take the action deleted on the application side when
> it becomes 0 bytes?
>
> > What is your block size. I don't think it make sense to put quota for
> less
> > then a blocksize.
> My block size is 64MByte(default setting).
> I understood that I should have set the quota larger than the block size.
>
> File size is 134217728/1024^2=128MByte.(larger than the block
> size(64MByte))
> Quota size is 256MByte(256*1024^2=268435456).(larger than the block
> size(64MByte))
> Number of replication is 2.
>
> # ls -ltr xaa
> -rw-r--r-- 1 root root 134217728 Jan 26 01:45 xaa
> # ls -ltr xab
> -rw-r--r-- 1 root root 134217728 Jan 26 01:45 xab
> # ./bin/hadoop dfsadmin -setSpaceQuota 256m /user/root/input/
> # ./bin/hadoop fs -count -q /user/root/input
> none inf 268435456 268435456
> 1 0 0 hdfs://drbd-test-vm03/user/root/input
> # ./bin/hadoop dfs -put xaa /user/root/input/
> # ./bin/hadoop fs -count -q /user/root/input
> none inf 268435456 0
> 1 1 134217728 hdfs://drbd-test-vm03/user/root/input
> # ./bin/hadoop dfs -put xab /user/root/input/
> # ./bin/hadoop fs -count -q /user/root/input
> none inf 268435456 -268435456
> 1 2 268435456 hdfs://drbd-test-vm03/user/root/input
>
> I thought that it became an error when the quota size was exceeded.
> However, it was a "-" mark without becoming an error.
> Why didn't the Space Quota setting(hard limit on the number of bytes) become
> effective?
>
> Best regards,
> Tadashi.
> > -----Original Message-----
> > From: Boris Shkolnik [mailto:borya@yahoo-inc.com]
> > Sent: Tuesday, January 26, 2010 5:20 AM
> > To: hdfs-user@hadoop.apache.org
> > Subject: Re: HDFS Quota
> >
> > >>> /user/root/input is exceeded: namespace quota=-1 file count=2,
> > From your example it looks like you have namespace quota (not diskspace),
> > i.e. quota for quantities of files (not size)
> >
> >
> > > Quota exceed exception,but it creates file of size 0.
> > Yes, I think this is by design. When HDFS creates a file, it doesn't know
> > how big the file will be. So diskspace quota is checked on writing to
> the
> > file (not on creation).
> >
> > >Q3：Why did input file's capacity exceed the limit(space quota)?
> > What is your block size. I don't think it make sense to put quota for
> less
> > then a blocksize.
> >
> > Boris.
> >
> > On 1/25/10 1:01 AM, "tatebet@nttdata.co.jp" <ta...@nttdata.co.jp>
> wrote:
> >
> > > Hi everyone,
> > >
> > > I set Space Quotas for the amount of space on HDFS.
> > > But I have some questions.
> > >
> > > ■Question
> > > Q1：Why does diskspace become MByet when the file of KByte is input?(1024
> > > calculations in HDFS)
> > > Q2：Is there a person who has information for this problem that quota
> exceed
> > > exception,but it creates file of size 0?
> > > Q3：Why did input file's capacity exceed the limit(space quota)?
> > >
> > >
> > > ■Details of question
> > > ★First time
> > > Even when I put file which was smaller than the capacity set in Space
> Quota,
> > > it became an error.
> > >
> > > The input file's capacity which I put is 64000=64kbyte.
> > >
> > > ・input file
> > > $ ls -ltr
> > > $ -rw-r--r-- 1 root root 64000 Jan 21 04:42 xaa
> > > $ du -h xaa
> > > 68K xaa
> > >
> > > ・hdfs-site.xml(replication)
> > > <property>
> > > <name>dfs.replication</name>
> > > <value>2</value>
> > > </property>
> > >
> > > ・Space Quota
> > > $ ./bin/hadoop dfsadmin -setSpaceQuota 192000 /user/root/input/
> > > $ ./bin/hadoop fs -count -q /user/root/input/
> > > none inf 192000 192000
> > 1
> > > 0 0 hdfs://drbd-test-vm03/user/root/input
> > > $ ./bin/hadoop dfs -put input/xaa /user/root/input/
> > > 10/01/21 19:35:58 WARN hdfs.DFSClient: DataStreamer Exception:
> > > org.apache.hadoop.hdfs.protocol.QuotaExceededException:
> > > org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> > > /user/root/input is exceeded: namespace quota=-1 file count=2,
> diskspace
> > > quota=192000 diskspace=134217728
> > > at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > > Method)
> > > at
> > >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructo
> rAc
> > cesso
> > > rImpl.java:39)
> > > at
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCo
> nst
> > ructo
> > > rAccessorImpl.java:27)
> > > at
> > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > at
> > >
> >
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteExcep
> tio
> > n.jav
> > > a:96)
> > > at
> > >
> >
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteExce
> pti
> > on.ja
> > > va:58)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(
> DFS
> > Clien
> > > t.java:2875)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream
> (DF
> > SClie
> > > nt.java:2755)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient
> .ja
> > va:20
> > > 46)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSC
> lie
> > nt.ja
> > > va:2232)
> > > Caused by: org.apache.hadoop.ipc.RemoteException:
> > > org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> > > /user/root/input is exceeded: namespace quota=-1 file count=2,
> diskspace
> > > quota=192000 diskspace=134217728
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQ
> uot
> > a(INo
> > > deDirectoryWithQuota.java:161)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.updateN
> umI
> > temsI
> > > nTree(INodeDirectoryWithQuota.java:134)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirec
> tor
> > y.jav
> > > a:859)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirector
> y.j
> > ava:2
> > > 65)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNa
> mes
> > ystem
> > > .java:1427)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock
> (FS
> > Names
> > > ystem.java:1274)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java
> :42
> > 2)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > at
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava
> > :39)
> > > at
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orI
> > mpl.j
> > > ava:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> > > at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > > at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > > at java.security.AccessController.doPrivileged(Native
> Method)
> > > at javax.security.auth.Subject.doAs(Subject.java:396)
> > > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> > >
> > > at org.apache.hadoop.ipc.Client.call(Client.java:739)
> > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> > > at $Proxy0.addBlock(Unknown Source)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > at
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava
> > :39)
> > > at
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orI
> > mpl.j
> > > ava:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at
> > >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryIn
> voc
> > ation
> > > Handler.java:82)
> > > at
> > >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocati
> onH
> > andle
> > > r.java:59)
> > > at $Proxy0.addBlock(Unknown Source)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(
> DFS
> > Clien
> > > t.java:2873)
> > > ... 3 more
> > >
> > > 10/01/21 19:35:58 WARN hdfs.DFSClient: Error Recovery for block null
> bad
> > > datanode[0] nodes == null
> > > 10/01/21 19:35:58 WARN hdfs.DFSClient: Could not get block locations.
> Source
> > > file "/user/root/input/xaa" - Aborting...
> > > put: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The
> quota of
> > > /user/root/input is exceeded: namespace quota=-1 file count=2,
> diskspace
> > > quota=192000 diskspace=134217728
> > >
> > > When I saw the above error messages,it was described as
> diskspace=134217728.
> > > Perhaps, I think that it is "134217728/1024^2=128MByte".
> > >
> > > I understood that this is why The quota(192000=192Kbyte) of
> /user/root/input
> > > was exceeded.
> > >
> > > Q1：Why does diskspace become MByet when the file of KByte is input?(1024
> > > calculations in HDFS)
> > >
> > > [root@drbd-test-vm03 current]# ./bin/hadoop dfs -lsr
> /user/root/input/xaa
> > > -rw-r--r-- 2 root supergroup 0 2010-01-21 19:35
> > > /user/root/input/xaa
> > >
> > > Quota exceed exception,but it creates file of size 0.
> > >
> > > It seems that this is the similar problem as the problem described in
> > > following URL.
> > > http://issues.apache.org/jira/browse/HDFS-172
> > >
> > > Has not this problem solved yet?
> > >
> > > Q2：Is there a person who has information for this problem that quota
> exceed
> > > exception,but it creates file of size 0?
> > >
> > > ★Since the second times
> > > The result was different the first time and the second times.
> > > It doesn't make an error even if it exceeds capacity since the second
> times.
> > >
> > > Imput file's capacity is 64000=64k.
> > > Reprication figure is 2.
> > > Calculated result
> > > 102400-(64000×2)=-25600<-- Why does it become a "-" mark without
> becoming
> > an
> > > error?
> > >
> > > I thought that the space quota is a limit on the number of bytes used
> by
> > files
> > > at that directory.
> > > However, the result shows the thing to which my assumption is wrong.
> > >
> > > Q3：Why did input file's capacity exceed the limit(space quota)?
> > >
> > > [root@drbd-test-vm03 current]# ./bin/hadoop dfsadmin -setSpaceQuota
> 100K
> > > /user/root/input/
> > > [root@drbd-test-vm03 current]# ./bin/hadoop fs -count -q
> /user/root/input/
> > > none inf 102400 102400
> > 1
> > > 0 0 hdfs://drbd-test-vm03/user/root/input
> > > [root@drbd-test-vm03 current]# ./bin/hadoop dfs -put xaa
> /user/root/input/
> > > [root@drbd-test-vm03 current]# ./bin/hadoop fs -count -q
> /user/root/input/
> > > none inf 102400 -25600
> > 1
> > > 1 64000 hdfs://drbd-test-vm03/user/root/input
> > >
> > > Best regards,
> > > Tadashi.
> > >
Re: HDFS Quota
Posted by Eli Collins <el...@cloudera.com>.
Hey Tadashi,
> I set HDFS Quota.
> I thought that it became an error when the quota size was exceeded.
> However, it became "-" mark without becoming an error.
This indicates the number of bytes you have violated the quota by.
> Why didn't the Space Quota setting become effective?
It normally is, perhaps you set the quota after already violating it.
HDFS-172 is not yet resolved.
Thanks,
Eli
RE: HDFS Quota
Posted by ta...@nttdata.co.jp.
Hi everyone,
I have the problems about HDFS Quota.
I set HDFS Quota.
I thought that it became an error when the quota size was exceeded.
However, it became "-" mark without becoming an error.
Why didn't the Space Quota setting become effective?
Is there the person who has information about this problems?
Best regards,
Tadashi
> -----Original Message-----
> From: tatebet@nttdata.co.jp [mailto:tatebet@nttdata.co.jp]
> Sent: Tuesday, January 26, 2010 4:05 PM
> To: hdfs-user@hadoop.apache.org
> Subject: RE: HDFS Quota
>
> Hi Boris,
>
> Thank you for your reply.
>
> > From your example it looks like you have namespace quota (not diskspace),
> > i.e. quota for quantities of files (not size)
> I used following command.
>
> $ ./bin/hadoop dfsadmin -setSpaceQuota 192000 /user/root/input/
>
> I thought this was a limit on total size(192000) of all the files
> under the directory tree(/user/root/input/).
>
> Does quota for quantities of files (not size) mean Name Quotas?
>
> I found the following sentences.
> Name Quotas is a hard limit on the number of file and directory.
> Space Quotas is a hard limit on the number of bytes used by files.
>
> Sorry,I'm confused about the difference between namespace and diskspace.
>
> > Yes, I think this is by design. When HDFS creates a file, it doesn't know
> > how big the file will be. So diskspace quota is checked on writing to
> the
> > file (not on creation).
> I understood that.
> However, I think that the system recognize that the file exists even by
> 0 bytes.
> Therefore, should I take the action deleted on the application side when
> it becomes 0 bytes?
>
> > What is your block size. I don't think it make sense to put quota for
> less
> > then a blocksize.
> My block size is 64MByte(default setting).
> I understood that I should have set the quota larger than the block size.
>
> File size is 134217728/1024^2=128MByte.(larger than the block
> size(64MByte))
> Quota size is 256MByte(256*1024^2=268435456).(larger than the block
> size(64MByte))
> Number of replication is 2.
>
> # ls -ltr xaa
> -rw-r--r-- 1 root root 134217728 Jan 26 01:45 xaa
> # ls -ltr xab
> -rw-r--r-- 1 root root 134217728 Jan 26 01:45 xab
> # ./bin/hadoop dfsadmin -setSpaceQuota 256m /user/root/input/
> # ./bin/hadoop fs -count -q /user/root/input
> none inf 268435456 268435456
> 1 0 0 hdfs://drbd-test-vm03/user/root/input
> # ./bin/hadoop dfs -put xaa /user/root/input/
> # ./bin/hadoop fs -count -q /user/root/input
> none inf 268435456 0
> 1 1 134217728 hdfs://drbd-test-vm03/user/root/input
> # ./bin/hadoop dfs -put xab /user/root/input/
> # ./bin/hadoop fs -count -q /user/root/input
> none inf 268435456 -268435456
> 1 2 268435456 hdfs://drbd-test-vm03/user/root/input
>
> I thought that it became an error when the quota size was exceeded.
> However, it was a "-" mark without becoming an error.
> Why didn't the Space Quota setting(hard limit on the number of bytes) become
> effective?
>
> Best regards,
> Tadashi.
> > -----Original Message-----
> > From: Boris Shkolnik [mailto:borya@yahoo-inc.com]
> > Sent: Tuesday, January 26, 2010 5:20 AM
> > To: hdfs-user@hadoop.apache.org
> > Subject: Re: HDFS Quota
> >
> > >>> /user/root/input is exceeded: namespace quota=-1 file count=2,
> > From your example it looks like you have namespace quota (not diskspace),
> > i.e. quota for quantities of files (not size)
> >
> >
> > > Quota exceed exception,but it creates file of size 0.
> > Yes, I think this is by design. When HDFS creates a file, it doesn't know
> > how big the file will be. So diskspace quota is checked on writing to
> the
> > file (not on creation).
> >
> > >Q3：Why did input file's capacity exceed the limit(space quota)?
> > What is your block size. I don't think it make sense to put quota for
> less
> > then a blocksize.
> >
> > Boris.
> >
> > On 1/25/10 1:01 AM, "tatebet@nttdata.co.jp" <ta...@nttdata.co.jp>
> wrote:
> >
> > > Hi everyone,
> > >
> > > I set Space Quotas for the amount of space on HDFS.
> > > But I have some questions.
> > >
> > > ■Question
> > > Q1：Why does diskspace become MByet when the file of KByte is input?(1024
> > > calculations in HDFS)
> > > Q2：Is there a person who has information for this problem that quota
> exceed
> > > exception,but it creates file of size 0?
> > > Q3：Why did input file's capacity exceed the limit(space quota)?
> > >
> > >
> > > ■Details of question
> > > ★First time
> > > Even when I put file which was smaller than the capacity set in Space
> Quota,
> > > it became an error.
> > >
> > > The input file's capacity which I put is 64000=64kbyte.
> > >
> > > ・input file
> > > $ ls -ltr
> > > $ -rw-r--r-- 1 root root 64000 Jan 21 04:42 xaa
> > > $ du -h xaa
> > > 68K xaa
> > >
> > > ・hdfs-site.xml(replication)
> > > <property>
> > > <name>dfs.replication</name>
> > > <value>2</value>
> > > </property>
> > >
> > > ・Space Quota
> > > $ ./bin/hadoop dfsadmin -setSpaceQuota 192000 /user/root/input/
> > > $ ./bin/hadoop fs -count -q /user/root/input/
> > > none inf 192000 192000
> > 1
> > > 0 0 hdfs://drbd-test-vm03/user/root/input
> > > $ ./bin/hadoop dfs -put input/xaa /user/root/input/
> > > 10/01/21 19:35:58 WARN hdfs.DFSClient: DataStreamer Exception:
> > > org.apache.hadoop.hdfs.protocol.QuotaExceededException:
> > > org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> > > /user/root/input is exceeded: namespace quota=-1 file count=2,
> diskspace
> > > quota=192000 diskspace=134217728
> > > at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > > Method)
> > > at
> > >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructo
> rAc
> > cesso
> > > rImpl.java:39)
> > > at
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCo
> nst
> > ructo
> > > rAccessorImpl.java:27)
> > > at
> > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > at
> > >
> >
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteExcep
> tio
> > n.jav
> > > a:96)
> > > at
> > >
> >
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteExce
> pti
> > on.ja
> > > va:58)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(
> DFS
> > Clien
> > > t.java:2875)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream
> (DF
> > SClie
> > > nt.java:2755)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient
> .ja
> > va:20
> > > 46)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSC
> lie
> > nt.ja
> > > va:2232)
> > > Caused by: org.apache.hadoop.ipc.RemoteException:
> > > org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> > > /user/root/input is exceeded: namespace quota=-1 file count=2,
> diskspace
> > > quota=192000 diskspace=134217728
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQ
> uot
> > a(INo
> > > deDirectoryWithQuota.java:161)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.updateN
> umI
> > temsI
> > > nTree(INodeDirectoryWithQuota.java:134)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirec
> tor
> > y.jav
> > > a:859)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirector
> y.j
> > ava:2
> > > 65)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNa
> mes
> > ystem
> > > .java:1427)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock
> (FS
> > Names
> > > ystem.java:1274)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java
> :42
> > 2)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > at
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava
> > :39)
> > > at
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orI
> > mpl.j
> > > ava:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> > > at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > > at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > > at java.security.AccessController.doPrivileged(Native
> Method)
> > > at javax.security.auth.Subject.doAs(Subject.java:396)
> > > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> > >
> > > at org.apache.hadoop.ipc.Client.call(Client.java:739)
> > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> > > at $Proxy0.addBlock(Unknown Source)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > at
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava
> > :39)
> > > at
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orI
> > mpl.j
> > > ava:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at
> > >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryIn
> voc
> > ation
> > > Handler.java:82)
> > > at
> > >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocati
> onH
> > andle
> > > r.java:59)
> > > at $Proxy0.addBlock(Unknown Source)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(
> DFS
> > Clien
> > > t.java:2873)
> > > ... 3 more
> > >
> > > 10/01/21 19:35:58 WARN hdfs.DFSClient: Error Recovery for block null
> bad
> > > datanode[0] nodes == null
> > > 10/01/21 19:35:58 WARN hdfs.DFSClient: Could not get block locations.
> Source
> > > file "/user/root/input/xaa" - Aborting...
> > > put: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The
> quota of
> > > /user/root/input is exceeded: namespace quota=-1 file count=2,
> diskspace
> > > quota=192000 diskspace=134217728
> > >
> > > When I saw the above error messages,it was described as
> diskspace=134217728.
> > > Perhaps, I think that it is "134217728/1024^2=128MByte".
> > >
> > > I understood that this is why The quota(192000=192Kbyte) of
> /user/root/input
> > > was exceeded.
> > >
> > > Q1：Why does diskspace become MByet when the file of KByte is input?(1024
> > > calculations in HDFS)
> > >
> > > [root@drbd-test-vm03 current]# ./bin/hadoop dfs -lsr
> /user/root/input/xaa
> > > -rw-r--r-- 2 root supergroup 0 2010-01-21 19:35
> > > /user/root/input/xaa
> > >
> > > Quota exceed exception,but it creates file of size 0.
> > >
> > > It seems that this is the similar problem as the problem described in
> > > following URL.
> > > http://issues.apache.org/jira/browse/HDFS-172
> > >
> > > Has not this problem solved yet?
> > >
> > > Q2：Is there a person who has information for this problem that quota
> exceed
> > > exception,but it creates file of size 0?
> > >
> > > ★Since the second times
> > > The result was different the first time and the second times.
> > > It doesn't make an error even if it exceeds capacity since the second
> times.
> > >
> > > Imput file's capacity is 64000=64k.
> > > Reprication figure is 2.
> > > Calculated result
> > > 102400-(64000×2)=-25600<-- Why does it become a "-" mark without
> becoming
> > an
> > > error?
> > >
> > > I thought that the space quota is a limit on the number of bytes used
> by
> > files
> > > at that directory.
> > > However, the result shows the thing to which my assumption is wrong.
> > >
> > > Q3：Why did input file's capacity exceed the limit(space quota)?
> > >
> > > [root@drbd-test-vm03 current]# ./bin/hadoop dfsadmin -setSpaceQuota
> 100K
> > > /user/root/input/
> > > [root@drbd-test-vm03 current]# ./bin/hadoop fs -count -q
> /user/root/input/
> > > none inf 102400 102400
> > 1
> > > 0 0 hdfs://drbd-test-vm03/user/root/input
> > > [root@drbd-test-vm03 current]# ./bin/hadoop dfs -put xaa
> /user/root/input/
> > > [root@drbd-test-vm03 current]# ./bin/hadoop fs -count -q
> /user/root/input/
> > > none inf 102400 -25600
> > 1
> > > 1 64000 hdfs://drbd-test-vm03/user/root/input
> > >
> > > Best regards,
> > > Tadashi.
> > >
RE: HDFS Quota
Posted by ta...@nttdata.co.jp.
Hi Boris,
Thank you for your reply.
> From your example it looks like you have namespace quota (not diskspace),
> i.e. quota for quantities of files (not size)
I used following command.
$ ./bin/hadoop dfsadmin -setSpaceQuota 192000 /user/root/input/
I thought this was a limit on total size(192000) of all the files
under the directory tree(/user/root/input/).
Does quota for quantities of files (not size) mean Name Quotas?
I found the following sentences.
Name Quotas is a hard limit on the number of file and directory.
Space Quotas is a hard limit on the number of bytes used by files.
Sorry,I'm confused about the difference between namespace and diskspace.
> Yes, I think this is by design. When HDFS creates a file, it doesn't know
> how big the file will be. So diskspace quota is checked on writing to the
> file (not on creation).
I understood that.
However, I think that the system recognize that the file exists even by 0 bytes.
Therefore, should I take the action deleted on the application side when it becomes 0 bytes?
> What is your block size. I don't think it make sense to put quota for less
> then a blocksize.
My block size is 64MByte(default setting).
I understood that I should have set the quota larger than the block size.
File size is 134217728/1024^2=128MByte.(larger than the block size(64MByte))
Quota size is 256MByte(256*1024^2=268435456).(larger than the block size(64MByte))
Number of replication is 2.
# ls -ltr xaa
-rw-r--r-- 1 root root 134217728 Jan 26 01:45 xaa
# ls -ltr xab
-rw-r--r-- 1 root root 134217728 Jan 26 01:45 xab
# ./bin/hadoop dfsadmin -setSpaceQuota 256m /user/root/input/
# ./bin/hadoop fs -count -q /user/root/input
none inf 268435456 268435456 1 0 0 hdfs://drbd-test-vm03/user/root/input
# ./bin/hadoop dfs -put xaa /user/root/input/
# ./bin/hadoop fs -count -q /user/root/input
none inf 268435456 0 1 1 134217728 hdfs://drbd-test-vm03/user/root/input
# ./bin/hadoop dfs -put xab /user/root/input/
# ./bin/hadoop fs -count -q /user/root/input
none inf 268435456 -268435456 1 2 268435456 hdfs://drbd-test-vm03/user/root/input
I thought that it became an error when the quota size was exceeded.
However, it was a "-" mark without becoming an error.
Why didn't the Space Quota setting(hard limit on the number of bytes) become effective?
Best regards,
Tadashi.
> -----Original Message-----
> From: Boris Shkolnik [mailto:borya@yahoo-inc.com]
> Sent: Tuesday, January 26, 2010 5:20 AM
> To: hdfs-user@hadoop.apache.org
> Subject: Re: HDFS Quota
>
> >>> /user/root/input is exceeded: namespace quota=-1 file count=2,
> From your example it looks like you have namespace quota (not diskspace),
> i.e. quota for quantities of files (not size)
>
>
> > Quota exceed exception,but it creates file of size 0.
> Yes, I think this is by design. When HDFS creates a file, it doesn't know
> how big the file will be. So diskspace quota is checked on writing to the
> file (not on creation).
>
> >Q3：Why did input file's capacity exceed the limit(space quota)?
> What is your block size. I don't think it make sense to put quota for less
> then a blocksize.
>
> Boris.
>
> On 1/25/10 1:01 AM, "tatebet@nttdata.co.jp" <ta...@nttdata.co.jp> wrote:
>
> > Hi everyone,
> >
> > I set Space Quotas for the amount of space on HDFS.
> > But I have some questions.
> >
> > ■Question
> > Q1：Why does diskspace become MByet when the file of KByte is input?(1024
> > calculations in HDFS)
> > Q2：Is there a person who has information for this problem that quota exceed
> > exception,but it creates file of size 0?
> > Q3：Why did input file's capacity exceed the limit(space quota)?
> >
> >
> > ■Details of question
> > ★First time
> > Even when I put file which was smaller than the capacity set in Space Quota,
> > it became an error.
> >
> > The input file's capacity which I put is 64000=64kbyte.
> >
> > ・input file
> > $ ls -ltr
> > $ -rw-r--r-- 1 root root 64000 Jan 21 04:42 xaa
> > $ du -h xaa
> > 68K xaa
> >
> > ・hdfs-site.xml(replication)
> > <property>
> > <name>dfs.replication</name>
> > <value>2</value>
> > </property>
> >
> > ・Space Quota
> > $ ./bin/hadoop dfsadmin -setSpaceQuota 192000 /user/root/input/
> > $ ./bin/hadoop fs -count -q /user/root/input/
> > none inf 192000 192000
> 1
> > 0 0 hdfs://drbd-test-vm03/user/root/input
> > $ ./bin/hadoop dfs -put input/xaa /user/root/input/
> > 10/01/21 19:35:58 WARN hdfs.DFSClient: DataStreamer Exception:
> > org.apache.hadoop.hdfs.protocol.QuotaExceededException:
> > org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> > /user/root/input is exceeded: namespace quota=-1 file count=2, diskspace
> > quota=192000 diskspace=134217728
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> > at
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAc
> cesso
> > rImpl.java:39)
> > at
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConst
> ructo
> > rAccessorImpl.java:27)
> > at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > at
> >
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteExceptio
> n.jav
> > a:96)
> > at
> >
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteExcepti
> on.ja
> > va:58)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFS
> Clien
> > t.java:2875)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DF
> SClie
> > nt.java:2755)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.ja
> va:20
> > 46)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClie
> nt.ja
> > va:2232)
> > Caused by: org.apache.hadoop.ipc.RemoteException:
> > org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> > /user/root/input is exceeded: namespace quota=-1 file count=2, diskspace
> > quota=192000 diskspace=134217728
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuot
> a(INo
> > deDirectoryWithQuota.java:161)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.updateNumI
> temsI
> > nTree(INodeDirectoryWithQuota.java:134)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirector
> y.jav
> > a:859)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.j
> ava:2
> > 65)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNames
> ystem
> > .java:1427)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FS
> Names
> > ystem.java:1274)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:42
> 2)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java
> :39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
> mpl.j
> > ava:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:739)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> > at $Proxy0.addBlock(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java
> :39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
> mpl.j
> > ava:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvoc
> ation
> > Handler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationH
> andle
> > r.java:59)
> > at $Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFS
> Clien
> > t.java:2873)
> > ... 3 more
> >
> > 10/01/21 19:35:58 WARN hdfs.DFSClient: Error Recovery for block null bad
> > datanode[0] nodes == null
> > 10/01/21 19:35:58 WARN hdfs.DFSClient: Could not get block locations. Source
> > file "/user/root/input/xaa" - Aborting...
> > put: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> > /user/root/input is exceeded: namespace quota=-1 file count=2, diskspace
> > quota=192000 diskspace=134217728
> >
> > When I saw the above error messages,it was described as diskspace=134217728.
> > Perhaps, I think that it is "134217728/1024^2=128MByte".
> >
> > I understood that this is why The quota(192000=192Kbyte) of /user/root/input
> > was exceeded.
> >
> > Q1：Why does diskspace become MByet when the file of KByte is input?(1024
> > calculations in HDFS)
> >
> > [root@drbd-test-vm03 current]# ./bin/hadoop dfs -lsr /user/root/input/xaa
> > -rw-r--r-- 2 root supergroup 0 2010-01-21 19:35
> > /user/root/input/xaa
> >
> > Quota exceed exception,but it creates file of size 0.
> >
> > It seems that this is the similar problem as the problem described in
> > following URL.
> > http://issues.apache.org/jira/browse/HDFS-172
> >
> > Has not this problem solved yet?
> >
> > Q2：Is there a person who has information for this problem that quota exceed
> > exception,but it creates file of size 0?
> >
> > ★Since the second times
> > The result was different the first time and the second times.
> > It doesn't make an error even if it exceeds capacity since the second times.
> >
> > Imput file's capacity is 64000=64k.
> > Reprication figure is 2.
> > Calculated result
> > 102400-(64000×2)=-25600<-- Why does it become a "-" mark without becoming
> an
> > error?
> >
> > I thought that the space quota is a limit on the number of bytes used by
> files
> > at that directory.
> > However, the result shows the thing to which my assumption is wrong.
> >
> > Q3：Why did input file's capacity exceed the limit(space quota)?
> >
> > [root@drbd-test-vm03 current]# ./bin/hadoop dfsadmin -setSpaceQuota 100K
> > /user/root/input/
> > [root@drbd-test-vm03 current]# ./bin/hadoop fs -count -q /user/root/input/
> > none inf 102400 102400
> 1
> > 0 0 hdfs://drbd-test-vm03/user/root/input
> > [root@drbd-test-vm03 current]# ./bin/hadoop dfs -put xaa /user/root/input/
> > [root@drbd-test-vm03 current]# ./bin/hadoop fs -count -q /user/root/input/
> > none inf 102400 -25600
> 1
> > 1 64000 hdfs://drbd-test-vm03/user/root/input
> >
> > Best regards,
> > Tadashi.
> >
Re: HDFS Quota
Posted by Boris Shkolnik <bo...@yahoo-inc.com>.
>>> /user/root/input is exceeded: namespace quota=-1 file count=2,
>From your example it looks like you have namespace quota (not diskspace),
i.e. quota for quantities of files (not size)
> Quota exceed exception,but it creates file of size 0.
Yes, I think this is by design. When HDFS creates a file, it doesn't know
how big the file will be. So diskspace quota is checked on writing to the
file (not on creation).
>Q3：Why did input file's capacity exceed the limit(space quota)?
What is your block size. I don't think it make sense to put quota for less
then a blocksize.
Boris.
On 1/25/10 1:01 AM, "tatebet@nttdata.co.jp" <ta...@nttdata.co.jp> wrote:
> Hi everyone,
>
> I set Space Quotas for the amount of space on HDFS.
> But I have some questions.
>
> ■Question
> Q1：Why does diskspace become MByet when the file of KByte is input?(1024
> calculations in HDFS)
> Q2：Is there a person who has information for this problem that quota exceed
> exception,but it creates file of size 0?
> Q3：Why did input file's capacity exceed the limit(space quota)?
>
>
> ■Details of question
> ★First time
> Even when I put file which was smaller than the capacity set in Space Quota,
> it became an error.
>
> The input file's capacity which I put is 64000=64kbyte.
>
> ・input file
> $ ls -ltr
> $ -rw-r--r-- 1 root root 64000 Jan 21 04:42 xaa
> $ du -h xaa
> 68K xaa
>
> ・hdfs-site.xml(replication)
> <property>
> <name>dfs.replication</name>
> <value>2</value>
> </property>
>
> ・Space Quota
> $ ./bin/hadoop dfsadmin -setSpaceQuota 192000 /user/root/input/
> $ ./bin/hadoop fs -count -q /user/root/input/
> none inf 192000 192000 1
> 0 0 hdfs://drbd-test-vm03/user/root/input
> $ ./bin/hadoop dfs -put input/xaa /user/root/input/
> 10/01/21 19:35:58 WARN hdfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.hdfs.protocol.QuotaExceededException:
> org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> /user/root/input is exceeded: namespace quota=-1 file count=2, diskspace
> quota=192000 diskspace=134217728
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccesso
> rImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructo
> rAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.jav
> a:96)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.ja
> va:58)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClien
> t.java:2875)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClie
> nt.java:2755)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:20
> 46)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.ja
> va:2232)
> Caused by: org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> /user/root/input is exceeded: namespace quota=-1 file count=2, diskspace
> quota=192000 diskspace=134217728
> at
> org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQuota(INo
> deDirectoryWithQuota.java:161)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.updateNumItemsI
> nTree(INodeDirectoryWithQuota.java:134)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.jav
> a:859)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:2
> 65)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNamesystem
> .java:1427)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNames
> ystem.java:1274)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
> ava:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:739)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> at $Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
> ava:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocation
> Handler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandle
> r.java:59)
> at $Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClien
> t.java:2873)
> ... 3 more
>
> 10/01/21 19:35:58 WARN hdfs.DFSClient: Error Recovery for block null bad
> datanode[0] nodes == null
> 10/01/21 19:35:58 WARN hdfs.DFSClient: Could not get block locations. Source
> file "/user/root/input/xaa" - Aborting...
> put: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> /user/root/input is exceeded: namespace quota=-1 file count=2, diskspace
> quota=192000 diskspace=134217728
>
> When I saw the above error messages,it was described as diskspace=134217728.
> Perhaps, I think that it is "134217728/1024^2=128MByte".
>
> I understood that this is why The quota(192000=192Kbyte) of /user/root/input
> was exceeded.
>
> Q1：Why does diskspace become MByet when the file of KByte is input?(1024
> calculations in HDFS)
>
> [root@drbd-test-vm03 current]# ./bin/hadoop dfs -lsr /user/root/input/xaa
> -rw-r--r-- 2 root supergroup 0 2010-01-21 19:35
> /user/root/input/xaa
>
> Quota exceed exception,but it creates file of size 0.
>
> It seems that this is the similar problem as the problem described in
> following URL.
> http://issues.apache.org/jira/browse/HDFS-172
>
> Has not this problem solved yet?
>
> Q2：Is there a person who has information for this problem that quota exceed
> exception,but it creates file of size 0?
>
> ★Since the second times
> The result was different the first time and the second times.
> It doesn't make an error even if it exceeds capacity since the second times.
>
> Imput file's capacity is 64000=64k.
> Reprication figure is 2.
> Calculated result
> 102400-(64000×2)=-25600<-- Why does it become a "-" mark without becoming an
> error?
>
> I thought that the space quota is a limit on the number of bytes used by files
> at that directory.
> However, the result shows the thing to which my assumption is wrong.
>
> Q3：Why did input file's capacity exceed the limit(space quota)?
>
> [root@drbd-test-vm03 current]# ./bin/hadoop dfsadmin -setSpaceQuota 100K
> /user/root/input/
> [root@drbd-test-vm03 current]# ./bin/hadoop fs -count -q /user/root/input/
> none inf 102400 102400 1
> 0 0 hdfs://drbd-test-vm03/user/root/input
> [root@drbd-test-vm03 current]# ./bin/hadoop dfs -put xaa /user/root/input/
> [root@drbd-test-vm03 current]# ./bin/hadoop fs -count -q /user/root/input/
> none inf 102400 -25600 1
> 1 64000 hdfs://drbd-test-vm03/user/root/input
>
> Best regards,
> Tadashi.
>