You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by ta...@nttdata.co.jp on 2010/02/03 10:20:31 UTC
RE: HDFS Quota
Hi Boris,
Thank you for your investigation
in spite of a busy schedule.
What situation is it?
Best regards,
Tadashi.
> -----Original Message-----
> From: tatebet@nttdata.co.jp [mailto:tatebet@nttdata.co.jp]
> Sent: Tuesday, January 26, 2010 4:05 PM
> To: hdfs-user@hadoop.apache.org
> Subject: RE: HDFS Quota
>
> Hi Boris,
>
> Thank you for your reply.
>
> > From your example it looks like you have namespace quota (not diskspace),
> > i.e. quota for quantities of files (not size)
> I used following command.
>
> $ ./bin/hadoop dfsadmin -setSpaceQuota 192000 /user/root/input/
>
> I thought this was a limit on total size(192000) of all the files
> under the directory tree(/user/root/input/).
>
> Does quota for quantities of files (not size) mean Name Quotas?
>
> I found the following sentences.
> Name Quotas is a hard limit on the number of file and directory.
> Space Quotas is a hard limit on the number of bytes used by files.
>
> Sorry,I'm confused about the difference between namespace and diskspace.
>
> > Yes, I think this is by design. When HDFS creates a file, it doesn't know
> > how big the file will be. So diskspace quota is checked on writing to
> the
> > file (not on creation).
> I understood that.
> However, I think that the system recognize that the file exists even by
> 0 bytes.
> Therefore, should I take the action deleted on the application side when
> it becomes 0 bytes?
>
> > What is your block size. I don't think it make sense to put quota for
> less
> > then a blocksize.
> My block size is 64MByte(default setting).
> I understood that I should have set the quota larger than the block size.
>
> File size is 134217728/1024^2=128MByte.(larger than the block
> size(64MByte))
> Quota size is 256MByte(256*1024^2=268435456).(larger than the block
> size(64MByte))
> Number of replication is 2.
>
> # ls -ltr xaa
> -rw-r--r-- 1 root root 134217728 Jan 26 01:45 xaa
> # ls -ltr xab
> -rw-r--r-- 1 root root 134217728 Jan 26 01:45 xab
> # ./bin/hadoop dfsadmin -setSpaceQuota 256m /user/root/input/
> # ./bin/hadoop fs -count -q /user/root/input
> none inf 268435456 268435456
> 1 0 0 hdfs://drbd-test-vm03/user/root/input
> # ./bin/hadoop dfs -put xaa /user/root/input/
> # ./bin/hadoop fs -count -q /user/root/input
> none inf 268435456 0
> 1 1 134217728 hdfs://drbd-test-vm03/user/root/input
> # ./bin/hadoop dfs -put xab /user/root/input/
> # ./bin/hadoop fs -count -q /user/root/input
> none inf 268435456 -268435456
> 1 2 268435456 hdfs://drbd-test-vm03/user/root/input
>
> I thought that it became an error when the quota size was exceeded.
> However, it was a "-" mark without becoming an error.
> Why didn't the Space Quota setting(hard limit on the number of bytes) become
> effective?
>
> Best regards,
> Tadashi.
> > -----Original Message-----
> > From: Boris Shkolnik [mailto:borya@yahoo-inc.com]
> > Sent: Tuesday, January 26, 2010 5:20 AM
> > To: hdfs-user@hadoop.apache.org
> > Subject: Re: HDFS Quota
> >
> > >>> /user/root/input is exceeded: namespace quota=-1 file count=2,
> > From your example it looks like you have namespace quota (not diskspace),
> > i.e. quota for quantities of files (not size)
> >
> >
> > > Quota exceed exception,but it creates file of size 0.
> > Yes, I think this is by design. When HDFS creates a file, it doesn't know
> > how big the file will be. So diskspace quota is checked on writing to
> the
> > file (not on creation).
> >
> > >Q3：Why did input file's capacity exceed the limit(space quota)?
> > What is your block size. I don't think it make sense to put quota for
> less
> > then a blocksize.
> >
> > Boris.
> >
> > On 1/25/10 1:01 AM, "tatebet@nttdata.co.jp" <ta...@nttdata.co.jp>
> wrote:
> >
> > > Hi everyone,
> > >
> > > I set Space Quotas for the amount of space on HDFS.
> > > But I have some questions.
> > >
> > > ■Question
> > > Q1：Why does diskspace become MByet when the file of KByte is input?(1024
> > > calculations in HDFS)
> > > Q2：Is there a person who has information for this problem that quota
> exceed
> > > exception,but it creates file of size 0?
> > > Q3：Why did input file's capacity exceed the limit(space quota)?
> > >
> > >
> > > ■Details of question
> > > ★First time
> > > Even when I put file which was smaller than the capacity set in Space
> Quota,
> > > it became an error.
> > >
> > > The input file's capacity which I put is 64000=64kbyte.
> > >
> > > ・input file
> > > $ ls -ltr
> > > $ -rw-r--r-- 1 root root 64000 Jan 21 04:42 xaa
> > > $ du -h xaa
> > > 68K xaa
> > >
> > > ・hdfs-site.xml(replication)
> > > <property>
> > > <name>dfs.replication</name>
> > > <value>2</value>
> > > </property>
> > >
> > > ・Space Quota
> > > $ ./bin/hadoop dfsadmin -setSpaceQuota 192000 /user/root/input/
> > > $ ./bin/hadoop fs -count -q /user/root/input/
> > > none inf 192000 192000
> > 1
> > > 0 0 hdfs://drbd-test-vm03/user/root/input
> > > $ ./bin/hadoop dfs -put input/xaa /user/root/input/
> > > 10/01/21 19:35:58 WARN hdfs.DFSClient: DataStreamer Exception:
> > > org.apache.hadoop.hdfs.protocol.QuotaExceededException:
> > > org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> > > /user/root/input is exceeded: namespace quota=-1 file count=2,
> diskspace
> > > quota=192000 diskspace=134217728
> > > at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > > Method)
> > > at
> > >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructo
> rAc
> > cesso
> > > rImpl.java:39)
> > > at
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCo
> nst
> > ructo
> > > rAccessorImpl.java:27)
> > > at
> > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> > > at
> > >
> >
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteExcep
> tio
> > n.jav
> > > a:96)
> > > at
> > >
> >
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteExce
> pti
> > on.ja
> > > va:58)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(
> DFS
> > Clien
> > > t.java:2875)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream
> (DF
> > SClie
> > > nt.java:2755)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient
> .ja
> > va:20
> > > 46)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSC
> lie
> > nt.ja
> > > va:2232)
> > > Caused by: org.apache.hadoop.ipc.RemoteException:
> > > org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> > > /user/root/input is exceeded: namespace quota=-1 file count=2,
> diskspace
> > > quota=192000 diskspace=134217728
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.verifyQ
> uot
> > a(INo
> > > deDirectoryWithQuota.java:161)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.INodeDirectoryWithQuota.updateN
> umI
> > temsI
> > > nTree(INodeDirectoryWithQuota.java:134)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirec
> tor
> > y.jav
> > > a:859)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirector
> y.j
> > ava:2
> > > 65)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.allocateBlock(FSNa
> mes
> > ystem
> > > .java:1427)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock
> (FS
> > Names
> > > ystem.java:1274)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java
> :42
> > 2)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > at
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava
> > :39)
> > > at
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orI
> > mpl.j
> > > ava:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> > > at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > > at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > > at java.security.AccessController.doPrivileged(Native
> Method)
> > > at javax.security.auth.Subject.doAs(Subject.java:396)
> > > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> > >
> > > at org.apache.hadoop.ipc.Client.call(Client.java:739)
> > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> > > at $Proxy0.addBlock(Unknown Source)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > at
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava
> > :39)
> > > at
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orI
> > mpl.j
> > > ava:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at
> > >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryIn
> voc
> > ation
> > > Handler.java:82)
> > > at
> > >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocati
> onH
> > andle
> > > r.java:59)
> > > at $Proxy0.addBlock(Unknown Source)
> > > at
> > >
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(
> DFS
> > Clien
> > > t.java:2873)
> > > ... 3 more
> > >
> > > 10/01/21 19:35:58 WARN hdfs.DFSClient: Error Recovery for block null
> bad
> > > datanode[0] nodes == null
> > > 10/01/21 19:35:58 WARN hdfs.DFSClient: Could not get block locations.
> Source
> > > file "/user/root/input/xaa" - Aborting...
> > > put: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The
> quota of
> > > /user/root/input is exceeded: namespace quota=-1 file count=2,
> diskspace
> > > quota=192000 diskspace=134217728
> > >
> > > When I saw the above error messages,it was described as
> diskspace=134217728.
> > > Perhaps, I think that it is "134217728/1024^2=128MByte".
> > >
> > > I understood that this is why The quota(192000=192Kbyte) of
> /user/root/input
> > > was exceeded.
> > >
> > > Q1：Why does diskspace become MByet when the file of KByte is input?(1024
> > > calculations in HDFS)
> > >
> > > [root@drbd-test-vm03 current]# ./bin/hadoop dfs -lsr
> /user/root/input/xaa
> > > -rw-r--r-- 2 root supergroup 0 2010-01-21 19:35
> > > /user/root/input/xaa
> > >
> > > Quota exceed exception,but it creates file of size 0.
> > >
> > > It seems that this is the similar problem as the problem described in
> > > following URL.
> > > http://issues.apache.org/jira/browse/HDFS-172
> > >
> > > Has not this problem solved yet?
> > >
> > > Q2：Is there a person who has information for this problem that quota
> exceed
> > > exception,but it creates file of size 0?
> > >
> > > ★Since the second times
> > > The result was different the first time and the second times.
> > > It doesn't make an error even if it exceeds capacity since the second
> times.
> > >
> > > Imput file's capacity is 64000=64k.
> > > Reprication figure is 2.
> > > Calculated result
> > > 102400-(64000×2)=-25600<-- Why does it become a "-" mark without
> becoming
> > an
> > > error?
> > >
> > > I thought that the space quota is a limit on the number of bytes used
> by
> > files
> > > at that directory.
> > > However, the result shows the thing to which my assumption is wrong.
> > >
> > > Q3：Why did input file's capacity exceed the limit(space quota)?
> > >
> > > [root@drbd-test-vm03 current]# ./bin/hadoop dfsadmin -setSpaceQuota
> 100K
> > > /user/root/input/
> > > [root@drbd-test-vm03 current]# ./bin/hadoop fs -count -q
> /user/root/input/
> > > none inf 102400 102400
> > 1
> > > 0 0 hdfs://drbd-test-vm03/user/root/input
> > > [root@drbd-test-vm03 current]# ./bin/hadoop dfs -put xaa
> /user/root/input/
> > > [root@drbd-test-vm03 current]# ./bin/hadoop fs -count -q
> /user/root/input/
> > > none inf 102400 -25600
> > 1
> > > 1 64000 hdfs://drbd-test-vm03/user/root/input
> > >
> > > Best regards,
> > > Tadashi.
> > >