You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Araceli Henley (JIRA)" <ji...@apache.org> on 2009/02/27 01:38:01 UTC
[jira] Created: (PIG-689) Error Handling - improve message for
hadoop18.3 quotas feature
Error Handling - improve message for hadoop18.3 quotas feature
--------------------------------------------------------------
Key: PIG-689
URL: https://issues.apache.org/jira/browse/PIG-689
Project: Pig
Issue Type: Bug
Environment: linux
hadoop18.3
Reporter: Araceli Henley
Priority: Minor
TEST: Set disk quota and attempt to create directory or load a file such that the quota is exceeded
RESULT: Throws an error 2998 that indicates an unhandled exception, but the hadoop message is correct.
EXPECTED: a message to the effect that the "disk quota was exceeed"
The error message from hadoop is correct and adequate, but pig is throwing a "2998" error which is an "unhandled internal error". It should throw "Quota exceed error".
Log shows:
ERROR 2998: Unhandled internal error. org.apache.hadoop.dfs.QuotaExceededException: The quota of /user/hadoopqa/foo is exceeded: quota=2 count=3
org.apache.hadoop.dfs.QuotaExceededException: org.apache.hadoop.dfs.QuotaExceededException: The quota of /user/hadoopqa/foo is exceeded: quota=2 count=3
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:52)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:2311)
at org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:477)
at org.apache.hadoop.dfs.DistributedFileSystem.create(DistributedFileSystem.java:178)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:503)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:484)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:391)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:213)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:150)
at org.apache.pig.backend.hadoop.datastorage.HPath.copy(HPath.java:86)
at org.apache.pig.backend.hadoop.datastorage.HPath.copy(HPath.java:165)
at org.apache.pig.tools.grunt.GruntParser.processCopyFromLocal(GruntParser.java:499)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:98)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:72)
at org.apache.pig.Main.main(Main.java:296)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.dfs.QuotaExceededException: The quota of /user/hadoopqa/foo is exceeded: quota=2 count=3
at org.apache.hadoop.dfs.INodeDirectoryWithQuota.verifyQuota(INode.java:782)
at org.apache.hadoop.dfs.INodeDirectoryWithQuota.updateNumItemsInTree(INode.java:761)
at org.apache.hadoop.dfs.FSDirectory.updateCount(FSDirectory.java:767)
at org.apache.hadoop.dfs.FSDirectory.addChild(FSDirectory.java:896)
at org.apache.hadoop.dfs.FSDirectory.addNode(FSDirectory.java:886)
at org.apache.hadoop.dfs.FSDirectory.addFile(FSDirectory.java:151)
at org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:1014)
at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:909)
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:284)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:890)
at org.apache.hadoop.ipc.Client.call(Client.java:716)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
at org.apache.hadoop.dfs.$Proxy0.create(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at org.apache.hadoop.dfs.$Proxy0.create(Unknown Source)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:2308)
... 14
TEST CASES
General setup:
hadoop --config /grid/2/pig_harness/tmp/hod dfs -mkdir /user/hadoopqa/foo
/hadoop --config /grid/2/pig_harness/tmp/hod dfsadmin -setQuota 2 /user/hadoopqa/foo
>From the pig shell test:
1) creation of directory
2) try loading a file
STEPS TO REPRODUCE:
Allocate Hod
2) Create directories and set quota using hadoop
/homes/hadoopqa> hadoop --config /grid/2/pig_harness/tmp/hod dfs -mkdir /user/hadoopqa/foo
/homes/hadoopqa> hadoop --config /grid/2/pig_harness/tmp/hod dfsadmin -setQuota 2 /user/hadoopqa/foo
/homes/hadoopqa> hadoop --config /grid/2/pig_harness/tmp/hod dfs -mkdir /user/hadoopqa/foo/foo1
/homes/hadoopqa> hadoop --config /grid/2/pig_harness/tmp/hod dfs -mkdir /user/hadoopqa/foo/foo2
mkdir: org.apache.hadoop.dfs.QuotaExceededException: The quota of /user/hadoopqa/foo is exceeded: quota=2 count=3/homes/hadoopqa> hadoop --config /grid/2/pig_harness/tmp/hod dfs -copyFromLocal /grid/2/pig/in/studentcolon10k /user/hadoopqa
/homes/hadoopqa> hadoop --config /grid/2/pig_harness/tmp/hod dfs -copyFromLocal /grid/2/pig/in/studentcolon10k /user/hadoopqa/foo
3) open grunt shell and load file into directory that does not have quota set, then into directory that does have quota set.
grunt> copyFromLocal /grid/2/pig/in/studentcolon10k /user/hadoopqa/studentcolon10K.p1
grunt> ls /user/hadoopqa/
hdfs://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/user/hadoopqa/foo <dir>
hdfs://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/user/hadoopqa/mapredsystem <dir>
hdfs://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/user/hadoopqa/studentcolon10K.p1<r 3> 218620
hdfs://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/user/hadoopqa/studentcolon10k<r 3> 218620
grunt> copyFromLocal /grid/2/pig/in/studentcolon10k /user/hadoopqa/foo/studentcolon10K.p1
2009-02-26 22:54:42,858 main ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. org.apache.hadoop.dfs.QuotaExceededException: The quota of /user/hadoopqa/foo is exceeded: quota=2 count=3
Details at logfile: /homes/hadoopqa/pig_1235688818725.log
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (PIG-689) Error Handling - improve message for
hadoop18.3 quotas feature
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich resolved PIG-689.
--------------------------------
Resolution: Fixed
The error message looks adequate to me since it does state that the problem is with the quota. We can't special case every error we encounter
> Error Handling - improve message for hadoop18.3 quotas feature
> --------------------------------------------------------------
>
> Key: PIG-689
> URL: https://issues.apache.org/jira/browse/PIG-689
> Project: Pig
> Issue Type: Bug
> Environment: linux
> hadoop18.3
> Reporter: Araceli Henley
> Priority: Minor
>
> TEST: Set disk quota and attempt to create directory or load a file such that the quota is exceeded
> RESULT: Throws an error 2998 that indicates an unhandled exception, but the hadoop message is correct.
> EXPECTED: a message to the effect that the "disk quota was exceeed"
> The error message from hadoop is correct and adequate, but pig is throwing a "2998" error which is an "unhandled internal error". It should throw "Quota exceed error".
> Log shows:
> ERROR 2998: Unhandled internal error. org.apache.hadoop.dfs.QuotaExceededException: The quota of /user/hadoopqa/foo is exceeded: quota=2 count=3
> org.apache.hadoop.dfs.QuotaExceededException: org.apache.hadoop.dfs.QuotaExceededException: The quota of /user/hadoopqa/foo is exceeded: quota=2 count=3
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
> at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:52)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:2311)
> at org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:477)
> at org.apache.hadoop.dfs.DistributedFileSystem.create(DistributedFileSystem.java:178)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:503)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:484)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:391)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:213)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:150)
> at org.apache.pig.backend.hadoop.datastorage.HPath.copy(HPath.java:86)
> at org.apache.pig.backend.hadoop.datastorage.HPath.copy(HPath.java:165)
> at org.apache.pig.tools.grunt.GruntParser.processCopyFromLocal(GruntParser.java:499)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:98)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:72)
> at org.apache.pig.Main.main(Main.java:296)
> Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.dfs.QuotaExceededException: The quota of /user/hadoopqa/foo is exceeded: quota=2 count=3
> at org.apache.hadoop.dfs.INodeDirectoryWithQuota.verifyQuota(INode.java:782)
> at org.apache.hadoop.dfs.INodeDirectoryWithQuota.updateNumItemsInTree(INode.java:761)
> at org.apache.hadoop.dfs.FSDirectory.updateCount(FSDirectory.java:767)
> at org.apache.hadoop.dfs.FSDirectory.addChild(FSDirectory.java:896)
> at org.apache.hadoop.dfs.FSDirectory.addNode(FSDirectory.java:886)
> at org.apache.hadoop.dfs.FSDirectory.addFile(FSDirectory.java:151)
> at org.apache.hadoop.dfs.FSNamesystem.startFileInternal(FSNamesystem.java:1014)
> at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:909)
> at org.apache.hadoop.dfs.NameNode.create(NameNode.java:284)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:890)
> at org.apache.hadoop.ipc.Client.call(Client.java:716)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
> at org.apache.hadoop.dfs.$Proxy0.create(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.create(Unknown Source)
> at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:2308)
> ... 14
> TEST CASES
> General setup:
> hadoop --config /grid/2/pig_harness/tmp/hod dfs -mkdir /user/hadoopqa/foo
> /hadoop --config /grid/2/pig_harness/tmp/hod dfsadmin -setQuota 2 /user/hadoopqa/foo
> From the pig shell test:
> 1) creation of directory
> 2) try loading a file
> STEPS TO REPRODUCE:
> Allocate Hod
> 2) Create directories and set quota using hadoop
> /homes/hadoopqa> hadoop --config /grid/2/pig_harness/tmp/hod dfs -mkdir /user/hadoopqa/foo
> /homes/hadoopqa> hadoop --config /grid/2/pig_harness/tmp/hod dfsadmin -setQuota 2 /user/hadoopqa/foo
> /homes/hadoopqa> hadoop --config /grid/2/pig_harness/tmp/hod dfs -mkdir /user/hadoopqa/foo/foo1
> /homes/hadoopqa> hadoop --config /grid/2/pig_harness/tmp/hod dfs -mkdir /user/hadoopqa/foo/foo2
> mkdir: org.apache.hadoop.dfs.QuotaExceededException: The quota of /user/hadoopqa/foo is exceeded: quota=2 count=3/homes/hadoopqa> hadoop --config /grid/2/pig_harness/tmp/hod dfs -copyFromLocal /grid/2/pig/in/studentcolon10k /user/hadoopqa
> /homes/hadoopqa> hadoop --config /grid/2/pig_harness/tmp/hod dfs -copyFromLocal /grid/2/pig/in/studentcolon10k /user/hadoopqa/foo
>
> 3) open grunt shell and load file into directory that does not have quota set, then into directory that does have quota set.
> grunt> copyFromLocal /grid/2/pig/in/studentcolon10k /user/hadoopqa/studentcolon10K.p1
> grunt> ls /user/hadoopqa/
> hdfs://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/user/hadoopqa/foo <dir>
> hdfs://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/user/hadoopqa/mapredsystem <dir>
> hdfs://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/user/hadoopqa/studentcolon10K.p1<r 3> 218620
> hdfs://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/user/hadoopqa/studentcolon10k<r 3> 218620
> grunt> copyFromLocal /grid/2/pig/in/studentcolon10k /user/hadoopqa/foo/studentcolon10K.p1
> 2009-02-26 22:54:42,858 main ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. org.apache.hadoop.dfs.QuotaExceededException: The quota of /user/hadoopqa/foo is exceeded: quota=2 count=3
> Details at logfile: /homes/hadoopqa/pig_1235688818725.log
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.