You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Cheolsoo Park (JIRA)" <ji...@apache.org> on 2012/10/24 04:52:11 UTC
[jira] [Updated] (PIG-2979) Pig.jar doesn't work with hadoop-2.0.x
[ https://issues.apache.org/jira/browse/PIG-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheolsoo Park updated PIG-2979:
-------------------------------
Description:
To reproduce, please do the following:
1) ensure that no hadoop is installed and therefore no hadoop classes are present in classpath.
2) ant clean jar -Dhadoopversion=23
3) ./bin/pig -x local
4) fail with the following error: (you may not see this if slf4j is available in classpath.)
{code}
cheolsoo@localhost:~/workspace/pig-trunk $cat /home/cheolsoo/workspace/pig-trunk/pig_1350687456711.log
Error before Pig is launched
----------------------------
ERROR 2998: Unhandled internal error. org/slf4j/LoggerFactory
java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
...
{code}
5) cp ./build/ivy/lib/Pig/slf4j-api-1.6.1.jar lib
6) ./bin/pig -x local
7) fail with the following error:
{code}
cheolsoo@localhost:~/workspace/pig-trunk $cat /home/cheolsoo/workspace/pig-trunk/pig_1350687052995.log
Error before Pig is launched
----------------------------
ERROR 2999: Unexpected internal error. Failed to create DataStorage
java.lang.RuntimeException: Failed to create DataStorage
at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:204)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:117)
at org.apache.pig.impl.PigContext.connect(PigContext.java:229)
at org.apache.pig.PigServer.<init>(PigServer.java:213)
at org.apache.pig.PigServer.<init>(PigServer.java:198)
at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:47)
at org.apache.pig.Main.run(Main.java:535)
at org.apache.pig.Main.main(Main.java:154)
Caused by: java.io.IOException: No FileSystem for scheme: file
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2130)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2137)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2176)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2158)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:302)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:158)
at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
... 9 more
================================================================================
{code}
8) cp ./build/ivy/lib/Pig/hadoop-common-2.0.0-alpha.jar lib
9) ./bin/pig -x local
10) works fine!
In fact, this is also an issue with running e2e test in local mode:
{code}
ant clean
ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir -Dharness.cluster.bin=hadoop_script test-e2e-deploy-local -Dhadoopversion=23
ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir -Dharness.cluster.bin=hadoop_script test-e2e-local -Dhadoopversion=23
{code}
The ant test-e2e-local fails with the following error:
{code}
java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
at org.apache.hadoop.security.authentication.util.KerberosName.<clinit>(KerberosName.java:42)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:211)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:274)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:531)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:512)
{code}
was:
To reproduce, please run on machine where no Hadoop is installed:
{code}
ant clean
ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir -Dharness.cluster.bin=hadoop_script test-e2e-deploy-local -Dhadoopversion=23
ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir -Dharness.cluster.bin=hadoop_script test-e2e-local -Dhadoopversion=23
{code}
The ant test-e2e-local fails with the following error:
{code}
java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
at org.apache.hadoop.security.authentication.util.KerberosName.<clinit>(KerberosName.java:42)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:211)
at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:274)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:531)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:512)
{code}
In fact, this is also an issue with running Pig in local mode with the fat jar where no hadoop dependencies are available in classpath. For example, the following command also fails with the same error:
{code}
ant clean jar -Dhadoopversion=23
./bin/pig -x local
{code}
Summary: Pig.jar doesn't work with hadoop-2.0.x (was: ant test-e2e-local fails due to missing run-time dependencies in classpath with hadoop-2.0.x)
> Pig.jar doesn't work with hadoop-2.0.x
> --------------------------------------
>
> Key: PIG-2979
> URL: https://issues.apache.org/jira/browse/PIG-2979
> Project: Pig
> Issue Type: Sub-task
> Reporter: Cheolsoo Park
> Assignee: Cheolsoo Park
> Fix For: 0.11
>
> Attachments: PIG-2979.patch
>
>
> To reproduce, please do the following:
> 1) ensure that no hadoop is installed and therefore no hadoop classes are present in classpath.
> 2) ant clean jar -Dhadoopversion=23
> 3) ./bin/pig -x local
> 4) fail with the following error: (you may not see this if slf4j is available in classpath.)
> {code}
> cheolsoo@localhost:~/workspace/pig-trunk $cat /home/cheolsoo/workspace/pig-trunk/pig_1350687456711.log
> Error before Pig is launched
> ----------------------------
> ERROR 2998: Unhandled internal error. org/slf4j/LoggerFactory
> java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
> ...
> {code}
> 5) cp ./build/ivy/lib/Pig/slf4j-api-1.6.1.jar lib
> 6) ./bin/pig -x local
> 7) fail with the following error:
> {code}
> cheolsoo@localhost:~/workspace/pig-trunk $cat /home/cheolsoo/workspace/pig-trunk/pig_1350687052995.log
> Error before Pig is launched
> ----------------------------
> ERROR 2999: Unexpected internal error. Failed to create DataStorage
> java.lang.RuntimeException: Failed to create DataStorage
> at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:75)
> at org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:204)
> at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:117)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:229)
> at org.apache.pig.PigServer.<init>(PigServer.java:213)
> at org.apache.pig.PigServer.<init>(PigServer.java:198)
> at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:47)
> at org.apache.pig.Main.run(Main.java:535)
> at org.apache.pig.Main.main(Main.java:154)
> Caused by: java.io.IOException: No FileSystem for scheme: file
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2130)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2137)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:80)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2176)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2158)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:302)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:158)
> at org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:72)
> ... 9 more
> ================================================================================
> {code}
> 8) cp ./build/ivy/lib/Pig/hadoop-common-2.0.0-alpha.jar lib
> 9) ./bin/pig -x local
> 10) works fine!
> In fact, this is also an issue with running e2e test in local mode:
> {code}
> ant clean
> ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir -Dharness.cluster.bin=hadoop_script test-e2e-deploy-local -Dhadoopversion=23
> ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir -Dharness.cluster.bin=hadoop_script test-e2e-local -Dhadoopversion=23
> {code}
> The ant test-e2e-local fails with the following error:
> {code}
> java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
> at org.apache.hadoop.security.authentication.util.KerberosName.<clinit>(KerberosName.java:42)
> at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:211)
> at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:274)
> at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:531)
> at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:512)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira