You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "dmitry d (JIRA)" <ji...@apache.org> on 2013/09/04 13:09:52 UTC
[jira] [Updated] (PIG-3450) error when pigrunner.run embedded
python stored in hdfs
[ https://issues.apache.org/jira/browse/PIG-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dmitry d updated PIG-3450:
--------------------------
Description:
I try to run embedded python from java program using PigRunner. Script file store in hdfs. And I get the following error:
Error before Pig is launched
----------------------------
ERROR 2999: Unexpected internal error. null
java.lang.NullPointerException
at org.apache.pig.impl.io.FileLocalizer.fetchFilesInternal(FileLocalizer.java:778)
at org.apache.pig.impl.io.FileLocalizer.fetchFiles(FileLocalizer.java:746)
at org.apache.pig.PigServer.registerJar(PigServer.java:458)
at org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:364)
at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:296)
at org.apache.pig.Main.runEmbeddedScript(Main.java:1018)
at org.apache.pig.Main.run(Main.java:427)
at org.apache.pig.PigRunner.run(PigRunner.java:49)
at ru.eastwind.ewsa.export.ExportByPig.export(ExportByPig.java:78)
at ru.eastwind.ewsa.export.RunExport.main(RunExport.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:491)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
================================================================================
No error when I run python script from local path.
My Java code:
PigStats stats = PigRunner.run(new String[] {"-f", "hdfs://hadoop-dev:8020/user/root/sample.py"}, null);
My Python code:
#!/usr/bin/python
import sys
print('hello from python');
I saw source code where error is happened - FileLocalizer.java:
FileSystem localFs = FileSystem.getLocal(conf);
FileSystem srcFs;
if ( (!"true".equals(properties.getProperty("pig.jars.relative.to.dfs"))
&& uri.getScheme() == null )||
uri.getScheme().equals("local") ) {
srcFs = localFs;
} else {
srcFs = path.getFileSystem(conf);
}
uri is local path to jython-standalone.jar. And uri.getScheme() return null. And uri.getScheme().equals("local") is throw exception
Maybe I must redefine some property to get jython-standalone.jar from hdfs?
was:
I try to run embedded python from java program using PigRunner. Script file store in hdfs. And I get the following error:
{quote}
Error before Pig is launched
----------------------------
ERROR 2999: Unexpected internal error. null
java.lang.NullPointerException
at org.apache.pig.impl.io.FileLocalizer.fetchFilesInternal(FileLocalizer.java:778)
at org.apache.pig.impl.io.FileLocalizer.fetchFiles(FileLocalizer.java:746)
at org.apache.pig.PigServer.registerJar(PigServer.java:458)
at org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:364)
at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:296)
at org.apache.pig.Main.runEmbeddedScript(Main.java:1018)
at org.apache.pig.Main.run(Main.java:427)
at org.apache.pig.PigRunner.run(PigRunner.java:49)
at ru.eastwind.ewsa.export.ExportByPig.export(ExportByPig.java:78)
at ru.eastwind.ewsa.export.RunExport.main(RunExport.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:491)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
================================================================================
{quote}
No error when I run python script from local path.
My Java code:
{quote}
PigStats stats = PigRunner.run(new String[] {"-f", "hdfs://hadoop-dev:8020/user/root/sample.py"}, null);
{quote}
My Python code:
{quote}
#!/usr/bin/python
import sys
print('hello from python');
{quote}
I saw source code where error is happened - FileLocalizer.java:
{quote}
FileSystem localFs = FileSystem.getLocal(conf);
FileSystem srcFs;
if ( (!"true".equals(properties.getProperty("pig.jars.relative.to.dfs"))
&& uri.getScheme() == null )||
uri.getScheme().equals("local") ) {
srcFs = localFs;
} else {
srcFs = path.getFileSystem(conf);
}
{quote}
uri is local path to jython-standalone.jar. And uri.getScheme() return null. And uri.getScheme().equals("local") is throw exception
Maybe I must redefine some property to get jython-standalone.jar from hdfs?
> error when pigrunner.run embedded python stored in hdfs
> -------------------------------------------------------
>
> Key: PIG-3450
> URL: https://issues.apache.org/jira/browse/PIG-3450
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10.0
> Reporter: dmitry d
>
> I try to run embedded python from java program using PigRunner. Script file store in hdfs. And I get the following error:
> Error before Pig is launched
> ----------------------------
> ERROR 2999: Unexpected internal error. null
> java.lang.NullPointerException
> at org.apache.pig.impl.io.FileLocalizer.fetchFilesInternal(FileLocalizer.java:778)
> at org.apache.pig.impl.io.FileLocalizer.fetchFiles(FileLocalizer.java:746)
> at org.apache.pig.PigServer.registerJar(PigServer.java:458)
> at org.apache.pig.scripting.jython.JythonScriptEngine.main(JythonScriptEngine.java:364)
> at org.apache.pig.scripting.ScriptEngine.run(ScriptEngine.java:296)
> at org.apache.pig.Main.runEmbeddedScript(Main.java:1018)
> at org.apache.pig.Main.run(Main.java:427)
> at org.apache.pig.PigRunner.run(PigRunner.java:49)
> at ru.eastwind.ewsa.export.ExportByPig.export(ExportByPig.java:78)
> at ru.eastwind.ewsa.export.RunExport.main(RunExport.java:37)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:491)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> ================================================================================
> No error when I run python script from local path.
> My Java code:
> PigStats stats = PigRunner.run(new String[] {"-f", "hdfs://hadoop-dev:8020/user/root/sample.py"}, null);
> My Python code:
> #!/usr/bin/python
> import sys
> print('hello from python');
> I saw source code where error is happened - FileLocalizer.java:
> FileSystem localFs = FileSystem.getLocal(conf);
> FileSystem srcFs;
> if ( (!"true".equals(properties.getProperty("pig.jars.relative.to.dfs"))
> && uri.getScheme() == null )||
> uri.getScheme().equals("local") ) {
> srcFs = localFs;
> } else {
> srcFs = path.getFileSystem(conf);
> }
> uri is local path to jython-standalone.jar. And uri.getScheme() return null. And uri.getScheme().equals("local") is throw exception
> Maybe I must redefine some property to get jython-standalone.jar from hdfs?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira