[jira] [Updated] (OOZIE-3321) PySpark example fails

["Daniel Becker (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OOZIE-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Becker updated OOZIE-3321:
---------------------------------
    Attachment: OOZIE-3321.2.patch

> PySpark example fails
> ---------------------
>
>                 Key: OOZIE-3321
>                 URL: https://issues.apache.org/jira/browse/OOZIE-3321
>             Project: Oozie
>          Issue Type: Bug
>          Components: examples
>    Affects Versions: 5.0.0, 5.1.0
>            Reporter: Daniel Becker
>            Assignee: Daniel Becker
>            Priority: Minor
>         Attachments: OOZIE-3321.1.patch, OOZIE-3321.2.patch
>
>
> The PySpark example fails with the following exception:
> {noformat}
> ACTION[0000001-180806145017687-oozie-oozi-W@spark-node] Launcher exception: Missing py4j and/or pyspark zip files. Please add them to the lib folder or to the Spark sharelib.
> org.apache.oozie.action.hadoop.OozieActionConfiguratorException: Missing py4j and/or pyspark zip files. Please add them to the lib folder or to the Spark sharelib.
> 	at org.apache.oozie.action.hadoop.SparkMain.getMatchingPyFile(SparkMain.java:151)
> 	at org.apache.oozie.action.hadoop.SparkMain.createPySparkLibFolder(SparkMain.java:132)
> 	at org.apache.oozie.action.hadoop.SparkMain.run(SparkMain.java:77)
> 	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:101)
> 	at org.apache.oozie.action.hadoop.SparkMain.main(SparkMain.java:60)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:410)
> 	at org.apache.oozie.action.hadoop.LauncherAM.access$300(LauncherAM.java:55)
> 	at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:223)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
> 	at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:217)
> 	at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
> 	at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)
> {noformat}
> This is because py4j-0.9-src.zip and pyspark.zip are not available. They are needed for the example to run, so we should either add them to the sharelib or if they are not important enough to be in the sharelib, we should add them to the example's lib directory. Currently these two files can be found in the codebase under sharelib/spark/src/test/resources. We should also decide what to do with these files there, to avoid duplication of resource files.
> If I add the files manually to either the sharelib or the lib directory of the example, the example runs successfilly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)