You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Adam Szita <sz...@cloudera.com> on 2017/03/28 14:13:15 UTC

Review Request 57995: Fixing HiveUDF e2e tests on Spark mode

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57995/
-----------------------------------------------------------

Review request for pig, Daniel Dai, liyun zhang, and Rohini Palaniswamy.


Bugs: PIG-5194
    https://issues.apache.org/jira/browse/PIG-5194


Repository: pig-git


Description
-------

Please refer to my detailed comment here: https://issues.apache.org/jira/browse/PIG-5194?focusedCommentId=15938532&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15938532
Also some more context for the Spark parts:
If we call sparkContext.addJar the jar file will be put on the classpath of Spark's MutableURLClassLoader, not on the classpath of the system classloader. This is something to keep in mind in the future.


Diffs
-----

  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 36813b27be1090b04d577829080e4b931c5eb950 
  src/org/apache/pig/builtin/HiveUDAF.java cf53d7c246e8690d88d9f74b7b6db128c7f60744 


Diff: https://reviews.apache.org/r/57995/diff/1/


Testing
-------

E2E HiveUDF 1..7 on Spark get fixed with this patch.


Thanks,

Adam Szita

Re: Review Request 57995: Fixing HiveUDF e2e tests on Spark mode

Posted by Adam Szita <sz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57995/
-----------------------------------------------------------

(Updated May 22, 2017, 9:23 a.m.)


Review request for pig, Daniel Dai, liyun zhang, Rohini Palaniswamy, and Xuefu Zhang.


Bugs: PIG-5194
    https://issues.apache.org/jira/browse/PIG-5194


Repository: pig-git


Description
-------

Please refer to my detailed comment here: https://issues.apache.org/jira/browse/PIG-5194?focusedCommentId=15938532&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15938532
Also some more context for the Spark parts:
If we call sparkContext.addJar the jar file will be put on the classpath of Spark's MutableURLClassLoader, not on the classpath of the system classloader. This is something to keep in mind in the future.


Diffs (updated)
-----

  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java c6351e01a48f297ea2e432401ffd65c4f27f8078 
  src/org/apache/pig/builtin/HiveUDAF.java cf53d7c246e8690d88d9f74b7b6db128c7f60744 


Diff: https://reviews.apache.org/r/57995/diff/2/

Changes: https://reviews.apache.org/r/57995/diff/1-2/


Testing
-------

E2E HiveUDF 1..7 on Spark get fixed with this patch.


Thanks,

Adam Szita