You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Koji Noguchi (Jira)" <ji...@apache.org> on 2021/10/12 19:50:00 UTC
[jira] [Updated] (PIG-5415) [spark] TestScriptLanguage conflict
between multiple SparkContext (after spark2.4 upgrade)
[ https://issues.apache.org/jira/browse/PIG-5415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated PIG-5415:
------------------------------
Attachment: pig-5415-v01.patch
* TestScriptLanguage.firstParallelTest
* TestScriptLanguage.runParallelTest
* TestScriptLanguage.runParallelTest2
They all started failing with the same error.
It seems like somewhere between spark2.1 to spark2.4, Spark started using "yarn" instead of "yarn-client".
This slight change broke SparkLauncher where we had "yarn-client" hard-coded. Attaching a quick fix.
> [spark] TestScriptLanguage conflict between multiple SparkContext (after spark2.4 upgrade)
> ------------------------------------------------------------------------------------------
>
> Key: PIG-5415
> URL: https://issues.apache.org/jira/browse/PIG-5415
> Project: Pig
> Issue Type: Bug
> Components: spark
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Attachments: pig-5415-v01.patch
>
>
> {noformat}
> 2021-10-12 17:54:40,073 [main] ERROR org.apache.pig.scripting.BoundScript - Pig pipeline failed to complete
> java.util.concurrent.ExecutionException: org.apache.pig.backend.executionengine.ExecException: ERROR 0: java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
> This stopped SparkContext was created at:
> org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
> org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.startSparkIfNeeded(SparkLauncher.java:640)
> org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.launchPig(SparkLauncher.java:184)
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290)
> org.apache.pig.PigServer.launchPlan(PigServer.java:1479)
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1464)
> org.apache.pig.PigServer.execute(PigServer.java:1453)
> org.apache.pig.PigServer.executeBatch(PigServer.java:489)
> org.apache.pig.PigServer.executeBatch(PigServer.java:472)
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:172)
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:235)
> org.apache.pig.scripting.BoundScript$MyCallable.call(BoundScript.java:347)
> org.apache.pig.scripting.BoundScript$MyCallable.call(BoundScript.java:323)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> The currently active SparkContext was created at:
> org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
> org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.startSparkIfNeeded(SparkLauncher.java:640)
> org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.launchPig(SparkLauncher.java:184)
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290)
> org.apache.pig.PigServer.launchPlan(PigServer.java:1479)
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1464)
> org.apache.pig.PigServer.execute(PigServer.java:1453)
> org.apache.pig.PigServer.executeBatch(PigServer.java:489)
> org.apache.pig.PigServer.executeBatch(PigServer.java:472)
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:172)
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:235)
> org.apache.pig.scripting.BoundScript$MyCallable.call(BoundScript.java:347)
> org.apache.pig.scripting.BoundScript$MyCallable.call(BoundScript.java:323)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)