You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Adam Szita (JIRA)" <ji...@apache.org> on 2017/09/01 13:18:01 UTC
[jira] [Created] (PIG-5297) Yarn-client mode doesn't work with
Spark 2
Adam Szita created PIG-5297:
-------------------------------
Summary: Yarn-client mode doesn't work with Spark 2
Key: PIG-5297
URL: https://issues.apache.org/jira/browse/PIG-5297
Project: Pig
Issue Type: Sub-task
Reporter: Adam Szita
Assignee: Adam Szita
When running tests in yarn-client mode that were built with Spark 2 I'm getting the following exception:
{code}
Caused by: java.lang.IllegalStateException: Library directory '...../pig/assembly/target/scala-2.11/jars' does not exist; make sure Spark is built.
at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
at org.apache.spark.launcher.CommandBuilderUtils.findJarsDir(CommandBuilderUtils.java:368)
at org.apache.spark.launcher.YarnCommandBuilderUtils$.findJarsDir(YarnCommandBuilderUtils.scala:38)
at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:558)
at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:882)
{code}
After overcoming this with symlinks and setting SPARK_MASTER I hit another issue:
{code}
Caused by: java.lang.NoSuchMethodError: io.netty.channel.DefaultFileRegion.<init>(Ljava/io/File;JJ)V
at org.apache.spark.network.buffer.FileSegmentManagedBuffer.convertToNetty(FileSegmentManagedBuffer.java:133)
at org.apache.spark.network.protocol.MessageEncoder.encode(MessageEncoder.java:58)
at org.apache.spark.network.protocol.MessageEncoder.encode(MessageEncoder.java:33)
at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:89)
{code}
I believe this will be an incompatibility between netty-all versions required by hadoop and spark..
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)