You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bigtop.apache.org by "Evans Ye (JIRA)" <ji...@apache.org> on 2018/12/24 08:49:00 UTC
[jira] [Created] (BIGTOP-3117) [Puppet] Can't run spark commands
for Spark on Yarn deployment
Evans Ye created BIGTOP-3117:
--------------------------------
Summary: [Puppet] Can't run spark commands for Spark on Yarn deployment
Key: BIGTOP-3117
URL: https://issues.apache.org/jira/browse/BIGTOP-3117
Project: Bigtop
Issue Type: Sub-task
Components: deployment, spark
Affects Versions: 1.3.0
Reporter: Evans Ye
Assignee: Evans Ye
Deployment settings:
{code}
cat vagrantconfig.yaml
...
memory_size: 4096
number_cpus: 1
box: "puppetlabs/centos-7.2-64-nocm"
repo: "http://repos.bigtop.apache.org/releases/1.3.0/centos/7/x86_64"
num_instances: 1
distro: centos
components: [hdfs, yarn, mapreduce, spark]
enable_local_repo: true
run_smoke_tests: false
smoke_test_components: [hdfs, yarn, mapreduce]
{code}
After deployed, execute spark commands:
{code}
[root@bigtop1 bin]# ./spark-shell
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:110)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)