You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bigtop.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/12/24 15:25:00 UTC

[jira] [Commented] (BIGTOP-3117) [Puppet] Can't run spark commands for Spark on Yarn deployment

    [ https://issues.apache.org/jira/browse/BIGTOP-3117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16728428#comment-16728428 ] 

ASF GitHub Bot commented on BIGTOP-3117:
----------------------------------------

GitHub user evans-ye opened a pull request:

    https://github.com/apache/bigtop/pull/433

    BIGTOP-3117. [Puppet] Can't run spark commands for Spark on Yarn deployment

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/evans-ye/bigtop BIGTOP-3117

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/bigtop/pull/433.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #433
    
----
commit 46c13d0b013b091caac05e2d64cdd124ba2e73a9
Author: Evans Ye <ev...@...>
Date:   2018-12-24T15:24:15Z

    BIGTOP-3117. [Puppet] Can't run spark commands for Spark on Yarn deployment

----


> [Puppet] Can't run spark commands for Spark on Yarn deployment
> --------------------------------------------------------------
>
>                 Key: BIGTOP-3117
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-3117
>             Project: Bigtop
>          Issue Type: Sub-task
>          Components: deployment, spark
>    Affects Versions: 1.3.0
>            Reporter: Evans Ye
>            Assignee: Evans Ye
>            Priority: Major
>
> Deployment settings:
> {code}
> cat vagrantconfig.yaml
> ...
> memory_size: 4096
> number_cpus: 1
> box: "puppetlabs/centos-7.2-64-nocm"
> repo: "http://repos.bigtop.apache.org/releases/1.3.0/centos/7/x86_64"
> num_instances: 1
> distro: centos
> components: [hdfs, yarn, mapreduce, spark]
> enable_local_repo: true
> run_smoke_tests: false
> smoke_test_components: [hdfs, yarn, mapreduce]
> {code}
> After deployed, execute spark commands:
> {code}
> [root@bigtop1 bin]# ./spark-shell
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
> 	at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
> 	at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
> 	at scala.Option.getOrElse(Option.scala:121)
> 	at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
> 	at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:110)
> 	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
> 	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> 	... 7 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)