You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Taylor, Ronald C" <Ro...@pnnl.gov> on 2016/02/13 04:14:02 UTC
new to Spark - trying to get a basic example to run - could use
some help
Hello folks,
This is my first msg to the list. New to Spark, and trying to run the SparkPi example shown in the Cloudera documentation. We have Cloudera 5.5.1 running on a small cluster at our lab, with Spark 1.5.
My trial invocation is given below. The output that I get *says* that I "SUCCEEDED" at the end. But - I don't get any screen output on the value of pi. I also tried a SecondarySort Spark program that I compiled and jarred from Dr. Parsian's Data Algorithms book. That program failed. So - I am focusing on getting SparkPi to work properly, to get started. Can somebody look at the screen output that I cut-and-pasted below and infer what I might be doing wrong?
Am I forgetting to set one or more environment variables? Or not setting such properly?
Here is the CLASSPATH value that I set:
CLASSPATH=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/*:/people/rtaylor/SparkWork/DataAlgUtils
Here is the settings of other environment variables:
HADOOP_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11
SPARK_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11
HADOOP_CLASSPATH='/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/*:$JAVA_HOME/lib/tools.jar'
SPARK_CLASSPATH='/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/*:$JAVA_HOME/lib/tools.jar:':'/people/rtaylor/SparkWork/DataAlgUtils'
I am not sure that those env vars are properly set (or if even all of them are needed). But that's what I'm currently using.
As I said, the invocation below appears to terminate with final status set to "SUCCEEDED". But - there is no screen output on the value of pi, which I understood would be shown. So - something appears to be going wrong. I went to the tracking URL given at the end, but could not access it.
I would very much appreciate some guidance!
- Ron Taylor
%%%%%
INVOCATION:
[rtaylor@bigdatann]$ spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --name RT_SparkPi /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar 10
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/slf4j-simple-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/livy-assembly.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/avro-tools-1.7.6-cdh5.5.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/pig-0.12.0-cdh5.5.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/02/12 18:16:59 INFO client.RMProxy: Connecting to ResourceManager at bigdatann.ib/172.17.115.18:8032
16/02/12 18:16:59 INFO yarn.Client: Requesting a new application from cluster with 15 NodeManagers
16/02/12 18:16:59 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (65536 MB per container)
16/02/12 18:16:59 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
16/02/12 18:16:59 INFO yarn.Client: Setting up container launch context for our AM
16/02/12 18:16:59 INFO yarn.Client: Setting up the launch environment for our AM container
16/02/12 18:16:59 INFO yarn.Client: Preparing resources for our AM container
16/02/12 18:17:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/02/12 18:17:00 INFO yarn.Client: Uploading resource file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/spark-assembly-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar -> hdfs://bigdatann.ib:8020/user/rtaylor/.sparkStaging/application_1454115464826_0070/spark-assembly-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar
16/02/12 18:17:21 INFO yarn.Client: Uploading resource file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar -> hdfs://bigdatann.ib:8020/user/rtaylor/.sparkStaging/application_1454115464826_0070/spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar
16/02/12 18:17:23 INFO yarn.Client: Uploading resource file:/tmp/spark-141bf8a4-2f4b-49d3-b041-61070107e4de/__spark_conf__8357851336386157291.zip -> hdfs://bigdatann.ib:8020/user/rtaylor/.sparkStaging/application_1454115464826_0070/__spark_conf__8357851336386157291.zip
16/02/12 18:17:23 INFO spark.SecurityManager: Changing view acls to: rtaylor
16/02/12 18:17:23 INFO spark.SecurityManager: Changing modify acls to: rtaylor
16/02/12 18:17:23 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(rtaylor); users with modify permissions: Set(rtaylor)
16/02/12 18:17:24 INFO yarn.Client: Submitting application 70 to ResourceManager
16/02/12 18:17:24 INFO impl.YarnClientImpl: Submitted application application_1454115464826_0070
16/02/12 18:17:25 INFO yarn.Client: Application report for application_1454115464826_0070 (state: ACCEPTED)
16/02/12 18:17:25 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.rtaylor
start time: 1455329844451
final status: UNDEFINED
tracking URL: http://bigdatann.ib:8088/proxy/application_1454115464826_0070/
user: rtaylor
16/02/12 18:17:26 INFO yarn.Client: Application report for application_1454115464826_0070 (state: ACCEPTED)
16/02/12 18:17:27 INFO yarn.Client: Application report for application_1454115464826_0070 (state: ACCEPTED)
16/02/12 18:17:28 INFO yarn.Client: Application report for application_1454115464826_0070 (state: ACCEPTED)
16/02/12 18:17:29 INFO yarn.Client: Application report for application_1454115464826_0070 (state: ACCEPTED)
16/02/12 18:17:30 INFO yarn.Client: Application report for application_1454115464826_0070 (state: ACCEPTED)
16/02/12 18:17:31 INFO yarn.Client: Application report for application_1454115464826_0070 (state: ACCEPTED)
16/02/12 18:17:32 INFO yarn.Client: Application report for application_1454115464826_0070 (state: ACCEPTED)
16/02/12 18:17:33 INFO yarn.Client: Application report for application_1454115464826_0070 (state: RUNNING)
16/02/12 18:17:33 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 172.17.115.8
ApplicationMaster RPC port: 0
queue: root.rtaylor
start time: 1455329844451
final status: UNDEFINED
tracking URL: http://bigdatann.ib:8088/proxy/application_1454115464826_0070/
user: rtaylor
16/02/12 18:17:34 INFO yarn.Client: Application report for application_1454115464826_0070 (state: RUNNING)
16/02/12 18:17:35 INFO yarn.Client: Application report for application_1454115464826_0070 (state: RUNNING)
16/02/12 18:17:36 INFO yarn.Client: Application report for application_1454115464826_0070 (state: RUNNING)
16/02/12 18:17:37 INFO yarn.Client: Application report for application_1454115464826_0070 (state: RUNNING)
16/02/12 18:17:38 INFO yarn.Client: Application report for application_1454115464826_0070 (state: RUNNING)
16/02/12 18:17:39 INFO yarn.Client: Application report for application_1454115464826_0070 (state: RUNNING)
16/02/12 18:17:40 INFO yarn.Client: Application report for application_1454115464826_0070 (state: RUNNING)
16/02/12 18:17:41 INFO yarn.Client: Application report for application_1454115464826_0070 (state: RUNNING)
16/02/12 18:17:42 INFO yarn.Client: Application report for application_1454115464826_0070 (state: RUNNING)
16/02/12 18:17:43 INFO yarn.Client: Application report for application_1454115464826_0070 (state: FINISHED)
16/02/12 18:17:43 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 172.17.115.8
ApplicationMaster RPC port: 0
queue: root.rtaylor
start time: 1455329844451
final status: SUCCEEDED
tracking URL: http://bigdatann.ib:8088/proxy/application_1454115464826_0070/A
user: rtaylor
16/02/12 18:17:43 INFO util.ShutdownHookManager: Shutdown hook called
16/02/12 18:17:43 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-141bf8a4-2f4b-49d3-b041-61070107e4de
[rtaylor@bigdatann SecondarySort]$
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Ronald C. Taylor, Ph.D.
Computational Biology & Bioinformatics Group
Pacific Northwest National Laboratory (U.S. Dept of Energy/Battelle)
Richland, WA 99352
phone: (509) 372-6568, email: ronald.taylor@pnnl.gov
web page: http://www.pnnl.gov/science/staff/staff_info.asp?staff_num=7048
Re: new to Spark - trying to get a basic example to run - could use
some help
Posted by Ted Yu <yu...@gmail.com>.
Maybe a comment should be added to SparkPi.scala, telling user to look for
the value in stdout log ?
Cheers
On Sat, Feb 13, 2016 at 3:12 AM, Chandeep Singh <ch...@gmail.com>
wrote:
> Try looking at stdout logs. I ran the exactly same job as you and did not
> see anything on the console as well but found it in stdout.
>
> [csingh@<> ~]$ spark-submit --class org.apache.spark.examples.SparkPi
> --master yarn --deploy-mode cluster --name RT_SparkPi
> /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar
> 10
>
> Log Type: stdout
>
> Log Upload Time: Sat Feb 13 11:00:08 +0000 2016
>
> Log Length: 23
>
> Pi is roughly 3.140224
>
>
> Hope that helps!
>
>
> On Sat, Feb 13, 2016 at 3:14 AM, Taylor, Ronald C <Ro...@pnnl.gov>
> wrote:
>
>> Hello folks,
>>
>> This is my first msg to the list. New to Spark, and trying to run the
>> SparkPi example shown in the Cloudera documentation. We have Cloudera
>> 5.5.1 running on a small cluster at our lab, with Spark 1.5.
>>
>> My trial invocation is given below. The output that I get **says** that
>> I “SUCCEEDED” at the end. But – I don’t get any screen output on the value
>> of pi. I also tried a SecondarySort Spark program that I compiled and
>> jarred from Dr. Parsian’s Data Algorithms book. That program failed. So –
>> I am focusing on getting SparkPi to work properly, to get started. Can
>> somebody look at the screen output that I cut-and-pasted below and infer
>> what I might be doing wrong?
>>
>> Am I forgetting to set one or more environment variables? Or not setting
>> such properly?
>>
>> Here is the CLASSPATH value that I set:
>>
>>
>> CLASSPATH=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/*:/people/rtaylor/SparkWork/DataAlgUtils
>>
>> Here is the settings of other environment variables:
>>
>> HADOOP_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11
>> SPARK_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11
>>
>> HADOOP_CLASSPATH='/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/*:$JAVA_HOME/lib/tools.jar'
>>
>> SPARK_CLASSPATH='/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/*:$JAVA_HOME/lib/tools.jar:':'/people/rtaylor/SparkWork/DataAlgUtils'
>>
>> I am not sure that those env vars are properly set (or if even all of
>> them are needed). But that’s what I’m currently using.
>>
>> As I said, the invocation below appears to terminate with final status
>> set to “SUCCEEDED”. But – there is no screen output on the value of pi,
>> which I understood would be shown. So – something appears to be going
>> wrong. I went to the tracking URL given at the end, but could not access it.
>>
>> I would very much appreciate some guidance!
>>
>>
>> - Ron Taylor
>>
>>
>> %%%%%
>>
>> INVOCATION:
>>
>> [rtaylor@bigdatann]$ spark-submit --class
>> org.apache.spark.examples.SparkPi --master yarn --deploy-mode
>> cluster --name RT_SparkPi
>> /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar
>> 10
>>
>> SLF4J: Class path contains multiple SLF4J bindings.
>> SLF4J: Found binding in
>> [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: Found binding in
>> [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/slf4j-simple-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: Found binding in
>> [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/livy-assembly.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: Found binding in
>> [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/avro-tools-1.7.6-cdh5.5.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: Found binding in
>> [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/pig-0.12.0-cdh5.5.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>> explanation.
>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>> 16/02/12 18:16:59 INFO client.RMProxy: Connecting to ResourceManager at
>> bigdatann.ib/172.17.115.18:8032
>> 16/02/12 18:16:59 INFO yarn.Client: Requesting a new application from
>> cluster with 15 NodeManagers
>> 16/02/12 18:16:59 INFO yarn.Client: Verifying our application has not
>> requested more than the maximum memory capability of the cluster (65536 MB
>> per container)
>> 16/02/12 18:16:59 INFO yarn.Client: Will allocate AM container, with 1408
>> MB memory including 384 MB overhead
>> 16/02/12 18:16:59 INFO yarn.Client: Setting up container launch context
>> for our AM
>> 16/02/12 18:16:59 INFO yarn.Client: Setting up the launch environment for
>> our AM container
>> 16/02/12 18:16:59 INFO yarn.Client: Preparing resources for our AM
>> container
>> 16/02/12 18:17:00 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop library for your platform... using builtin-java classes where
>> applicable
>> 16/02/12 18:17:00 INFO yarn.Client: Uploading resource
>> file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/spark-assembly-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar
>> ->
>> hdfs://bigdatann.ib:8020/user/rtaylor/.sparkStaging/application_1454115464826_0070/spark-assembly-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar
>> 16/02/12 18:17:21 INFO yarn.Client: Uploading resource
>> file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar
>> ->
>> hdfs://bigdatann.ib:8020/user/rtaylor/.sparkStaging/application_1454115464826_0070/spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar
>> 16/02/12 18:17:23 INFO yarn.Client: Uploading resource
>> file:/tmp/spark-141bf8a4-2f4b-49d3-b041-61070107e4de/__spark_conf__8357851336386157291.zip
>> ->
>> hdfs://bigdatann.ib:8020/user/rtaylor/.sparkStaging/application_1454115464826_0070/__spark_conf__8357851336386157291.zip
>> 16/02/12 18:17:23 INFO spark.SecurityManager: Changing view acls to:
>> rtaylor
>> 16/02/12 18:17:23 INFO spark.SecurityManager: Changing modify acls to:
>> rtaylor
>> 16/02/12 18:17:23 INFO spark.SecurityManager: SecurityManager:
>> authentication disabled; ui acls disabled; users with view permissions:
>> Set(rtaylor); users with modify permissions: Set(rtaylor)
>> 16/02/12 18:17:24 INFO yarn.Client: Submitting application 70 to
>> ResourceManager
>> 16/02/12 18:17:24 INFO impl.YarnClientImpl: Submitted application
>> application_1454115464826_0070
>> 16/02/12 18:17:25 INFO yarn.Client: Application report for
>> application_1454115464826_0070 (state: ACCEPTED)
>> 16/02/12 18:17:25 INFO yarn.Client:
>> client token: N/A
>> diagnostics: N/A
>> ApplicationMaster host: N/A
>> ApplicationMaster RPC port: -1
>> queue: root.rtaylor
>> start time: 1455329844451
>> final status: UNDEFINED
>> tracking URL:
>> http://bigdatann.ib:8088/proxy/application_1454115464826_0070/
>> user: rtaylor
>> 16/02/12 18:17:26 INFO yarn.Client: Application report for
>> application_1454115464826_0070 (state: ACCEPTED)
>> 16/02/12 18:17:27 INFO yarn.Client: Application report for
>> application_1454115464826_0070 (state: ACCEPTED)
>> 16/02/12 18:17:28 INFO yarn.Client: Application report for
>> application_1454115464826_0070 (state: ACCEPTED)
>> 16/02/12 18:17:29 INFO yarn.Client: Application report for
>> application_1454115464826_0070 (state: ACCEPTED)
>> 16/02/12 18:17:30 INFO yarn.Client: Application report for
>> application_1454115464826_0070 (state: ACCEPTED)
>> 16/02/12 18:17:31 INFO yarn.Client: Application report for
>> application_1454115464826_0070 (state: ACCEPTED)
>> 16/02/12 18:17:32 INFO yarn.Client: Application report for
>> application_1454115464826_0070 (state: ACCEPTED)
>> 16/02/12 18:17:33 INFO yarn.Client: Application report for
>> application_1454115464826_0070 (state: RUNNING)
>> 16/02/12 18:17:33 INFO yarn.Client:
>> client token: N/A
>> diagnostics: N/A
>> ApplicationMaster host: 172.17.115.8
>> ApplicationMaster RPC port: 0
>> queue: root.rtaylor
>> start time: 1455329844451
>> final status: UNDEFINED
>> tracking URL:
>> http://bigdatann.ib:8088/proxy/application_1454115464826_0070/
>> user: rtaylor
>> 16/02/12 18:17:34 INFO yarn.Client: Application report for
>> application_1454115464826_0070 (state: RUNNING)
>> 16/02/12 18:17:35 INFO yarn.Client: Application report for
>> application_1454115464826_0070 (state: RUNNING)
>> 16/02/12 18:17:36 INFO yarn.Client: Application report for
>> application_1454115464826_0070 (state: RUNNING)
>> 16/02/12 18:17:37 INFO yarn.Client: Application report for
>> application_1454115464826_0070 (state: RUNNING)
>> 16/02/12 18:17:38 INFO yarn.Client: Application report for
>> application_1454115464826_0070 (state: RUNNING)
>> 16/02/12 18:17:39 INFO yarn.Client: Application report for
>> application_1454115464826_0070 (state: RUNNING)
>> 16/02/12 18:17:40 INFO yarn.Client: Application report for
>> application_1454115464826_0070 (state: RUNNING)
>> 16/02/12 18:17:41 INFO yarn.Client: Application report for
>> application_1454115464826_0070 (state: RUNNING)
>> 16/02/12 18:17:42 INFO yarn.Client: Application report for
>> application_1454115464826_0070 (state: RUNNING)
>> 16/02/12 18:17:43 INFO yarn.Client: Application report for
>> application_1454115464826_0070 (state: FINISHED)
>> 16/02/12 18:17:43 INFO yarn.Client:
>> client token: N/A
>> diagnostics: N/A
>> ApplicationMaster host: 172.17.115.8
>> ApplicationMaster RPC port: 0
>> queue: root.rtaylor
>> start time: 1455329844451
>> final status: SUCCEEDED
>> tracking URL:
>> http://bigdatann.ib:8088/proxy/application_1454115464826_0070/A
>> user: rtaylor
>> 16/02/12 18:17:43 INFO util.ShutdownHookManager: Shutdown hook called
>> 16/02/12 18:17:43 INFO util.ShutdownHookManager: Deleting directory
>> /tmp/spark-141bf8a4-2f4b-49d3-b041-61070107e4de
>> [rtaylor@bigdatann SecondarySort]$
>>
>> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
>>
>>
>> Ronald C. Taylor, Ph.D.
>> Computational Biology & Bioinformatics Group
>> Pacific Northwest National Laboratory (U.S. Dept of Energy/Battelle)
>> Richland, WA 99352
>> phone: (509) 372-6568, email: ronald.taylor@pnnl.gov
>> web page:
>> http://www.pnnl.gov/science/staff/staff_info.asp?staff_num=7048
>>
>>
>>
>>
>
>
Re: new to Spark - trying to get a basic example to run - could use
some help
Posted by Chandeep Singh <ch...@gmail.com>.
Try looking at stdout logs. I ran the exactly same job as you and did not
see anything on the console as well but found it in stdout.
[csingh@<> ~]$ spark-submit --class org.apache.spark.examples.SparkPi
--master yarn --deploy-mode cluster --name RT_SparkPi
/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar
10
Log Type: stdout
Log Upload Time: Sat Feb 13 11:00:08 +0000 2016
Log Length: 23
Pi is roughly 3.140224
Hope that helps!
On Sat, Feb 13, 2016 at 3:14 AM, Taylor, Ronald C <Ro...@pnnl.gov>
wrote:
> Hello folks,
>
> This is my first msg to the list. New to Spark, and trying to run the
> SparkPi example shown in the Cloudera documentation. We have Cloudera
> 5.5.1 running on a small cluster at our lab, with Spark 1.5.
>
> My trial invocation is given below. The output that I get **says** that I
> “SUCCEEDED” at the end. But – I don’t get any screen output on the value of
> pi. I also tried a SecondarySort Spark program that I compiled and jarred
> from Dr. Parsian’s Data Algorithms book. That program failed. So – I am
> focusing on getting SparkPi to work properly, to get started. Can somebody
> look at the screen output that I cut-and-pasted below and infer what I
> might be doing wrong?
>
> Am I forgetting to set one or more environment variables? Or not setting
> such properly?
>
> Here is the CLASSPATH value that I set:
>
>
> CLASSPATH=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/*:/people/rtaylor/SparkWork/DataAlgUtils
>
> Here is the settings of other environment variables:
>
> HADOOP_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11
> SPARK_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11
>
> HADOOP_CLASSPATH='/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/*:$JAVA_HOME/lib/tools.jar'
>
> SPARK_CLASSPATH='/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/*:$JAVA_HOME/lib/tools.jar:':'/people/rtaylor/SparkWork/DataAlgUtils'
>
> I am not sure that those env vars are properly set (or if even all of them
> are needed). But that’s what I’m currently using.
>
> As I said, the invocation below appears to terminate with final status set
> to “SUCCEEDED”. But – there is no screen output on the value of pi, which I
> understood would be shown. So – something appears to be going wrong. I went
> to the tracking URL given at the end, but could not access it.
>
> I would very much appreciate some guidance!
>
>
> - Ron Taylor
>
>
> %%%%%
>
> INVOCATION:
>
> [rtaylor@bigdatann]$ spark-submit --class
> org.apache.spark.examples.SparkPi --master yarn --deploy-mode
> cluster --name RT_SparkPi
> /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar
> 10
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/slf4j-simple-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/livy-assembly.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/avro-tools-1.7.6-cdh5.5.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/pig-0.12.0-cdh5.5.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 16/02/12 18:16:59 INFO client.RMProxy: Connecting to ResourceManager at
> bigdatann.ib/172.17.115.18:8032
> 16/02/12 18:16:59 INFO yarn.Client: Requesting a new application from
> cluster with 15 NodeManagers
> 16/02/12 18:16:59 INFO yarn.Client: Verifying our application has not
> requested more than the maximum memory capability of the cluster (65536 MB
> per container)
> 16/02/12 18:16:59 INFO yarn.Client: Will allocate AM container, with 1408
> MB memory including 384 MB overhead
> 16/02/12 18:16:59 INFO yarn.Client: Setting up container launch context
> for our AM
> 16/02/12 18:16:59 INFO yarn.Client: Setting up the launch environment for
> our AM container
> 16/02/12 18:16:59 INFO yarn.Client: Preparing resources for our AM
> container
> 16/02/12 18:17:00 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 16/02/12 18:17:00 INFO yarn.Client: Uploading resource
> file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/spark-assembly-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar
> ->
> hdfs://bigdatann.ib:8020/user/rtaylor/.sparkStaging/application_1454115464826_0070/spark-assembly-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar
> 16/02/12 18:17:21 INFO yarn.Client: Uploading resource
> file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar
> ->
> hdfs://bigdatann.ib:8020/user/rtaylor/.sparkStaging/application_1454115464826_0070/spark-examples-1.5.0-cdh5.5.1-hadoop2.6.0-cdh5.5.1.jar
> 16/02/12 18:17:23 INFO yarn.Client: Uploading resource
> file:/tmp/spark-141bf8a4-2f4b-49d3-b041-61070107e4de/__spark_conf__8357851336386157291.zip
> ->
> hdfs://bigdatann.ib:8020/user/rtaylor/.sparkStaging/application_1454115464826_0070/__spark_conf__8357851336386157291.zip
> 16/02/12 18:17:23 INFO spark.SecurityManager: Changing view acls to:
> rtaylor
> 16/02/12 18:17:23 INFO spark.SecurityManager: Changing modify acls to:
> rtaylor
> 16/02/12 18:17:23 INFO spark.SecurityManager: SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(rtaylor); users with modify permissions: Set(rtaylor)
> 16/02/12 18:17:24 INFO yarn.Client: Submitting application 70 to
> ResourceManager
> 16/02/12 18:17:24 INFO impl.YarnClientImpl: Submitted application
> application_1454115464826_0070
> 16/02/12 18:17:25 INFO yarn.Client: Application report for
> application_1454115464826_0070 (state: ACCEPTED)
> 16/02/12 18:17:25 INFO yarn.Client:
> client token: N/A
> diagnostics: N/A
> ApplicationMaster host: N/A
> ApplicationMaster RPC port: -1
> queue: root.rtaylor
> start time: 1455329844451
> final status: UNDEFINED
> tracking URL:
> http://bigdatann.ib:8088/proxy/application_1454115464826_0070/
> user: rtaylor
> 16/02/12 18:17:26 INFO yarn.Client: Application report for
> application_1454115464826_0070 (state: ACCEPTED)
> 16/02/12 18:17:27 INFO yarn.Client: Application report for
> application_1454115464826_0070 (state: ACCEPTED)
> 16/02/12 18:17:28 INFO yarn.Client: Application report for
> application_1454115464826_0070 (state: ACCEPTED)
> 16/02/12 18:17:29 INFO yarn.Client: Application report for
> application_1454115464826_0070 (state: ACCEPTED)
> 16/02/12 18:17:30 INFO yarn.Client: Application report for
> application_1454115464826_0070 (state: ACCEPTED)
> 16/02/12 18:17:31 INFO yarn.Client: Application report for
> application_1454115464826_0070 (state: ACCEPTED)
> 16/02/12 18:17:32 INFO yarn.Client: Application report for
> application_1454115464826_0070 (state: ACCEPTED)
> 16/02/12 18:17:33 INFO yarn.Client: Application report for
> application_1454115464826_0070 (state: RUNNING)
> 16/02/12 18:17:33 INFO yarn.Client:
> client token: N/A
> diagnostics: N/A
> ApplicationMaster host: 172.17.115.8
> ApplicationMaster RPC port: 0
> queue: root.rtaylor
> start time: 1455329844451
> final status: UNDEFINED
> tracking URL:
> http://bigdatann.ib:8088/proxy/application_1454115464826_0070/
> user: rtaylor
> 16/02/12 18:17:34 INFO yarn.Client: Application report for
> application_1454115464826_0070 (state: RUNNING)
> 16/02/12 18:17:35 INFO yarn.Client: Application report for
> application_1454115464826_0070 (state: RUNNING)
> 16/02/12 18:17:36 INFO yarn.Client: Application report for
> application_1454115464826_0070 (state: RUNNING)
> 16/02/12 18:17:37 INFO yarn.Client: Application report for
> application_1454115464826_0070 (state: RUNNING)
> 16/02/12 18:17:38 INFO yarn.Client: Application report for
> application_1454115464826_0070 (state: RUNNING)
> 16/02/12 18:17:39 INFO yarn.Client: Application report for
> application_1454115464826_0070 (state: RUNNING)
> 16/02/12 18:17:40 INFO yarn.Client: Application report for
> application_1454115464826_0070 (state: RUNNING)
> 16/02/12 18:17:41 INFO yarn.Client: Application report for
> application_1454115464826_0070 (state: RUNNING)
> 16/02/12 18:17:42 INFO yarn.Client: Application report for
> application_1454115464826_0070 (state: RUNNING)
> 16/02/12 18:17:43 INFO yarn.Client: Application report for
> application_1454115464826_0070 (state: FINISHED)
> 16/02/12 18:17:43 INFO yarn.Client:
> client token: N/A
> diagnostics: N/A
> ApplicationMaster host: 172.17.115.8
> ApplicationMaster RPC port: 0
> queue: root.rtaylor
> start time: 1455329844451
> final status: SUCCEEDED
> tracking URL:
> http://bigdatann.ib:8088/proxy/application_1454115464826_0070/A
> user: rtaylor
> 16/02/12 18:17:43 INFO util.ShutdownHookManager: Shutdown hook called
> 16/02/12 18:17:43 INFO util.ShutdownHookManager: Deleting directory
> /tmp/spark-141bf8a4-2f4b-49d3-b041-61070107e4de
> [rtaylor@bigdatann SecondarySort]$
>
> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
>
>
> Ronald C. Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
> Pacific Northwest National Laboratory (U.S. Dept of Energy/Battelle)
> Richland, WA 99352
> phone: (509) 372-6568, email: ronald.taylor@pnnl.gov
> web page: http://www.pnnl.gov/science/staff/staff_info.asp?staff_num=7048
>
>
>
>