[jira] [Commented] (BIGTOP-1658) puppet recipe updates for spark 1.2

["YoungWoo Kim (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BIGTOP-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313391#comment-14313391 ] 

YoungWoo Kim commented on BIGTOP-1658:
--------------------------------------

>From BIGTOP-1648, Apache Spark on Bigtop updated version to 1.2.1. This is notes regarding installation of the spark-history-server and spark-thriftserver.

0) Install Spark history server
{noformat}
# yum install -y spark-history-server
{noformat}

1) To run spark-history-server,  make sure you've created a event log directory on HDFS:
{noformat}
su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -mkdir -p /var/log/spark/apps'
su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chmod -R 1777 /var/log/spark/apps'
su -s /bin/bash hdfs -c '/usr/bin/hadoop fs -chown spark:spark /var/log/spark/apps'
{noformat}

2) Create '/etc/spark/spark-default.conf':
{noformat}
cd /etc/spark/conf
cp spark-default.conf.template spark-default.conf
{noformat}

Edit the spark-default.conf:
{code}
spark.master                     spark://SPARK-MASTER-HOSTNAME:7077
spark.eventLog.enabled           true
spark.eventLog.dir               hdfs://HDFS-NN-HOSTNAME:8020/var/log/spark/apps/
{code}

3) Run spark examples with yarn:
{noformat}
# service spark-history-server start
export HADOOP_CONF_DIR=/etc/hadoop/conf

spark-submit --class org.apache.spark.examples.SparkPi --deploy-mode client --master yarn /usr/lib/spark/lib/spark-examples_2.10-1.2.0.jar 2
{noformat}

4) Browse the spark history server:
http://HOSTNAME:18082/

----

Installation of spark-thriftserver (Optional) 

if you want to use Spark SQL' JDBC access through hive thrift server, you have to run spark-thriftserver

1) Edit hive-site.xml for Spark SQL
{noformat}
vi /etc/spark/conf/hive-site.xml
{noformat}
And make sure following properties have setted up properly. (in my case, I use MySQL for database)
{code}

<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:mysql://localhost/hive?createDatabaseIfNotExist=true</value>
  <description>JDBC connect string for a JDBC metastore</description>
</property>
 
<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.jdbc.Driver</value>
  <description>Driver class name for a JDBC metastore</description>
</property>
 
<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>username</value>
  <description>username to use against metastore database</description>
</property>
 
<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>mypassword</value>
  <description>password to use against metastore database</description>
</property>

{code}

and then, run spark-thriftserver service(default port is 10000):
{noformat}
# service spark-thriftserver start
{noformat}

Using beeline client, you can access JDBC connection for spark thriftserver:
{noformat}
$SPARK_HOME/bin/beeline -u jdbc:hive2://THRIFT-HOSTNAME:10000
{noformat}

> puppet recipe updates for spark 1.2
> -----------------------------------
>
>                 Key: BIGTOP-1658
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-1658
>             Project: Bigtop
>          Issue Type: Bug
>          Components: spark
>    Affects Versions: 0.9.0
>            Reporter: jay vyas
>             Fix For: 0.9.0
>
>
> In BIGTOP-1648 we upgraded and now there are some updates we need to do for puppet recipes .  This is a critical blocker for release, as we want to have puppet recipes as first-class citizens reflecting the correct deployment of anything in bigtop.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)