[jira] [Resolved] (KYLIN-2421) Add spark engine to Integration Test

["Shaofeng SHI (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KYLIN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shaofeng SHI resolved KYLIN-2421.
---------------------------------
    Resolution: Fixed

> Add spark engine to Integration Test
> ------------------------------------
>
>                 Key: KYLIN-2421
>                 URL: https://issues.apache.org/jira/browse/KYLIN-2421
>             Project: Kylin
>          Issue Type: Sub-task
>          Components: Job Engine
>    Affects Versions: v2.0.0
>            Reporter: Shaofeng SHI
>            Assignee: Shaofeng SHI
>             Fix For: v2.0.0
>
>
> Let Jenkins to test Spark cubing daily. I have changed the engine type for "ci_inner_join_cube" from "2" (MRV2) to "4" (spark).
> After this change, you need do a couple changes manually in your HDP 2.4 sandbox to finish the Integration Test:
> 1. manually install spark-1.6.3-bin-hadoop2.6 in a local folder like /usr/local/spark
> cd /usr/local
> tar -zxvf spark-1.6.3-bin-hadoop2.6.tgz
> ln -s spark-1.6.3-bin-hadoop2.6 spark
> 2. manually upload the spark-assembly jar to HDFS as /kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar (avoid repeatedly uploading the jar to HDFS):
> hadoop fs -mkdir /kylin/spark/
> hadoop fs -put /usr/local/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar /kylin/spark/
> 3. manually create local temp folder for hbase client (if it doesn't exist):
> mkdir -p /hadoop/hbase/local/jars
> chmod 777 /hadoop/hbase/local/jars
> 4. Specify SPARK_HOME property when run mvn:
> mvn verify -fae -Dhdp.version=2.4.0.0-169 -DSPARK_HOME=/usr/local/spark -P sandbox



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)