[jira] [Comment Edited] (BIGTOP-1657) bigtop-deploy puppets to support tez

["Evans Ye (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BIGTOP-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14388813#comment-14388813 ] 

Evans Ye edited comment on BIGTOP-1657 at 5/5/15 3:17 PM:
----------------------------------------------------------

Since tez require libraries to be uploaded to hdfs, I add the upload logic into init-hcfs.json and init-hcfs.groovy.
The patch also added tez conf into hadoop-env.sh, set hive.execution.engine to tez, and set mapreduce framework to yarn-tez when tez is deployed.
This makes the testing of this patch a little bit complicated. Here's the steps test I used:

* {{./gradlew hadoop-yum bigtop-utils-yum}} to build new hadoop and bigtop-utils after patch applied
* update the bigtop-deploy/vm/vagrant-puppet-docker/vagrantconfig.yaml:
{code}
...
components: [hadoop, yarn, hive, tez]
...
enable_local_repo: true
{code}
* cd bigtop-deploy/vm/vagrant-puppet-docekr; ./docker-hadoop -d -c 1 (takes 4mins in my environment)
* vagrant ssh bigtop1
* su - jenkins
* hadoop fs -put /etc/passwd
* hive (get into the hive shell)
* create table t1 (s string) location '/user/jenkins'
* select * from t1 order by s;
* Should see a fancy tez execution table :)




was (Author: evans_ye):
Since tez require libraries to be uploaded to hdfs, I add the upload logic into init-hcfs.json and init-hcfs.groovy.
The patch also added tez conf into hadoop-env.sh, set hive.execution.engine to tez, and set mapreduce framework to yarn-tez when tez is deployed.
This makes the testing of this patch a little bit complicated. Here's the steps test I used:

* {{./gradlew hadoop-yum bigtop-utils-yum}} to build new hadoop and bigtop-utils after patch applied
* update the bigtop-deploy/vm/vagrant-puppet-docker/vagrantconfig.yaml:
{code}
...
components: [hadoop, yarn, hive, tez]
...
enable_local_repo: true
{code}
* cd bigtop-deploy/vm/vagrant-puppet-docekr; ./docker-hadoop -d -c 1 (takes 4mins in my environment)
* vagrant ssh bigtop1
* su - jenkins
* hadoop fs -put /etc/passwd
* hive (get into the hive shell)
* create table t1 (s string) location 'user/jenkins'
* select * from t1 order by s;
* Should see a fancy tez execution table :)



> bigtop-deploy puppets to support tez
> ------------------------------------
>
>                 Key: BIGTOP-1657
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-1657
>             Project: Bigtop
>          Issue Type: Bug
>    Affects Versions: 0.8.0
>            Reporter: Olaf Flebbe
>            Assignee: Evans Ye
>             Fix For: 1.0.0
>
>         Attachments: BIGTOP-1657.patch, BIGTOP-1657.patch
>
>
> If tez configuration is selected: supply yarn-tez to mapred.xml and select hive.execution.engine=tez and others in hive-site.xml configuration



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)