[jira] [Comment Edited] (KYLIN-1082) Hive dependencies should be add to tmpjars

["Zhong Yanghong (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KYLIN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110855#comment-15110855 ] 

Zhong Yanghong edited comment on KYLIN-1082 at 1/21/16 4:25 PM:
----------------------------------------------------------------

Thank you for you guys. Finally my initial version is done. I mainly changed the file engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java.
The additional function is to filter the system level property called "kylin.hive.dependency" to get interested hive dependency jar names which will later be added to the hadoop's special property called "tmpjars" whose related jars will be uploaded to each datanode running mapreduce work. 
To set the value for "kylin.hive.dependency", for each platform there is a way. For development and testing machines which don't install hive and run the "DebugTomcat.java" to start KYLIN, just add the System.setProperty("kylin.hive.dependency","XXXX") in "DebugTomcat.java". While for the sandbox which have installed hive and run shell script kylin.sh to start KYLIN, the shell script find-hive-dependency.sh run in kylin.sh will automatically set the property.
To add additional helpful jars, there is another way. In the file "kylin.properties", we can set a property called "kylin.job.mr.lib.dir". Then AbstractHadoopJob.java will parse out all of the jars and files under this self-defined directory including the subdirectory and add them to "tmpjars".



was (Author: yaho):
Thank you for you guys. Finally my initial version is done. I mainly changed the file engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java.
The additional function is to filter the system level property called "kylin.hive.dependency" to get interested hive dependency jar names which will later be added to the hadoop's special property called "tmpjars" whose related jars will be uploaded to each datanode running mapreduce work. 
To set the value for "kylin.hive.dependency", for each platform there is a way. For development and testing machines which don't install hive and run the "DebugTomcat.java" to start KYLIN, just add the System.setProperty("kylin.hive.dependency","XXXX"). While for the sandbox which have install hive and run kylin.sh to start KYLIN, the shell script find-hive-dependency.sh run in kylin.sh will automatically set the property.
To add additional helpful jars, there is another way. In the file "kylin.properties", we can set a property called "kylin.job.mr.lib.dir". Then AbstractHadoopJob.java will parse out all of the jars and files under this self-defined directory including the subdirectory and add them to "tmpjars".


> Hive dependencies should be add to tmpjars
> ------------------------------------------
>
>                 Key: KYLIN-1082
>                 URL: https://issues.apache.org/jira/browse/KYLIN-1082
>             Project: Kylin
>          Issue Type: Bug
>            Reporter: liyang
>            Assignee: Zhong Yanghong
>              Labels: newbie
>
> Currently kylin assume all data nodes have hive deployment at exact same FS location. However, a better position is to think hive as a client side app. Then we need to ship hive jar with MR job every time.
> This make deploy kylin a lot easier in cluster that does not have hive on all data nodes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)