You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@kylin.apache.org by "fengYu (JIRA)" <ji...@apache.org> on 2016/09/21 11:48:20 UTC

[jira] [Commented] (KYLIN-1839) improvement set classpath before submitting mr job

    [ https://issues.apache.org/jira/browse/KYLIN-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15509663#comment-15509663 ] 

fengYu commented on KYLIN-1839:
-------------------------------

I think the patch is useable. I think tmpjars and tmpfiles which used for MR job is process level, So I cache them all.

> improvement set classpath before submitting mr job
> --------------------------------------------------
>
>                 Key: KYLIN-1839
>                 URL: https://issues.apache.org/jira/browse/KYLIN-1839
>             Project: Kylin
>          Issue Type: Improvement
>          Components: Job Engine
>    Affects Versions: v1.5.2
>            Reporter: fengYu
>            Assignee: fengYu
>         Attachments: 0001-KYLIN-1839-support-extend-lib-from-HDFS-and-cache-tm.patch
>
>
> in setClasspath, kylin will alway find hive jars from hive dependency using regex, however, this will not change in one process lifetime, so I cache the location of tmpjars and tmpfiles.
> What is more, support extends user lib setting to hdfs path rather than only support local filesystem which will cause upload jars every time if DistributedCache do not exist.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)