[jira] [Comment Edited] (KYLIN-2865) wrong fs when use two cluster

["Shaofeng SHI (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KYLIN-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16166281#comment-16166281 ] 

Shaofeng SHI edited comment on KYLIN-2865 at 9/14/17 1:38 PM:
--------------------------------------------------------------

Write to HBase cluster should be better because the meta store is in HBase; From another perspective, Kylin should be closer to the HBase cluster, putting the large files to HBase cluster's HDFS is also better than putting to computing cluster. So Kylin 2.1 is better than 2.0 on this.

If the deployment is R/W separated, when upgrade to 2.1, need do a migration for the metadata file on hdfs, e.g:
hadoop distcp hdfs://hive-hdfs:8020/kylin/kylin_default_instance/resources/* hdfs://hbase-hdfs:8020/kylin/kylin_default_instance/resources/

For the error you mentioned in the email, that is a bug; I had made the change (only add one line) in a local branch. When it passed the integration testing, I will push it. 


was (Author: shaofengshi):
Write to HBase cluster should be better because the meta store is in HBase; From another perspective, Kylin should be closer to the HBase cluster, putting the large files to HBase cluster's HDFS is also better than putting to computing cluster. So Kylin 2.1 is better than 2.0 on this.

For the error you mentioned in the email, that is a bug; I had made the change (only add one line) in a local branch. When it passed the integration testing, I will push it. 

> wrong fs when use two cluster
> -----------------------------
>
>                 Key: KYLIN-2865
>                 URL: https://issues.apache.org/jira/browse/KYLIN-2865
>             Project: Kylin
>          Issue Type: Bug
>          Components: Storage - HBase
>    Affects Versions: v2.1.0
>         Environment: kylin:2.1.0 
> hadoop:2.7.3 
> hive:1.2.1 
> hbase: 1.2.5 
>            Reporter: yzq
>            Assignee: Shaofeng SHI
>              Labels: patch
>             Fix For: Future
>
>         Attachments: fix_address_conflict.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Hbase is running on  separate cluster with hive and mapreduce, then it will throw the
> exception. See the link for details http://apache-kylin.74782.x6.nabble.com/wrong-fs-when-use-two-cluster-td8985.html
> Except that, I found that kylin-2.0 is not compatible is kylin-2.1 which
> will cause query failed. In function writeLargeCellToHdfs(...) , the
> kylin-2.1 will write content to cluster of mapreduce. In kylin-2.0, the
> destination is hbase's cluster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)