[jira] [Updated] (KYLIN-957) Support HBase in a separate cluster

["sunyerui (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KYLIN-957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

sunyerui updated KYLIN-957:
---------------------------
    Attachment: KYLIN-957-0.7-staging.patch

Here is our patch to support HBase deployed on another hadoop cluster.

Some explaining for this patch:
1. Add config 'kylin.hadoop.cluster.fs' and 'kylin.hbase.cluster.fs' in default kylin.properties. Leave empty if using one cluster and local default config; Fill it with filesystem qualified if using another cluster;
2. set fs.defaultFs as hadoop cluster fs for hive and mapreduce jobs, make sure intermediate hive table and cuboid files written on hadoop cluster;
3. make paths fully-qualified which used in jobs involves two clusters, for example, RangeKeyDistributionJob, and CubeHFileJob, make sure rowkey_stats files and hfiles written on hbase cluster directly;
4. set fs.defaultFs as hbase cluster fs in HadoopUtil.newHBaseConfiguration, and add HadoopUtils.getCurrentHBaseConfiguration, used by other hbase-related jobs;


This patch has been tested on our testing environment, and Cube Building&Merging works well. 

But it did NOT passed all UT by 'mvn clean test', failed on Query module:
{quote}
CombinationTest>KylinQueryTest.testCommonQuery:170->KylinTestBase.execAndCompQuery:368 row count (table=query result of query35) expected:<176\[8\]> but was:<176\[7\]>
  CombinationTest>KylinQueryTest.testCommonQuery:170->KylinTestBase.execAndCompQuery:368 row count (table=query result of query35) expected:<149\[4\]> but was:<149\[3\]>
  CombinationTest>KylinQueryTest.testCommonQuery:170->KylinTestBase.execAndCompQuery:368 row count (table=query result of query35) expected:<176\[8\]> but was:<176\[7\]>
  CombinationTest>KylinQueryTest.testCommonQuery:170->KylinTestBase.execAndCompQuery:368 row count (table=query result of query35) expected:<149\[4\]> but was:<149\[3\]>
  CombinationTest>KylinQueryTest.testCommonQuery:170->KylinTestBase.execAndCompQuery:368 row count (table=query result of query35) expected:<176\[8\]> but was:<176\[7\]>
  CombinationTest>KylinQueryTest.testCommonQuery:170->KylinTestBase.execAndCompQuery:368 row count (table=query result of query35) expected:<149\[4\]> but was:<149\[3\]>
{quote} 
I'm not sure these failed tests were caused by this patch,  I'll continue to work on this issue.
And I'm looking forward to some helps from community, any help will be very appreciated.


> Support HBase in a separate cluster
> -----------------------------------
>
>                 Key: KYLIN-957
>                 URL: https://issues.apache.org/jira/browse/KYLIN-957
>             Project: Kylin
>          Issue Type: Improvement
>          Components: General
>    Affects Versions: v0.7.2
>            Reporter: Dayue Gao
>              Labels: features
>         Attachments: KYLIN-957-0.7-staging.patch
>
>
> Currently Kylin assumes that HBase is deployed in the same cluster where Hive tables resides, which is not necessarily the case.
> We should support Kylin to write cubes to HBase in another cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)