You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@kylin.apache.org by Wei Li <lw...@qq.com> on 2017/11/24 04:13:59 UTC

回复：RE: Apache Kylin - Hbase Cube Name Space

Hi Manoj,
I also need hbase namespace feature, and I build the package successfully.




------------------ 原始邮件 ------------------
发件人: "Kumar, Manoj H";<ma...@jpmorgan.com>;
发送时间: 2017年11月24日(星期五) 中午11:56
收件人: "user@kylin.apache.org"<us...@kylin.apache.org>;

主题: RE: Apache Kylin - Hbase Cube Name Space



  
Can someone upload Beta Jar file to Github so that other folks can use it? I am sure many folks have done applied the patch at their locally but it will be  good to use the Updated Jar file instead of building it again & again. Can you pls. someone help on this? I need this issue to be fixed asap.
 
 
 
Appreciate your faster response in this regard.
 
 
  
Regards,
 
Manoj
 
 
 
   
From: Kumar, Manoj H 
 Sent: Thursday, November 23, 2017 8:08 PM
 To: user@kylin.apache.org
 Subject: RE: Apache Kylin - Hbase Cube Name Space
 
 
 
 
 
How to do the patch.. I cant built it locally as This project is not added to JPMC Maven Repo yet. Can you send the binary & I can do the update?
 
 
 
Regards,
 
Manoj
 
 
 
From: ShaoFeng Shi [mailto:shaofengshi@apache.org] 
 Sent: Thursday, November 23, 2017 8:03 PM
 To: user
 Subject: Re: Apache Kylin - Hbase Cube Name Space
 
 
  
Hi Manoj, 
  
 
 
  
This feature is not released yet, see: https://issues.apache.org/jira/browse/KYLIN-2846
 
  
But the patch is available; you can build a new Kylin binary package by your own with the patches.
 
 
  
 
  
2017-11-23 21:49 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
   
Its pointing to Default name – How we can change the Habse Namespace so that we can build the code  successfully. Pls. advise. It has failed at Create HT Table step.
 
 
 
org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions (user=a_rcmo_nd@NAEAST.AD.JPMORGANCHASE.COM,  scope=default, params=[namespace=default,table=default:KYLIN_S1W12SO8AC,family=F1],action=CREATE)
 
 
 
Regards,
 
Manoj
 
 
 
From: ShaoFeng Shi [mailto:shaofengshi@apache.org] 
 Sent: Wednesday, November 22, 2017 8:53 AM
 To: user
 Subject: Re: How to Update the Cube Structure
 
 
  
The intermediate hive table will be cleaned in the last step "Hive  Cleanup" automatically.
  
 
 
  
If your job is failed in the between, then the last step won't be executed by Kylin. In this case,  you can manually clean them with the the  storage clean up cli.
 
 
  
 
  
2017-11-22 11:13 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
   
Np Mind, I got confused with Cube’s Partition column. Once Cube is build out, Can we clean Hive Intermediate  flat table? Also, How can we know which flat belongs particular cube? How to schedule the cube build process – taking incremental data from hive source table
 
 
 
Regards,
 
Manoj
 
 
 
From: ShaoFeng Shi [mailto:shaofengshi@apache.org] 
 Sent: Wednesday, November 22, 2017 6:52 AM
   

 To: user
 Subject: Re: How to Update the Cube Structure
 
 
   
 
  
Hi Manoj,
  
 
 
  
The intermediate flat table is not partitioned, always; it has all files in one folder; I'm wondering what's the purpose to make it partitioned?
 
 
  
 
  
2017-11-22 1:12 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
   
Thanks… one quick question – How to make intermediate flat table partitioned one?? Do we have any  parameter in kylin.properties?
 
 
 
Regards,
 
Manoj
 
 
 
From: ShaoFeng Shi [mailto:shaofengshi@apache.org] 
 Sent: Tuesday, November 21, 2017 8:02 PM
   

 To: user
 Subject: Re: How to Update the Cube Structure
 
 
   
 
  
This document can help:
  
 
 
  
https://kylin.apache.org/docs21/howto/howto_optimize_build.html
 
  
 
 
 
  
 
  
2017-11-21 21:36 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
   
Thanks for the inputs.. It make sense..
 
 
 
As now I am running on Dev Cluster & need to know optimize parameter for Kylin so that Cube building  process runs fast. Can you pls. tell what are properties I need to change in kylin.properties file – To optimize the cube build process.. I have 180 millions of data on fact table.
 
 
 
Regards,
 
Manoj
 
 
 
From: ShaoFeng Shi [mailto:shaofengshi@apache.org] 
 Sent: Tuesday, November 21, 2017 3:23 PM
 To: user
 Subject: Re: How to Update the Cube Structure
   
 
  
To be safe, you can clone the existing to a new cube, make changes on the new cube, and then rebuild it.
 
  
 
  
2017-11-21 14:53 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
   
In case of Data Source table has DDL changes – either new Column added or Existing Column data type  changed. How Can we do Cube data model update & reflecting new column data to Cube? Pls. advise.
 
 
 
Regards,
 
Manoj
 
 
 
From: ShaoFeng Shi [mailto:shaofengshi@apache.org] 
 Sent: Monday, November 20, 2017 7:03 PM
 To: user
 Subject: Re: Apache Kyine - kylin.env.hdfs-working-dir
 
 
  
It seems that the script still try to create "/kylin" on HDFS, not in the configured path.
  
 
 
  
Please double check and make sure this line is not commented:
 
  
 
 
  
kylin.env.hdfs-working-dir=/tenants/rft/rcmo/kylin
 
 
  
 
  
2017-11-20 14:00 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
   
I am installing kyline 2.10 on Dev Cluster(Cloudera) . Although I have given 777 permission on kylin Folder but still giving error. Can you pls. help on this? Whats missing here?
 
 
 
-bash-4.1$ hdfs dfs -mkdir /tenants/rft/rcmo/kylin
 
-bash-4.1$ hdfs dfs -chmod 777 /tenants/rft/rcmo/kylin
 
 
 
Kyline properties.conf - #kylin.env.hdfs-working-dir=/tenants/rft/rcmo/kylin
 
 
 
-bash-4.1$ ./check-env.sh
 
Retrieving hadoop conf dir...
 
KYLIN_HOME is set to /apps/rft/rcmo/apps/kylin/apache-kylin-2.1.0-bin-cdh57
 
mkdir: Permission denied: user=a_rcmo_nd, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x
 
Failed to create /kylin. Please make sure the user has right to access /kylin
 
 
 
Regards,
 
Manoj
 
 
 
 
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages.  If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
 
 
 

 
 
  
 
 
 
-- 
     
Best regards,
  
 
 
  
Shaofeng Shi 史少锋
 
  
 
 
 
 
 
 
 
 
 
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages.  If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
 
 
 

 
 
  
 
 
 
-- 
     
Best regards,
  
 
 
  
Shaofeng Shi 史少锋
 
  
 
 
 
 
 
 
 
 
 
 
   
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages.  If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
 
 
 
 
 

 
 
  
 
 
 
-- 
     
Best regards,
  
 
 
  
Shaofeng Shi 史少锋
 
  
 
 
 
 
 
 
 
 
 
 
   
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages.  If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
 
 
 
 
 

 
 
  
 
 
 
-- 
     
Best regards,
  
 
 
  
Shaofeng Shi 史少锋
 
  
 
 
 
 
 
 
 
 
 
 
   
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages.  If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
 
 
 
 
 

 
 
  
 
 
 
-- 
     
Best regards,
  
 
 
  
Shaofeng Shi 史少锋
 
  
 
 
 
 
 
 
 
 
 
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages.  If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
 
 
 

 
 
  
 
 
 
-- 
     
Best regards,
  
 
 
  
Shaofeng Shi 史少锋
 
  
 
 
 
 
 
 
 
 
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not  the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
 
 
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.