You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by "Kumar, Manoj H" <ma...@jpmorgan.com> on 2017/11/21 06:53:08 UTC
How to Update the Cube Structure
In case of Data Source table has DDL changes – either new Column added or Existing Column data type changed. How Can we do Cube data model update & reflecting new column data to Cube? Pls. advise.
Regards,
Manoj
From: ShaoFeng Shi [mailto:shaofengshi@apache.org]
Sent: Monday, November 20, 2017 7:03 PM
To: user
Subject: Re: Apache Kyine - kylin.env.hdfs-working-dir
It seems that the script still try to create "/kylin" on HDFS, not in the configured path.
Please double check and make sure this line is not commented:
kylin.env.hdfs-working-dir=/tenants/rft/rcmo/kylin
2017-11-20 14:00 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>>:
I am installing kyline 2.10 on Dev Cluster(Cloudera) . Although I have given 777 permission on kylin Folder but still giving error. Can you pls. help on this? Whats missing here?
-bash-4.1$ hdfs dfs -mkdir /tenants/rft/rcmo/kylin
-bash-4.1$ hdfs dfs -chmod 777 /tenants/rft/rcmo/kylin
Kyline properties.conf - #kylin.env.hdfs-working-dir=/tenants/rft/rcmo/kylin
-bash-4.1$ ./check-env.sh
Retrieving hadoop conf dir...
KYLIN_HOME is set to /apps/rft/rcmo/apps/kylin/apache-kylin-2.1.0-bin-cdh57
mkdir: Permission denied: user=a_rcmo_nd, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x
Failed to create /kylin. Please make sure the user has right to access /kylin
Regards,
Manoj
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
--
Best regards,
Shaofeng Shi 史少锋
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
Re: How to Update the Cube Structure
Posted by ShaoFeng Shi <sh...@apache.org>.
The intermediate hive table will be cleaned in the last step "Hive Cleanup"
automatically.
If your job is failed in the between, then the last step won't be executed
by Kylin. In this case, you can manually clean them with the the storage
<https://kylin.apache.org/docs21/howto/howto_cleanup_storage.html> clean up
cli <https://kylin.apache.org/docs21/howto/howto_cleanup_storage.html>.
2017-11-22 11:13 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
> Np Mind, I got confused with Cube’s Partition column. Once Cube is build
> out, Can we clean Hive Intermediate flat table? Also, How can we know which
> flat belongs particular cube? How to schedule the cube build process –
> taking incremental data from hive source table
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
> *Sent:* Wednesday, November 22, 2017 6:52 AM
>
> *To:* user
> *Subject:* Re: How to Update the Cube Structure
>
>
>
> Hi Manoj,
>
>
>
> The intermediate flat table is not partitioned, always; it has all files
> in one folder; I'm wondering what's the purpose to make it partitioned?
>
>
>
> 2017-11-22 1:12 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
>
> Thanks… one quick question – How to make intermediate flat table
> partitioned one?? Do we have any parameter in kylin.properties?
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
> *Sent:* Tuesday, November 21, 2017 8:02 PM
>
>
> *To:* user
> *Subject:* Re: How to Update the Cube Structure
>
>
>
> This document can help:
>
>
>
> https://kylin.apache.org/docs21/howto/howto_optimize_build.html
>
>
>
>
>
> 2017-11-21 21:36 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
>
> Thanks for the inputs.. It make sense..
>
>
>
> As now I am running on Dev Cluster & need to know optimize parameter for
> Kylin so that Cube building process runs fast. Can you pls. tell what are
> properties I need to change in kylin.properties file – To optimize the cube
> build process.. I have 180 millions of data on fact table.
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
> *Sent:* Tuesday, November 21, 2017 3:23 PM
> *To:* user
> *Subject:* Re: How to Update the Cube Structure
>
>
>
> To be safe, you can clone the existing to a new cube, make changes on the
> new cube, and then rebuild it.
>
>
>
> 2017-11-21 14:53 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
>
> In case of Data Source table has DDL changes – either new Column added or
> Existing Column data type changed. How Can we do Cube data model update &
> reflecting new column data to Cube? Pls. advise.
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
> *Sent:* Monday, November 20, 2017 7:03 PM
> *To:* user
> *Subject:* Re: Apache Kyine - kylin.env.hdfs-working-dir
>
>
>
> It seems that the script still try to create "/kylin" on HDFS, not in the
> configured path.
>
>
>
> Please double check and make sure this line is not commented:
>
>
>
> kylin.env.hdfs-working-dir=/tenants/rft/rcmo/kylin
>
>
>
> 2017-11-20 14:00 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
>
> I am installing kyline 2.10 on Dev Cluster(Cloudera) . Although I have
> given 777 permission on kylin Folder but still giving error. Can you pls.
> help on this? Whats missing here?
>
>
>
> -bash-4.1$ hdfs dfs -mkdir /tenants/rft/rcmo/kylin
>
> -bash-4.1$ hdfs dfs -chmod 777 /tenants/rft/rcmo/kylin
>
>
>
> Kyline properties.conf - #kylin.env.hdfs-working-dir=/
> tenants/rft/rcmo/kylin
>
>
>
> -bash-4.1$ ./check-env.sh
>
> Retrieving hadoop conf dir...
>
> KYLIN_HOME is set to /apps/rft/rcmo/apps/kylin/
> apache-kylin-2.1.0-bin-cdh57
>
> mkdir: Permission denied: user=a_rcmo_nd, access=WRITE,
> inode="/":hdfs:supergroup:drwxr-xr-x
>
> Failed to create /kylin. Please make sure the user has right to access
> /kylin
>
>
>
> Regards,
>
> Manoj
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
--
Best regards,
Shaofeng Shi 史少锋
RE: How to Update the Cube Structure
Posted by "Kumar, Manoj H" <ma...@jpmorgan.com>.
Np Mind, I got confused with Cube’s Partition column. Once Cube is build out, Can we clean Hive Intermediate flat table? Also, How can we know which flat belongs particular cube? How to schedule the cube build process – taking incremental data from hive source table
Regards,
Manoj
From: ShaoFeng Shi [mailto:shaofengshi@apache.org]
Sent: Wednesday, November 22, 2017 6:52 AM
To: user
Subject: Re: How to Update the Cube Structure
Hi Manoj,
The intermediate flat table is not partitioned, always; it has all files in one folder; I'm wondering what's the purpose to make it partitioned?
2017-11-22 1:12 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>>:
Thanks… one quick question – How to make intermediate flat table partitioned one?? Do we have any parameter in kylin.properties?
Regards,
Manoj
From: ShaoFeng Shi [mailto:shaofengshi@apache.org<ma...@apache.org>]
Sent: Tuesday, November 21, 2017 8:02 PM
To: user
Subject: Re: How to Update the Cube Structure
This document can help:
https://kylin.apache.org/docs21/howto/howto_optimize_build.html
2017-11-21 21:36 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>>:
Thanks for the inputs.. It make sense..
As now I am running on Dev Cluster & need to know optimize parameter for Kylin so that Cube building process runs fast. Can you pls. tell what are properties I need to change in kylin.properties file – To optimize the cube build process.. I have 180 millions of data on fact table.
Regards,
Manoj
From: ShaoFeng Shi [mailto:shaofengshi@apache.org<ma...@apache.org>]
Sent: Tuesday, November 21, 2017 3:23 PM
To: user
Subject: Re: How to Update the Cube Structure
To be safe, you can clone the existing to a new cube, make changes on the new cube, and then rebuild it.
2017-11-21 14:53 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>>:
In case of Data Source table has DDL changes – either new Column added or Existing Column data type changed. How Can we do Cube data model update & reflecting new column data to Cube? Pls. advise.
Regards,
Manoj
From: ShaoFeng Shi [mailto:shaofengshi@apache.org<ma...@apache.org>]
Sent: Monday, November 20, 2017 7:03 PM
To: user
Subject: Re: Apache Kyine - kylin.env.hdfs-working-dir
It seems that the script still try to create "/kylin" on HDFS, not in the configured path.
Please double check and make sure this line is not commented:
kylin.env.hdfs-working-dir=/tenants/rft/rcmo/kylin
2017-11-20 14:00 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>>:
I am installing kyline 2.10 on Dev Cluster(Cloudera) . Although I have given 777 permission on kylin Folder but still giving error. Can you pls. help on this? Whats missing here?
-bash-4.1$ hdfs dfs -mkdir /tenants/rft/rcmo/kylin
-bash-4.1$ hdfs dfs -chmod 777 /tenants/rft/rcmo/kylin
Kyline properties.conf - #kylin.env.hdfs-working-dir=/tenants/rft/rcmo/kylin
-bash-4.1$ ./check-env.sh
Retrieving hadoop conf dir...
KYLIN_HOME is set to /apps/rft/rcmo/apps/kylin/apache-kylin-2.1.0-bin-cdh57
mkdir: Permission denied: user=a_rcmo_nd, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x
Failed to create /kylin. Please make sure the user has right to access /kylin
Regards,
Manoj
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
--
Best regards,
Shaofeng Shi 史少锋
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
--
Best regards,
Shaofeng Shi 史少锋
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
--
Best regards,
Shaofeng Shi 史少锋
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
--
Best regards,
Shaofeng Shi 史少锋
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
Re: How to Update the Cube Structure
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Manoj,
The intermediate flat table is not partitioned, always; it has all files in
one folder; I'm wondering what's the purpose to make it partitioned?
2017-11-22 1:12 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
> Thanks… one quick question – How to make intermediate flat table
> partitioned one?? Do we have any parameter in kylin.properties?
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
> *Sent:* Tuesday, November 21, 2017 8:02 PM
>
> *To:* user
> *Subject:* Re: How to Update the Cube Structure
>
>
>
> This document can help:
>
>
>
> https://kylin.apache.org/docs21/howto/howto_optimize_build.html
>
>
>
>
>
> 2017-11-21 21:36 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
>
> Thanks for the inputs.. It make sense..
>
>
>
> As now I am running on Dev Cluster & need to know optimize parameter for
> Kylin so that Cube building process runs fast. Can you pls. tell what are
> properties I need to change in kylin.properties file – To optimize the cube
> build process.. I have 180 millions of data on fact table.
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
> *Sent:* Tuesday, November 21, 2017 3:23 PM
> *To:* user
> *Subject:* Re: How to Update the Cube Structure
>
>
>
> To be safe, you can clone the existing to a new cube, make changes on the
> new cube, and then rebuild it.
>
>
>
> 2017-11-21 14:53 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
>
> In case of Data Source table has DDL changes – either new Column added or
> Existing Column data type changed. How Can we do Cube data model update &
> reflecting new column data to Cube? Pls. advise.
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
> *Sent:* Monday, November 20, 2017 7:03 PM
> *To:* user
> *Subject:* Re: Apache Kyine - kylin.env.hdfs-working-dir
>
>
>
> It seems that the script still try to create "/kylin" on HDFS, not in the
> configured path.
>
>
>
> Please double check and make sure this line is not commented:
>
>
>
> kylin.env.hdfs-working-dir=/tenants/rft/rcmo/kylin
>
>
>
> 2017-11-20 14:00 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
>
> I am installing kyline 2.10 on Dev Cluster(Cloudera) . Although I have
> given 777 permission on kylin Folder but still giving error. Can you pls.
> help on this? Whats missing here?
>
>
>
> -bash-4.1$ hdfs dfs -mkdir /tenants/rft/rcmo/kylin
>
> -bash-4.1$ hdfs dfs -chmod 777 /tenants/rft/rcmo/kylin
>
>
>
> Kyline properties.conf - #kylin.env.hdfs-working-dir=/
> tenants/rft/rcmo/kylin
>
>
>
> -bash-4.1$ ./check-env.sh
>
> Retrieving hadoop conf dir...
>
> KYLIN_HOME is set to /apps/rft/rcmo/apps/kylin/
> apache-kylin-2.1.0-bin-cdh57
>
> mkdir: Permission denied: user=a_rcmo_nd, access=WRITE,
> inode="/":hdfs:supergroup:drwxr-xr-x
>
> Failed to create /kylin. Please make sure the user has right to access
> /kylin
>
>
>
> Regards,
>
> Manoj
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
--
Best regards,
Shaofeng Shi 史少锋
RE: How to Update the Cube Structure
Posted by "Kumar, Manoj H" <ma...@jpmorgan.com>.
Thanks… one quick question – How to make intermediate flat table partitioned one?? Do we have any parameter in kylin.properties?
Regards,
Manoj
From: ShaoFeng Shi [mailto:shaofengshi@apache.org]
Sent: Tuesday, November 21, 2017 8:02 PM
To: user
Subject: Re: How to Update the Cube Structure
This document can help:
https://kylin.apache.org/docs21/howto/howto_optimize_build.html
2017-11-21 21:36 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>>:
Thanks for the inputs.. It make sense..
As now I am running on Dev Cluster & need to know optimize parameter for Kylin so that Cube building process runs fast. Can you pls. tell what are properties I need to change in kylin.properties file – To optimize the cube build process.. I have 180 millions of data on fact table.
Regards,
Manoj
From: ShaoFeng Shi [mailto:shaofengshi@apache.org<ma...@apache.org>]
Sent: Tuesday, November 21, 2017 3:23 PM
To: user
Subject: Re: How to Update the Cube Structure
To be safe, you can clone the existing to a new cube, make changes on the new cube, and then rebuild it.
2017-11-21 14:53 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>>:
In case of Data Source table has DDL changes – either new Column added or Existing Column data type changed. How Can we do Cube data model update & reflecting new column data to Cube? Pls. advise.
Regards,
Manoj
From: ShaoFeng Shi [mailto:shaofengshi@apache.org<ma...@apache.org>]
Sent: Monday, November 20, 2017 7:03 PM
To: user
Subject: Re: Apache Kyine - kylin.env.hdfs-working-dir
It seems that the script still try to create "/kylin" on HDFS, not in the configured path.
Please double check and make sure this line is not commented:
kylin.env.hdfs-working-dir=/tenants/rft/rcmo/kylin
2017-11-20 14:00 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>>:
I am installing kyline 2.10 on Dev Cluster(Cloudera) . Although I have given 777 permission on kylin Folder but still giving error. Can you pls. help on this? Whats missing here?
-bash-4.1$ hdfs dfs -mkdir /tenants/rft/rcmo/kylin
-bash-4.1$ hdfs dfs -chmod 777 /tenants/rft/rcmo/kylin
Kyline properties.conf - #kylin.env.hdfs-working-dir=/tenants/rft/rcmo/kylin
-bash-4.1$ ./check-env.sh
Retrieving hadoop conf dir...
KYLIN_HOME is set to /apps/rft/rcmo/apps/kylin/apache-kylin-2.1.0-bin-cdh57
mkdir: Permission denied: user=a_rcmo_nd, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x
Failed to create /kylin. Please make sure the user has right to access /kylin
Regards,
Manoj
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
--
Best regards,
Shaofeng Shi 史少锋
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
--
Best regards,
Shaofeng Shi 史少锋
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
--
Best regards,
Shaofeng Shi 史少锋
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
Re: How to Update the Cube Structure
Posted by ShaoFeng Shi <sh...@apache.org>.
This document can help:
https://kylin.apache.org/docs21/howto/howto_optimize_build.html
2017-11-21 21:36 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
> Thanks for the inputs.. It make sense..
>
>
>
> As now I am running on Dev Cluster & need to know optimize parameter for
> Kylin so that Cube building process runs fast. Can you pls. tell what are
> properties I need to change in kylin.properties file – To optimize the cube
> build process.. I have 180 millions of data on fact table.
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
> *Sent:* Tuesday, November 21, 2017 3:23 PM
> *To:* user
> *Subject:* Re: How to Update the Cube Structure
>
>
>
> To be safe, you can clone the existing to a new cube, make changes on the
> new cube, and then rebuild it.
>
>
>
> 2017-11-21 14:53 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
>
> In case of Data Source table has DDL changes – either new Column added or
> Existing Column data type changed. How Can we do Cube data model update &
> reflecting new column data to Cube? Pls. advise.
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
> *Sent:* Monday, November 20, 2017 7:03 PM
> *To:* user
> *Subject:* Re: Apache Kyine - kylin.env.hdfs-working-dir
>
>
>
> It seems that the script still try to create "/kylin" on HDFS, not in the
> configured path.
>
>
>
> Please double check and make sure this line is not commented:
>
>
>
> kylin.env.hdfs-working-dir=/tenants/rft/rcmo/kylin
>
>
>
> 2017-11-20 14:00 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
>
> I am installing kyline 2.10 on Dev Cluster(Cloudera) . Although I have
> given 777 permission on kylin Folder but still giving error. Can you pls.
> help on this? Whats missing here?
>
>
>
> -bash-4.1$ hdfs dfs -mkdir /tenants/rft/rcmo/kylin
>
> -bash-4.1$ hdfs dfs -chmod 777 /tenants/rft/rcmo/kylin
>
>
>
> Kyline properties.conf - #kylin.env.hdfs-working-dir=/
> tenants/rft/rcmo/kylin
>
>
>
> -bash-4.1$ ./check-env.sh
>
> Retrieving hadoop conf dir...
>
> KYLIN_HOME is set to /apps/rft/rcmo/apps/kylin/
> apache-kylin-2.1.0-bin-cdh57
>
> mkdir: Permission denied: user=a_rcmo_nd, access=WRITE,
> inode="/":hdfs:supergroup:drwxr-xr-x
>
> Failed to create /kylin. Please make sure the user has right to access
> /kylin
>
>
>
> Regards,
>
> Manoj
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
--
Best regards,
Shaofeng Shi 史少锋
RE: How to Update the Cube Structure
Posted by "Kumar, Manoj H" <ma...@jpmorgan.com>.
Thanks for the inputs.. It make sense..
As now I am running on Dev Cluster & need to know optimize parameter for Kylin so that Cube building process runs fast. Can you pls. tell what are properties I need to change in kylin.properties file – To optimize the cube build process.. I have 180 millions of data on fact table.
Regards,
Manoj
From: ShaoFeng Shi [mailto:shaofengshi@apache.org]
Sent: Tuesday, November 21, 2017 3:23 PM
To: user
Subject: Re: How to Update the Cube Structure
To be safe, you can clone the existing to a new cube, make changes on the new cube, and then rebuild it.
2017-11-21 14:53 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>>:
In case of Data Source table has DDL changes – either new Column added or Existing Column data type changed. How Can we do Cube data model update & reflecting new column data to Cube? Pls. advise.
Regards,
Manoj
From: ShaoFeng Shi [mailto:shaofengshi@apache.org<ma...@apache.org>]
Sent: Monday, November 20, 2017 7:03 PM
To: user
Subject: Re: Apache Kyine - kylin.env.hdfs-working-dir
It seems that the script still try to create "/kylin" on HDFS, not in the configured path.
Please double check and make sure this line is not commented:
kylin.env.hdfs-working-dir=/tenants/rft/rcmo/kylin
2017-11-20 14:00 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>>:
I am installing kyline 2.10 on Dev Cluster(Cloudera) . Although I have given 777 permission on kylin Folder but still giving error. Can you pls. help on this? Whats missing here?
-bash-4.1$ hdfs dfs -mkdir /tenants/rft/rcmo/kylin
-bash-4.1$ hdfs dfs -chmod 777 /tenants/rft/rcmo/kylin
Kyline properties.conf - #kylin.env.hdfs-working-dir=/tenants/rft/rcmo/kylin
-bash-4.1$ ./check-env.sh
Retrieving hadoop conf dir...
KYLIN_HOME is set to /apps/rft/rcmo/apps/kylin/apache-kylin-2.1.0-bin-cdh57
mkdir: Permission denied: user=a_rcmo_nd, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x
Failed to create /kylin. Please make sure the user has right to access /kylin
Regards,
Manoj
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
--
Best regards,
Shaofeng Shi 史少锋
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
--
Best regards,
Shaofeng Shi 史少锋
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
Re: How to Update the Cube Structure
Posted by ShaoFeng Shi <sh...@apache.org>.
To be safe, you can clone the existing to a new cube, make changes on the
new cube, and then rebuild it.
2017-11-21 14:53 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
> In case of Data Source table has DDL changes – either new Column added or
> Existing Column data type changed. How Can we do Cube data model update &
> reflecting new column data to Cube? Pls. advise.
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
> *Sent:* Monday, November 20, 2017 7:03 PM
> *To:* user
> *Subject:* Re: Apache Kyine - kylin.env.hdfs-working-dir
>
>
>
> It seems that the script still try to create "/kylin" on HDFS, not in the
> configured path.
>
>
>
> Please double check and make sure this line is not commented:
>
>
>
> kylin.env.hdfs-working-dir=/tenants/rft/rcmo/kylin
>
>
>
> 2017-11-20 14:00 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
>
> I am installing kyline 2.10 on Dev Cluster(Cloudera) . Although I have
> given 777 permission on kylin Folder but still giving error. Can you pls.
> help on this? Whats missing here?
>
>
>
> -bash-4.1$ hdfs dfs -mkdir /tenants/rft/rcmo/kylin
>
> -bash-4.1$ hdfs dfs -chmod 777 /tenants/rft/rcmo/kylin
>
>
>
> Kyline properties.conf - #kylin.env.hdfs-working-dir=/
> tenants/rft/rcmo/kylin
>
>
>
> -bash-4.1$ ./check-env.sh
>
> Retrieving hadoop conf dir...
>
> KYLIN_HOME is set to /apps/rft/rcmo/apps/kylin/
> apache-kylin-2.1.0-bin-cdh57
>
> mkdir: Permission denied: user=a_rcmo_nd, access=WRITE,
> inode="/":hdfs:supergroup:drwxr-xr-x
>
> Failed to create /kylin. Please make sure the user has right to access
> /kylin
>
>
>
> Regards,
>
> Manoj
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
--
Best regards,
Shaofeng Shi 史少锋