You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by 梅秋莹 <32...@qq.com> on 2020/01/10 05:56:46 UTC
回复： Kylin support view as lookup table？
Thank you very much! This is compelete kylin log. You can see the error message at 2020-01-08 16:04:12. If kylin dose not support view as a lookup table, is there only one way to solve the problem? That is, creating entity table.
------------------ 原始邮件 ------------------
发件人: "codingforfun@126.com"<codingforfun@126.com>;
发送时间: 2020年1月9日(星期四) 下午2:39
收件人: "user"<user@kylin.apache.org>;
主题: Re: Kylin support view as lookup table？
First, kylin does support using view as a lookup table. During the build process, there is a special step to handle the case where the view is a lookup table, called Materialize Hive View in Lookup Tables, it's main process is as follows:
From your logs, the error occurred in step Build Dimension Dictionary, at this time step Materialize Hive View in Lookup Tables has been completed。
Can you provide a more complete kylin.log or reproduce the error on the sample dataset？
在 2020年1月9日，14:08，梅秋莹 <3281438086@qq.com> 写道：
Dear Kyliners:
1. Environment: kylin 2.6.3
2. Question:
In some articles, I notice that kylin support view as lookup table. In my model and cube, there is field “fpdate” and “date” in fact table, and the field "fpdate" and "date" both represnt date.
So I created a PAYMENT_DATE_DIM as entity table and a view called BILLING_PERIOD_DIM. These two table structures are identical.
But when I build the cube, in the "Build Dimension Dictionary" step, kylin returned an error like this "hive view 替代物理表 build cube 时 java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.io.Text". How can I solve the problem? If kylin have some usage restriction？
<2D01CF17@FE604852.6EC3165E.jpg>
<2F04CD10@1BE0A81B.6EC3165E.jpg>
Re: Kylin support view as lookup table？
Posted by 朱卫斌 <co...@126.com>.
Sorry, I didn't understand what the root cause is and how to solve it. Can you describe it in more detail?
| |
weibin0516
|
|
codingforfun@126.com
Best wishes !
|
签名由网易邮箱大师定制
On 01/17/2020 14:28，梅秋莹<32...@qq.com> wrote：
Hi! We have solved the problem which was discribed blow. When create lookup table view, we used clause like this:
CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
[DISABLE REWRITE]
[COMMENT materialized_view_comment]
[PARTITIONED ON (col_name, ...)]
[CLUSTERED ON (col_name, ...) | DISTRIBUTED ON (col_name, ...) SORTED ON (col_name, ...)]
[
[ROW FORMAT row_format]
[STORED AS file_format]
| STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
]
[LOCATION hdfs_path]
[TBLPROPERTIES (property_name=property_value, ...)]
AS SELECT ...;
But I also have a question , what will be affected if we created a MATERIALIZED VIEW other than a logical view? Or is there other point for attention? Thank you very much!
------------------ 原始邮件 ------------------
发件人: "3281438086"<32...@qq.com>;
发送时间: 2020年1月13日(星期一) 晚上6:12
收件人: "codingforfun"<us...@kylin.apache.org>;
主题: 回复： Kylin support view as lookup table？
Hi! Thank you for your tips. I will check the configurations. And the dimension views are as following. In the last email,you said maybe some property settings are missing, please tell me where I can find the instructions or relative information when I use view as dim.
发件人: "codingforfun"<co...@126.com>;
发送时间: 2020年1月13日(星期一) 下午4:42
收件人: "user"<us...@kylin.apache.org>;
主题: Re: Kylin support view as lookup table？
Hi, I checked the logs and found 6 Materialize Hive View in Lookup Tables steps
and found all of them SUCCEED
I suspect that some property settings are missing when create the table in step Materialize Hive View in Lookup Tables，
such as
[ROW FORMAT row_format][STORED AS file_format]
I want to see the detail schema of these dim views, you provide the results of：
show create table DWS.COMPANY_DIM;
show create table DWS.REGION_DIM;
show create table DWS.BILLING_PERIOD_DIM;
show create table DWS.PAYMENT_DATE_DIM;
show create table DWS.PAYMENT_TIME_DIM;
在 2020年1月12日，22:33，codingforfun@126.com 写道：
Thanks, i will see tomorrow.
<PastedGraphic-2.tiff>
在 2020年1月10日，13:56，梅秋莹 <32...@qq.com> 写道：
Thank you very much! This is compelete kylin log. You can see the error message at 2020-01-08 16:04:12. If kylin dose not support view as a lookup table, is there only one way to solve the problem? That is, creating entity table.
------------------ 原始邮件 ------------------
发件人: "codingforfun@126.com"<co...@126.com>;
发送时间: 2020年1月9日(星期四) 下午2:39
收件人: "user"<us...@kylin.apache.org>;
主题: Re: Kylin support view as lookup table？
First, kylin does support using view as a lookup table. During the build process, there is a special step to handle the case where the view is a lookup table, called Materialize Hive View in Lookup Tables, it's main process is as follows:
<2A...@05841551.1E12185E.jpg>
From your logs, the error occurred in step Build Dimension Dictionary, at this time step Materialize Hive View in Lookup Tables has been completed。
Can you provide a more complete kylin.log or reproduce the error on the sample dataset？
<ED...@DD5F0153.1E12185E.jpg>
在 2020年1月9日，14:08，梅秋莹 <32...@qq.com> 写道：
Dear Kyliners:
1. Environment: kylin 2.6.3
2. Question:
In some articles, I notice that kylin support view as lookup table. In my model and cube, there is field “fpdate” and “date” in fact table, and the field "fpdate" and "date" both represnt date.
So I created a PAYMENT_DATE_DIM as entity table and a view called BILLING_PERIOD_DIM. These two table structures are identical.
But when I build the cube, in the "Build Dimension Dictionary" step, kylin returned an error like this "hive view 替代物理表 build cube 时 java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.io.Text". How can I solve the problem? If kylin have some usage restriction？
<2D...@FE604852.6EC3165E.jpg>
<2F...@1BE0A81B.6EC3165E.jpg>
<ED...@05841551.1E12185E.jpg><kylin.rar>
回复： Kylin support view as lookup table？
Posted by 梅秋莹 <32...@qq.com>.
Hi! We have solved the problem which was discribed blow. When create lookup table view, we used clause like this:
CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
[DISABLE REWRITE]
[COMMENT materialized_view_comment]
[PARTITIONED ON (col_name, ...)]
[CLUSTERED ON (col_name, ...) | DISTRIBUTED ON (col_name, ...) SORTED ON (col_name, ...)]
[
[ROW FORMAT row_format]
[STORED AS file_format]
| STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
]
[LOCATION hdfs_path]
[TBLPROPERTIES (property_name=property_value, ...)]
AS SELECT ...;
But I also have a question , what will be affected if we created a MATERIALIZED VIEW other than a logical view? Or is there other point for attention? Thank you very much!
------------------ 原始邮件 ------------------
发件人: "3281438086"<3281438086@qq.com>;
发送时间: 2020年1月13日(星期一) 晚上6:12
收件人: "codingforfun"<user@kylin.apache.org>;
主题: 回复： Kylin support view as lookup table？
Hi! Thank you for your tips. I will check the configurations. And the dimension views are as following. In the last email,you said maybe some property settings are missing, please tell me where I can find the instructions or relative information when I use view as dim.
发件人: "codingforfun"<codingforfun@126.com>;
发送时间: 2020年1月13日(星期一) 下午4:42
收件人: "user"<user@kylin.apache.org>;
主题: Re: Kylin support view as lookup table？
Hi, I checked the logs and found 6 Materialize Hive View in Lookup Tables steps
and found all of them SUCCEED
I suspect that some property settings are missing when create the table in step Materialize Hive View in Lookup Tables，
such as
[ROW FORMAT row_format] [STORED AS file_format] I want to see the detail schema of these dim views, you provide the results of：
show create table DWS.COMPANY_DIM;
show create table DWS.REGION_DIM;
show create table DWS.BILLING_PERIOD_DIM;
show create table DWS.PAYMENT_DATE_DIM;
show create table DWS.PAYMENT_TIME_DIM;
在 2020年1月12日，22:33，codingforfun@126.com 写道：
Thanks, i will see tomorrow.
<PastedGraphic-2.tiff>
在 2020年1月10日，13:56，梅秋莹 <3281438086@qq.com> 写道：
Thank you very much! This is compelete kylin log. You can see the error message at 2020-01-08 16:04:12. If kylin dose not support view as a lookup table, is there only one way to solve the problem? That is, creating entity table.
------------------ 原始邮件 ------------------
发件人: "codingforfun@126.com"<codingforfun@126.com>;
发送时间: 2020年1月9日(星期四) 下午2:39
收件人: "user"<user@kylin.apache.org>;
主题: Re: Kylin support view as lookup table？
First, kylin does support using view as a lookup table. During the build process, there is a special step to handle the case where the view is a lookup table, called Materialize Hive View in Lookup Tables, it's main process is as follows:
<2A915CDD@05841551.1E12185E.jpg>
From your logs, the error occurred in step Build Dimension Dictionary, at this time step Materialize Hive View in Lookup Tables has been completed。
Can you provide a more complete kylin.log or reproduce the error on the sample dataset？
<EDA00F33@DD5F0153.1E12185E.jpg>
在 2020年1月9日，14:08，梅秋莹 <3281438086@qq.com> 写道：
Dear Kyliners:
1. Environment: kylin 2.6.3
2. Question:
In some articles, I notice that kylin support view as lookup table. In my model and cube, there is field “fpdate” and “date” in fact table, and the field "fpdate" and "date" both represnt date.
So I created a PAYMENT_DATE_DIM as entity table and a view called BILLING_PERIOD_DIM. These two table structures are identical.
But when I build the cube, in the "Build Dimension Dictionary" step, kylin returned an error like this "hive view 替代物理表 build cube 时 java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.io.Text". How can I solve the problem? If kylin have some usage restriction？
<2D01CF17@FE604852.6EC3165E.jpg>
<2F04CD10@1BE0A81B.6EC3165E.jpg>
<EDA00F33@DD5F0153.1E12185E.jpg><2A915CDD@05841551.1E12185E.jpg><kylin.rar>
回复： Kylin support view as lookup table？
Posted by 梅秋莹 <32...@qq.com>.
Hi! Thank you for your tips. I will check the configurations. And the dimension views are as following. In the last email,you said maybe some property settings are missing, please tell me where I can find the instructions or relative information when I use view as dim.
发件人: "codingforfun"<codingforfun@126.com>;
发送时间: 2020年1月13日(星期一) 下午4:42
收件人: "user"<user@kylin.apache.org>;
主题: Re: Kylin support view as lookup table？
Hi, I checked the logs and found 6 Materialize Hive View in Lookup Tables steps
and found all of them SUCCEED
I suspect that some property settings are missing when create the table in step Materialize Hive View in Lookup Tables，
such as
[ROW FORMAT row_format] [STORED AS file_format] I want to see the detail schema of these dim views, you provide the results of：
show create table DWS.COMPANY_DIM;
show create table DWS.REGION_DIM;
show create table DWS.BILLING_PERIOD_DIM;
show create table DWS.PAYMENT_DATE_DIM;
show create table DWS.PAYMENT_TIME_DIM;
在 2020年1月12日，22:33，codingforfun@126.com 写道：
Thanks, i will see tomorrow.
<PastedGraphic-2.tiff>
在 2020年1月10日，13:56，梅秋莹 <3281438086@qq.com> 写道：
Thank you very much! This is compelete kylin log. You can see the error message at 2020-01-08 16:04:12. If kylin dose not support view as a lookup table, is there only one way to solve the problem? That is, creating entity table.
------------------ 原始邮件 ------------------
发件人: "codingforfun@126.com"<codingforfun@126.com>;
发送时间: 2020年1月9日(星期四) 下午2:39
收件人: "user"<user@kylin.apache.org>;
主题: Re: Kylin support view as lookup table？
First, kylin does support using view as a lookup table. During the build process, there is a special step to handle the case where the view is a lookup table, called Materialize Hive View in Lookup Tables, it's main process is as follows:
<2A915CDD@05841551.1E12185E.jpg>
From your logs, the error occurred in step Build Dimension Dictionary, at this time step Materialize Hive View in Lookup Tables has been completed。
Can you provide a more complete kylin.log or reproduce the error on the sample dataset？
<EDA00F33@DD5F0153.1E12185E.jpg>
在 2020年1月9日，14:08，梅秋莹 <3281438086@qq.com> 写道：
Dear Kyliners:
1. Environment: kylin 2.6.3
2. Question:
In some articles, I notice that kylin support view as lookup table. In my model and cube, there is field “fpdate” and “date” in fact table, and the field "fpdate" and "date" both represnt date.
So I created a PAYMENT_DATE_DIM as entity table and a view called BILLING_PERIOD_DIM. These two table structures are identical.
But when I build the cube, in the "Build Dimension Dictionary" step, kylin returned an error like this "hive view 替代物理表 build cube 时 java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.io.Text". How can I solve the problem? If kylin have some usage restriction？
<2D01CF17@FE604852.6EC3165E.jpg>
<2F04CD10@1BE0A81B.6EC3165E.jpg>
<EDA00F33@DD5F0153.1E12185E.jpg><2A915CDD@05841551.1E12185E.jpg><kylin.rar>
Re: Kylin support view as lookup table？
Posted by "codingforfun@126.com" <co...@126.com>.
Hi, I checked the logs and found 6 Materialize Hive View in Lookup Tables steps
and found all of them SUCCEED
I suspect that some property settings are missing when create the table in step Materialize Hive View in Lookup Tables，
such as
[ROW FORMAT row_format]
[STORED AS file_format]
I want to see the detail schema of these dim views, you provide the results of：
show create table DWS.COMPANY_DIM;
show create table DWS.REGION_DIM;
show create table DWS.BILLING_PERIOD_DIM;
show create table DWS.PAYMENT_DATE_DIM;
show create table DWS.PAYMENT_TIME_DIM;
> 在 2020年1月12日，22:33，codingforfun@126.com 写道：
>
> Thanks, i will see tomorrow.
>
> <PastedGraphic-2.tiff>
>
>> 在 2020年1月10日，13:56，梅秋莹 <3281438086@qq.com <ma...@qq.com>> 写道：
>>
>> Thank you very much! This is compelete kylin log. You can see the error message at 2020-01-08 16:04:12. If kylin dose not support view as a lookup table, is there only one way to solve the problem? That is, creating entity table.
>>
>>
>> ------------------ 原始邮件 ------------------
>> 发件人: "codingforfun@126.com <ma...@126.com>"<codingforfun@126.com <ma...@126.com>>;
>> 发送时间: 2020年1月9日(星期四) 下午2:39
>> 收件人: "user"<user@kylin.apache.org <ma...@kylin.apache.org>>;
>> 主题: Re: Kylin support view as lookup table？
>>
>> First, kylin does support using view as a lookup table. During the build process, there is a special step to handle the case where the view is a lookup table, called Materialize Hive View in Lookup Tables, it's main process is as follows:
>>
>> <2A...@05841551.1E12185E.jpg>
>> From your logs, the error occurred in step Build Dimension Dictionary, at this time step Materialize Hive View in Lookup Tables has been completed。
>>
>> Can you provide a more complete kylin.log or reproduce the error on the sample dataset？
>>
>>
>> <ED...@DD5F0153.1E12185E.jpg>
>>
>>> 在 2020年1月9日，14:08，梅秋莹 <3281438086@qq.com <ma...@qq.com>> 写道：
>>>
>>> Dear Kyliners:
>>>
>>> 1. Environment: kylin 2.6.3
>>> 2. Question:
>>>
>>> In some articles, I notice that kylin support view as lookup table. In my model and cube, there is field “fpdate” and “date” in fact table, and the field "fpdate" and "date" both represnt date.
>>> So I created a PAYMENT_DATE_DIM as entity table and a view called BILLING_PERIOD_DIM. These two table structures are identical.
>>> But when I build the cube, in the "Build Dimension Dictionary" step, kylin returned an error like this "hive view 替代物理表 build cube 时 java.lang.ClassCastException: org.apache.hadoop.io <http://org.apache.hadoop.io/>.LongWritable cannot be cast to org.apache.hadoop.io.Text". How can I solve the problem? If kylin have some usage restriction？
>>>
>>> <2D...@FE604852.6EC3165E.jpg>
>>> <2F...@1BE0A81B.6EC3165E.jpg>
>>>
>>
>> <ED...@05841551.1E12185E.jpg><kylin.rar>
Re: Kylin support view as lookup table？
Posted by "codingforfun@126.com" <co...@126.com>.
Thanks, i will see tomorrow.
> 在 2020年1月10日，13:56，梅秋莹 <32...@qq.com> 写道：
>
> Thank you very much! This is compelete kylin log. You can see the error message at 2020-01-08 16:04:12. If kylin dose not support view as a lookup table, is there only one way to solve the problem? That is, creating entity table.
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "codingforfun@126.com <ma...@126.com>"<codingforfun@126.com <ma...@126.com>>;
> 发送时间: 2020年1月9日(星期四) 下午2:39
> 收件人: "user"<user@kylin.apache.org <ma...@kylin.apache.org>>;
> 主题: Re: Kylin support view as lookup table？
>
> First, kylin does support using view as a lookup table. During the build process, there is a special step to handle the case where the view is a lookup table, called Materialize Hive View in Lookup Tables, it's main process is as follows:
>
> <2A...@05841551.1E12185E.jpg>
> From your logs, the error occurred in step Build Dimension Dictionary, at this time step Materialize Hive View in Lookup Tables has been completed。
>
> Can you provide a more complete kylin.log or reproduce the error on the sample dataset？
>
>
> <ED...@DD5F0153.1E12185E.jpg>
>
>> 在 2020年1月9日，14:08，梅秋莹 <3281438086@qq.com <ma...@qq.com>> 写道：
>>
>> Dear Kyliners:
>>
>> 1. Environment: kylin 2.6.3
>> 2. Question:
>>
>> In some articles, I notice that kylin support view as lookup table. In my model and cube, there is field “fpdate” and “date” in fact table, and the field "fpdate" and "date" both represnt date.
>> So I created a PAYMENT_DATE_DIM as entity table and a view called BILLING_PERIOD_DIM. These two table structures are identical.
>> But when I build the cube, in the "Build Dimension Dictionary" step, kylin returned an error like this "hive view 替代物理表 build cube 时 java.lang.ClassCastException: org.apache.hadoop.io <http://org.apache.hadoop.io/>.LongWritable cannot be cast to org.apache.hadoop.io.Text". How can I solve the problem? If kylin have some usage restriction？
>>
>> <2D...@FE604852.6EC3165E.jpg>
>> <2F...@1BE0A81B.6EC3165E.jpg>
>>
>
> <ED...@05841551.1E12185E.jpg><kylin.rar>