You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "XiaoXiang Yu (JIRA)" <ji...@apache.org> on 2019/06/12 06:58:00 UTC
[jira] [Issue Comment Deleted] (KYLIN-4038) Lookup tables is empty
when build on ORC tables.
[ https://issues.apache.org/jira/browse/KYLIN-4038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
XiaoXiang Yu updated KYLIN-4038:
--------------------------------
Comment: was deleted
(was: {code}$ watch org.apache.kylin.dict.lookup.SnapshotTable takeSnapshot "\{params,target,returnObj}" -x 2
Press Q or Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 172 ms.
ts=2019-06-12 02:35:45; [cost=235.194326ms] result=@ArrayList[
@Object[][
@HiveTable[hive: database=[DEFAULT], table=[KYLIN_ACCOUNT_SIDA]],
@TableDesc[TableDesc\{name='KYLIN_ACCOUNT_SIDA', columns=[ColumnDesc{id='1', name='ACCOUNT_ID', datatype='bigint', comment='null'}, ColumnDesc\{id='2', name='ACCOUNT_BUYER_LEVEL', datatype='integer', comment='null'}, ColumnDesc\{id='3', name='ACCOUNT_SELLER_LEVEL', datatype='integer', comment='null'}, ColumnDesc\{id='4', name='ACCOUNT_COUNTRY', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='5', name='ACCOUNT_CONTACT', datatype='varchar(256)', comment='null'}], sourceType=0, tableType='MANAGED_TABLE', database=DatabaseDesc [name=DEFAULT], identity='DEFAULT.KYLIN_ACCOUNT_SIDA'}],
],
@SnapshotTable[
STORAGE_TYPE_METASTORE=@String[metaStore],
tableName=@String[DEFAULT.KYLIN_ACCOUNT_SIDA],
signature=@TableSignature[FileSignature [path=hdfs://hdp30.hortonworks.com:8020/warehouse/tablespace/managed/hive/kylin_account_sida, size=0, lastModifiedTime=1560234144865]],
useDictionary=@Boolean[true],
lastBuildTime=@Long[0],
rowIndices=@ArrayList[isEmpty=true;size=0],
dict=@TrieDictionary[org.apache.kylin.dict.TrieDictionary@663163f9],
NULL_STR=@String[ ÿ],
],
null,
]
$
$
$ watch org.apache.kylin.dict.lookup.SnapshotTable takeSnapshot "\{params,target,returnObj}" -x 2
Press Q or Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 123 ms.
ts=2019-06-12 02:40:06; [cost=611.591702ms] result=@ArrayList[
@Object[][
@HiveTable[hive: database=[DEFAULT], table=[KYLIN_CAL_DT]],
@TableDesc[TableDesc\{name='KYLIN_CAL_DT', columns=[ColumnDesc{id='1', name='CAL_DT', datatype='date', comment='null'}, ColumnDesc\{id='2', name='YEAR_BEG_DT', datatype='date', comment='null'}, ColumnDesc\{id='3', name='QTR_BEG_DT', datatype='date', comment='null'}, ColumnDesc\{id='4', name='MONTH_BEG_DT', datatype='date', comment='null'}, ColumnDesc\{id='5', name='WEEK_BEG_DT', datatype='date', comment='null'}, ColumnDesc\{id='6', name='AGE_FOR_YEAR_ID', datatype='smallint', comment='null'}, ColumnDesc\{id='7', name='AGE_FOR_QTR_ID', datatype='smallint', comment='null'}, ColumnDesc\{id='8', name='AGE_FOR_MONTH_ID', datatype='smallint', comment='null'}, ColumnDesc\{id='9', name='AGE_FOR_WEEK_ID', datatype='smallint', comment='null'}, ColumnDesc\{id='10', name='AGE_FOR_DT_ID', datatype='smallint', comment='null'}, ColumnDesc\{id='11', name='AGE_FOR_RTL_YEAR_ID', datatype='smallint', comment='null'}, ColumnDesc\{id='12', name='AGE_FOR_RTL_QTR_ID', datatype='smallint', comment='null'}, ColumnDesc\{id='13', name='AGE_FOR_RTL_MONTH_ID', datatype='smallint', comment='null'}, ColumnDesc\{id='14', name='AGE_FOR_RTL_WEEK_ID', datatype='smallint', comment='null'}, ColumnDesc\{id='15', name='AGE_FOR_CS_WEEK_ID', datatype='smallint', comment='null'}, ColumnDesc\{id='16', name='DAY_OF_CAL_ID', datatype='integer', comment='null'}, ColumnDesc\{id='17', name='DAY_OF_YEAR_ID', datatype='smallint', comment='null'}, ColumnDesc\{id='18', name='DAY_OF_QTR_ID', datatype='smallint', comment='null'}, ColumnDesc\{id='19', name='DAY_OF_MONTH_ID', datatype='smallint', comment='null'}, ColumnDesc\{id='20', name='DAY_OF_WEEK_ID', datatype='integer', comment='null'}, ColumnDesc\{id='21', name='WEEK_OF_YEAR_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='22', name='WEEK_OF_CAL_ID', datatype='integer', comment='null'}, ColumnDesc\{id='23', name='MONTH_OF_QTR_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='24', name='MONTH_OF_YEAR_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='25', name='MONTH_OF_CAL_ID', datatype='smallint', comment='null'}, ColumnDesc\{id='26', name='QTR_OF_YEAR_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='27', name='QTR_OF_CAL_ID', datatype='smallint', comment='null'}, ColumnDesc\{id='28', name='YEAR_OF_CAL_ID', datatype='smallint', comment='null'}, ColumnDesc\{id='29', name='YEAR_END_DT', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='30', name='QTR_END_DT', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='31', name='MONTH_END_DT', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='32', name='WEEK_END_DT', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='33', name='CAL_DT_NAME', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='34', name='CAL_DT_DESC', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='35', name='CAL_DT_SHORT_NAME', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='36', name='YTD_YN_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='37', name='QTD_YN_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='38', name='MTD_YN_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='39', name='WTD_YN_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='40', name='SEASON_BEG_DT', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='41', name='DAY_IN_YEAR_COUNT', datatype='smallint', comment='null'}, ColumnDesc\{id='42', name='DAY_IN_QTR_COUNT', datatype='tinyint', comment='null'}, ColumnDesc\{id='43', name='DAY_IN_MONTH_COUNT', datatype='tinyint', comment='null'}, ColumnDesc\{id='44', name='DAY_IN_WEEK_COUNT', datatype='tinyint', comment='null'}, ColumnDesc\{id='45', name='RTL_YEAR_BEG_DT', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='46', name='RTL_QTR_BEG_DT', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='47', name='RTL_MONTH_BEG_DT', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='48', name='RTL_WEEK_BEG_DT', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='49', name='CS_WEEK_BEG_DT', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='50', name='CAL_DATE', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='51', name='DAY_OF_WEEK', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='52', name='MONTH_ID', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='53', name='PRD_DESC', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='54', name='PRD_FLAG', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='55', name='PRD_ID', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='56', name='PRD_IND', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='57', name='QTR_DESC', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='58', name='QTR_ID', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='59', name='QTR_IND', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='60', name='RETAIL_WEEK', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='61', name='RETAIL_YEAR', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='62', name='RETAIL_START_DATE', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='63', name='RETAIL_WK_END_DATE', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='64', name='WEEK_IND', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='65', name='WEEK_NUM_DESC', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='66', name='WEEK_BEG_DATE', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='67', name='WEEK_END_DATE', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='68', name='WEEK_IN_YEAR_ID', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='69', name='WEEK_ID', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='70', name='WEEK_BEG_END_DESC_MDY', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='71', name='WEEK_BEG_END_DESC_MD', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='72', name='YEAR_ID', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='73', name='YEAR_IND', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='74', name='CAL_DT_MNS_1YEAR_DT', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='75', name='CAL_DT_MNS_2YEAR_DT', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='76', name='CAL_DT_MNS_1QTR_DT', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='77', name='CAL_DT_MNS_2QTR_DT', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='78', name='CAL_DT_MNS_1MONTH_DT', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='79', name='CAL_DT_MNS_2MONTH_DT', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='80', name='CAL_DT_MNS_1WEEK_DT', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='81', name='CAL_DT_MNS_2WEEK_DT', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='82', name='CURR_CAL_DT_MNS_1YEAR_YN_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='83', name='CURR_CAL_DT_MNS_2YEAR_YN_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='84', name='CURR_CAL_DT_MNS_1QTR_YN_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='85', name='CURR_CAL_DT_MNS_2QTR_YN_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='86', name='CURR_CAL_DT_MNS_1MONTH_YN_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='87', name='CURR_CAL_DT_MNS_2MONTH_YN_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='88', name='CURR_CAL_DT_MNS_1WEEK_YN_IND', datatype='tinyint', comment='null'}, ColumnDesc\{id='89', name='CURR_CAL_DT_MNS_2WEEK_YN_IND', datatype='tinyint', comment='null'}, ColumnDesc\{id='90', name='RTL_MONTH_OF_RTL_YEAR_ID', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='91', name='RTL_QTR_OF_RTL_YEAR_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='92', name='RTL_WEEK_OF_RTL_YEAR_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='93', name='SEASON_OF_YEAR_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='94', name='YTM_YN_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='95', name='YTQ_YN_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='96', name='YTW_YN_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='97', name='KYLIN_CAL_DT_CRE_DATE', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='98', name='KYLIN_CAL_DT_CRE_USER', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='99', name='KYLIN_CAL_DT_UPD_DATE', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='100', name='KYLIN_CAL_DT_UPD_USER', datatype='varchar(256)', comment='null'}], sourceType=0, tableType='MANAGED_TABLE', database=DatabaseDesc [name=DEFAULT], identity='DEFAULT.KYLIN_CAL_DT'}],
],
@SnapshotTable[
STORAGE_TYPE_METASTORE=@String[metaStore],
tableName=@String[DEFAULT.KYLIN_CAL_DT],
signature=@TableSignature[FileSignature [path=hdfs://hdp30.hortonworks.com:8020/warehouse/tablespace/managed/hive/kylin_cal_dt, size=0, lastModifiedTime=1560320723405]],
useDictionary=@Boolean[true],
lastBuildTime=@Long[0],
rowIndices=@ArrayList[isEmpty=false;size=731],
dict=@TrieDictionary[org.apache.kylin.dict.TrieDictionary@e4ea8c92],
NULL_STR=@String[ ÿ],
],
null,
]
ts=2019-06-12 02:40:07; [cost=247.320587ms] result=@ArrayList[
@Object[][
@HiveTable[hive: database=[DEFAULT], table=[KYLIN_CATEGORY_GROUPINGS]],
@TableDesc[TableDesc\{name='KYLIN_CATEGORY_GROUPINGS', columns=[ColumnDesc{id='1', name='LEAF_CATEG_ID', datatype='bigint', comment='null'}, ColumnDesc\{id='2', name='LEAF_CATEG_NAME', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='3', name='SITE_ID', datatype='integer', comment='null'}, ColumnDesc\{id='4', name='CATEG_BUSN_MGR', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='5', name='CATEG_BUSN_UNIT', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='6', name='REGN_CATEG', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='7', name='USER_DEFINED_FIELD1', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='8', name='USER_DEFINED_FIELD3', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='9', name='KYLIN_GROUPINGS_CRE_DATE', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='10', name='KYLIN_GROUPINGS_UPD_DATE', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='11', name='KYLIN_GROUPINGS_CRE_USER', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='12', name='KYLIN_GROUPINGS_UPD_USER', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='13', name='META_CATEG_ID', datatype='decimal(19,4)', comment='null'}, ColumnDesc\{id='14', name='META_CATEG_NAME', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='15', name='CATEG_LVL2_ID', datatype='decimal(19,4)', comment='null'}, ColumnDesc\{id='16', name='CATEG_LVL3_ID', datatype='decimal(19,4)', comment='null'}, ColumnDesc\{id='17', name='CATEG_LVL4_ID', datatype='decimal(19,4)', comment='null'}, ColumnDesc\{id='18', name='CATEG_LVL5_ID', datatype='decimal(19,4)', comment='null'}, ColumnDesc\{id='19', name='CATEG_LVL6_ID', datatype='decimal(19,4)', comment='null'}, ColumnDesc\{id='20', name='CATEG_LVL7_ID', datatype='decimal(19,4)', comment='null'}, ColumnDesc\{id='21', name='CATEG_LVL2_NAME', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='22', name='CATEG_LVL3_NAME', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='23', name='CATEG_LVL4_NAME', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='24', name='CATEG_LVL5_NAME', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='25', name='CATEG_LVL6_NAME', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='26', name='CATEG_LVL7_NAME', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='27', name='CATEG_FLAGS', datatype='decimal(19,4)', comment='null'}, ColumnDesc\{id='28', name='ADULT_CATEG_YN', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='29', name='DOMAIN_ID', datatype='decimal(19,4)', comment='null'}, ColumnDesc\{id='30', name='USER_DEFINED_FIELD5', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='31', name='VCS_ID', datatype='decimal(19,4)', comment='null'}, ColumnDesc\{id='32', name='GCS_ID', datatype='decimal(19,4)', comment='null'}, ColumnDesc\{id='33', name='MOVE_TO', datatype='decimal(19,4)', comment='null'}, ColumnDesc\{id='34', name='SAP_CATEGORY_ID', datatype='decimal(19,4)', comment='null'}, ColumnDesc\{id='35', name='SRC_ID', datatype='tinyint', comment='null'}, ColumnDesc\{id='36', name='BSNS_VRTCL_NAME', datatype='varchar(256)', comment='null'}], sourceType=0, tableType='MANAGED_TABLE', database=DatabaseDesc [name=DEFAULT], identity='DEFAULT.KYLIN_CATEGORY_GROUPINGS'}],
],
@SnapshotTable[
STORAGE_TYPE_METASTORE=@String[metaStore],
tableName=@String[DEFAULT.KYLIN_CATEGORY_GROUPINGS],
signature=@TableSignature[FileSignature [path=hdfs://hdp30.hortonworks.com:8020/warehouse/tablespace/managed/hive/kylin_category_groupings, size=0, lastModifiedTime=1560320724280]],
useDictionary=@Boolean[true],
lastBuildTime=@Long[0],
rowIndices=@ArrayList[isEmpty=false;size=144],
dict=@TrieDictionary[org.apache.kylin.dict.TrieDictionary@9004172d],
NULL_STR=@String[ ÿ],
],
null,
]
ts=2019-06-12 02:40:08; [cost=292.788882ms] result=@ArrayList[
@Object[][
@HiveTable[hive: database=[DEFAULT], table=[KYLIN_ACCOUNT]],
@TableDesc[TableDesc\{name='KYLIN_ACCOUNT', columns=[ColumnDesc{id='1', name='ACCOUNT_ID', datatype='bigint', comment='null'}, ColumnDesc\{id='2', name='ACCOUNT_BUYER_LEVEL', datatype='integer', comment='null'}, ColumnDesc\{id='3', name='ACCOUNT_SELLER_LEVEL', datatype='integer', comment='null'}, ColumnDesc\{id='4', name='ACCOUNT_COUNTRY', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='5', name='ACCOUNT_CONTACT', datatype='varchar(256)', comment='null'}], sourceType=0, tableType='MANAGED_TABLE', database=DatabaseDesc [name=DEFAULT], identity='DEFAULT.KYLIN_ACCOUNT'}],
],
@SnapshotTable[
STORAGE_TYPE_METASTORE=@String[metaStore],
tableName=@String[DEFAULT.KYLIN_ACCOUNT],
signature=@TableSignature[FileSignature [path=hdfs://hdp30.hortonworks.com:8020/warehouse/tablespace/managed/hive/kylin_account, size=0, lastModifiedTime=1560320722712]],
useDictionary=@Boolean[true],
lastBuildTime=@Long[0],
rowIndices=@ArrayList[isEmpty=false;size=10000],
dict=@TrieDictionary[org.apache.kylin.dict.TrieDictionary@470c01d2],
NULL_STR=@String[ ÿ],
],
null,
]
ts=2019-06-12 02:40:08; [cost=227.225103ms] result=@ArrayList[
@Object[][
@HiveTable[hive: database=[DEFAULT], table=[KYLIN_COUNTRY]],
@TableDesc[TableDesc\{name='KYLIN_COUNTRY', columns=[ColumnDesc{id='1', name='COUNTRY', datatype='varchar(256)', comment='null'}, ColumnDesc\{id='2', name='LATITUDE', datatype='double', comment='null'}, ColumnDesc\{id='3', name='LONGITUDE', datatype='double', comment='null'}, ColumnDesc\{id='4', name='NAME', datatype='varchar(256)', comment='null'}], sourceType=0, tableType='MANAGED_TABLE', database=DatabaseDesc [name=DEFAULT], identity='DEFAULT.KYLIN_COUNTRY'}],
],
@SnapshotTable[
STORAGE_TYPE_METASTORE=@String[metaStore],
tableName=@String[DEFAULT.KYLIN_COUNTRY],
signature=@TableSignature[FileSignature [path=hdfs://hdp30.hortonworks.com:8020/warehouse/tablespace/managed/hive/kylin_country, size=0, lastModifiedTime=1560320723080]],
useDictionary=@Boolean[true],
lastBuildTime=@Long[0],
rowIndices=@ArrayList[isEmpty=false;size=244],
dict=@TrieDictionary[org.apache.kylin.dict.TrieDictionary@cdf49bc6],
NULL_STR=@String[ ÿ],
],
null,
]
{code})
> Lookup tables is empty when build on ORC tables.
> ------------------------------------------------
>
> Key: KYLIN-4038
> URL: https://issues.apache.org/jira/browse/KYLIN-4038
> Project: Kylin
> Issue Type: Bug
> Affects Versions: v2.6.2
> Environment: HDP 3 cluster.
> Reporter: Alexander
> Assignee: XiaoXiang Yu
> Priority: Blocker
> Attachments: image-2019-06-11-16-12-09-472.png
>
>
> Create tables, simple copy of learn_kylin project.
> {code:java}
> // Create tables
> create table kylin_sales_sida (trans_id bigint,part_dt date,lstg_format_name string,leaf_categ_id bigint,lstg_site_id int,slr_segment_cd smallint,price decimal(19,4),item_count bigint,seller_id bigint,buyer_id bigint,ops_user_id string,ops_region string) stored as orc tblproperties ("orc.compress"="SNAPPY");
> insert into table kylin_sales_sida select * from kylin_sales;
> create table kylin_country_sida (country string,latitude double,longitude double,name string) stored as orc tblproperties ("orc.compress"="SNAPPY");
> insert into table kylin_country_sida select * from kylin_country;
> create table kylin_category_groupings_sida (leaf_categ_id bigint,leaf_categ_name string,site_id int,categ_busn_mgr string,categ_busn_unit string,regn_categ string,user_defined_field1 string,user_defined_field3 string,kylin_groupings_cre_date string,kylin_groupings_upd_date string,kylin_groupings_cre_user string,kylin_groupings_upd_user string,meta_categ_id decimal(10,0),meta_categ_name string,categ_lvl2_id decimal(10,0),categ_lvl3_id decimal(10,0),categ_lvl4_id decimal(10,0),categ_lvl5_id decimal(10,0),categ_lvl6_id decimal(10,0),categ_lvl7_id decimal(10,0),categ_lvl2_name string,categ_lvl3_name string,categ_lvl4_name string,categ_lvl5_name string,categ_lvl6_name string,categ_lvl7_name string,categ_flags decimal(10,0),adult_categ_yn string,domain_id decimal(10,0),user_defined_field5 string,vcs_id decimal(10,0),gcs_id decimal(10,0),move_to decimal(10,0),sap_category_id decimal(10,0),src_id tinyint,bsns_vrtcl_name string) stored as orc tblproperties ("orc.compress"="SNAPPY");
> insert into table kylin_category_groupings_sida select * from kylin_category_groupings;
> create table kylin_cal_dt_sida (cal_dt date,year_beg_dt date,qtr_beg_dt date,month_beg_dt date,week_beg_dt date,age_for_year_id smallint,age_for_qtr_id smallint,age_for_month_id smallint,age_for_week_id smallint,age_for_dt_id smallint,age_for_rtl_year_id smallint,age_for_rtl_qtr_id smallint,age_for_rtl_month_id smallint,age_for_rtl_week_id smallint,age_for_cs_week_id smallint,day_of_cal_id int,day_of_year_id smallint,day_of_qtr_id smallint,day_of_month_id smallint,day_of_week_id int,week_of_year_id tinyint,week_of_cal_id int,month_of_qtr_id tinyint,month_of_year_id tinyint,month_of_cal_id smallint,qtr_of_year_id tinyint,qtr_of_cal_id smallint,year_of_cal_id smallint,year_end_dt string,qtr_end_dt string,month_end_dt string,week_end_dt string,cal_dt_name string,cal_dt_desc string,cal_dt_short_name string,ytd_yn_id tinyint,qtd_yn_id tinyint,mtd_yn_id tinyint,wtd_yn_id tinyint,season_beg_dt string,day_in_year_count smallint,day_in_qtr_count tinyint,day_in_month_count tinyint,day_in_week_count tinyint,rtl_year_beg_dt string,rtl_qtr_beg_dt string,rtl_month_beg_dt string,rtl_week_beg_dt string,cs_week_beg_dt string,cal_date string,day_of_week string,month_id string,prd_desc string,prd_flag string,prd_id string,prd_ind string,qtr_desc string,qtr_id string,qtr_ind string,retail_week string,retail_year string,retail_start_date string,retail_wk_end_date string,week_ind string,week_num_desc string,week_beg_date string,week_end_date string,week_in_year_id string,week_id string,week_beg_end_desc_mdy string,week_beg_end_desc_md string,year_id string,year_ind string,cal_dt_mns_1year_dt string,cal_dt_mns_2year_dt string,cal_dt_mns_1qtr_dt string,cal_dt_mns_2qtr_dt string,cal_dt_mns_1month_dt string,cal_dt_mns_2month_dt string,cal_dt_mns_1week_dt string,cal_dt_mns_2week_dt string,curr_cal_dt_mns_1year_yn_id tinyint,curr_cal_dt_mns_2year_yn_id tinyint,curr_cal_dt_mns_1qtr_yn_id tinyint,curr_cal_dt_mns_2qtr_yn_id tinyint,curr_cal_dt_mns_1month_yn_id tinyint,curr_cal_dt_mns_2month_yn_id tinyint,curr_cal_dt_mns_1week_yn_ind tinyint,curr_cal_dt_mns_2week_yn_ind tinyint,rtl_month_of_rtl_year_id string,rtl_qtr_of_rtl_year_id tinyint,rtl_week_of_rtl_year_id tinyint,season_of_year_id tinyint,ytm_yn_id tinyint,ytq_yn_id tinyint,ytw_yn_id tinyint,kylin_cal_dt_cre_date string,kylin_cal_dt_cre_user string,kylin_cal_dt_upd_date string,kylin_cal_dt_upd_user string) stored as orc tblproperties ("orc.compress"="SNAPPY");
> insert into table kylin_cal_dt_sida select * from kylin_cal_dt;
> create table kylin_account_sida (account_id bigint,account_buyer_level int,account_seller_level int,account_country string,account_contact string) stored as orc tblproperties ("orc.compress"="SNAPPY");
> insert into table kylin_account_sida select * from kylin_account;{code}
> Create new project and data model, a manual copy of test cube learn_kylin. Add Lookup tables to HBase.
> Build cube, and notice what the lookup tables builds empty:
> {code:java}
> // Result of Convert Lookup Table to HFile: step
> 2019-06-06 11:12:23,172 DEBUG [Scheduler 1312561281 Job a501b13d-1026-8fbf-5695-80483044f23b-580] common.HadoopCmdOutput:100 : Counters: 39
> File System Counters
> FILE: Number of bytes read=0
> FILE: Number of bytes written=323522
> FILE: Number of read operations=0
> FILE: Number of large read operations=0
> FILE: Number of write operations=0
> HDFS: Number of bytes read=0
> HDFS: Number of bytes written=0
> HDFS: Number of read operations=1
> HDFS: Number of large read operations=0
> HDFS: Number of write operations=0
> Job Counters
> Launched reduce tasks=1
> Total time spent by all reduces in occupied slots (ms)=21696
> Total time spent by all reduce tasks (ms)=3616
> Total vcore-milliseconds taken by all reduce tasks=3616
> Total megabyte-milliseconds taken by all reduce tasks=22216704
> Map-Reduce Framework
> Combine input records=0
> Combine output records=0
> Reduce input groups=0
> Reduce shuffle bytes=0
> Reduce input records=0
> Reduce output records=0
> Spilled Records=0
> Shuffled Maps =0
> Failed Shuffles=0
> Merged Map outputs=0
> GC time elapsed (ms)=71
> CPU time spent (ms)=840
> Physical memory (bytes) snapshot=216272896
> Virtual memory (bytes) snapshot=7290363904
> Total committed heap usage (bytes)=191365120
> Peak Reduce Physical memory (bytes)=216272896
> Peak Reduce Virtual memory (bytes)=7290363904
> Shuffle Errors
> BAD_ID=0
> CONNECTION=0
> IO_ERROR=0
> WRONG_LENGTH=0
> WRONG_MAP=0
> WRONG_REDUCE=0
> File Output Format Counters
> Bytes Written=0
> 2019-06-06 11:12:23,172 DEBUG [Scheduler 1312561281 Job a501b13d-1026-8fbf-5695-80483044f23b-580] common.HadoopCmdOutput:107 : outputFolder is hdfs://********:8020/kylin/kylin_metadata/kylin-a501b13d-1026-8fbf-5695-80483044f23b/DEFAULT.KYLIN_ACCOUNT_SIDA/hfile
> 2019-06-06 11:12:23,173 DEBUG [Scheduler 1312561281 Job a501b13d-1026-8fbf-5695-80483044f23b-580] common.HadoopCmdOutput:112 : Seems no counter found for hdfs
> 2019-06-06 11:12:23,202 INFO [Scheduler 1312561281 Job a501b13d-1026-8fbf-5695-80483044f23b-580] execution.ExecutableManager:453 : job id:a501b13d-1026-8fbf-5695-80483044f23b-05 from RUNNING to SUCCEED
> {code}
> As result any Query with lookup tables data goes to be incorrect. As they are missed.
> (In Build Dimension Dictionary step the same thing, empty lookup data goes into cube calculation)
> If i run fo each tables compact major:
> {code:java}
> // Compact Major on orc tables
> ALTER TABLE DEFAULT.KYLIN_ACCOUNT_SIDA COMPACT 'MAJOR';
> ... ALTER all tables{code}
> After that, a can rerun cube build. The result will be error in MR job:
> {code:java}
> // MR Exception
> 2019-06-07 11:51:54,482 INFO [Scheduler 1312561281 Job 6d0f79da-7902-1b2c-a49e-fa6b6190911c-580] orc.OrcInputFormat:1737 : ORC pushdown predicate: null
> 2019-06-07 11:51:55,052 INFO [ORC_GET_SPLITS #7] impl.OrcCodecPool:56 : Got brand-new codec SNAPPY
> 2019-06-07 11:51:55,263 INFO [Scheduler 1312561281 Job 6d0f79da-7902-1b2c-a49e-fa6b6190911c-580] orc.OrcInputFormat:1855 : FooterCacheHitRatio: 0/1
> 2019-06-07 11:51:55,297 INFO [Scheduler 1312561281 Job 6d0f79da-7902-1b2c-a49e-fa6b6190911c-580] mapreduce.JobSubmitter:202 : number of splits:1
> 2019-06-07 11:51:55,329 INFO [Scheduler 1312561281 Job 6d0f79da-7902-1b2c-a49e-fa6b6190911c-580] mapreduce.JobSubmitter:298 : Submitting tokens for job: job_1559213270190_0372
> 2019-06-07 11:51:55,329 INFO [Scheduler 1312561281 Job 6d0f79da-7902-1b2c-a49e-fa6b6190911c-580] mapreduce.JobSubmitter:299 : Executing with tokens: []
> 2019-06-07 11:51:55,572 INFO [Scheduler 1312561281 Job 6d0f79da-7902-1b2c-a49e-fa6b6190911c-580] impl.YarnClientImpl:306 : Submitted application application_1559213270190_0372
> 2019-06-07 11:51:55,575 INFO [Scheduler 1312561281 Job 6d0f79da-7902-1b2c-a49e-fa6b6190911c-580] mapreduce.Job:1574 : The url to track the job: http://******:8088/proxy/application_1559213270190_0372/
> 2019-06-07 11:51:58,408 INFO [FetcherRunner 833282134-77] threadpool.DefaultFetcherRunner:94 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 229 already succeed, 61 error, 0 discarded, 1 others
> 2019-06-07 11:52:28,408 INFO [FetcherRunner 833282134-77] threadpool.DefaultFetcherRunner:94 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 229 already succeed, 61 error, 0 discarded, 1 others
> 2019-06-07 11:52:31,361 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
> 2019-06-07 11:52:58,407 INFO [FetcherRunner 833282134-77] threadpool.DefaultFetcherRunner:94 : Job Fetcher: 1 should running, 1 actual running, 0 stopped, 0 ready, 229 already succeed, 61 error, 0 discarded, 1 others
> 2019-06-07 11:53:05,805 DEBUG [Scheduler 1312561281 Job 6d0f79da-7902-1b2c-a49e-fa6b6190911c-580] common.HadoopCmdOutput:100 : Counters: 10
> Job Counters
> Failed map tasks=4
> Killed reduce tasks=1
> Launched map tasks=4
> Other local map tasks=3
> Data-local map tasks=1
> Total time spent by all maps in occupied slots (ms)=136272
> Total time spent by all reduces in occupied slots (ms)=0
> Total time spent by all map tasks (ms)=45424
> Total vcore-milliseconds taken by all map tasks=45424
> Total megabyte-milliseconds taken by all map tasks=139542528
> 2019-06-07 11:53:05,806 DEBUG [Scheduler 1312561281 Job 6d0f79da-7902-1b2c-a49e-fa6b6190911c-580] common.HadoopCmdOutput:107 : outputFolder is hdfs://apachai1.apm.local:8020/kylin/kylin_metadata/kylin-6d0f79da-7902-1b2c-a49e-fa6b6190911c/DEFAULT.KYLIN_ACCOUNT_SIDA/hfile
> 2019-06-07 11:53:05,807 DEBUG [Scheduler 1312561281 Job 6d0f79da-7902-1b2c-a49e-fa6b6190911c-580] common.HadoopCmdOutput:112 : Seems no counter found for hdfs
> 2019-06-07 11:53:05,808 WARN [Scheduler 1312561281 Job 6d0f79da-7902-1b2c-a49e-fa6b6190911c-580] common.HadoopCmdOutput:119 : Job Diagnostics:Task failed task_1559213270190_0372_m_000000
> Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0
> 2019-06-07 11:53:05,994 WARN [Scheduler 1312561281 Job 6d0f79da-7902-1b2c-a49e-fa6b6190911c-580] common.HadoopCmdOutput:124 : Failure task Diagnostics:
> 2019-06-07 11:53:05,994 WARN [Scheduler 1312561281 Job 6d0f79da-7902-1b2c-a49e-fa6b6190911c-580] common.HadoopCmdOutput:127 : Error: java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.LongWritable
> at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableLongObjectInspector.getPrimitiveJavaObject(WritableLongObjectInspector.java:46)
> at org.apache.hive.hcatalog.data.HCatRecordSerDe.serializePrimitiveField(HCatRecordSerDe.java:278)
> at org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeField(HCatRecordSerDe.java:199)
> at org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:53)
> at org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:97)
> at org.apache.hive.hcatalog.mapreduce.HCatRecordReader.nextKeyValue(HCatRecordReader.java:204)
> at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:568)
> at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> 2019-06-07 11:53:06,004 INFO [Scheduler 1312561281 Job 6d0f79da-7902-1b2c-a49e-fa6b6190911c-580] execution.ExecutableManager:453 : job id:6d0f79da-7902-1b2c-a49e-fa6b6190911c-05 from RUNNING to ERROR
> 2019-06-07 11:53:06,005 ERROR [Scheduler 1312561281 Job 6d0f79da-7902-1b2c-a49e-fa6b6190911c-580] execution.AbstractExecutable:167 : error running Executable: CubingJob{id=6d0f79da-7902-1b2c-a49e-fa6b6190911c, name=BUILD CUBE - kylin_sales_cube_orc - 20120101000000_20140101000000 - MSK 2019-06-07 11:36:49, state=RUNNING}
> 2019-06-07 11:53:06,008 DEBUG [pool-7-thread-1] cachesync.Broadcaster:116 : Servers in the cluster: [localhost:7070]
> 2019-06-07 11:53:06,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:126 : Announcing new broadcast to all: BroadcastEvent{entity=execute_output, event=update, cacheKey=6d0f79da-7902-1b2c-a49e-fa6b6190911c}
> 2019-06-07 11:53:06,012 INFO [Scheduler 1312561281 Job 6d0f79da-7902-1b2c-a49e-fa6b6190911c-580] execution.ExecutableManager:453 : job id:6d0f79da-7902-1b2c-a49e-fa6b6190911c from RUNNING to ERROR
> 2019-06-07 11:53:06,012 DEBUG [pool-7-thread-1] cachesync.Broadcaster:116 : Servers in the cluster: [localhost:7070]
> 2019-06-07 11:53:06,012 DEBUG [http-nio-7070-exec-1] cachesync.Broadcaster:246 : Broadcasting UPDATE, execute_output, 6d0f79da-7902-1b2c-a49e-fa6b6190911c
> 2019-06-07 11:53:06,012 DEBUG [Scheduler 1312561281 Job 6d0f79da-7902-1b2c-a49e-fa6b6190911c-580] execution.AbstractExecutable:318 : no need to send email, user list is empty
> 2019-06-07 11:53:06,012 DEBUG [pool-7-thread-1] cachesync.Broadcaster:126 : Announcing new broadcast to all: BroadcastEvent{entity=execute_output, event=update, cacheKey=6d0f79da-7902-1b2c-a49e-fa6b6190911c}
> 2019-06-07 11:53:06,013 DEBUG [http-nio-7070-exec-1] cachesync.Broadcaster:280 : Done broadcasting UPDATE, execute_output, 6d0f79da-7902-1b2c-a49e-fa6b6190911c
> 2019-06-07 11:53:06,014 ERROR [pool-11-thread-10] threadpool.DefaultScheduler:116 : ExecuteException job:6d0f79da-7902-1b2c-a49e-fa6b6190911c
> org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.engine.mr.exception.MapReduceException: Counters: 10
> Job Counters
> Failed map tasks=4
> Killed reduce tasks=1
> Launched map tasks=4
> Other local map tasks=3
> Data-local map tasks=1
> Total time spent by all maps in occupied slots (ms)=136272
> Total time spent by all reduces in occupied slots (ms)=0
> Total time spent by all map tasks (ms)=45424
> Total vcore-milliseconds taken by all map tasks=45424
> Total megabyte-milliseconds taken by all map tasks=139542528
> Job Diagnostics:Task failed task_1559213270190_0372_m_000000
> Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0
> Failure task Diagnostics:
> Error: java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.LongWritable
> at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableLongObjectInspector.getPrimitiveJavaObject(WritableLongObjectInspector.java:46)
> at org.apache.hive.hcatalog.data.HCatRecordSerDe.serializePrimitiveField(HCatRecordSerDe.java:278)
> at org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeField(HCatRecordSerDe.java:199)
> at org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:53)
> at org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:97)
> at org.apache.hive.hcatalog.mapreduce.HCatRecordReader.nextKeyValue(HCatRecordReader.java:204)
> at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:568)
> at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:180)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.engine.mr.exception.MapReduceException: Counters: 10
> Job Counters
> Failed map tasks=4
> Killed reduce tasks=1
> Launched map tasks=4
> Other local map tasks=3
> Data-local map tasks=1
> Total time spent by all maps in occupied slots (ms)=136272
> Total time spent by all reduces in occupied slots (ms)=0
> Total time spent by all map tasks (ms)=45424
> Total vcore-milliseconds taken by all map tasks=45424
> Total megabyte-milliseconds taken by all map tasks=139542528
> Job Diagnostics:Task failed task_1559213270190_0372_m_000000
> Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0
> Failure task Diagnostics:
> Error: java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.LongWritable
> at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableLongObjectInspector.getPrimitiveJavaObject(WritableLongObjectInspector.java:46)
> at org.apache.hive.hcatalog.data.HCatRecordSerDe.serializePrimitiveField(HCatRecordSerDe.java:278)
> at org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeField(HCatRecordSerDe.java:199)
> at org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:53)
> at org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:97)
> at org.apache.hive.hcatalog.mapreduce.HCatRecordReader.nextKeyValue(HCatRecordReader.java:204)
> at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:568)
> at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:180)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
> ... 4 more
> Caused by: org.apache.kylin.engine.mr.exception.MapReduceException: Counters: 10
> Job Counters
> Failed map tasks=4
> Killed reduce tasks=1
> Launched map tasks=4
> Other local map tasks=3
> Data-local map tasks=1
> Total time spent by all maps in occupied slots (ms)=136272
> Total time spent by all reduces in occupied slots (ms)=0
> Total time spent by all map tasks (ms)=45424
> Total vcore-milliseconds taken by all map tasks=45424
> Total megabyte-milliseconds taken by all map tasks=139542528
> Job Diagnostics:Task failed task_1559213270190_0372_m_000000
> Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0
> Failure task Diagnostics:
> Error: java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.LongWritable
> at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableLongObjectInspector.getPrimitiveJavaObject(WritableLongObjectInspector.java:46)
> at org.apache.hive.hcatalog.data.HCatRecordSerDe.serializePrimitiveField(HCatRecordSerDe.java:278)
> at org.apache.hive.hcatalog.data.HCatRecordSerDe.serializeField(HCatRecordSerDe.java:199)
> at org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:53)
> at org.apache.hive.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:97)
> at org.apache.hive.hcatalog.mapreduce.HCatRecordReader.nextKeyValue(HCatRecordReader.java:204)
> at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:568)
> at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173)
> at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
> ... 6 more
> {code}
> Looks like the problem ib HCatalog that used by Kylin to access Hive tables... Please help, with solution finding...
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)