You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by "Kumar, Manoj H" <ma...@jpmorgan.com> on 2017/12/08 10:08:37 UTC
Dups records
Getting Dups record error while building dimension cuboids
TSLGENBUHIERARCHY Table has unique set of records of comprising from ID 2 to 13. I have one look up table which has data of Flatten Hierarchy & All are having derived dimension. Can you pls. help in why I am getting the Dups error?
[cid:image001.png@01D3703A.985ECC80]
java.lang.IllegalStateException: The table: TSLGENBUHIERARCHY_SAPRK Dup key found, key=[null], value1=[TOP,TOTAL MARKET,HOUSTON,HOUSTON MARKETS,HOUSTON CENTRAL,HOUSTON MAIN,null,null,null,null,null,JP MORGAN CHASE & CO.,null,null,null,null,null,null,null,null,null,null,JP MORGAN CHASE EX CONSOL ENT.,null,TOTAL GROUPS-EX STAFF & UNDI,CONSUMER & COMMUNITY BANKING,CONSUMER & BUSINESS BANKING,CONSUMER BANKING,CONSUMER BANK W/O CWM,MARKETS, NATIONAL SALES, SAFE ACT,1,2017-11-01 02:50:10.0,null,9999-12-31 00:00:00.0,1238000,0201,6,null,1689017,1683810,1685426,3053087,1672365,1865068,null,null,null,null,null,1689008,null,null,null,null,null,null,null,null,null,null,1687999,null,1687987,1683683,1683201,1689618,1654880,1683216,579999,577143,577920,570223,559604,528466,null,null,null,null,null,579990,null,null,null,null,null,null,null,null,null,null,579980,null,579965,577006,576344,580691], value2=[TOP,TOTAL MARKET,NEW JERSEY REGION,NEW JERSEY MARKETS,NJ PRINCETON,null,null,null,null,null,null,JP MORGAN CHASE & CO.,null,null,null,null,null,null,null,null,null,null,JP MORGAN CHASE EX CONSOL ENT.,null,TOTAL GROUPS-EX STAFF & UNDI,CONSUMER & COMMUNITY BANKING,CONSUMER & BUSINESS BANKING,CONSUMER BANKING,CONSUMER BANK W/O CWM,MARKETS, NATIONAL SALES, SAFE ACT,10,2017-11-01 02:50:10.0,null,9999-12-31 00:00:00.0,1238001,0802,6,null,1689017,1683810,1699104,3053239,1656691,null,null,null,null,null,null,1689008,null,null,null,null,null,null,null,null,null,null,1687999,null,1687987,1683683,1683201,1689618,1654880,1683216,579999,577143,587040,570795,527763,null,null,null,null,null,null,579990,null,null,null,null,null,null,null,null,null,null,579980,null,579965,577006,576344,580691]
at org.apache.kylin.dict.lookup.LookupTable.initRow(LookupTable.java:86)
at org.apache.kylin.dict.lookup.LookupTable.init(LookupTable.java:69)
at org.apache.kylin.dict.lookup.LookupStringTable.init(LookupStringTable.java:79)
at org.apache.kylin.dict.lookup.LookupTable.<init>(LookupTable.java:57)
at org.apache.kylin.dict.lookup.LookupStringTable.<init>(LookupStringTable.java:65)
at org.apache.kylin.cube.CubeManager.getLookupTable(CubeManager.java:686)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:98)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:54)
at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:66)
Regards,
Manoj
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
RE: Dups records
Posted by "Kumar, Manoj H" <ma...@jpmorgan.com>.
Thanks for your reply.. Yes it was related to Table column which was not refreshed as per new format. Hive was getting old value.
Regards,
Manoj
From: ShaoFeng Shi [mailto:shaofengshi@apache.org]
Sent: Friday, December 08, 2017 7:26 PM
To: user
Subject: Re: Dups records
It will check the uniqueness of the join keys with fact table; Please check your data model, how many columns are joining with the fact table? It seems to be one, and that column has the null value.
I couldn't see your model, and also don't know your table structure; just take more checks and searches; This question has been asked several times before.
2017-12-08 20:45 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>>:
Can someone give advise on this urgently? These are columns on Lookup table?
BUnodeName1 ,
BUnodeName2 ,
BUnodeName3 ,
BUnodeName4,BUnodeName5 ,
BUnodeName6 ,
BUnodeName7 ,
BUnodeName8,id,sslsid,corporateBUEntityId,TSLBUHIERARCHY.BUSINESS_UNIT_ID, TSLBUHIERARCHY.NM_LONG, TSLBUSINESSUNIT.EntityID,
TSLBUSTANDARDHIR_STAGE
["BUNODENAME1","BUNODENAME3","BUNODENAME2","BUNODENAME5","BUNODENAME4","BUNODENAME7","BUNODENAME6","ENTITYID","ID","SSLSID"]
BUNodeName columns are repated one but if we take all columns combine together its unique. Why Kylin is giving Dups error on this look up table? I am failed to understand what set of columns have to be unique?
Join Column – ID & -SSLSID with Fact table & BUNode columns are not joined with Fact table. Whats Criteria for getting distinct Columns on Look up tables?
Regards,
Manoj
From: Kumar, Manoj H
Sent: Friday, December 08, 2017 3:39 PM
To: user@kylin.apache.org<ma...@kylin.apache.org>
Subject: Dups records
Getting Dups record error while building dimension cuboids
TSLGENBUHIERARCHY Table has unique set of records of comprising from ID 2 to 13. I have one look up table which has data of Flatten Hierarchy & All are having derived dimension. Can you pls. help in why I am getting the Dups error?
java.lang.IllegalStateException: The table: TSLGENBUHIERARCHY_SAPRK Dup key found, key=[null], value1=[TOP,TOTAL MARKET,HOUSTON,HOUSTON MARKETS,HOUSTON CENTRAL,HOUSTON MAIN,null,null,null,null,null,JP MORGAN CHASE & CO.,null,null,null,null,null,null,null,null,null,null,JP MORGAN CHASE EX CONSOL ENT.,null,TOTAL GROUPS-EX STAFF & UNDI,CONSUMER & COMMUNITY BANKING,CONSUMER & BUSINESS BANKING,CONSUMER BANKING,CONSUMER BANK W/O CWM,MARKETS, NATIONAL SALES, SAFE ACT,1,2017-11-01 02:50:10.0,null,9999-12-31 00:00:00.0,1238000,0201,6,null,1689017,1683810,1685426,3053087,1672365,1865068,null,null,null,null,null,1689008,null,null,null,null,null,null,null,null,null,null,1687999,null,1687987,1683683,1683201,1689618,1654880,1683216,579999,577143,577920,570223,559604,528466,null,null,null,null,null,579990,null,null,null,null,null,null,null,null,null,null,579980,null,579965,577006,576344,580691], value2=[TOP,TOTAL MARKET,NEW JERSEY REGION,NEW JERSEY MARKETS,NJ PRINCETON,null,null,null,null,null,null,JP MORGAN CHASE & CO.,null,null,null,null,null,null,null,null,null,null,JP MORGAN CHASE EX CONSOL ENT.,null,TOTAL GROUPS-EX STAFF & UNDI,CONSUMER & COMMUNITY BANKING,CONSUMER & BUSINESS BANKING,CONSUMER BANKING,CONSUMER BANK W/O CWM,MARKETS, NATIONAL SALES, SAFE ACT,10,2017-11-01 02:50:10.0,null,9999-12-31 00:00:00.0,1238001,0802,6,null,1689017,1683810,1699104,3053239,1656691,null,null,null,null,null,null,1689008,null,null,null,null,null,null,null,null,null,null,1687999,null,1687987,1683683,1683201,1689618,1654880,1683216,579999,577143,587040,570795,527763,null,null,null,null,null,null,579990,null,null,null,null,null,null,null,null,null,null,579980,null,579965,577006,576344,580691]
at org.apache.kylin.dict.lookup.LookupTable.initRow(LookupTable.java:86)
at org.apache.kylin.dict.lookup.LookupTable.init(LookupTable.java:69)
at org.apache.kylin.dict.lookup.LookupStringTable.init(LookupStringTable.java:79)
at org.apache.kylin.dict.lookup.LookupTable.<init>(LookupTable.java:57)
at org.apache.kylin.dict.lookup.LookupStringTable.<init>(LookupStringTable.java:65)
at org.apache.kylin.cube.CubeManager.getLookupTable(CubeManager.java:686)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:98)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:54)
at org.apache.kylin.engine.mr<https://secureweb.jpmchase.net/readonly/http:/org.apache.kylin.engine.mr>.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:66)
Regards,
Manoj
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
--
Best regards,
Shaofeng Shi 史少锋
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
Re: Dups records
Posted by ShaoFeng Shi <sh...@apache.org>.
It will check the uniqueness of the join keys with fact table; Please check
your data model, how many columns are joining with the fact table? It seems
to be one, and that column has the null value.
I couldn't see your model, and also don't know your table structure; just
take more checks and searches; This question has been asked several times
before.
2017-12-08 20:45 GMT+08:00 Kumar, Manoj H <ma...@jpmorgan.com>:
> Can someone give advise on this urgently? These are columns on Lookup
> table?
>
>
>
> BUnodeName1 ,
>
> BUnodeName2 ,
>
> BUnodeName3 ,
>
> BUnodeName4,BUnodeName5 ,
>
> BUnodeName6 ,
>
> BUnodeName7 ,
>
> BUnodeName8,id,sslsid,corporateBUEntityId,TSLBUHIERARCHY.BUSINESS_UNIT_ID,
> TSLBUHIERARCHY.NM_LONG, TSLBUSINESSUNIT.EntityID,
>
>
>
> TSLBUSTANDARDHIR_STAGE
>
> ["BUNODENAME1","BUNODENAME3","BUNODENAME2","BUNODENAME5","
> BUNODENAME4","BUNODENAME7","BUNODENAME6","ENTITYID","ID","SSLSID"]
>
>
>
>
>
> BUNodeName columns are repated one but if we take all columns combine
> together its unique. Why Kylin is giving Dups error on this look up table?
> I am failed to understand what set of columns have to be unique?
>
>
>
> Join Column – ID & -SSLSID with Fact table & BUNode columns are not joined
> with Fact table. Whats Criteria for getting distinct Columns on Look up
> tables?
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* Kumar, Manoj H
> *Sent:* Friday, December 08, 2017 3:39 PM
> *To:* user@kylin.apache.org
> *Subject:* Dups records
>
>
>
> Getting Dups record error while building dimension cuboids
>
>
>
> TSLGENBUHIERARCHY Table has unique set of records of comprising from ID 2
> to 13. I have one look up table which has data of Flatten Hierarchy & All
> are having derived dimension. Can you pls. help in why I am getting the
> Dups error?
>
>
>
>
>
>
>
>
>
> java.lang.IllegalStateException: The table: TSLGENBUHIERARCHY_SAPRK Dup
> key found, key=[null], value1=[TOP,TOTAL MARKET,HOUSTON,HOUSTON
> MARKETS,HOUSTON CENTRAL,HOUSTON MAIN,null,null,null,null,null,JP MORGAN
> CHASE & CO.,null,null,null,null,null,null,null,null,null,null,JP MORGAN
> CHASE EX CONSOL ENT.,null,TOTAL GROUPS-EX STAFF & UNDI,CONSUMER & COMMUNITY
> BANKING,CONSUMER & BUSINESS BANKING,CONSUMER BANKING,CONSUMER BANK W/O
> CWM,MARKETS, NATIONAL SALES, SAFE ACT,1,2017-11-01
> 02:50:10.0,null,9999-12-31 00:00:00.0,1238000,0201,6,
> null,1689017,1683810,1685426,3053087,1672365,1865068,null,
> null,null,null,null,1689008,null,null,null,null,null,null,
> null,null,null,null,1687999,null,1687987,1683683,1683201,
> 1689618,1654880,1683216,579999,577143,577920,570223,
> 559604,528466,null,null,null,null,null,579990,null,null,
> null,null,null,null,null,null,null,null,579980,null,579965,577006,576344,580691],
> value2=[TOP,TOTAL MARKET,NEW JERSEY REGION,NEW JERSEY MARKETS,NJ
> PRINCETON,null,null,null,null,null,null,JP MORGAN CHASE &
> CO.,null,null,null,null,null,null,null,null,null,null,JP MORGAN CHASE EX
> CONSOL ENT.,null,TOTAL GROUPS-EX STAFF & UNDI,CONSUMER & COMMUNITY
> BANKING,CONSUMER & BUSINESS BANKING,CONSUMER BANKING,CONSUMER BANK W/O
> CWM,MARKETS, NATIONAL SALES, SAFE ACT,10,2017-11-01
> 02:50:10.0,null,9999-12-31 00:00:00.0,1238001,0802,6,
> null,1689017,1683810,1699104,3053239,1656691,null,null,
> null,null,null,null,1689008,null,null,null,null,null,null,
> null,null,null,null,1687999,null,1687987,1683683,1683201,
> 1689618,1654880,1683216,579999,577143,587040,570795,
> 527763,null,null,null,null,null,null,579990,null,null,
> null,null,null,null,null,null,null,null,579980,null,579965,
> 577006,576344,580691]
>
> at org.apache.kylin.dict.lookup.LookupTable.initRow(
> LookupTable.java:86)
>
> at org.apache.kylin.dict.lookup.LookupTable.init(LookupTable.
> java:69)
>
> at org.apache.kylin.dict.lookup.LookupStringTable.init(
> LookupStringTable.java:79)
>
> at org.apache.kylin.dict.lookup.LookupTable.<init>(
> LookupTable.java:57)
>
> at org.apache.kylin.dict.lookup.LookupStringTable.<init>(
> LookupStringTable.java:65)
>
> at org.apache.kylin.cube.CubeManager.getLookupTable(
> CubeManager.java:686)
>
> at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.
> processSegment(DictionaryGeneratorCLI.java:98)
>
> at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.
> processSegment(DictionaryGeneratorCLI.java:54)
>
> at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(
> CreateDictionaryJob.java:66)
>
>
>
>
>
> Regards,
>
> Manoj
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
--
Best regards,
Shaofeng Shi 史少锋
RE: Dups records
Posted by "Kumar, Manoj H" <ma...@jpmorgan.com>.
Can someone give advise on this urgently? These are columns on Lookup table?
BUnodeName1 ,
BUnodeName2 ,
BUnodeName3 ,
BUnodeName4,BUnodeName5 ,
BUnodeName6 ,
BUnodeName7 ,
BUnodeName8,id,sslsid,corporateBUEntityId,TSLBUHIERARCHY.BUSINESS_UNIT_ID, TSLBUHIERARCHY.NM_LONG, TSLBUSINESSUNIT.EntityID,
TSLBUSTANDARDHIR_STAGE
["BUNODENAME1","BUNODENAME3","BUNODENAME2","BUNODENAME5","BUNODENAME4","BUNODENAME7","BUNODENAME6","ENTITYID","ID","SSLSID"]
BUNodeName columns are repated one but if we take all columns combine together its unique. Why Kylin is giving Dups error on this look up table? I am failed to understand what set of columns have to be unique?
Join Column - ID & -SSLSID with Fact table & BUNode columns are not joined with Fact table. Whats Criteria for getting distinct Columns on Look up tables?
Regards,
Manoj
From: Kumar, Manoj H
Sent: Friday, December 08, 2017 3:39 PM
To: user@kylin.apache.org
Subject: Dups records
Getting Dups record error while building dimension cuboids
TSLGENBUHIERARCHY Table has unique set of records of comprising from ID 2 to 13. I have one look up table which has data of Flatten Hierarchy & All are having derived dimension. Can you pls. help in why I am getting the Dups error?
java.lang.IllegalStateException: The table: TSLGENBUHIERARCHY_SAPRK Dup key found, key=[null], value1=[TOP,TOTAL MARKET,HOUSTON,HOUSTON MARKETS,HOUSTON CENTRAL,HOUSTON MAIN,null,null,null,null,null,JP MORGAN CHASE & CO.,null,null,null,null,null,null,null,null,null,null,JP MORGAN CHASE EX CONSOL ENT.,null,TOTAL GROUPS-EX STAFF & UNDI,CONSUMER & COMMUNITY BANKING,CONSUMER & BUSINESS BANKING,CONSUMER BANKING,CONSUMER BANK W/O CWM,MARKETS, NATIONAL SALES, SAFE ACT,1,2017-11-01 02:50:10.0,null,9999-12-31 00:00:00.0,1238000,0201,6,null,1689017,1683810,1685426,3053087,1672365,1865068,null,null,null,null,null,1689008,null,null,null,null,null,null,null,null,null,null,1687999,null,1687987,1683683,1683201,1689618,1654880,1683216,579999,577143,577920,570223,559604,528466,null,null,null,null,null,579990,null,null,null,null,null,null,null,null,null,null,579980,null,579965,577006,576344,580691], value2=[TOP,TOTAL MARKET,NEW JERSEY REGION,NEW JERSEY MARKETS,NJ PRINCETON,null,null,null,null,null,null,JP MORGAN CHASE & CO.,null,null,null,null,null,null,null,null,null,null,JP MORGAN CHASE EX CONSOL ENT.,null,TOTAL GROUPS-EX STAFF & UNDI,CONSUMER & COMMUNITY BANKING,CONSUMER & BUSINESS BANKING,CONSUMER BANKING,CONSUMER BANK W/O CWM,MARKETS, NATIONAL SALES, SAFE ACT,10,2017-11-01 02:50:10.0,null,9999-12-31 00:00:00.0,1238001,0802,6,null,1689017,1683810,1699104,3053239,1656691,null,null,null,null,null,null,1689008,null,null,null,null,null,null,null,null,null,null,1687999,null,1687987,1683683,1683201,1689618,1654880,1683216,579999,577143,587040,570795,527763,null,null,null,null,null,null,579990,null,null,null,null,null,null,null,null,null,null,579980,null,579965,577006,576344,580691]
at org.apache.kylin.dict.lookup.LookupTable.initRow(LookupTable.java:86)
at org.apache.kylin.dict.lookup.LookupTable.init(LookupTable.java:69)
at org.apache.kylin.dict.lookup.LookupStringTable.init(LookupStringTable.java:79)
at org.apache.kylin.dict.lookup.LookupTable.<init>(LookupTable.java:57)
at org.apache.kylin.dict.lookup.LookupStringTable.<init>(LookupStringTable.java:65)
at org.apache.kylin.cube.CubeManager.getLookupTable(CubeManager.java:686)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:98)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:54)
at org.apache.kylin.engine.mr<https://secureweb.jpmchase.net/readonly/http:/org.apache.kylin.engine.mr>.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:66)
Regards,
Manoj
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.