You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by Tero Paananen <te...@gmail.com> on 2015/06/26 16:03:24 UTC
Many-to-many relationships
Does Kylin support many-to-many type relationships on lookup tables?
Our data structure is as follows.
We have a fact table, let's call it Document.
The Document has a unique id document.id
There is a lookup table that contains many entries per document, let's
call it Document_Rel.
Document_Rel.doc_id
Document_Rel.rel_id
Document_Rel.rel_name
The data looks like this:
Document:
id = 1
id = 2
Document_Rel:
doc_id = 1, rel_id = 1, rel_name = foo
doc_id = 1, rel_id = 2, rel_name = bar
doc_id = 1, rel_id = 3, rel_name = sna
doc_id = 1, rel_id = 4, rel_name = fu
doc_id = 2, rel_id = 1, rel_name = foo
doc_id = 2, rel_id = 5, rel_name = humpty
There are multiple relationships like this for a single Document, e.g.
Document_Rel2, Document_Rel3.
I'm getting errors on cube builds:
java.lang.IllegalStateException: Dup key found, key=[1],
value1=[1,1,foo], value2=[1,2,bar]
at org.apache.kylin.dict.lookup.LookupTable.initRow(LookupTable.java:81)
at org.apache.kylin.dict.lookup.LookupTable.init(LookupTable.java:64)
at org.apache.kylin.dict.lookup.LookupTable.<init>(LookupTable.java:52)
at org.apache.kylin.dict.lookup.LookupStringTable.<init>(LookupStringTable.java:32)
at org.apache.kylin.cube.CubeManager.getLookupTable(CubeManager.java:419)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:63)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42)
at org.apache.kylin.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:53)
-TPP
Re: Many-to-many relationships
Posted by Sarnath <st...@gmail.com>.
Good one!
Re: Many-to-many relationships
Posted by Li Yang <li...@apache.org>.
An alternative is join your Document & Rel in a HIVE VIEW, and use the view
as Kylin's fact table. Then you need no lookup table in Kylin any more.
On Fri, Jun 26, 2015 at 11:23 PM, Tero Paananen <te...@gmail.com>
wrote:
> On Fri, Jun 26, 2015 at 11:21 AM, Luke Han <lu...@gmail.com> wrote:
> > No, Kylin supports only star schema so far, one fact table and some
> lookup tables.
> > Please re-architect the data model to star schema.
>
> That's not possible for our data.
>
> -TPP
>
Re: Many-to-many relationships
Posted by Tero Paananen <te...@gmail.com>.
On Fri, Jun 26, 2015 at 11:21 AM, Luke Han <lu...@gmail.com> wrote:
> No, Kylin supports only star schema so far, one fact table and some lookup tables.
> Please re-architect the data model to star schema.
That's not possible for our data.
-TPP
Re: Many-to-many relationships
Posted by Luke Han <lu...@gmail.com>.
No, Kylin supports only star schema so far, one fact table and some lookup tables.
Please re-architect the data model to star schema.
Thanks.
Regards!
Luke Han
_____________________________
From: Tero Paananen <te...@gmail.com>
Sent: 星期五, 六月 26, 2015 22:03
Subject: Many-to-many relationships
To: <de...@kylin.incubator.apache.org>
Does Kylin support many-to-many type relationships on lookup tables?
Our data structure is as follows.
We have a fact table, let's call it Document.
The Document has a unique id document.id
There is a lookup table that contains many entries per document, let's
call it Document_Rel.
Document_Rel.doc_id
Document_Rel.rel_id
Document_Rel.rel_name
The data looks like this:
Document:
id = 1
id = 2
Document_Rel:
doc_id = 1, rel_id = 1, rel_name = foo
doc_id = 1, rel_id = 2, rel_name = bar
doc_id = 1, rel_id = 3, rel_name = sna
doc_id = 1, rel_id = 4, rel_name = fu
doc_id = 2, rel_id = 1, rel_name = foo
doc_id = 2, rel_id = 5, rel_name = humpty
There are multiple relationships like this for a single Document, e.g.
Document_Rel2, Document_Rel3.
I'm getting errors on cube builds:
java.lang.IllegalStateException: Dup key found, key=[1],
value1=[1,1,foo], value2=[1,2,bar]
at org.apache.kylin.dict.lookup.LookupTable.initRow(LookupTable.java:81)
at org.apache.kylin.dict.lookup.LookupTable.init(LookupTable.java:64)
at org.apache.kylin.dict.lookup.LookupTable.<init>(LookupTable.java:52)
at org.apache.kylin.dict.lookup.LookupStringTable.<init>(LookupStringTable.java:32)
at org.apache.kylin.cube.CubeManager.getLookupTable(CubeManager.java:419)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:63)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42)
at org.apache.kylin.job.hadoop.dict.CreateDictionaryJob.run(CreateDictionaryJob.java:53)
-TPP