You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by raghu <ra...@gmail.com> on 2020/02/01 00:28:14 UTC
Kylin cube build failing at #12 Step Name: Build N-Dimension Cuboid
Data : 22-Dimension
One of our cubes is failing at "#12 Step Name: Build N-Dimension Cuboid Data
: 22-Dimension" step and irrespective of how much i bump up the mapper and
reducer memory and other configs, the jobs fails at this step everytime.
Looked at the yarn application logs and it doesnt give any useful info
there. Can someone help.
The cube has 8 measures 27 dimensions. Source data is ~42 million.
FYI.. this cube has been running fine for a few months now, its just started
failing a few days back and the data has not increased by much as well.
Kylin version= 1.6.0
Hortonworks = 2.4.0
Yarn MR Error:
java.lang.RuntimeException: java.io.IOException: I failed to find the one of
the right cookies.
at
org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:61)
at
org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:30)
at org.apache.kylin.measure.MeasureCodec.decode(MeasureCodec.java:97)
at
org.apache.kylin.measure.BufferedMeasureCodec.decode(BufferedMeasureCodec.java:79)
at
org.apache.kylin.engine.mr.steps.CuboidReducer.reduce(CuboidReducer.java:93)
at
org.apache.kylin.engine.mr.steps.CuboidReducer.reduce(CuboidReducer.java:42)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1688)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1637)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.io.IOException: I failed to find the one of the right
cookies.
at
org.roaringbitmap.buffer.MutableRoaringArray.deserialize(MutableRoaringArray.java:218)
at
org.roaringbitmap.buffer.MutableRoaringBitmap.deserialize(MutableRoaringBitmap.java:829)
at
org.apache.kylin.measure.bitmap.BitmapCounter.readRegisters(BitmapCounter.java:113)
at
org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:59)
... 17 more
2020-01-31 21:49:16,955 INFO [IPC Server handler 24 on 38174]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from
attempt_1568236776824_87076_m_000900_0: Error: java.lang.RuntimeException:
java.io.IOException: I failed to find the one of the right cookies.
at
org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:61)
at
org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:30)
at org.apache.kylin.measure.MeasureCodec.decode(MeasureCodec.java:97)
at
org.apache.kylin.measure.BufferedMeasureCodec.decode(BufferedMeasureCodec.java:79)
at
org.apache.kylin.engine.mr.steps.CuboidReducer.reduce(CuboidReducer.java:93)
at
org.apache.kylin.engine.mr.steps.CuboidReducer.reduce(CuboidReducer.java:42)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1688)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1637)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.io.IOException: I failed to find the one of the right
cookies.
at
org.roaringbitmap.buffer.MutableRoaringArray.deserialize(MutableRoaringArray.java:218)
at
org.roaringbitmap.buffer.MutableRoaringBitmap.deserialize(MutableRoaringBitmap.java:829)
at
org.apache.kylin.measure.bitmap.BitmapCounter.readRegisters(BitmapCounter.java:113)
at
org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:59)
... 17 more
--
Sent from: http://apache-kylin.74782.x6.nabble.com/
Re: Kylin cube build failing at #12 Step Name: Build N-Dimension Cuboid Data : 22-Dimension
Posted by Raghu Ram Reddy Medapati <ra...@gmail.com>.
How do we decrease the cardinality of a column?, We need those two ID columns as measures and we want precise count, so im not sure how we can decrease the cardinality of the columns.
These two columns are only added in measures and not in Aggregation groups and Row keys, then how do we change the encodings type? When we select a column in measure with precise count distinct , the default return type is "bitmap" and there is no option to change it.
Any suggestions will greatly help.
On 2020/02/01 13:06:13, ShaoFeng Shi <sh...@apache.org> wrote:
> I have no idea either; Kylin 1.6 is a very old version. We didn't get such
> reporting before.
>
> BTW, you may need to look at the cube design (as you already increase the
> node memory to a very big number), especially for the UHC dimensions. Try
> to decrease the cardinality, or using other encodings like fixed-length
> encoding.
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Email: shaofengshi@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscribe@kylin.apache.org
> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
>
>
>
>
> Raghu Ram Reddy Medapati <ra...@gmail.com> 于2020年2月1日周六 下午8:06写道：
>
> > Hi Shi,
> >
> > Thanks for the quick reply.
> > I haven't changed any configurations lately. We always had
> > "kylin.hbase.default.compression.codec=snappy" and this has been there for
> > 3 years.
> > All the other cubes build fine and we are able to query this cube and
> > other cubes fine as well.
> > The data has increased by only about 200K records for this cube and
> > suddenly it started failing at "Build DimensionDictionary" step this week.
> > I bumped up the ec2 instance of job node from 64 GB to 160 GB and the build
> > dimension dictionary step is running fine now but the cube is failing at
> > "#12 Step Name: Build N-Dimension Cuboid Data : 22-Dimension"
> > Any help would be greatly appreciated.
> >
> > On 2020/02/01 02:06:45, ShaoFeng Shi <sh...@apache.org> wrote:
> > > I encountered the same error just a couple of days ago on Kylin 3.0. But
> > I
> > > don't think it is the same root cause, because my error is happened in
> > > query time. And the reason is the different compression settings:
> > >
> > > https://issues.apache.org/jira/browse/KYLIN-4363
> > >
> > > Did you make any code/configuration change for Kylin or Hadoop in the
> > > between?
> > >
> > > Best regards,
> > >
> > > Shaofeng Shi 史少锋
> > > Apache Kylin PMC
> > > Email: shaofengshi@apache.org
> > >
> > > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> > > Join Kylin user mail group: user-subscribe@kylin.apache.org
> > > Join Kylin dev mail group: dev-subscribe@kylin.apache.org
> > >
> > >
> > >
> > >
> > > raghu <ra...@gmail.com> 于2020年2月1日周六 上午8:30写道：
> > >
> > > > One of our cubes is failing at "#12 Step Name: Build N-Dimension Cuboid
> > > > Data
> > > > : 22-Dimension" step and irrespective of how much i bump up the mapper
> > and
> > > > reducer memory and other configs, the jobs fails at this step
> > everytime.
> > > > Looked at the yarn application logs and it doesnt give any useful info
> > > > there. Can someone help.
> > > > The cube has 8 measures 27 dimensions. Source data is ~42 million.
> > > > FYI.. this cube has been running fine for a few months now, its just
> > > > started
> > > > failing a few days back and the data has not increased by much as well.
> > > > Kylin version= 1.6.0
> > > > Hortonworks = 2.4.0
> > > >
> > > > Yarn MR Error:
> > > > java.lang.RuntimeException: java.io.IOException: I failed to find the
> > one
> > > > of
> > > > the right cookies.
> > > > at
> > > >
> > > >
> > org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:61)
> > > > at
> > > >
> > > >
> > org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:30)
> > > > at
> > > > org.apache.kylin.measure.MeasureCodec.decode(MeasureCodec.java:97)
> > > > at
> > > >
> > > >
> > org.apache.kylin.measure.BufferedMeasureCodec.decode(BufferedMeasureCodec.java:79)
> > > > at
> > > > org.apache.kylin.engine.mr
> > > > .steps.CuboidReducer.reduce(CuboidReducer.java:93)
> > > > at
> > > > org.apache.kylin.engine.mr
> > > > .steps.CuboidReducer.reduce(CuboidReducer.java:42)
> > > > at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
> > > > at
> > > > org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1688)
> > > > at
> > > >
> > > >
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1637)
> > > > at
> > > >
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489)
> > > > at
> > > >
> > org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
> > > > at
> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> > > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> > > > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > at javax.security.auth.Subject.doAs(Subject.java:422)
> > > > at
> > > >
> > > >
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
> > > > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> > > > Caused by: java.io.IOException: I failed to find the one of the right
> > > > cookies.
> > > > at
> > > >
> > > >
> > org.roaringbitmap.buffer.MutableRoaringArray.deserialize(MutableRoaringArray.java:218)
> > > > at
> > > >
> > > >
> > org.roaringbitmap.buffer.MutableRoaringBitmap.deserialize(MutableRoaringBitmap.java:829)
> > > > at
> > > >
> > > >
> > org.apache.kylin.measure.bitmap.BitmapCounter.readRegisters(BitmapCounter.java:113)
> > > > at
> > > >
> > > >
> > org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:59)
> > > > ... 17 more
> > > >
> > > > 2020-01-31 21:49:16,955 INFO [IPC Server handler 24 on 38174]
> > > > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report
> > from
> > > > attempt_1568236776824_87076_m_000900_0: Error:
> > java.lang.RuntimeException:
> > > > java.io.IOException: I failed to find the one of the right cookies.
> > > > at
> > > >
> > > >
> > org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:61)
> > > > at
> > > >
> > > >
> > org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:30)
> > > > at
> > > > org.apache.kylin.measure.MeasureCodec.decode(MeasureCodec.java:97)
> > > > at
> > > >
> > > >
> > org.apache.kylin.measure.BufferedMeasureCodec.decode(BufferedMeasureCodec.java:79)
> > > > at
> > > > org.apache.kylin.engine.mr
> > > > .steps.CuboidReducer.reduce(CuboidReducer.java:93)
> > > > at
> > > > org.apache.kylin.engine.mr
> > > > .steps.CuboidReducer.reduce(CuboidReducer.java:42)
> > > > at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
> > > > at
> > > > org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1688)
> > > > at
> > > >
> > > >
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1637)
> > > > at
> > > >
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489)
> > > > at
> > > >
> > org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
> > > > at
> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> > > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> > > > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > at javax.security.auth.Subject.doAs(Subject.java:422)
> > > > at
> > > >
> > > >
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
> > > > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> > > > Caused by: java.io.IOException: I failed to find the one of the right
> > > > cookies.
> > > > at
> > > >
> > > >
> > org.roaringbitmap.buffer.MutableRoaringArray.deserialize(MutableRoaringArray.java:218)
> > > > at
> > > >
> > > >
> > org.roaringbitmap.buffer.MutableRoaringBitmap.deserialize(MutableRoaringBitmap.java:829)
> > > > at
> > > >
> > > >
> > org.apache.kylin.measure.bitmap.BitmapCounter.readRegisters(BitmapCounter.java:113)
> > > > at
> > > >
> > > >
> > org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:59)
> > > > ... 17 more
> > > >
> > > >
> > > > --
> > > > Sent from: http://apache-kylin.74782.x6.nabble.com/
> > > >
> > >
> >
>
Re: Kylin cube build failing at #12 Step Name: Build N-Dimension
Cuboid Data : 22-Dimension
Posted by ShaoFeng Shi <sh...@apache.org>.
I have no idea either; Kylin 1.6 is a very old version. We didn't get such
reporting before.
BTW, you may need to look at the cube design (as you already increase the
node memory to a very big number), especially for the UHC dimensions. Try
to decrease the cardinality, or using other encodings like fixed-length
encoding.
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org
Raghu Ram Reddy Medapati <ra...@gmail.com> 于2020年2月1日周六 下午8:06写道：
> Hi Shi,
>
> Thanks for the quick reply.
> I haven't changed any configurations lately. We always had
> "kylin.hbase.default.compression.codec=snappy" and this has been there for
> 3 years.
> All the other cubes build fine and we are able to query this cube and
> other cubes fine as well.
> The data has increased by only about 200K records for this cube and
> suddenly it started failing at "Build DimensionDictionary" step this week.
> I bumped up the ec2 instance of job node from 64 GB to 160 GB and the build
> dimension dictionary step is running fine now but the cube is failing at
> "#12 Step Name: Build N-Dimension Cuboid Data : 22-Dimension"
> Any help would be greatly appreciated.
>
> On 2020/02/01 02:06:45, ShaoFeng Shi <sh...@apache.org> wrote:
> > I encountered the same error just a couple of days ago on Kylin 3.0. But
> I
> > don't think it is the same root cause, because my error is happened in
> > query time. And the reason is the different compression settings:
> >
> > https://issues.apache.org/jira/browse/KYLIN-4363
> >
> > Did you make any code/configuration change for Kylin or Hadoop in the
> > between?
> >
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> > Apache Kylin PMC
> > Email: shaofengshi@apache.org
> >
> > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> > Join Kylin user mail group: user-subscribe@kylin.apache.org
> > Join Kylin dev mail group: dev-subscribe@kylin.apache.org
> >
> >
> >
> >
> > raghu <ra...@gmail.com> 于2020年2月1日周六 上午8:30写道：
> >
> > > One of our cubes is failing at "#12 Step Name: Build N-Dimension Cuboid
> > > Data
> > > : 22-Dimension" step and irrespective of how much i bump up the mapper
> and
> > > reducer memory and other configs, the jobs fails at this step
> everytime.
> > > Looked at the yarn application logs and it doesnt give any useful info
> > > there. Can someone help.
> > > The cube has 8 measures 27 dimensions. Source data is ~42 million.
> > > FYI.. this cube has been running fine for a few months now, its just
> > > started
> > > failing a few days back and the data has not increased by much as well.
> > > Kylin version= 1.6.0
> > > Hortonworks = 2.4.0
> > >
> > > Yarn MR Error:
> > > java.lang.RuntimeException: java.io.IOException: I failed to find the
> one
> > > of
> > > the right cookies.
> > > at
> > >
> > >
> org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:61)
> > > at
> > >
> > >
> org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:30)
> > > at
> > > org.apache.kylin.measure.MeasureCodec.decode(MeasureCodec.java:97)
> > > at
> > >
> > >
> org.apache.kylin.measure.BufferedMeasureCodec.decode(BufferedMeasureCodec.java:79)
> > > at
> > > org.apache.kylin.engine.mr
> > > .steps.CuboidReducer.reduce(CuboidReducer.java:93)
> > > at
> > > org.apache.kylin.engine.mr
> > > .steps.CuboidReducer.reduce(CuboidReducer.java:42)
> > > at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
> > > at
> > > org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1688)
> > > at
> > >
> > >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1637)
> > > at
> > >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489)
> > > at
> > >
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
> > > at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> > > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at javax.security.auth.Subject.doAs(Subject.java:422)
> > > at
> > >
> > >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
> > > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> > > Caused by: java.io.IOException: I failed to find the one of the right
> > > cookies.
> > > at
> > >
> > >
> org.roaringbitmap.buffer.MutableRoaringArray.deserialize(MutableRoaringArray.java:218)
> > > at
> > >
> > >
> org.roaringbitmap.buffer.MutableRoaringBitmap.deserialize(MutableRoaringBitmap.java:829)
> > > at
> > >
> > >
> org.apache.kylin.measure.bitmap.BitmapCounter.readRegisters(BitmapCounter.java:113)
> > > at
> > >
> > >
> org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:59)
> > > ... 17 more
> > >
> > > 2020-01-31 21:49:16,955 INFO [IPC Server handler 24 on 38174]
> > > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report
> from
> > > attempt_1568236776824_87076_m_000900_0: Error:
> java.lang.RuntimeException:
> > > java.io.IOException: I failed to find the one of the right cookies.
> > > at
> > >
> > >
> org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:61)
> > > at
> > >
> > >
> org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:30)
> > > at
> > > org.apache.kylin.measure.MeasureCodec.decode(MeasureCodec.java:97)
> > > at
> > >
> > >
> org.apache.kylin.measure.BufferedMeasureCodec.decode(BufferedMeasureCodec.java:79)
> > > at
> > > org.apache.kylin.engine.mr
> > > .steps.CuboidReducer.reduce(CuboidReducer.java:93)
> > > at
> > > org.apache.kylin.engine.mr
> > > .steps.CuboidReducer.reduce(CuboidReducer.java:42)
> > > at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
> > > at
> > > org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1688)
> > > at
> > >
> > >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1637)
> > > at
> > >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489)
> > > at
> > >
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
> > > at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> > > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at javax.security.auth.Subject.doAs(Subject.java:422)
> > > at
> > >
> > >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
> > > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> > > Caused by: java.io.IOException: I failed to find the one of the right
> > > cookies.
> > > at
> > >
> > >
> org.roaringbitmap.buffer.MutableRoaringArray.deserialize(MutableRoaringArray.java:218)
> > > at
> > >
> > >
> org.roaringbitmap.buffer.MutableRoaringBitmap.deserialize(MutableRoaringBitmap.java:829)
> > > at
> > >
> > >
> org.apache.kylin.measure.bitmap.BitmapCounter.readRegisters(BitmapCounter.java:113)
> > > at
> > >
> > >
> org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:59)
> > > ... 17 more
> > >
> > >
> > > --
> > > Sent from: http://apache-kylin.74782.x6.nabble.com/
> > >
> >
>
Re: Kylin cube build failing at #12 Step Name: Build N-Dimension Cuboid Data : 22-Dimension
Posted by Raghu Ram Reddy Medapati <ra...@gmail.com>.
Hi Shi,
Thanks for the quick reply.
I haven't changed any configurations lately. We always had "kylin.hbase.default.compression.codec=snappy" and this has been there for 3 years.
All the other cubes build fine and we are able to query this cube and other cubes fine as well.
The data has increased by only about 200K records for this cube and suddenly it started failing at "Build DimensionDictionary" step this week. I bumped up the ec2 instance of job node from 64 GB to 160 GB and the build dimension dictionary step is running fine now but the cube is failing at "#12 Step Name: Build N-Dimension Cuboid Data : 22-Dimension"
Any help would be greatly appreciated.
On 2020/02/01 02:06:45, ShaoFeng Shi <sh...@apache.org> wrote:
> I encountered the same error just a couple of days ago on Kylin 3.0. But I
> don't think it is the same root cause, because my error is happened in
> query time. And the reason is the different compression settings:
>
> https://issues.apache.org/jira/browse/KYLIN-4363
>
> Did you make any code/configuration change for Kylin or Hadoop in the
> between?
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Email: shaofengshi@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscribe@kylin.apache.org
> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
>
>
>
>
> raghu <ra...@gmail.com> 于2020年2月1日周六 上午8:30写道：
>
> > One of our cubes is failing at "#12 Step Name: Build N-Dimension Cuboid
> > Data
> > : 22-Dimension" step and irrespective of how much i bump up the mapper and
> > reducer memory and other configs, the jobs fails at this step everytime.
> > Looked at the yarn application logs and it doesnt give any useful info
> > there. Can someone help.
> > The cube has 8 measures 27 dimensions. Source data is ~42 million.
> > FYI.. this cube has been running fine for a few months now, its just
> > started
> > failing a few days back and the data has not increased by much as well.
> > Kylin version= 1.6.0
> > Hortonworks = 2.4.0
> >
> > Yarn MR Error:
> > java.lang.RuntimeException: java.io.IOException: I failed to find the one
> > of
> > the right cookies.
> > at
> >
> > org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:61)
> > at
> >
> > org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:30)
> > at
> > org.apache.kylin.measure.MeasureCodec.decode(MeasureCodec.java:97)
> > at
> >
> > org.apache.kylin.measure.BufferedMeasureCodec.decode(BufferedMeasureCodec.java:79)
> > at
> > org.apache.kylin.engine.mr
> > .steps.CuboidReducer.reduce(CuboidReducer.java:93)
> > at
> > org.apache.kylin.engine.mr
> > .steps.CuboidReducer.reduce(CuboidReducer.java:42)
> > at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
> > at
> > org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1688)
> > at
> >
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1637)
> > at
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489)
> > at
> > org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:422)
> > at
> >
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
> > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> > Caused by: java.io.IOException: I failed to find the one of the right
> > cookies.
> > at
> >
> > org.roaringbitmap.buffer.MutableRoaringArray.deserialize(MutableRoaringArray.java:218)
> > at
> >
> > org.roaringbitmap.buffer.MutableRoaringBitmap.deserialize(MutableRoaringBitmap.java:829)
> > at
> >
> > org.apache.kylin.measure.bitmap.BitmapCounter.readRegisters(BitmapCounter.java:113)
> > at
> >
> > org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:59)
> > ... 17 more
> >
> > 2020-01-31 21:49:16,955 INFO [IPC Server handler 24 on 38174]
> > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from
> > attempt_1568236776824_87076_m_000900_0: Error: java.lang.RuntimeException:
> > java.io.IOException: I failed to find the one of the right cookies.
> > at
> >
> > org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:61)
> > at
> >
> > org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:30)
> > at
> > org.apache.kylin.measure.MeasureCodec.decode(MeasureCodec.java:97)
> > at
> >
> > org.apache.kylin.measure.BufferedMeasureCodec.decode(BufferedMeasureCodec.java:79)
> > at
> > org.apache.kylin.engine.mr
> > .steps.CuboidReducer.reduce(CuboidReducer.java:93)
> > at
> > org.apache.kylin.engine.mr
> > .steps.CuboidReducer.reduce(CuboidReducer.java:42)
> > at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
> > at
> > org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1688)
> > at
> >
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1637)
> > at
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489)
> > at
> > org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:422)
> > at
> >
> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
> > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> > Caused by: java.io.IOException: I failed to find the one of the right
> > cookies.
> > at
> >
> > org.roaringbitmap.buffer.MutableRoaringArray.deserialize(MutableRoaringArray.java:218)
> > at
> >
> > org.roaringbitmap.buffer.MutableRoaringBitmap.deserialize(MutableRoaringBitmap.java:829)
> > at
> >
> > org.apache.kylin.measure.bitmap.BitmapCounter.readRegisters(BitmapCounter.java:113)
> > at
> >
> > org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:59)
> > ... 17 more
> >
> >
> > --
> > Sent from: http://apache-kylin.74782.x6.nabble.com/
> >
>
Re: Kylin cube build failing at #12 Step Name: Build N-Dimension
Cuboid Data : 22-Dimension
Posted by raghu <ra...@gmail.com>.
Thanks for your reply.
Nope, nothing has changed, i mean we didn't change any kylin configuration
recently.
We are using snappy compression for hbase tables
"kylin.hbase.default.compression.codec=snappy" and this has been there for
almost 3 years.
All the other cubes are running fine with same configs.
This cube has been having issues with build dimension dictionary step,
there are a couple of measures of UHC columns on this cube with ~40 million
records. To make the cube build, i bumped up the kylin job node from 64 GB
to 160 GB node and the build dimension step is running fine now but the cube
is failing at "#12 Step Name: Build N-Dimension Cuboid Data : 22-Dimension"
Also another weird thing is the cube has been building fine until last week,
the data increased by only about 200K records.
Any help is greatly appreciated.
--
Sent from: http://apache-kylin.74782.x6.nabble.com/
Re: Kylin cube build failing at #12 Step Name: Build N-Dimension
Cuboid Data : 22-Dimension
Posted by ShaoFeng Shi <sh...@apache.org>.
I encountered the same error just a couple of days ago on Kylin 3.0. But I
don't think it is the same root cause, because my error is happened in
query time. And the reason is the different compression settings:
https://issues.apache.org/jira/browse/KYLIN-4363
Did you make any code/configuration change for Kylin or Hadoop in the
between?
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org
raghu <ra...@gmail.com> 于2020年2月1日周六 上午8:30写道：
> One of our cubes is failing at "#12 Step Name: Build N-Dimension Cuboid
> Data
> : 22-Dimension" step and irrespective of how much i bump up the mapper and
> reducer memory and other configs, the jobs fails at this step everytime.
> Looked at the yarn application logs and it doesnt give any useful info
> there. Can someone help.
> The cube has 8 measures 27 dimensions. Source data is ~42 million.
> FYI.. this cube has been running fine for a few months now, its just
> started
> failing a few days back and the data has not increased by much as well.
> Kylin version= 1.6.0
> Hortonworks = 2.4.0
>
> Yarn MR Error:
> java.lang.RuntimeException: java.io.IOException: I failed to find the one
> of
> the right cookies.
> at
>
> org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:61)
> at
>
> org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:30)
> at
> org.apache.kylin.measure.MeasureCodec.decode(MeasureCodec.java:97)
> at
>
> org.apache.kylin.measure.BufferedMeasureCodec.decode(BufferedMeasureCodec.java:79)
> at
> org.apache.kylin.engine.mr
> .steps.CuboidReducer.reduce(CuboidReducer.java:93)
> at
> org.apache.kylin.engine.mr
> .steps.CuboidReducer.reduce(CuboidReducer.java:42)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
> at
> org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1688)
> at
>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1637)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: java.io.IOException: I failed to find the one of the right
> cookies.
> at
>
> org.roaringbitmap.buffer.MutableRoaringArray.deserialize(MutableRoaringArray.java:218)
> at
>
> org.roaringbitmap.buffer.MutableRoaringBitmap.deserialize(MutableRoaringBitmap.java:829)
> at
>
> org.apache.kylin.measure.bitmap.BitmapCounter.readRegisters(BitmapCounter.java:113)
> at
>
> org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:59)
> ... 17 more
>
> 2020-01-31 21:49:16,955 INFO [IPC Server handler 24 on 38174]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from
> attempt_1568236776824_87076_m_000900_0: Error: java.lang.RuntimeException:
> java.io.IOException: I failed to find the one of the right cookies.
> at
>
> org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:61)
> at
>
> org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:30)
> at
> org.apache.kylin.measure.MeasureCodec.decode(MeasureCodec.java:97)
> at
>
> org.apache.kylin.measure.BufferedMeasureCodec.decode(BufferedMeasureCodec.java:79)
> at
> org.apache.kylin.engine.mr
> .steps.CuboidReducer.reduce(CuboidReducer.java:93)
> at
> org.apache.kylin.engine.mr
> .steps.CuboidReducer.reduce(CuboidReducer.java:42)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
> at
> org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1688)
> at
>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1637)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: java.io.IOException: I failed to find the one of the right
> cookies.
> at
>
> org.roaringbitmap.buffer.MutableRoaringArray.deserialize(MutableRoaringArray.java:218)
> at
>
> org.roaringbitmap.buffer.MutableRoaringBitmap.deserialize(MutableRoaringBitmap.java:829)
> at
>
> org.apache.kylin.measure.bitmap.BitmapCounter.readRegisters(BitmapCounter.java:113)
> at
>
> org.apache.kylin.measure.bitmap.BitmapSerializer.deserialize(BitmapSerializer.java:59)
> ... 17 more
>
>
> --
> Sent from: http://apache-kylin.74782.x6.nabble.com/
>