You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "Alexander Sterligov (JIRA)" <ji...@apache.org> on 2017/07/22 10:56:00 UTC
[jira] [Updated] (KYLIN-2749) Merge may remove old segments without
saving merged segment
[ https://issues.apache.org/jira/browse/KYLIN-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Sterligov updated KYLIN-2749:
---------------------------------------
Summary: Merge may remove old segments without saving merged segment (was: Merge may remove merged segments before proper register of merged segment)
> Merge may remove old segments without saving merged segment
> -----------------------------------------------------------
>
> Key: KYLIN-2749
> URL: https://issues.apache.org/jira/browse/KYLIN-2749
> Project: Kylin
> Issue Type: Bug
> Affects Versions: v2.0.0
> Reporter: Alexander Sterligov
>
> Merge started to work on last 7 segments.
> During the process hbase had a failure because of spot-instances shutdown in Amazon. Data was not lost, because it is at S3.
> I stopped kylin and did hbase hbck --repair. During the report of repair I didn't see any information about lost data, just redistribution of regions.
> Then after kylin was started I cannot query data from the last 7 query segments with error:
> {quote}
> Caused by: java.lang.RuntimeException: org.apache.hadoop.hbase.TableNotFoundException: Table 'KYLIN_7MMHCHKVVB' was not found, got: KYLIN_7H3WSPX1UJ.
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:67)
> at org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:31)
> at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at com.google.common.collect.Iterators$6.hasNext(Iterators.java:583)
> at org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator$2.hasNext(SegmentCubeTupleIterator.java:116)
> at org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator.hasNext(SegmentCubeTupleIterator.java:149)
> at com.google.common.collect.Iterators$6.hasNext(Iterators.java:582)
> at org.apache.kylin.storage.gtrecord.SequentialCubeTupleIterator.hasNext(SequentialCubeTupleIterator.java:129)
> at org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:67)
> at Baz$1$1.moveNext(Unknown Source)
> at org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:826)
> at org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:761)
> at org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
> at Baz.bind(Unknown Source)
> at org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:331)
> at org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:294)
> at org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:553)
> at org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:544)
> at org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:193)
> at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
> at org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:607)
> at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:600)
> at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:615)
> at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:148)
> ... 77 more
> Caused by: org.apache.hadoop.hbase.TableNotFoundException: Table 'KYLIN_7MMHCHKVVB' was not found, got: KYLIN_7H3WSPX1UJ.
> at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1310)
> at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1189)
> at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1173)
> at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1130)
> at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:965)
> at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83)
> at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:505)
> at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:721)
> at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:691)
> at org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1796)
> at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1751)
> at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1.run(CubeHBaseEndpointRPC.java:182)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ... 1 more
> {quote}
> hbase really doesn't contain tables for the last 7 segments and I didn't call any cleanup jobs.
> It looks like Merge removed old segment tables, without ensuring that merged segment is well presented in hbase.
> I'm going to continue to investigate this problem and will post more details next week.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)