[jira] [Created] (KYLIN-2749) Merge may remove merged segments before proper register of merged segment

["Alexander Sterligov (JIRA)" <ji...@apache.org>]

Alexander Sterligov created KYLIN-2749:
------------------------------------------

             Summary: Merge may remove merged segments before proper register of merged segment
                 Key: KYLIN-2749
                 URL: https://issues.apache.org/jira/browse/KYLIN-2749
             Project: Kylin
          Issue Type: Bug
    Affects Versions: v2.0.0
            Reporter: Alexander Sterligov


Merge started to work on last 7 segments.
During the process hbase had a failure because of spot-instances shutdown in Amazon. Data was not lost, because it is at S3.
I stopped kylin and did hbase hbck --repair. During the report of repair I didn't see any information about lost data, just redistribution of regions.

Then after kylin was started I cannot query data from the last 7 query segments with error:
{quote}
Caused by: java.lang.RuntimeException: org.apache.hadoop.hbase.TableNotFoundException: Table 'KYLIN_7MMHCHKVVB' was not found, got: KYLIN_7H3WSPX1UJ.
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:67)
        at org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:31)
        at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
        at com.google.common.collect.Iterators$6.hasNext(Iterators.java:583)
        at org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator$2.hasNext(SegmentCubeTupleIterator.java:116)
        at org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator.hasNext(SegmentCubeTupleIterator.java:149)
        at com.google.common.collect.Iterators$6.hasNext(Iterators.java:582)
        at org.apache.kylin.storage.gtrecord.SequentialCubeTupleIterator.hasNext(SequentialCubeTupleIterator.java:129)
        at org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:67)
        at Baz$1$1.moveNext(Unknown Source)
        at org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:826)
        at org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:761)
        at org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
        at Baz.bind(Unknown Source)
        at org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:331)
        at org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:294)
        at org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:553)
        at org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:544)
        at org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:193)
        at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
        at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
        at org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:607)
        at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:600)
        at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:615)
        at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:148)
        ... 77 more
Caused by: org.apache.hadoop.hbase.TableNotFoundException: Table 'KYLIN_7MMHCHKVVB' was not found, got: KYLIN_7H3WSPX1UJ.
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1310)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1189)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1173)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1130)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:965)
        at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83)
        at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:505)
        at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:721)
        at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:691)
        at org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1796)
        at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1751)
        at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1.run(CubeHBaseEndpointRPC.java:182)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        ... 1 more
{quote}

hbase really doesn't contain tables for the last 7 segments and I didn't call any cleanup jobs.
It looks like Merge removed old segment tables, without ensuring that merged segment is well presented in hbase.

I'm going to continue to investigate this problem and will post more details next week.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)