You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Clément Guillaume <cl...@guillaume.bzh> on 2017/12/21 20:28:10 UTC
assert in hbase 1.4
I'm upgrading my projects from 1.2.6 to 1.4, and I have a system assertion
failure (in a failsafe test that run with system assertion enable)
java.lang.AssertionError
at
org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:484)
~[hbase-shaded-client-1.4.0.jar:1.4.0]
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:312)
~[hbase-shaded-client-1.4.0.jar:1.4.0]
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1324)
~[hbase-shaded-client-1.4.0.jar:1.4.0]
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1221)
~[hbase-shaded-client-1.4.0.jar:1.4.0]
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncProcess.java:1063)
~[hbase-shaded-client-1.4.0.jar:1.4.0]
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:980)
~[hbase-shaded-client-1.4.0.jar:1.4.0]
at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$200(AsyncProcess.java:667)
~[hbase-shaded-client-1.4.0.jar:1.4.0]
at
org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:649)
~[hbase-shaded-client-1.4.0.jar:1.4.0]
at
org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:612)
~[hbase-shaded-client-1.4.0.jar:1.4.0]
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.processBatchCallback(ConnectionManager.java:2392)
~[hbase-shaded-client-1.4.0.jar:1.4.0]
at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.java:950)
~[hbase-shaded-client-1.4.0.jar:1.4.0]
I have one Put action that look like that :
{"totalColumns":1,"row":"a7ttxdQY9euS-KEaPIMvKsRUWANCmigsaiT8_-V0Sgc\\x00visited\\x00\\xFF\\xFF\\xFE\\x9F\\x85Dh\\x19","families":{"a":[{"qualifier":"value","vlen":17,"tag":[],"timestamp":9223372036854775807}]}}
The cluster is a local standalone instance.
I haven't dig into it yet, but it looks like the data is correctly written
in hbase if I disable assertion.
Any idea of what's causing the issue?
Thank you.
Re: assert in hbase 1.4
Posted by Ted Yu <yu...@gmail.com>.
Is it possible to come up with unit test which shows the assertion ?
Probably you can mimic the contents of hbase:meta when writing the unit
test.
Thanks
On Thu, Dec 21, 2017 at 12:28 PM, Clément Guillaume <cl...@guillaume.bzh>
wrote:
> I'm upgrading my projects from 1.2.6 to 1.4, and I have a system assertion
> failure (in a failsafe test that run with system assertion enable)
>
> java.lang.AssertionError
> at
> org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:
> 484)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> at
> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:312)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> at
> org.apache.hadoop.hbase.client.ConnectionManager$
> HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1324)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> at
> org.apache.hadoop.hbase.client.ConnectionManager$
> HConnectionImplementation.locateRegion(ConnectionManager.java:1221)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.
> findAllLocationsOrFail(AsyncProcess.java:1063)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.
> groupAndSendMultiAction(AsyncProcess.java:980)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$
> 200(AsyncProcess.java:667)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submitAll(
> AsyncProcess.java:649)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submitAll(
> AsyncProcess.java:612)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> at
> org.apache.hadoop.hbase.client.ConnectionManager$
> HConnectionImplementation.processBatchCallback(
> ConnectionManager.java:2392)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.java:950)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>
> I have one Put action that look like that :
> {"totalColumns":1,"row":"a7ttxdQY9euS-KEaPIMvKsRUWANCmigsaiT8_-
> V0Sgc\\x00visited\\x00\\xFF\\xFF\\xFE\\x9F\\x85Dh\\x19","
> families":{"a":[{"qualifier":"value","vlen":17,"tag":[],"timestamp":
> 9223372036854775807}]}}
> The cluster is a local standalone instance.
>
> I haven't dig into it yet, but it looks like the data is correctly written
> in hbase if I disable assertion.
> Any idea of what's causing the issue?
>
> Thank you.
>
Re: assert in hbase 1.4
Posted by Ted Yu <yu...@gmail.com>.
Thanks, Clément
On Wed, Jan 3, 2018 at 2:54 PM, Clément Guillaume <cl...@guillaume.bzh>
wrote:
> Done https://issues.apache.org/jira/browse/HBASE-19700
>
> 2017-12-31 8:16 GMT-08:00 Ted Yu <yu...@gmail.com>:
>
>> Clément:
>> Since you have reproduced the assertion error, can you log a JIRA ?
>>
>> Thanks
>>
>> On Sun, Dec 31, 2017 at 7:46 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> I modified the test so that the unit test runs against branch-1.4 :
>>>
>>> + Configuration hbaseConfig = TEST_UTIL.getConfiguration();
>>> + //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM,
>>> "hbase123.docker");
>>>
>>> However the test runs to completion without assertion - meaning the
>>> assertion wouldn't be triggered in homogeneous cluster.
>>>
>>> Here were the last few lines in the test output:
>>>
>>> 2017-12-31 07:33:36,865 INFO [Time-limited test]
>>> client.HBaseAdmin$CreateTableFuture(793): Created AssertionTest
>>> table created
>>> 2017-12-31 07:33:36,871 DEBUG [Time-limited test]
>>> client.ClientScanner(248): Advancing internal scanner to startKey at
>>> '\x01\x02\x03\x04\x05\x06\x07', inclusive
>>> 2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3]
>>> ipc.RpcConnection(126): Use SIMPLE authentication for service
>>> ClientService, sasl=false
>>> 2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3]
>>> ipc.BlockingRpcConnection(431): Connecting to
>>> tyus-mbp.attlocal.net/192.168.1.64:61585
>>> 2017-12-31 07:33:36,875 DEBUG [RpcServer.listener,port=61585]
>>> ipc.RpcServer$Listener(920): RpcServer.listener,port=61585: connection from
>>> 192.168.1.64:61633; # active connections: 3
>>> 2017-12-31 07:33:36,875 INFO [RpcServer.reader=0,bindAddress=
>>> tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1809): Auth
>>> successful for tyu (auth:SIMPLE)
>>> 2017-12-31 07:33:36,875 INFO [RpcServer.reader=0,bindAddress=
>>> tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1839):
>>> Connection from 192.168.1.64 port: 61633 with version info: version:
>>> "1.4.1-SNAPSHOT" url: "git://TYus-MBP.attlocal.net/Users/tyu/1.4-hbase"
>>> revision: "a7383851b9e62f23eb6f06eb23f09967f636c985" user: "tyu" date:
>>> "Sun Dec 31 07:32:58 PST 2017" src_checksum: "39bdbbe40374e2094f53ce77af539671"
>>> version_major: 1 version_minor: 4
>>> 2017-12-31 07:33:36,917 INFO [main] hbase.ResourceChecker(171): after:
>>> client.TestHCM#test Thread=325 (was 305)
>>>
>>> I assume you pointed the test at a 1.2.3 cluster which triggered the
>>> assertion.
>>>
>>> It seems the patch which introduced the assertion didn't take into
>>> account client running against older release cluster deployment.
>>>
>>> FYI
>>>
>>> On Sun, Dec 31, 2017 at 6:09 AM, Clément Guillaume <
>>> clement@guillaume.bzh> wrote:
>>>
>>>> I have consistent failure with this code (a table with pre-splits):
>>>>
>>>> @Test
>>>>> public void test() throws IOException{
>>>>> Configuration hbaseConfig = HBaseConfiguration.create();
>>>>> hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker");
>>>>> Connection connection = ConnectionFactory.createConnec
>>>>> tion(hbaseConfig);
>>>>>
>>>>
>>>>
>>>> TableName tableName = TableName.valueOf("AssertionTest");
>>>>> Admin admin = connection.getAdmin();
>>>>>
>>>>> if(!admin.tableExists(tableName)){
>>>>> HTableDescriptor htable = new HTableDescriptor(tableName);
>>>>> htable.addFamily(new HColumnDescriptor(new byte[]{(byte)'a'}));
>>>>> byte[][] splitPoints = {{1, 2, 3, 4, 5, 6, 7}};
>>>>> admin.createTable(htable, splitPoints);
>>>>> System.out.println("table created");
>>>>> }
>>>>>
>>>>> Table table = connection.getTable(tableName);
>>>>> ResultScanner scanner = table.getScanner(new Scan());
>>>>> scanner.iterator().hasNext(); // Exception thrown here
>>>>> }
>>>>
>>>>
>>>> 2017-12-30 5:04 GMT-08:00 Ted Yu <yu...@gmail.com>:
>>>>
>>>>> bq. a table has been created with pre-splits
>>>>>
>>>>> Unless you record the initial number of regions somewhere, there is no
>>>>> built-in mechanism to provide the answer.
>>>>>
>>>>> Even if you cannot reproduce the assertion consistently, as long as
>>>>> the chance of reproduction is high, the unit test would still be useful.
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Sat, Dec 30, 2017 at 3:31 AM, Clément Guillaume <
>>>>> clement@guillaume.bzh> wrote:
>>>>>
>>>>>> I dropped all the tables in my test cluster. Now I'm inconstantly
>>>>>> able to reproduce the issue. I'm digging more.
>>>>>>
>>>>>> I have a related question: is it possible to check if a table has
>>>>>> been created with pre-splits?
>>>>>>
>>>>>> I forgot to mention that I'm connecting to a version 1.2.3 cluster,
>>>>>> but I think this should be ok according to this post
>>>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/2017
>>>>>> 11.mbox/%3C5BAAC90F-31D8-4A5F-B9E4-BA61FF4CD40E%40gmail.com%3E
>>>>>>
>>>>>> 2017-12-25 17:10 GMT-08:00 Ted Yu <yu...@gmail.com>:
>>>>>>
>>>>>>> I got the following if I use your code directly:
>>>>>>>
>>>>>>> testAssert(org.apache.hadoop.hbase.client.TestHCM) Time elapsed:
>>>>>>> 0.096 sec <<< ERROR!
>>>>>>> java.io.UncheckedIOException: org.apache.hadoop.hbase.TableNotFoundException:
>>>>>>> AssertionTest
>>>>>>> at org.apache.hadoop.hbase.client.TestHCM.testAssert(TestHCM.ja
>>>>>>> va:243)
>>>>>>>
>>>>>>> If I create the table at the beginning of the test, the test passes
>>>>>>> without triggering assertion.
>>>>>>>
>>>>>>> I tried both master branch and branch-1.4
>>>>>>>
>>>>>>> As I mentioned previously, the assertion in your case was probably
>>>>>>> related to the formation of row keys in hbase:meta table.
>>>>>>> See if you can reproduce the assertion by presplitting the table,
>>>>>>> etc.
>>>>>>>
>>>>>>> Here is the diff I used based on branch-1.4
>>>>>>>
>>>>>>> diff --git a/hbase-server/src/test/java/o
>>>>>>> rg/apache/hadoop/hbase/client/TestHCM.java
>>>>>>> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>>>>> TestHCM.java
>>>>>>> index a5e3a65..2944c8e 100644
>>>>>>> --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>>>>> TestHCM.java
>>>>>>> +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>>>>> TestHCM.java
>>>>>>> @@ -246,6 +246,18 @@ public class TestHCM {
>>>>>>> TEST_UTIL.shutdownMiniCluster();
>>>>>>> }
>>>>>>>
>>>>>>> + @Test
>>>>>>> + public void testAssert() throws IOException{
>>>>>>> + Configuration hbaseConfig = TEST_UTIL.getConfiguration();
>>>>>>> + final TableName tableName = TableName.valueOf("assert");
>>>>>>> + TEST_UTIL.createTable(tableName, FAM_NAM).close();
>>>>>>> + String zkQuorum = "hbase123.docker";
>>>>>>> + //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>>>>>>> + Connection connection = ConnectionFactory.createConnec
>>>>>>> tion(hbaseConfig);
>>>>>>> + Table table = connection.getTable(tableName);
>>>>>>> + ResultScanner scanner = table.getScanner(new Scan());
>>>>>>> + scanner.iterator().hasNext(); // Exception thrown here
>>>>>>> + }
>>>>>>>
>>>>>>> private static int getHConnectionManagerCacheSize(){
>>>>>>> return HConnectionTestingUtility.getConnectionCount();
>>>>>>>
>>>>>>> On Mon, Dec 25, 2017 at 4:16 PM, Clément Guillaume <
>>>>>>> clement@guillaume.bzh> wrote:
>>>>>>>
>>>>>>>> Here is a unit test, note that the AssertionTest table can exit or
>>>>>>>> not
>>>>>>>> exit, assertion will fail un both case.
>>>>>>>>
>>>>>>>> > @Test
>>>>>>>> > public void test() throws IOException{
>>>>>>>> > Configuration hbaseConfig = HBaseConfiguration.create();
>>>>>>>> > String zkQuorum = "hbase123.docker";
>>>>>>>> > hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>>>>>>>> > Connection connection = ConnectionFactory.createConnec
>>>>>>>> tion(hbaseConfig);
>>>>>>>> > Table table = connection.getTable(TableName.
>>>>>>>> valueOf("AssertionTest"));
>>>>>>>> > ResultScanner scanner = table.getScanner(new Scan());
>>>>>>>> > scanner.iterator().hasNext(); // Exception thrown here
>>>>>>>> > }
>>>>>>>> >
>>>>>>>>
>>>>>>>> PS: I'm not receiving email when a response is posted here. I'm not
>>>>>>>> subscribed at the hbase-user mailing list but I would have hope
>>>>>>>> that answer
>>>>>>>> get sent to me as direct email.
>>>>>>>>
>>>>>>>> 2017-12-21 12:28 GMT-08:00 Clément Guillaume <clement@guillaume.bzh
>>>>>>>> >:
>>>>>>>>
>>>>>>>> > I'm upgrading my projects from 1.2.6 to 1.4, and I have a system
>>>>>>>> assertion
>>>>>>>> > failure (in a failsafe test that run with system assertion enable)
>>>>>>>> >
>>>>>>>> > java.lang.AssertionError
>>>>>>>> > at org.apache.hadoop.hbase.client.ClientScanner.
>>>>>>>> > loadCache(ClientScanner.java:484) ~[hbase-shaded-client-1.4.0.ja
>>>>>>>> r:1.4.0]
>>>>>>>> > at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScan
>>>>>>>> ner.java:312)
>>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>>>>> > HConnectionImplementation.locateRegionInMeta(ConnectionManag
>>>>>>>> er.java:1324)
>>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>>>>> > HConnectionImplementation.locateRegion(ConnectionManager.jav
>>>>>>>> a:1221)
>>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu
>>>>>>>> reImpl.
>>>>>>>> > findAllLocationsOrFail(AsyncProcess.java:1063)
>>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu
>>>>>>>> reImpl.
>>>>>>>> > groupAndSendMultiAction(AsyncProcess.java:980)
>>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$
>>>>>>>> > AsyncRequestFutureImpl.access$200(AsyncProcess.java:667)
>>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>>>>>>>> rocess.java:649)
>>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>>>>>>>> rocess.java:612)
>>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>>>>> > HConnectionImplementation.processBatchCallback(ConnectionMan
>>>>>>>> ager.java:2392)
>>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>>> > at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.j
>>>>>>>> ava:950)
>>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>>> >
>>>>>>>> > I have one Put action that look like that :
>>>>>>>> {"totalColumns":1,"row":"
>>>>>>>> > a7ttxdQY9euS-KEaPIMvKsRUWANCmigsaiT8_-V0Sgc\\x00visited\\x00
>>>>>>>> \\xFF\\
>>>>>>>> > xFF\\xFE\\x9F\\x85Dh\\x19","families":{"a":[{"qualifier":"
>>>>>>>> > value","vlen":17,"tag":[],"timestamp":9223372036854775807}]}}
>>>>>>>> > The cluster is a local standalone instance.
>>>>>>>> >
>>>>>>>> > I haven't dig into it yet, but it looks like the data is
>>>>>>>> correctly written
>>>>>>>> > in hbase if I disable assertion.
>>>>>>>> > Any idea of what's causing the issue?
>>>>>>>> >
>>>>>>>> > Thank you.
>>>>>>>> >
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: assert in hbase 1.4
Posted by Clément Guillaume <cl...@guillaume.bzh>.
Done https://issues.apache.org/jira/browse/HBASE-19700
2017-12-31 8:16 GMT-08:00 Ted Yu <yu...@gmail.com>:
> Clément:
> Since you have reproduced the assertion error, can you log a JIRA ?
>
> Thanks
>
> On Sun, Dec 31, 2017 at 7:46 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> I modified the test so that the unit test runs against branch-1.4 :
>>
>> + Configuration hbaseConfig = TEST_UTIL.getConfiguration();
>> + //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker");
>>
>> However the test runs to completion without assertion - meaning the
>> assertion wouldn't be triggered in homogeneous cluster.
>>
>> Here were the last few lines in the test output:
>>
>> 2017-12-31 07:33:36,865 INFO [Time-limited test]
>> client.HBaseAdmin$CreateTableFuture(793): Created AssertionTest
>> table created
>> 2017-12-31 07:33:36,871 DEBUG [Time-limited test]
>> client.ClientScanner(248): Advancing internal scanner to startKey at
>> '\x01\x02\x03\x04\x05\x06\x07', inclusive
>> 2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3]
>> ipc.RpcConnection(126): Use SIMPLE authentication for service
>> ClientService, sasl=false
>> 2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3]
>> ipc.BlockingRpcConnection(431): Connecting to
>> tyus-mbp.attlocal.net/192.168.1.64:61585
>> 2017-12-31 07:33:36,875 DEBUG [RpcServer.listener,port=61585]
>> ipc.RpcServer$Listener(920): RpcServer.listener,port=61585: connection from
>> 192.168.1.64:61633; # active connections: 3
>> 2017-12-31 07:33:36,875 INFO [RpcServer.reader=0,bindAddress=
>> tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1809): Auth
>> successful for tyu (auth:SIMPLE)
>> 2017-12-31 07:33:36,875 INFO [RpcServer.reader=0,bindAddress=
>> tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1839):
>> Connection from 192.168.1.64 port: 61633 with version info: version:
>> "1.4.1-SNAPSHOT" url: "git://TYus-MBP.attlocal.net/Users/tyu/1.4-hbase"
>> revision: "a7383851b9e62f23eb6f06eb23f09967f636c985" user: "tyu" date:
>> "Sun Dec 31 07:32:58 PST 2017" src_checksum: "39bdbbe40374e2094f53ce77af539671"
>> version_major: 1 version_minor: 4
>> 2017-12-31 07:33:36,917 INFO [main] hbase.ResourceChecker(171): after:
>> client.TestHCM#test Thread=325 (was 305)
>>
>> I assume you pointed the test at a 1.2.3 cluster which triggered the
>> assertion.
>>
>> It seems the patch which introduced the assertion didn't take into
>> account client running against older release cluster deployment.
>>
>> FYI
>>
>> On Sun, Dec 31, 2017 at 6:09 AM, Clément Guillaume <clement@guillaume.bzh
>> > wrote:
>>
>>> I have consistent failure with this code (a table with pre-splits):
>>>
>>> @Test
>>>> public void test() throws IOException{
>>>> Configuration hbaseConfig = HBaseConfiguration.create();
>>>> hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker");
>>>> Connection connection = ConnectionFactory.createConnec
>>>> tion(hbaseConfig);
>>>>
>>>
>>>
>>> TableName tableName = TableName.valueOf("AssertionTest");
>>>> Admin admin = connection.getAdmin();
>>>>
>>>> if(!admin.tableExists(tableName)){
>>>> HTableDescriptor htable = new HTableDescriptor(tableName);
>>>> htable.addFamily(new HColumnDescriptor(new byte[]{(byte)'a'}));
>>>> byte[][] splitPoints = {{1, 2, 3, 4, 5, 6, 7}};
>>>> admin.createTable(htable, splitPoints);
>>>> System.out.println("table created");
>>>> }
>>>>
>>>> Table table = connection.getTable(tableName);
>>>> ResultScanner scanner = table.getScanner(new Scan());
>>>> scanner.iterator().hasNext(); // Exception thrown here
>>>> }
>>>
>>>
>>> 2017-12-30 5:04 GMT-08:00 Ted Yu <yu...@gmail.com>:
>>>
>>>> bq. a table has been created with pre-splits
>>>>
>>>> Unless you record the initial number of regions somewhere, there is no
>>>> built-in mechanism to provide the answer.
>>>>
>>>> Even if you cannot reproduce the assertion consistently, as long as the
>>>> chance of reproduction is high, the unit test would still be useful.
>>>>
>>>> Cheers
>>>>
>>>> On Sat, Dec 30, 2017 at 3:31 AM, Clément Guillaume <
>>>> clement@guillaume.bzh> wrote:
>>>>
>>>>> I dropped all the tables in my test cluster. Now I'm inconstantly able
>>>>> to reproduce the issue. I'm digging more.
>>>>>
>>>>> I have a related question: is it possible to check if a table has been
>>>>> created with pre-splits?
>>>>>
>>>>> I forgot to mention that I'm connecting to a version 1.2.3 cluster,
>>>>> but I think this should be ok according to this post
>>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/2017
>>>>> 11.mbox/%3C5BAAC90F-31D8-4A5F-B9E4-BA61FF4CD40E%40gmail.com%3E
>>>>>
>>>>> 2017-12-25 17:10 GMT-08:00 Ted Yu <yu...@gmail.com>:
>>>>>
>>>>>> I got the following if I use your code directly:
>>>>>>
>>>>>> testAssert(org.apache.hadoop.hbase.client.TestHCM) Time elapsed:
>>>>>> 0.096 sec <<< ERROR!
>>>>>> java.io.UncheckedIOException: org.apache.hadoop.hbase.TableNotFoundException:
>>>>>> AssertionTest
>>>>>> at org.apache.hadoop.hbase.client.TestHCM.testAssert(TestHCM.ja
>>>>>> va:243)
>>>>>>
>>>>>> If I create the table at the beginning of the test, the test passes
>>>>>> without triggering assertion.
>>>>>>
>>>>>> I tried both master branch and branch-1.4
>>>>>>
>>>>>> As I mentioned previously, the assertion in your case was probably
>>>>>> related to the formation of row keys in hbase:meta table.
>>>>>> See if you can reproduce the assertion by presplitting the table, etc.
>>>>>>
>>>>>> Here is the diff I used based on branch-1.4
>>>>>>
>>>>>> diff --git a/hbase-server/src/test/java/o
>>>>>> rg/apache/hadoop/hbase/client/TestHCM.java
>>>>>> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>>>> TestHCM.java
>>>>>> index a5e3a65..2944c8e 100644
>>>>>> --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>>>> TestHCM.java
>>>>>> +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>>>> TestHCM.java
>>>>>> @@ -246,6 +246,18 @@ public class TestHCM {
>>>>>> TEST_UTIL.shutdownMiniCluster();
>>>>>> }
>>>>>>
>>>>>> + @Test
>>>>>> + public void testAssert() throws IOException{
>>>>>> + Configuration hbaseConfig = TEST_UTIL.getConfiguration();
>>>>>> + final TableName tableName = TableName.valueOf("assert");
>>>>>> + TEST_UTIL.createTable(tableName, FAM_NAM).close();
>>>>>> + String zkQuorum = "hbase123.docker";
>>>>>> + //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>>>>>> + Connection connection = ConnectionFactory.createConnec
>>>>>> tion(hbaseConfig);
>>>>>> + Table table = connection.getTable(tableName);
>>>>>> + ResultScanner scanner = table.getScanner(new Scan());
>>>>>> + scanner.iterator().hasNext(); // Exception thrown here
>>>>>> + }
>>>>>>
>>>>>> private static int getHConnectionManagerCacheSize(){
>>>>>> return HConnectionTestingUtility.getConnectionCount();
>>>>>>
>>>>>> On Mon, Dec 25, 2017 at 4:16 PM, Clément Guillaume <
>>>>>> clement@guillaume.bzh> wrote:
>>>>>>
>>>>>>> Here is a unit test, note that the AssertionTest table can exit or
>>>>>>> not
>>>>>>> exit, assertion will fail un both case.
>>>>>>>
>>>>>>> > @Test
>>>>>>> > public void test() throws IOException{
>>>>>>> > Configuration hbaseConfig = HBaseConfiguration.create();
>>>>>>> > String zkQuorum = "hbase123.docker";
>>>>>>> > hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>>>>>>> > Connection connection = ConnectionFactory.createConnec
>>>>>>> tion(hbaseConfig);
>>>>>>> > Table table = connection.getTable(TableName.
>>>>>>> valueOf("AssertionTest"));
>>>>>>> > ResultScanner scanner = table.getScanner(new Scan());
>>>>>>> > scanner.iterator().hasNext(); // Exception thrown here
>>>>>>> > }
>>>>>>> >
>>>>>>>
>>>>>>> PS: I'm not receiving email when a response is posted here. I'm not
>>>>>>> subscribed at the hbase-user mailing list but I would have hope that
>>>>>>> answer
>>>>>>> get sent to me as direct email.
>>>>>>>
>>>>>>> 2017-12-21 12:28 GMT-08:00 Clément Guillaume <clement@guillaume.bzh
>>>>>>> >:
>>>>>>>
>>>>>>> > I'm upgrading my projects from 1.2.6 to 1.4, and I have a system
>>>>>>> assertion
>>>>>>> > failure (in a failsafe test that run with system assertion enable)
>>>>>>> >
>>>>>>> > java.lang.AssertionError
>>>>>>> > at org.apache.hadoop.hbase.client.ClientScanner.
>>>>>>> > loadCache(ClientScanner.java:484) ~[hbase-shaded-client-1.4.0.ja
>>>>>>> r:1.4.0]
>>>>>>> > at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScan
>>>>>>> ner.java:312)
>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>>>> > HConnectionImplementation.locateRegionInMeta(ConnectionManag
>>>>>>> er.java:1324)
>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>>>> > HConnectionImplementation.locateRegion(ConnectionManager.jav
>>>>>>> a:1221)
>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu
>>>>>>> reImpl.
>>>>>>> > findAllLocationsOrFail(AsyncProcess.java:1063)
>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu
>>>>>>> reImpl.
>>>>>>> > groupAndSendMultiAction(AsyncProcess.java:980)
>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$
>>>>>>> > AsyncRequestFutureImpl.access$200(AsyncProcess.java:667)
>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>>>>>>> rocess.java:649)
>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>>>>>>> rocess.java:612)
>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>>>> > HConnectionImplementation.processBatchCallback(ConnectionMan
>>>>>>> ager.java:2392)
>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>> > at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.j
>>>>>>> ava:950)
>>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>>> >
>>>>>>> > I have one Put action that look like that :
>>>>>>> {"totalColumns":1,"row":"
>>>>>>> > a7ttxdQY9euS-KEaPIMvKsRUWANCmigsaiT8_-V0Sgc\\x00visited\\x00
>>>>>>> \\xFF\\
>>>>>>> > xFF\\xFE\\x9F\\x85Dh\\x19","families":{"a":[{"qualifier":"
>>>>>>> > value","vlen":17,"tag":[],"timestamp":9223372036854775807}]}}
>>>>>>> > The cluster is a local standalone instance.
>>>>>>> >
>>>>>>> > I haven't dig into it yet, but it looks like the data is correctly
>>>>>>> written
>>>>>>> > in hbase if I disable assertion.
>>>>>>> > Any idea of what's causing the issue?
>>>>>>> >
>>>>>>> > Thank you.
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: assert in hbase 1.4
Posted by Ted Yu <yu...@gmail.com>.
Clément:
Since you have reproduced the assertion error, can you log a JIRA ?
Thanks
On Sun, Dec 31, 2017 at 7:46 AM, Ted Yu <yu...@gmail.com> wrote:
> I modified the test so that the unit test runs against branch-1.4 :
>
> + Configuration hbaseConfig = TEST_UTIL.getConfiguration();
> + //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker");
>
> However the test runs to completion without assertion - meaning the
> assertion wouldn't be triggered in homogeneous cluster.
>
> Here were the last few lines in the test output:
>
> 2017-12-31 07:33:36,865 INFO [Time-limited test] client.HBaseAdmin$CreateTableFuture(793):
> Created AssertionTest
> table created
> 2017-12-31 07:33:36,871 DEBUG [Time-limited test]
> client.ClientScanner(248): Advancing internal scanner to startKey at
> '\x01\x02\x03\x04\x05\x06\x07', inclusive
> 2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3]
> ipc.RpcConnection(126): Use SIMPLE authentication for service
> ClientService, sasl=false
> 2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3]
> ipc.BlockingRpcConnection(431): Connecting to
> tyus-mbp.attlocal.net/192.168.1.64:61585
> 2017-12-31 07:33:36,875 DEBUG [RpcServer.listener,port=61585]
> ipc.RpcServer$Listener(920): RpcServer.listener,port=61585: connection from
> 192.168.1.64:61633; # active connections: 3
> 2017-12-31 07:33:36,875 INFO [RpcServer.reader=0,bindAddress=
> tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1809): Auth
> successful for tyu (auth:SIMPLE)
> 2017-12-31 07:33:36,875 INFO [RpcServer.reader=0,bindAddress=
> tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1839):
> Connection from 192.168.1.64 port: 61633 with version info: version:
> "1.4.1-SNAPSHOT" url: "git://TYus-MBP.attlocal.net/Users/tyu/1.4-hbase"
> revision: "a7383851b9e62f23eb6f06eb23f09967f636c985" user: "tyu" date:
> "Sun Dec 31 07:32:58 PST 2017" src_checksum: "
> 39bdbbe40374e2094f53ce77af539671" version_major: 1 version_minor: 4
> 2017-12-31 07:33:36,917 INFO [main] hbase.ResourceChecker(171): after:
> client.TestHCM#test Thread=325 (was 305)
>
> I assume you pointed the test at a 1.2.3 cluster which triggered the
> assertion.
>
> It seems the patch which introduced the assertion didn't take into account
> client running against older release cluster deployment.
>
> FYI
>
> On Sun, Dec 31, 2017 at 6:09 AM, Clément Guillaume <cl...@guillaume.bzh>
> wrote:
>
>> I have consistent failure with this code (a table with pre-splits):
>>
>> @Test
>>> public void test() throws IOException{
>>> Configuration hbaseConfig = HBaseConfiguration.create();
>>> hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker");
>>> Connection connection = ConnectionFactory.createConnection(hbaseConfig);
>>>
>>
>>
>> TableName tableName = TableName.valueOf("AssertionTest");
>>> Admin admin = connection.getAdmin();
>>>
>>> if(!admin.tableExists(tableName)){
>>> HTableDescriptor htable = new HTableDescriptor(tableName);
>>> htable.addFamily(new HColumnDescriptor(new byte[]{(byte)'a'}));
>>> byte[][] splitPoints = {{1, 2, 3, 4, 5, 6, 7}};
>>> admin.createTable(htable, splitPoints);
>>> System.out.println("table created");
>>> }
>>>
>>> Table table = connection.getTable(tableName);
>>> ResultScanner scanner = table.getScanner(new Scan());
>>> scanner.iterator().hasNext(); // Exception thrown here
>>> }
>>
>>
>> 2017-12-30 5:04 GMT-08:00 Ted Yu <yu...@gmail.com>:
>>
>>> bq. a table has been created with pre-splits
>>>
>>> Unless you record the initial number of regions somewhere, there is no
>>> built-in mechanism to provide the answer.
>>>
>>> Even if you cannot reproduce the assertion consistently, as long as the
>>> chance of reproduction is high, the unit test would still be useful.
>>>
>>> Cheers
>>>
>>> On Sat, Dec 30, 2017 at 3:31 AM, Clément Guillaume <
>>> clement@guillaume.bzh> wrote:
>>>
>>>> I dropped all the tables in my test cluster. Now I'm inconstantly able
>>>> to reproduce the issue. I'm digging more.
>>>>
>>>> I have a related question: is it possible to check if a table has been
>>>> created with pre-splits?
>>>>
>>>> I forgot to mention that I'm connecting to a version 1.2.3 cluster, but
>>>> I think this should be ok according to this post
>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/2017
>>>> 11.mbox/%3C5BAAC90F-31D8-4A5F-B9E4-BA61FF4CD40E%40gmail.com%3E
>>>>
>>>> 2017-12-25 17:10 GMT-08:00 Ted Yu <yu...@gmail.com>:
>>>>
>>>>> I got the following if I use your code directly:
>>>>>
>>>>> testAssert(org.apache.hadoop.hbase.client.TestHCM) Time elapsed:
>>>>> 0.096 sec <<< ERROR!
>>>>> java.io.UncheckedIOException: org.apache.hadoop.hbase.TableNotFoundException:
>>>>> AssertionTest
>>>>> at org.apache.hadoop.hbase.client.TestHCM.testAssert(TestHCM.java:243)
>>>>>
>>>>> If I create the table at the beginning of the test, the test passes
>>>>> without triggering assertion.
>>>>>
>>>>> I tried both master branch and branch-1.4
>>>>>
>>>>> As I mentioned previously, the assertion in your case was probably
>>>>> related to the formation of row keys in hbase:meta table.
>>>>> See if you can reproduce the assertion by presplitting the table, etc.
>>>>>
>>>>> Here is the diff I used based on branch-1.4
>>>>>
>>>>> diff --git a/hbase-server/src/test/java/o
>>>>> rg/apache/hadoop/hbase/client/TestHCM.java
>>>>> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>>> TestHCM.java
>>>>> index a5e3a65..2944c8e 100644
>>>>> --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>>> TestHCM.java
>>>>> +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>>> TestHCM.java
>>>>> @@ -246,6 +246,18 @@ public class TestHCM {
>>>>> TEST_UTIL.shutdownMiniCluster();
>>>>> }
>>>>>
>>>>> + @Test
>>>>> + public void testAssert() throws IOException{
>>>>> + Configuration hbaseConfig = TEST_UTIL.getConfiguration();
>>>>> + final TableName tableName = TableName.valueOf("assert");
>>>>> + TEST_UTIL.createTable(tableName, FAM_NAM).close();
>>>>> + String zkQuorum = "hbase123.docker";
>>>>> + //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>>>>> + Connection connection = ConnectionFactory.createConnec
>>>>> tion(hbaseConfig);
>>>>> + Table table = connection.getTable(tableName);
>>>>> + ResultScanner scanner = table.getScanner(new Scan());
>>>>> + scanner.iterator().hasNext(); // Exception thrown here
>>>>> + }
>>>>>
>>>>> private static int getHConnectionManagerCacheSize(){
>>>>> return HConnectionTestingUtility.getConnectionCount();
>>>>>
>>>>> On Mon, Dec 25, 2017 at 4:16 PM, Clément Guillaume <
>>>>> clement@guillaume.bzh> wrote:
>>>>>
>>>>>> Here is a unit test, note that the AssertionTest table can exit or not
>>>>>> exit, assertion will fail un both case.
>>>>>>
>>>>>> > @Test
>>>>>> > public void test() throws IOException{
>>>>>> > Configuration hbaseConfig = HBaseConfiguration.create();
>>>>>> > String zkQuorum = "hbase123.docker";
>>>>>> > hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>>>>>> > Connection connection = ConnectionFactory.createConnec
>>>>>> tion(hbaseConfig);
>>>>>> > Table table = connection.getTable(TableName.
>>>>>> valueOf("AssertionTest"));
>>>>>> > ResultScanner scanner = table.getScanner(new Scan());
>>>>>> > scanner.iterator().hasNext(); // Exception thrown here
>>>>>> > }
>>>>>> >
>>>>>>
>>>>>> PS: I'm not receiving email when a response is posted here. I'm not
>>>>>> subscribed at the hbase-user mailing list but I would have hope that
>>>>>> answer
>>>>>> get sent to me as direct email.
>>>>>>
>>>>>> 2017-12-21 12:28 GMT-08:00 Clément Guillaume <cl...@guillaume.bzh>:
>>>>>>
>>>>>> > I'm upgrading my projects from 1.2.6 to 1.4, and I have a system
>>>>>> assertion
>>>>>> > failure (in a failsafe test that run with system assertion enable)
>>>>>> >
>>>>>> > java.lang.AssertionError
>>>>>> > at org.apache.hadoop.hbase.client.ClientScanner.
>>>>>> > loadCache(ClientScanner.java:484) ~[hbase-shaded-client-1.4.0.ja
>>>>>> r:1.4.0]
>>>>>> > at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScan
>>>>>> ner.java:312)
>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>>> > HConnectionImplementation.locateRegionInMeta(ConnectionManag
>>>>>> er.java:1324)
>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>>> > HConnectionImplementation.locateRegion(ConnectionManager.java:1221)
>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu
>>>>>> reImpl.
>>>>>> > findAllLocationsOrFail(AsyncProcess.java:1063)
>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu
>>>>>> reImpl.
>>>>>> > groupAndSendMultiAction(AsyncProcess.java:980)
>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$
>>>>>> > AsyncRequestFutureImpl.access$200(AsyncProcess.java:667)
>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>>>>>> rocess.java:649)
>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>>>>>> rocess.java:612)
>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>>> > HConnectionImplementation.processBatchCallback(ConnectionMan
>>>>>> ager.java:2392)
>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>> > at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.j
>>>>>> ava:950)
>>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>>> >
>>>>>> > I have one Put action that look like that :
>>>>>> {"totalColumns":1,"row":"
>>>>>> > a7ttxdQY9euS-KEaPIMvKsRUWANCmigsaiT8_-V0Sgc\\x00visited\\x00\\xFF\\
>>>>>> > xFF\\xFE\\x9F\\x85Dh\\x19","families":{"a":[{"qualifier":"
>>>>>> > value","vlen":17,"tag":[],"timestamp":9223372036854775807}]}}
>>>>>> > The cluster is a local standalone instance.
>>>>>> >
>>>>>> > I haven't dig into it yet, but it looks like the data is correctly
>>>>>> written
>>>>>> > in hbase if I disable assertion.
>>>>>> > Any idea of what's causing the issue?
>>>>>> >
>>>>>> > Thank you.
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: assert in hbase 1.4
Posted by Ted Yu <yu...@gmail.com>.
I modified the test so that the unit test runs against branch-1.4 :
+ Configuration hbaseConfig = TEST_UTIL.getConfiguration();
+ //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker");
However the test runs to completion without assertion - meaning the
assertion wouldn't be triggered in homogeneous cluster.
Here were the last few lines in the test output:
2017-12-31 07:33:36,865 INFO [Time-limited test]
client.HBaseAdmin$CreateTableFuture(793): Created AssertionTest
table created
2017-12-31 07:33:36,871 DEBUG [Time-limited test]
client.ClientScanner(248): Advancing internal scanner to startKey at
'\x01\x02\x03\x04\x05\x06\x07', inclusive
2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3]
ipc.RpcConnection(126): Use SIMPLE authentication for service
ClientService, sasl=false
2017-12-31 07:33:36,873 DEBUG [hconnection-0x6e0982ab-shared--pool24-t3]
ipc.BlockingRpcConnection(431): Connecting to
tyus-mbp.attlocal.net/192.168.1.64:61585
2017-12-31 07:33:36,875 DEBUG [RpcServer.listener,port=61585]
ipc.RpcServer$Listener(920): RpcServer.listener,port=61585: connection from
192.168.1.64:61633; # active connections: 3
2017-12-31 07:33:36,875 INFO [RpcServer.reader=0,bindAddress=
tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1809): Auth
successful for tyu (auth:SIMPLE)
2017-12-31 07:33:36,875 INFO [RpcServer.reader=0,bindAddress=
tyus-mbp.attlocal.net,port=61585] ipc.RpcServer$Connection(1839):
Connection from 192.168.1.64 port: 61633 with version info: version:
"1.4.1-SNAPSHOT" url: "git://TYus-MBP.attlocal.net/Users/tyu/1.4-hbase"
revision: "a7383851b9e62f23eb6f06eb23f09967f636c985" user: "tyu" date: "Sun
Dec 31 07:32:58 PST 2017" src_checksum: "39bdbbe40374e2094f53ce77af539671"
version_major: 1 version_minor: 4
2017-12-31 07:33:36,917 INFO [main] hbase.ResourceChecker(171): after:
client.TestHCM#test Thread=325 (was 305)
I assume you pointed the test at a 1.2.3 cluster which triggered the
assertion.
It seems the patch which introduced the assertion didn't take into account
client running against older release cluster deployment.
FYI
On Sun, Dec 31, 2017 at 6:09 AM, Clément Guillaume <cl...@guillaume.bzh>
wrote:
> I have consistent failure with this code (a table with pre-splits):
>
> @Test
>> public void test() throws IOException{
>> Configuration hbaseConfig = HBaseConfiguration.create();
>> hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker");
>> Connection connection = ConnectionFactory.createConnection(hbaseConfig);
>>
>
>
> TableName tableName = TableName.valueOf("AssertionTest");
>> Admin admin = connection.getAdmin();
>>
>> if(!admin.tableExists(tableName)){
>> HTableDescriptor htable = new HTableDescriptor(tableName);
>> htable.addFamily(new HColumnDescriptor(new byte[]{(byte)'a'}));
>> byte[][] splitPoints = {{1, 2, 3, 4, 5, 6, 7}};
>> admin.createTable(htable, splitPoints);
>> System.out.println("table created");
>> }
>>
>> Table table = connection.getTable(tableName);
>> ResultScanner scanner = table.getScanner(new Scan());
>> scanner.iterator().hasNext(); // Exception thrown here
>> }
>
>
> 2017-12-30 5:04 GMT-08:00 Ted Yu <yu...@gmail.com>:
>
>> bq. a table has been created with pre-splits
>>
>> Unless you record the initial number of regions somewhere, there is no
>> built-in mechanism to provide the answer.
>>
>> Even if you cannot reproduce the assertion consistently, as long as the
>> chance of reproduction is high, the unit test would still be useful.
>>
>> Cheers
>>
>> On Sat, Dec 30, 2017 at 3:31 AM, Clément Guillaume <clement@guillaume.bzh
>> > wrote:
>>
>>> I dropped all the tables in my test cluster. Now I'm inconstantly able
>>> to reproduce the issue. I'm digging more.
>>>
>>> I have a related question: is it possible to check if a table has been
>>> created with pre-splits?
>>>
>>> I forgot to mention that I'm connecting to a version 1.2.3 cluster, but
>>> I think this should be ok according to this post
>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/2017
>>> 11.mbox/%3C5BAAC90F-31D8-4A5F-B9E4-BA61FF4CD40E%40gmail.com%3E
>>>
>>> 2017-12-25 17:10 GMT-08:00 Ted Yu <yu...@gmail.com>:
>>>
>>>> I got the following if I use your code directly:
>>>>
>>>> testAssert(org.apache.hadoop.hbase.client.TestHCM) Time elapsed:
>>>> 0.096 sec <<< ERROR!
>>>> java.io.UncheckedIOException: org.apache.hadoop.hbase.TableNotFoundException:
>>>> AssertionTest
>>>> at org.apache.hadoop.hbase.client.TestHCM.testAssert(TestHCM.java:243)
>>>>
>>>> If I create the table at the beginning of the test, the test passes
>>>> without triggering assertion.
>>>>
>>>> I tried both master branch and branch-1.4
>>>>
>>>> As I mentioned previously, the assertion in your case was probably
>>>> related to the formation of row keys in hbase:meta table.
>>>> See if you can reproduce the assertion by presplitting the table, etc.
>>>>
>>>> Here is the diff I used based on branch-1.4
>>>>
>>>> diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
>>>> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>> TestHCM.java
>>>> index a5e3a65..2944c8e 100644
>>>> --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>> TestHCM.java
>>>> +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>>> TestHCM.java
>>>> @@ -246,6 +246,18 @@ public class TestHCM {
>>>> TEST_UTIL.shutdownMiniCluster();
>>>> }
>>>>
>>>> + @Test
>>>> + public void testAssert() throws IOException{
>>>> + Configuration hbaseConfig = TEST_UTIL.getConfiguration();
>>>> + final TableName tableName = TableName.valueOf("assert");
>>>> + TEST_UTIL.createTable(tableName, FAM_NAM).close();
>>>> + String zkQuorum = "hbase123.docker";
>>>> + //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>>>> + Connection connection = ConnectionFactory.createConnec
>>>> tion(hbaseConfig);
>>>> + Table table = connection.getTable(tableName);
>>>> + ResultScanner scanner = table.getScanner(new Scan());
>>>> + scanner.iterator().hasNext(); // Exception thrown here
>>>> + }
>>>>
>>>> private static int getHConnectionManagerCacheSize(){
>>>> return HConnectionTestingUtility.getConnectionCount();
>>>>
>>>> On Mon, Dec 25, 2017 at 4:16 PM, Clément Guillaume <
>>>> clement@guillaume.bzh> wrote:
>>>>
>>>>> Here is a unit test, note that the AssertionTest table can exit or not
>>>>> exit, assertion will fail un both case.
>>>>>
>>>>> > @Test
>>>>> > public void test() throws IOException{
>>>>> > Configuration hbaseConfig = HBaseConfiguration.create();
>>>>> > String zkQuorum = "hbase123.docker";
>>>>> > hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>>>>> > Connection connection = ConnectionFactory.createConnec
>>>>> tion(hbaseConfig);
>>>>> > Table table = connection.getTable(TableName.
>>>>> valueOf("AssertionTest"));
>>>>> > ResultScanner scanner = table.getScanner(new Scan());
>>>>> > scanner.iterator().hasNext(); // Exception thrown here
>>>>> > }
>>>>> >
>>>>>
>>>>> PS: I'm not receiving email when a response is posted here. I'm not
>>>>> subscribed at the hbase-user mailing list but I would have hope that
>>>>> answer
>>>>> get sent to me as direct email.
>>>>>
>>>>> 2017-12-21 12:28 GMT-08:00 Clément Guillaume <cl...@guillaume.bzh>:
>>>>>
>>>>> > I'm upgrading my projects from 1.2.6 to 1.4, and I have a system
>>>>> assertion
>>>>> > failure (in a failsafe test that run with system assertion enable)
>>>>> >
>>>>> > java.lang.AssertionError
>>>>> > at org.apache.hadoop.hbase.client.ClientScanner.
>>>>> > loadCache(ClientScanner.java:484) ~[hbase-shaded-client-1.4.0.ja
>>>>> r:1.4.0]
>>>>> > at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScan
>>>>> ner.java:312)
>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>> > HConnectionImplementation.locateRegionInMeta(ConnectionManag
>>>>> er.java:1324)
>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>> > HConnectionImplementation.locateRegion(ConnectionManager.java:1221)
>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu
>>>>> reImpl.
>>>>> > findAllLocationsOrFail(AsyncProcess.java:1063)
>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu
>>>>> reImpl.
>>>>> > groupAndSendMultiAction(AsyncProcess.java:980)
>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$
>>>>> > AsyncRequestFutureImpl.access$200(AsyncProcess.java:667)
>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>>>>> rocess.java:649)
>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>>>>> rocess.java:612)
>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>>> > HConnectionImplementation.processBatchCallback(ConnectionMan
>>>>> ager.java:2392)
>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>> > at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.j
>>>>> ava:950)
>>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>>> >
>>>>> > I have one Put action that look like that : {"totalColumns":1,"row":"
>>>>> > a7ttxdQY9euS-KEaPIMvKsRUWANCmigsaiT8_-V0Sgc\\x00visited\\x00\\xFF\\
>>>>> > xFF\\xFE\\x9F\\x85Dh\\x19","families":{"a":[{"qualifier":"
>>>>> > value","vlen":17,"tag":[],"timestamp":9223372036854775807}]}}
>>>>> > The cluster is a local standalone instance.
>>>>> >
>>>>> > I haven't dig into it yet, but it looks like the data is correctly
>>>>> written
>>>>> > in hbase if I disable assertion.
>>>>> > Any idea of what's causing the issue?
>>>>> >
>>>>> > Thank you.
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>
Re: assert in hbase 1.4
Posted by Clément Guillaume <cl...@guillaume.bzh>.
I have consistent failure with this code (a table with pre-splits):
@Test
> public void test() throws IOException{
> Configuration hbaseConfig = HBaseConfiguration.create();
> hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, "hbase123.docker");
> Connection connection = ConnectionFactory.createConnection(hbaseConfig);
>
TableName tableName = TableName.valueOf("AssertionTest");
> Admin admin = connection.getAdmin();
>
> if(!admin.tableExists(tableName)){
> HTableDescriptor htable = new HTableDescriptor(tableName);
> htable.addFamily(new HColumnDescriptor(new byte[]{(byte)'a'}));
> byte[][] splitPoints = {{1, 2, 3, 4, 5, 6, 7}};
> admin.createTable(htable, splitPoints);
> System.out.println("table created");
> }
>
> Table table = connection.getTable(tableName);
> ResultScanner scanner = table.getScanner(new Scan());
> scanner.iterator().hasNext(); // Exception thrown here
> }
2017-12-30 5:04 GMT-08:00 Ted Yu <yu...@gmail.com>:
> bq. a table has been created with pre-splits
>
> Unless you record the initial number of regions somewhere, there is no
> built-in mechanism to provide the answer.
>
> Even if you cannot reproduce the assertion consistently, as long as the
> chance of reproduction is high, the unit test would still be useful.
>
> Cheers
>
> On Sat, Dec 30, 2017 at 3:31 AM, Clément Guillaume <cl...@guillaume.bzh>
> wrote:
>
>> I dropped all the tables in my test cluster. Now I'm inconstantly able to
>> reproduce the issue. I'm digging more.
>>
>> I have a related question: is it possible to check if a table has been
>> created with pre-splits?
>>
>> I forgot to mention that I'm connecting to a version 1.2.3 cluster, but I
>> think this should be ok according to this post http://mail-archives.apac
>> he.org/mod_mbox/hbase-dev/201711.mbox/%3C5BAAC90F-31D8-4A5F-
>> B9E4-BA61FF4CD40E%40gmail.com%3E
>>
>> 2017-12-25 17:10 GMT-08:00 Ted Yu <yu...@gmail.com>:
>>
>>> I got the following if I use your code directly:
>>>
>>> testAssert(org.apache.hadoop.hbase.client.TestHCM) Time elapsed: 0.096
>>> sec <<< ERROR!
>>> java.io.UncheckedIOException: org.apache.hadoop.hbase.TableNotFoundException:
>>> AssertionTest
>>> at org.apache.hadoop.hbase.client.TestHCM.testAssert(TestHCM.java:243)
>>>
>>> If I create the table at the beginning of the test, the test passes
>>> without triggering assertion.
>>>
>>> I tried both master branch and branch-1.4
>>>
>>> As I mentioned previously, the assertion in your case was probably
>>> related to the formation of row keys in hbase:meta table.
>>> See if you can reproduce the assertion by presplitting the table, etc.
>>>
>>> Here is the diff I used based on branch-1.4
>>>
>>> diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
>>> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
>>> index a5e3a65..2944c8e 100644
>>> --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>> TestHCM.java
>>> +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>>> TestHCM.java
>>> @@ -246,6 +246,18 @@ public class TestHCM {
>>> TEST_UTIL.shutdownMiniCluster();
>>> }
>>>
>>> + @Test
>>> + public void testAssert() throws IOException{
>>> + Configuration hbaseConfig = TEST_UTIL.getConfiguration();
>>> + final TableName tableName = TableName.valueOf("assert");
>>> + TEST_UTIL.createTable(tableName, FAM_NAM).close();
>>> + String zkQuorum = "hbase123.docker";
>>> + //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>>> + Connection connection = ConnectionFactory.createConnec
>>> tion(hbaseConfig);
>>> + Table table = connection.getTable(tableName);
>>> + ResultScanner scanner = table.getScanner(new Scan());
>>> + scanner.iterator().hasNext(); // Exception thrown here
>>> + }
>>>
>>> private static int getHConnectionManagerCacheSize(){
>>> return HConnectionTestingUtility.getConnectionCount();
>>>
>>> On Mon, Dec 25, 2017 at 4:16 PM, Clément Guillaume <
>>> clement@guillaume.bzh> wrote:
>>>
>>>> Here is a unit test, note that the AssertionTest table can exit or not
>>>> exit, assertion will fail un both case.
>>>>
>>>> > @Test
>>>> > public void test() throws IOException{
>>>> > Configuration hbaseConfig = HBaseConfiguration.create();
>>>> > String zkQuorum = "hbase123.docker";
>>>> > hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>>>> > Connection connection = ConnectionFactory.createConnec
>>>> tion(hbaseConfig);
>>>> > Table table = connection.getTable(TableName.
>>>> valueOf("AssertionTest"));
>>>> > ResultScanner scanner = table.getScanner(new Scan());
>>>> > scanner.iterator().hasNext(); // Exception thrown here
>>>> > }
>>>> >
>>>>
>>>> PS: I'm not receiving email when a response is posted here. I'm not
>>>> subscribed at the hbase-user mailing list but I would have hope that
>>>> answer
>>>> get sent to me as direct email.
>>>>
>>>> 2017-12-21 12:28 GMT-08:00 Clément Guillaume <cl...@guillaume.bzh>:
>>>>
>>>> > I'm upgrading my projects from 1.2.6 to 1.4, and I have a system
>>>> assertion
>>>> > failure (in a failsafe test that run with system assertion enable)
>>>> >
>>>> > java.lang.AssertionError
>>>> > at org.apache.hadoop.hbase.client.ClientScanner.
>>>> > loadCache(ClientScanner.java:484) ~[hbase-shaded-client-1.4.0.ja
>>>> r:1.4.0]
>>>> > at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScan
>>>> ner.java:312)
>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>> > HConnectionImplementation.locateRegionInMeta(ConnectionManag
>>>> er.java:1324)
>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>> > HConnectionImplementation.locateRegion(ConnectionManager.java:1221)
>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu
>>>> reImpl.
>>>> > findAllLocationsOrFail(AsyncProcess.java:1063)
>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutu
>>>> reImpl.
>>>> > groupAndSendMultiAction(AsyncProcess.java:980)
>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>> > at org.apache.hadoop.hbase.client.AsyncProcess$
>>>> > AsyncRequestFutureImpl.access$200(AsyncProcess.java:667)
>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>>>> rocess.java:649)
>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>>>> rocess.java:612)
>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>>> > HConnectionImplementation.processBatchCallback(ConnectionMan
>>>> ager.java:2392)
>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>> > at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.j
>>>> ava:950)
>>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>>> >
>>>> > I have one Put action that look like that : {"totalColumns":1,"row":"
>>>> > a7ttxdQY9euS-KEaPIMvKsRUWANCmigsaiT8_-V0Sgc\\x00visited\\x00\\xFF\\
>>>> > xFF\\xFE\\x9F\\x85Dh\\x19","families":{"a":[{"qualifier":"
>>>> > value","vlen":17,"tag":[],"timestamp":9223372036854775807}]}}
>>>> > The cluster is a local standalone instance.
>>>> >
>>>> > I haven't dig into it yet, but it looks like the data is correctly
>>>> written
>>>> > in hbase if I disable assertion.
>>>> > Any idea of what's causing the issue?
>>>> >
>>>> > Thank you.
>>>> >
>>>>
>>>
>>>
>>
>
Re: assert in hbase 1.4
Posted by Ted Yu <yu...@gmail.com>.
bq. a table has been created with pre-splits
Unless you record the initial number of regions somewhere, there is no
built-in mechanism to provide the answer.
Even if you cannot reproduce the assertion consistently, as long as the
chance of reproduction is high, the unit test would still be useful.
Cheers
On Sat, Dec 30, 2017 at 3:31 AM, Clément Guillaume <cl...@guillaume.bzh>
wrote:
> I dropped all the tables in my test cluster. Now I'm inconstantly able to
> reproduce the issue. I'm digging more.
>
> I have a related question: is it possible to check if a table has been
> created with pre-splits?
>
> I forgot to mention that I'm connecting to a version 1.2.3 cluster, but I
> think this should be ok according to this post http://mail-archives.apache
> .org/mod_mbox/hbase-dev/201711.mbox/%3C5BAAC90F-31D8-
> 4A5F-B9E4-BA61FF4CD40E%40gmail.com%3E
>
> 2017-12-25 17:10 GMT-08:00 Ted Yu <yu...@gmail.com>:
>
>> I got the following if I use your code directly:
>>
>> testAssert(org.apache.hadoop.hbase.client.TestHCM) Time elapsed: 0.096
>> sec <<< ERROR!
>> java.io.UncheckedIOException: org.apache.hadoop.hbase.TableNotFoundException:
>> AssertionTest
>> at org.apache.hadoop.hbase.client.TestHCM.testAssert(TestHCM.java:243)
>>
>> If I create the table at the beginning of the test, the test passes
>> without triggering assertion.
>>
>> I tried both master branch and branch-1.4
>>
>> As I mentioned previously, the assertion in your case was probably
>> related to the formation of row keys in hbase:meta table.
>> See if you can reproduce the assertion by presplitting the table, etc.
>>
>> Here is the diff I used based on branch-1.4
>>
>> diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
>> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
>> index a5e3a65..2944c8e 100644
>> --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>> TestHCM.java
>> +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/
>> TestHCM.java
>> @@ -246,6 +246,18 @@ public class TestHCM {
>> TEST_UTIL.shutdownMiniCluster();
>> }
>>
>> + @Test
>> + public void testAssert() throws IOException{
>> + Configuration hbaseConfig = TEST_UTIL.getConfiguration();
>> + final TableName tableName = TableName.valueOf("assert");
>> + TEST_UTIL.createTable(tableName, FAM_NAM).close();
>> + String zkQuorum = "hbase123.docker";
>> + //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>> + Connection connection = ConnectionFactory.createConnec
>> tion(hbaseConfig);
>> + Table table = connection.getTable(tableName);
>> + ResultScanner scanner = table.getScanner(new Scan());
>> + scanner.iterator().hasNext(); // Exception thrown here
>> + }
>>
>> private static int getHConnectionManagerCacheSize(){
>> return HConnectionTestingUtility.getConnectionCount();
>>
>> On Mon, Dec 25, 2017 at 4:16 PM, Clément Guillaume <clement@guillaume.bzh
>> > wrote:
>>
>>> Here is a unit test, note that the AssertionTest table can exit or not
>>> exit, assertion will fail un both case.
>>>
>>> > @Test
>>> > public void test() throws IOException{
>>> > Configuration hbaseConfig = HBaseConfiguration.create();
>>> > String zkQuorum = "hbase123.docker";
>>> > hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>>> > Connection connection = ConnectionFactory.createConnec
>>> tion(hbaseConfig);
>>> > Table table = connection.getTable(TableName.valueOf("AssertionTest"));
>>> > ResultScanner scanner = table.getScanner(new Scan());
>>> > scanner.iterator().hasNext(); // Exception thrown here
>>> > }
>>> >
>>>
>>> PS: I'm not receiving email when a response is posted here. I'm not
>>> subscribed at the hbase-user mailing list but I would have hope that
>>> answer
>>> get sent to me as direct email.
>>>
>>> 2017-12-21 12:28 GMT-08:00 Clément Guillaume <cl...@guillaume.bzh>:
>>>
>>> > I'm upgrading my projects from 1.2.6 to 1.4, and I have a system
>>> assertion
>>> > failure (in a failsafe test that run with system assertion enable)
>>> >
>>> > java.lang.AssertionError
>>> > at org.apache.hadoop.hbase.client.ClientScanner.
>>> > loadCache(ClientScanner.java:484) ~[hbase-shaded-client-1.4.0.ja
>>> r:1.4.0]
>>> > at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScan
>>> ner.java:312)
>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>> > HConnectionImplementation.locateRegionInMeta(ConnectionManag
>>> er.java:1324)
>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>> > HConnectionImplementation.locateRegion(ConnectionManager.java:1221)
>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.
>>> > findAllLocationsOrFail(AsyncProcess.java:1063)
>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.
>>> > groupAndSendMultiAction(AsyncProcess.java:980)
>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>> > at org.apache.hadoop.hbase.client.AsyncProcess$
>>> > AsyncRequestFutureImpl.access$200(AsyncProcess.java:667)
>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>>> rocess.java:649)
>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>>> rocess.java:612)
>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>>> > HConnectionImplementation.processBatchCallback(ConnectionMan
>>> ager.java:2392)
>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>> > at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.j
>>> ava:950)
>>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>>> >
>>> > I have one Put action that look like that : {"totalColumns":1,"row":"
>>> > a7ttxdQY9euS-KEaPIMvKsRUWANCmigsaiT8_-V0Sgc\\x00visited\\x00\\xFF\\
>>> > xFF\\xFE\\x9F\\x85Dh\\x19","families":{"a":[{"qualifier":"
>>> > value","vlen":17,"tag":[],"timestamp":9223372036854775807}]}}
>>> > The cluster is a local standalone instance.
>>> >
>>> > I haven't dig into it yet, but it looks like the data is correctly
>>> written
>>> > in hbase if I disable assertion.
>>> > Any idea of what's causing the issue?
>>> >
>>> > Thank you.
>>> >
>>>
>>
>>
>
Re: assert in hbase 1.4
Posted by Clément Guillaume <cl...@guillaume.bzh>.
I dropped all the tables in my test cluster. Now I'm inconstantly able to
reproduce the issue. I'm digging more.
I have a related question: is it possible to check if a table has been
created with pre-splits?
I forgot to mention that I'm connecting to a version 1.2.3 cluster, but I
think this should be ok according to this post http://mail-archives.apache.
org/mod_mbox/hbase-dev/201711.mbox/%3C5BAAC90F-31D8-4A5F-B9E4-BA61FF4CD40E%
40gmail.com%3E
2017-12-25 17:10 GMT-08:00 Ted Yu <yu...@gmail.com>:
> I got the following if I use your code directly:
>
> testAssert(org.apache.hadoop.hbase.client.TestHCM) Time elapsed: 0.096
> sec <<< ERROR!
> java.io.UncheckedIOException: org.apache.hadoop.hbase.TableNotFoundException:
> AssertionTest
> at org.apache.hadoop.hbase.client.TestHCM.testAssert(TestHCM.java:243)
>
> If I create the table at the beginning of the test, the test passes
> without triggering assertion.
>
> I tried both master branch and branch-1.4
>
> As I mentioned previously, the assertion in your case was probably related
> to the formation of row keys in hbase:meta table.
> See if you can reproduce the assertion by presplitting the table, etc.
>
> Here is the diff I used based on branch-1.4
>
> diff --git a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
> index a5e3a65..2944c8e 100644
> --- a/hbase-server/src/test/java/org/apache/hadoop/hbase/
> client/TestHCM.java
> +++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/
> client/TestHCM.java
> @@ -246,6 +246,18 @@ public class TestHCM {
> TEST_UTIL.shutdownMiniCluster();
> }
>
> + @Test
> + public void testAssert() throws IOException{
> + Configuration hbaseConfig = TEST_UTIL.getConfiguration();
> + final TableName tableName = TableName.valueOf("assert");
> + TEST_UTIL.createTable(tableName, FAM_NAM).close();
> + String zkQuorum = "hbase123.docker";
> + //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
> + Connection connection = ConnectionFactory.
> createConnection(hbaseConfig);
> + Table table = connection.getTable(tableName);
> + ResultScanner scanner = table.getScanner(new Scan());
> + scanner.iterator().hasNext(); // Exception thrown here
> + }
>
> private static int getHConnectionManagerCacheSize(){
> return HConnectionTestingUtility.getConnectionCount();
>
> On Mon, Dec 25, 2017 at 4:16 PM, Clément Guillaume <cl...@guillaume.bzh>
> wrote:
>
>> Here is a unit test, note that the AssertionTest table can exit or not
>> exit, assertion will fail un both case.
>>
>> > @Test
>> > public void test() throws IOException{
>> > Configuration hbaseConfig = HBaseConfiguration.create();
>> > String zkQuorum = "hbase123.docker";
>> > hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
>> > Connection connection = ConnectionFactory.createConnec
>> tion(hbaseConfig);
>> > Table table = connection.getTable(TableName.valueOf("AssertionTest"));
>> > ResultScanner scanner = table.getScanner(new Scan());
>> > scanner.iterator().hasNext(); // Exception thrown here
>> > }
>> >
>>
>> PS: I'm not receiving email when a response is posted here. I'm not
>> subscribed at the hbase-user mailing list but I would have hope that
>> answer
>> get sent to me as direct email.
>>
>> 2017-12-21 12:28 GMT-08:00 Clément Guillaume <cl...@guillaume.bzh>:
>>
>> > I'm upgrading my projects from 1.2.6 to 1.4, and I have a system
>> assertion
>> > failure (in a failsafe test that run with system assertion enable)
>> >
>> > java.lang.AssertionError
>> > at org.apache.hadoop.hbase.client.ClientScanner.
>> > loadCache(ClientScanner.java:484) ~[hbase-shaded-client-1.4.0.ja
>> r:1.4.0]
>> > at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScan
>> ner.java:312)
>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>> > HConnectionImplementation.locateRegionInMeta(ConnectionManag
>> er.java:1324)
>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>> > HConnectionImplementation.locateRegion(ConnectionManager.java:1221)
>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.
>> > findAllLocationsOrFail(AsyncProcess.java:1063)
>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.
>> > groupAndSendMultiAction(AsyncProcess.java:980)
>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>> > at org.apache.hadoop.hbase.client.AsyncProcess$
>> > AsyncRequestFutureImpl.access$200(AsyncProcess.java:667)
>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>> rocess.java:649)
>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncP
>> rocess.java:612)
>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>> > at org.apache.hadoop.hbase.client.ConnectionManager$
>> > HConnectionImplementation.processBatchCallback(ConnectionMan
>> ager.java:2392)
>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>> > at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.java:950)
>> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>> >
>> > I have one Put action that look like that : {"totalColumns":1,"row":"
>> > a7ttxdQY9euS-KEaPIMvKsRUWANCmigsaiT8_-V0Sgc\\x00visited\\x00\\xFF\\
>> > xFF\\xFE\\x9F\\x85Dh\\x19","families":{"a":[{"qualifier":"
>> > value","vlen":17,"tag":[],"timestamp":9223372036854775807}]}}
>> > The cluster is a local standalone instance.
>> >
>> > I haven't dig into it yet, but it looks like the data is correctly
>> written
>> > in hbase if I disable assertion.
>> > Any idea of what's causing the issue?
>> >
>> > Thank you.
>> >
>>
>
>
Re: assert in hbase 1.4
Posted by Ted Yu <yu...@gmail.com>.
I got the following if I use your code directly:
testAssert(org.apache.hadoop.hbase.client.TestHCM) Time elapsed: 0.096
sec <<< ERROR!
java.io.UncheckedIOException:
org.apache.hadoop.hbase.TableNotFoundException: AssertionTest
at org.apache.hadoop.hbase.client.TestHCM.testAssert(TestHCM.java:243)
If I create the table at the beginning of the test, the test passes without
triggering assertion.
I tried both master branch and branch-1.4
As I mentioned previously, the assertion in your case was probably related
to the formation of row keys in hbase:meta table.
See if you can reproduce the assertion by presplitting the table, etc.
Here is the diff I used based on branch-1.4
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
index a5e3a65..2944c8e 100644
--- a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
+++ b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
@@ -246,6 +246,18 @@ public class TestHCM {
TEST_UTIL.shutdownMiniCluster();
}
+ @Test
+ public void testAssert() throws IOException{
+ Configuration hbaseConfig = TEST_UTIL.getConfiguration();
+ final TableName tableName = TableName.valueOf("assert");
+ TEST_UTIL.createTable(tableName, FAM_NAM).close();
+ String zkQuorum = "hbase123.docker";
+ //hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
+ Connection connection =
ConnectionFactory.createConnection(hbaseConfig);
+ Table table = connection.getTable(tableName);
+ ResultScanner scanner = table.getScanner(new Scan());
+ scanner.iterator().hasNext(); // Exception thrown here
+ }
private static int getHConnectionManagerCacheSize(){
return HConnectionTestingUtility.getConnectionCount();
On Mon, Dec 25, 2017 at 4:16 PM, Clément Guillaume <cl...@guillaume.bzh>
wrote:
> Here is a unit test, note that the AssertionTest table can exit or not
> exit, assertion will fail un both case.
>
> > @Test
> > public void test() throws IOException{
> > Configuration hbaseConfig = HBaseConfiguration.create();
> > String zkQuorum = "hbase123.docker";
> > hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
> > Connection connection = ConnectionFactory.createConnection(hbaseConfig);
> > Table table = connection.getTable(TableName.valueOf("AssertionTest"));
> > ResultScanner scanner = table.getScanner(new Scan());
> > scanner.iterator().hasNext(); // Exception thrown here
> > }
> >
>
> PS: I'm not receiving email when a response is posted here. I'm not
> subscribed at the hbase-user mailing list but I would have hope that answer
> get sent to me as direct email.
>
> 2017-12-21 12:28 GMT-08:00 Clément Guillaume <cl...@guillaume.bzh>:
>
> > I'm upgrading my projects from 1.2.6 to 1.4, and I have a system
> assertion
> > failure (in a failsafe test that run with system assertion enable)
> >
> > java.lang.AssertionError
> > at org.apache.hadoop.hbase.client.ClientScanner.
> > loadCache(ClientScanner.java:484) ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> > at org.apache.hadoop.hbase.client.ClientScanner.next(
> ClientScanner.java:312)
> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> > at org.apache.hadoop.hbase.client.ConnectionManager$
> > HConnectionImplementation.locateRegionInMeta(
> ConnectionManager.java:1324)
> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> > at org.apache.hadoop.hbase.client.ConnectionManager$
> > HConnectionImplementation.locateRegion(ConnectionManager.java:1221)
> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.
> > findAllLocationsOrFail(AsyncProcess.java:1063)
> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> > at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.
> > groupAndSendMultiAction(AsyncProcess.java:980)
> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> > at org.apache.hadoop.hbase.client.AsyncProcess$
> > AsyncRequestFutureImpl.access$200(AsyncProcess.java:667)
> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(
> AsyncProcess.java:649)
> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> > at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(
> AsyncProcess.java:612)
> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> > at org.apache.hadoop.hbase.client.ConnectionManager$
> > HConnectionImplementation.processBatchCallback(
> ConnectionManager.java:2392)
> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> > at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.java:950)
> > ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> >
> > I have one Put action that look like that : {"totalColumns":1,"row":"
> > a7ttxdQY9euS-KEaPIMvKsRUWANCmigsaiT8_-V0Sgc\\x00visited\\x00\\xFF\\
> > xFF\\xFE\\x9F\\x85Dh\\x19","families":{"a":[{"qualifier":"
> > value","vlen":17,"tag":[],"timestamp":9223372036854775807}]}}
> > The cluster is a local standalone instance.
> >
> > I haven't dig into it yet, but it looks like the data is correctly
> written
> > in hbase if I disable assertion.
> > Any idea of what's causing the issue?
> >
> > Thank you.
> >
>
Re: assert in hbase 1.4
Posted by Clément Guillaume <cl...@guillaume.bzh>.
Here is a unit test, note that the AssertionTest table can exit or not
exit, assertion will fail un both case.
> @Test
> public void test() throws IOException{
> Configuration hbaseConfig = HBaseConfiguration.create();
> String zkQuorum = "hbase123.docker";
> hbaseConfig.set(HConstants.ZOOKEEPER_QUORUM, zkQuorum);
> Connection connection = ConnectionFactory.createConnection(hbaseConfig);
> Table table = connection.getTable(TableName.valueOf("AssertionTest"));
> ResultScanner scanner = table.getScanner(new Scan());
> scanner.iterator().hasNext(); // Exception thrown here
> }
>
PS: I'm not receiving email when a response is posted here. I'm not
subscribed at the hbase-user mailing list but I would have hope that answer
get sent to me as direct email.
2017-12-21 12:28 GMT-08:00 Clément Guillaume <cl...@guillaume.bzh>:
> I'm upgrading my projects from 1.2.6 to 1.4, and I have a system assertion
> failure (in a failsafe test that run with system assertion enable)
>
> java.lang.AssertionError
> at org.apache.hadoop.hbase.client.ClientScanner.
> loadCache(ClientScanner.java:484) ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:312)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> at org.apache.hadoop.hbase.client.ConnectionManager$
> HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1324)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> at org.apache.hadoop.hbase.client.ConnectionManager$
> HConnectionImplementation.locateRegion(ConnectionManager.java:1221)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.
> findAllLocationsOrFail(AsyncProcess.java:1063)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.
> groupAndSendMultiAction(AsyncProcess.java:980)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> at org.apache.hadoop.hbase.client.AsyncProcess$
> AsyncRequestFutureImpl.access$200(AsyncProcess.java:667)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:649)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:612)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> at org.apache.hadoop.hbase.client.ConnectionManager$
> HConnectionImplementation.processBatchCallback(ConnectionManager.java:2392)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
> at org.apache.hadoop.hbase.client.HTable.batchCallback(HTable.java:950)
> ~[hbase-shaded-client-1.4.0.jar:1.4.0]
>
> I have one Put action that look like that : {"totalColumns":1,"row":"
> a7ttxdQY9euS-KEaPIMvKsRUWANCmigsaiT8_-V0Sgc\\x00visited\\x00\\xFF\\
> xFF\\xFE\\x9F\\x85Dh\\x19","families":{"a":[{"qualifier":"
> value","vlen":17,"tag":[],"timestamp":9223372036854775807}]}}
> The cluster is a local standalone instance.
>
> I haven't dig into it yet, but it looks like the data is correctly written
> in hbase if I disable assertion.
> Any idea of what's causing the issue?
>
> Thank you.
>