You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Akshat Mahajan <am...@brightedge.com> on 2017/02/08 02:29:17 UTC
Hbase REST returns invalid status code
We are seeing exceptions where Hbase REST is returning an invalid status code on writes, causing our Python processes to error out on trying to make a write (the following error is thrown by the Python starbase library, but is not caused by starbase directly):
`requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))`
There are no traces nor errors reported in the REST log at the time of the exception. We know that this is consistent, as we retry thrice with exponential backoff (2 seconds initially).
Our REST threads are allowed to grow dynamically from 100 to 9000 at worst; our connections typically do not exceed these values, and did not at the time.
Are there circumstances where Hbase REST will deliberately refuse to honor a connection or respond with invalid codes? Or should I continue looking for more environmental reasons? Note that at the time REST was documenting wait on a number of AsyncProcesses on several tasks after having recently closed a table - not sure how useful that information is. A log snippet at the time from Hbase REST is below:
```
2017-02-07 17:07:35,312 INFO [196541187@qtp-1054818236-2084] client.HBaseAdmin: Started disable of urls_table_50007_20170207170255
2017-02-07 17:07:43,294 INFO [196541187@qtp-1054818236-2084] client.HBaseAdmin: Disabled urls_table_50007_20170207170255
2017-02-07 17:07:44,093 INFO [196541187@qtp-1054818236-2084] client.HBaseAdmin: Deleted urls_table_50007_20170207170255
2017-02-07 17:07:53,238 INFO [115866407@qtp-1054818236-2339] client.HBaseAdmin: Started disable of urls_table_50001_20170207170502
2017-02-07 17:07:59,953 INFO [115866407@qtp-1054818236-2339] client.HBaseAdmin: Disabled urls_table_50001_20170207170502
2017-02-07 17:08:02,128 INFO [115866407@qtp-1054818236-2339] client.HBaseAdmin: Deleted urls_table_50001_20170207170502
2017-02-07 17:08:04,585 INFO [328489809@qtp-1054818236-2207] client.HBaseAdmin: Started disable of urls_table_50023_20170207170455
2017-02-07 17:08:04,791 INFO [1227614359@qtp-1054818236-647] client.HBaseAdmin: Started disable of urls_table_50028_20170207170311
2017-02-07 17:08:05,664 INFO [1252180146@qtp-1054818236-143] client.HBaseAdmin: Started disable of urls_table_50036_20170207170458
2017-02-07 17:08:07,783 INFO [328489809@qtp-1054818236-2207] client.HBaseAdmin: Disabled urls_table_50023_20170207170455
2017-02-07 17:08:14,066 INFO [1474938506@qtp-1054818236-2246] client.AsyncProcess: #1361530, waiting for some tasks to finish. Expected max=0, tasksInProgress=7
2017-02-07 17:08:14,167 INFO [865578926@qtp-1054818236-1712] client.AsyncProcess: #1361562, waiting for some tasks to finish. Expected max=0, tasksInProgress=7
2017-02-07 17:08:14,267 INFO [1276865244@qtp-1054818236-1407] client.AsyncProcess: #1361556, waiting for some tasks to finish. Expected max=0, tasksInProgress=7
2017-02-07 17:08:14,267 INFO [1368235409@qtp-1054818236-315] client.AsyncProcess: #1361671, waiting for some tasks to finish. Expected max=0, tasksInProgress=7
```
Akshat
Re: Hbase REST returns invalid status code
Posted by Ted Yu <yu...@gmail.com>.
Which release of hbase are you using ?
I was searching using the snippet from AsyncProcess and found HBASE-14431.
FYI
On Tue, Feb 7, 2017 at 6:29 PM, Akshat Mahajan <am...@brightedge.com>
wrote:
> We are seeing exceptions where Hbase REST is returning an invalid status
> code on writes, causing our Python processes to error out on trying to make
> a write (the following error is thrown by the Python starbase library, but
> is not caused by starbase directly):
>
> `requests.exceptions.ConnectionError: ('Connection aborted.',
> BadStatusLine("''",))`
>
> There are no traces nor errors reported in the REST log at the time of the
> exception. We know that this is consistent, as we retry thrice with
> exponential backoff (2 seconds initially).
>
> Our REST threads are allowed to grow dynamically from 100 to 9000 at
> worst; our connections typically do not exceed these values, and did not at
> the time.
>
> Are there circumstances where Hbase REST will deliberately refuse to honor
> a connection or respond with invalid codes? Or should I continue looking
> for more environmental reasons? Note that at the time REST was documenting
> wait on a number of AsyncProcesses on several tasks after having recently
> closed a table - not sure how useful that information is. A log snippet at
> the time from Hbase REST is below:
>
> ```
> 2017-02-07 17:07:35,312 INFO [196541187@qtp-1054818236-2084]
> client.HBaseAdmin: Started disable of urls_table_50007_20170207170255
> 2017-02-07 17:07:43,294 INFO [196541187@qtp-1054818236-2084]
> client.HBaseAdmin: Disabled urls_table_50007_20170207170255
> 2017-02-07 17:07:44,093 INFO [196541187@qtp-1054818236-2084]
> client.HBaseAdmin: Deleted urls_table_50007_20170207170255
> 2017-02-07 17:07:53,238 INFO [115866407@qtp-1054818236-2339]
> client.HBaseAdmin: Started disable of urls_table_50001_20170207170502
> 2017-02-07 17:07:59,953 INFO [115866407@qtp-1054818236-2339]
> client.HBaseAdmin: Disabled urls_table_50001_20170207170502
> 2017-02-07 17:08:02,128 INFO [115866407@qtp-1054818236-2339]
> client.HBaseAdmin: Deleted urls_table_50001_20170207170502
> 2017-02-07 17:08:04,585 INFO [328489809@qtp-1054818236-2207]
> client.HBaseAdmin: Started disable of urls_table_50023_20170207170455
> 2017-02-07 17:08:04,791 INFO [1227614359@qtp-1054818236-647]
> client.HBaseAdmin: Started disable of urls_table_50028_20170207170311
> 2017-02-07 17:08:05,664 INFO [1252180146@qtp-1054818236-143]
> client.HBaseAdmin: Started disable of urls_table_50036_20170207170458
> 2017-02-07 17:08:07,783 INFO [328489809@qtp-1054818236-2207]
> client.HBaseAdmin: Disabled urls_table_50023_20170207170455
> 2017-02-07 17:08:14,066 INFO [1474938506@qtp-1054818236-2246]
> client.AsyncProcess: #1361530, waiting for some tasks to finish. Expected
> max=0, tasksInProgress=7
> 2017-02-07 17:08:14,167 INFO [865578926@qtp-1054818236-1712]
> client.AsyncProcess: #1361562, waiting for some tasks to finish. Expected
> max=0, tasksInProgress=7
> 2017-02-07 17:08:14,267 INFO [1276865244@qtp-1054818236-1407]
> client.AsyncProcess: #1361556, waiting for some tasks to finish. Expected
> max=0, tasksInProgress=7
> 2017-02-07 17:08:14,267 INFO [1368235409@qtp-1054818236-315]
> client.AsyncProcess: #1361671, waiting for some tasks to finish. Expected
> max=0, tasksInProgress=7
> ```
>
> Akshat
>