Phoenix Performance Improvement

[Azhar Shaikh <az...@gmail.com>]

Hi,

We have an Hbase with 5 million records which has 3 region-servers(8 Core
and 32 GB Ram each) with 20 regions. On Daily basis we have around 10k of
records feed into our hbase table. We are performing queries on Hbase table
via Java application using Phoenix query engine. Since few days we are
facing an issue that whenever we execute multiple queries via phoenix it is
not able to fetch the results within 30ms due to which we are now getting
timeout error on our application side.


When our data was in range of 2-3 million we had not notice such issue but
as the data has increased we have started facing timeout when we have
multiple users fetching data via application.

We have done below testing but there is no improvement noticed

1. Moved from SAN storage to SSD storage
2. Increased CPU from 4 cores to 8 cores on each Region-servers
3. Scaled Region Servers from 3 to 5 but no improvement
4. Tried various combination of SALT Bucket(10,20,40)
5. Increased hfile.block.cache.size from 40% to 60%
6. Increasing Phoenic scanner timeout value
7. Added secondary Index

Even after performing above steps we have not seen any improvement in
phoenix performance. Request you to please let us know how we can improvise
the performance of phoenix.

Hbase Version - 1.2.3
Phoenix Version - 4.8

Your help is greatly appreciated.




--
Sent from: http://apache-phoenix-user-list.1124778.n5.nabble.com/

Re: Phoenix Performance Improvement

[Thomas D'Silva <td...@salesforce.com>]

Can you provide the schema of the table and the queries you are running?

On Tue, Jan 15, 2019 at 12:35 PM Azharuddin Shaikh <az...@gmail.com>
wrote:

> Thanks Pedro for your response.
>
> Actually we have an status column which consists of various status and we
> are executing queries which select queries which are fetching data from
> Hbase tables basis on status clause in the query and same is displayed on
> the application dashboard.
>
> An average daily ingestion is of 10k rows  to the existing table which is
> currently at 5M+.
>
> We had tried secondary index to avoid full scan on table but when queries
> are executed parallelly then timeout occurs after 30ms.
>
> It is working fine if we have only 1instance of an application running but
> if there are multiple application instances then we are experiencing
> timeouts.
>
>
> On Tue, 15 Jan, 2019, 9:47 PM Pedro Boado, <pb...@apache.org> wrote:
>
>> What type of queries are being thrown to the cluster? What's the average
>> row size? 5M rows seems a tiny table size.  30ms is OK for scans over a few
>> thousand records, but maybe not for full table scans.
>>
>

Re: Phoenix Performance Improvement

[Azharuddin Shaikh <az...@gmail.com>]

Thanks Pedro for your response.

Actually we have an status column which consists of various status and we
are executing queries which select queries which are fetching data from
Hbase tables basis on status clause in the query and same is displayed on
the application dashboard.

An average daily ingestion is of 10k rows  to the existing table which is
currently at 5M+.

We had tried secondary index to avoid full scan on table but when queries
are executed parallelly then timeout occurs after 30ms.

It is working fine if we have only 1instance of an application running but
if there are multiple application instances then we are experiencing
timeouts.


On Tue, 15 Jan, 2019, 9:47 PM Pedro Boado, <pb...@apache.org> wrote:

> What type of queries are being thrown to the cluster? What's the average
> row size? 5M rows seems a tiny table size.  30ms is OK for scans over a few
> thousand records, but maybe not for full table scans.
>

Re: Phoenix Performance Improvement

[Pedro Boado <pb...@apache.org>]

What type of queries are being thrown to the cluster? What's the average
row size? 5M rows seems a tiny table size.  30ms is OK for scans over a few
thousand records, but maybe not for full table scans.