You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by Mac Fang <ma...@gmail.com> on 2017/04/12 07:12:27 UTC
Fwd: Weird High Read throuput of SYSTEM.STATS
Hi, Guys,
We are notices some weird high read throuput of the HBase,
The HBase Read Rate
And the SYSTEM.STATS Read Rate
During that time frame, the system did not have a high QPS. However, it did
import some data (some millions) data via the "Phoenix Bulk Load".
The question is what does phoenix do with the SYSTEM.STATS table when we do
"Bulk Load"?
We did not find any clues while we looked into the code. Any hints ?
--
regards
macf
Re: Weird High Read throuput of SYSTEM.STATS
Posted by Mac Fang <ma...@gmail.com>.
Is there any configuration that I can disable / decrease the frequencies
of cache refreshing. I was noticed by a "phoenix.stats.updateFrequency",
but i am not 100% sure if it is reasonable to change this which can avoid
frequent cache refresh
On Fri, Apr 21, 2017 at 1:32 PM, Ankit Singhal <an...@gmail.com>
wrote:
> During bulkload, compaction might be getting triggered which is generating
> new stats ( and delete the stale one's with a scan) then the table
> cache(containg stats) will also get refreshed (with another scan).
> Depending upon the no. of time the compactions is triggered and no. of
> times the stats needs to be cached again might be correlated with your
> observation.
>
> On Wed, Apr 19, 2017 at 4:10 AM, Sergey Soldatov <sergeysoldatov@gmail.com
> > wrote:
>
>> The bulk load itself should not cause any read activity on SYSTEM.STATS.
>> But if you run something like select count(1) from table to check whether
>> the bulk load was successful, it may cause read activity on SYSTEM.STATS
>> table.
>>
>> Thanks,
>> Sergey
>>
>> On Wed, Apr 12, 2017 at 10:38 PM, Mac Fang <ma...@gmail.com>
>> wrote:
>>
>>> Josh,
>>>
>>> Thanks for the reply,
>>> It is a CDH vendor version (4.7.0_1.3.0). Mutations are reasonable when
>>> we do bulk load. Question is why we have the high read load of the
>>> SYSTEM.STATS table.
>>>
>>>
>>> On Thu, Apr 13, 2017 at 11:42 AM, Josh Elser <jo...@gmail.com>
>>> wrote:
>>>
>>>> What version of Phoenix are you using? (an Apache release? some
>>>> vendors' packaging?)
>>>>
>>>> Academically speaking, when you bulk load some data, the stats table
>>>> should get updated (otherwise the stats are wrong until a compaction
>>>> occurs), but I can't specifically point you at a line of code that is doing
>>>> this (nor am I 100% positive it happens).
>>>>
>>>>
>>>> On Wed, Apr 12, 2017 at 3:12 AM, Mac Fang <ma...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi, Guys,
>>>>>
>>>>>
>>>>> We are notices some weird high read throuput of the HBase,
>>>>>
>>>>> The HBase Read Rate
>>>>>
>>>>>
>>>>>
>>>>> And the SYSTEM.STATS Read Rate
>>>>>
>>>>>
>>>>>
>>>>> During that time frame, the system did not have a high QPS. However,
>>>>> it did import some data (some millions) data via the "Phoenix Bulk Load".
>>>>>
>>>>> The question is what does phoenix do with the SYSTEM.STATS table when
>>>>> we do "Bulk Load"?
>>>>>
>>>>> We did not find any clues while we looked into the code. Any hints ?
>>>>>
>>>>>
>>>>> --
>>>>> regards
>>>>> macf
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> regards
>>> macf
>>>
>>
>>
>
--
regards
macf
Re: Weird High Read throuput of SYSTEM.STATS
Posted by Ankit Singhal <an...@gmail.com>.
During bulkload, compaction might be getting triggered which is generating
new stats ( and delete the stale one's with a scan) then the table
cache(containg stats) will also get refreshed (with another scan).
Depending upon the no. of time the compactions is triggered and no. of
times the stats needs to be cached again might be correlated with your
observation.
On Wed, Apr 19, 2017 at 4:10 AM, Sergey Soldatov <se...@gmail.com>
wrote:
> The bulk load itself should not cause any read activity on SYSTEM.STATS.
> But if you run something like select count(1) from table to check whether
> the bulk load was successful, it may cause read activity on SYSTEM.STATS
> table.
>
> Thanks,
> Sergey
>
> On Wed, Apr 12, 2017 at 10:38 PM, Mac Fang <ma...@gmail.com>
> wrote:
>
>> Josh,
>>
>> Thanks for the reply,
>> It is a CDH vendor version (4.7.0_1.3.0). Mutations are reasonable when
>> we do bulk load. Question is why we have the high read load of the
>> SYSTEM.STATS table.
>>
>>
>> On Thu, Apr 13, 2017 at 11:42 AM, Josh Elser <jo...@gmail.com>
>> wrote:
>>
>>> What version of Phoenix are you using? (an Apache release? some vendors'
>>> packaging?)
>>>
>>> Academically speaking, when you bulk load some data, the stats table
>>> should get updated (otherwise the stats are wrong until a compaction
>>> occurs), but I can't specifically point you at a line of code that is doing
>>> this (nor am I 100% positive it happens).
>>>
>>>
>>> On Wed, Apr 12, 2017 at 3:12 AM, Mac Fang <ma...@gmail.com>
>>> wrote:
>>>
>>>> Hi, Guys,
>>>>
>>>>
>>>> We are notices some weird high read throuput of the HBase,
>>>>
>>>> The HBase Read Rate
>>>>
>>>>
>>>>
>>>> And the SYSTEM.STATS Read Rate
>>>>
>>>>
>>>>
>>>> During that time frame, the system did not have a high QPS. However, it
>>>> did import some data (some millions) data via the "Phoenix Bulk Load".
>>>>
>>>> The question is what does phoenix do with the SYSTEM.STATS table when
>>>> we do "Bulk Load"?
>>>>
>>>> We did not find any clues while we looked into the code. Any hints ?
>>>>
>>>>
>>>> --
>>>> regards
>>>> macf
>>>>
>>>
>>>
>>
>>
>> --
>> regards
>> macf
>>
>
>
Re: Weird High Read throuput of SYSTEM.STATS
Posted by Sergey Soldatov <se...@gmail.com>.
The bulk load itself should not cause any read activity on SYSTEM.STATS.
But if you run something like select count(1) from table to check whether
the bulk load was successful, it may cause read activity on SYSTEM.STATS
table.
Thanks,
Sergey
On Wed, Apr 12, 2017 at 10:38 PM, Mac Fang <ma...@gmail.com>
wrote:
> Josh,
>
> Thanks for the reply,
> It is a CDH vendor version (4.7.0_1.3.0). Mutations are reasonable when we
> do bulk load. Question is why we have the high read load of the
> SYSTEM.STATS table.
>
>
> On Thu, Apr 13, 2017 at 11:42 AM, Josh Elser <jo...@gmail.com> wrote:
>
>> What version of Phoenix are you using? (an Apache release? some vendors'
>> packaging?)
>>
>> Academically speaking, when you bulk load some data, the stats table
>> should get updated (otherwise the stats are wrong until a compaction
>> occurs), but I can't specifically point you at a line of code that is doing
>> this (nor am I 100% positive it happens).
>>
>>
>> On Wed, Apr 12, 2017 at 3:12 AM, Mac Fang <ma...@gmail.com>
>> wrote:
>>
>>> Hi, Guys,
>>>
>>>
>>> We are notices some weird high read throuput of the HBase,
>>>
>>> The HBase Read Rate
>>>
>>>
>>>
>>> And the SYSTEM.STATS Read Rate
>>>
>>>
>>>
>>> During that time frame, the system did not have a high QPS. However, it
>>> did import some data (some millions) data via the "Phoenix Bulk Load".
>>>
>>> The question is what does phoenix do with the SYSTEM.STATS table when we
>>> do "Bulk Load"?
>>>
>>> We did not find any clues while we looked into the code. Any hints ?
>>>
>>>
>>> --
>>> regards
>>> macf
>>>
>>
>>
>
>
> --
> regards
> macf
>
Re: Weird High Read throuput of SYSTEM.STATS
Posted by Josh Elser <jo...@gmail.com>.
Thanks for the version info. I'm not sure about what's included in CDH's
packaging -- maybe someone else knows.
I understand that your question was about the read load. However, the
old guideposts likely needed to be updated with the new files' stats.
Thus, even though you're writing data to your table, to update the stats
for that table, it would involve a read of the *existing* stats data.
But, again, that's just a guess :)
Mac Fang wrote:
> Josh,
>
> Thanks for the reply,
> It is a CDH vendor version (4.7.0_1.3.0). Mutations are reasonable when
> we do bulk load. Question is why we have the high read load of the
> SYSTEM.STATS table.
>
>
> On Thu, Apr 13, 2017 at 11:42 AM, Josh Elser <josh.elser@gmail.com
> <ma...@gmail.com>> wrote:
>
> What version of Phoenix are you using? (an Apache release? some
> vendors' packaging?)
>
> Academically speaking, when you bulk load some data, the stats table
> should get updated (otherwise the stats are wrong until a compaction
> occurs), but I can't specifically point you at a line of code that
> is doing this (nor am I 100% positive it happens).
>
>
> On Wed, Apr 12, 2017 at 3:12 AM, Mac Fang
> <mac.fangzhen.dev@gmail.com <ma...@gmail.com>> wrote:
>
> Hi, Guys,
>
>
> We are notices some weird high read throuput of the HBase,
>
> The HBase Read Rate
>
>
> \u200b
> And the SYSTEM.STATS Read Rate
>
>
> \u200b
> During that time frame, the system did not have a high QPS.
> However, it did import some data (some millions) data via the
> "Phoenix Bulk Load".
>
> The question is what does phoenix do with the SYSTEM.STATS table
> when we do "Bulk Load"?
>
> We did not find any clues while we looked into the code. Any
> hints ?
>
>
> --
> regards
> macf
>
>
>
>
>
> --
> regards
> macf
Re: Weird High Read throuput of SYSTEM.STATS
Posted by Mac Fang <ma...@gmail.com>.
Josh,
Thanks for the reply,
It is a CDH vendor version (4.7.0_1.3.0). Mutations are reasonable when we
do bulk load. Question is why we have the high read load of the
SYSTEM.STATS table.
On Thu, Apr 13, 2017 at 11:42 AM, Josh Elser <jo...@gmail.com> wrote:
> What version of Phoenix are you using? (an Apache release? some vendors'
> packaging?)
>
> Academically speaking, when you bulk load some data, the stats table
> should get updated (otherwise the stats are wrong until a compaction
> occurs), but I can't specifically point you at a line of code that is doing
> this (nor am I 100% positive it happens).
>
>
> On Wed, Apr 12, 2017 at 3:12 AM, Mac Fang <ma...@gmail.com>
> wrote:
>
>> Hi, Guys,
>>
>>
>> We are notices some weird high read throuput of the HBase,
>>
>> The HBase Read Rate
>>
>>
>>
>> And the SYSTEM.STATS Read Rate
>>
>>
>>
>> During that time frame, the system did not have a high QPS. However, it
>> did import some data (some millions) data via the "Phoenix Bulk Load".
>>
>> The question is what does phoenix do with the SYSTEM.STATS table when we
>> do "Bulk Load"?
>>
>> We did not find any clues while we looked into the code. Any hints ?
>>
>>
>> --
>> regards
>> macf
>>
>
>
--
regards
macf
Re: Weird High Read throuput of SYSTEM.STATS
Posted by Josh Elser <jo...@gmail.com>.
What version of Phoenix are you using? (an Apache release? some vendors'
packaging?)
Academically speaking, when you bulk load some data, the stats table should
get updated (otherwise the stats are wrong until a compaction occurs), but
I can't specifically point you at a line of code that is doing this (nor am
I 100% positive it happens).
On Wed, Apr 12, 2017 at 3:12 AM, Mac Fang <ma...@gmail.com>
wrote:
> Hi, Guys,
>
>
> We are notices some weird high read throuput of the HBase,
>
> The HBase Read Rate
>
>
>
> And the SYSTEM.STATS Read Rate
>
>
>
> During that time frame, the system did not have a high QPS. However, it
> did import some data (some millions) data via the "Phoenix Bulk Load".
>
> The question is what does phoenix do with the SYSTEM.STATS table when we
> do "Bulk Load"?
>
> We did not find any clues while we looked into the code. Any hints ?
>
>
> --
> regards
> macf
>