You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by Sumit Nigam <su...@yahoo.com> on 2016/03/22 15:19:10 UTC
Secondary index memory footprint
Hi,
I am trying to estimate what (if any) are the implications of accumulating data in phoenix secondary index. I have a secondary index on 3 columns and would like to know if anyone has an idea of how to estimate memory footprint of secondary index (if any) based on number of entries in data table. The index is global and not local.
Plus, what is the impact of setting IN_MEMORY=true when index is created?
Thanks,Sumit
Re: Secondary index memory footprint
Posted by Anil Gupta <an...@gmail.com>.
I dont know the meaning in_memory=false in context of phoenix. In context of hbase, that entire data will be kept in memory( not just block cache). I m guessing, Phoenix is also doing same thing.
By data size on disk, you can roughly estimate memory footprint. For example, You can load one index into memory by setting in_memory flag and see how much memory it took. Then u can estimate memory usage for other index on basis of your experiment.
Sent from my iPhone
> On Mar 22, 2016, at 9:05 AM, Sumit Nigam <su...@yahoo.com> wrote:
>
> Thank you Anil.
>
> Any inputs on what does IN_MEMORY = false do at index creation time?
>
> I get the separate table part. But not sure I understood why would memory footprint be equal to data size. Would this secondary index reside entirely in memory? I guess only parts of it would reside on block cache, right?
>
> Thanks for such a prompt reply.
> Sumit
>
>
>
> From: anil gupta <an...@gmail.com>
> To: "user@phoenix.apache.org" <us...@phoenix.apache.org>; Sumit Nigam <su...@yahoo.com>
> Sent: Tuesday, March 22, 2016 9:11 PM
> Subject: Re: Secondary index memory footprint
>
> Global indexes are stored in a separate hbase table. So, you can estimate memory footprint by looking at the data size of that index currently.
>
> HTH,
> Anil Gupta
>
> On Tue, Mar 22, 2016 at 7:19 AM, Sumit Nigam <su...@yahoo.com> wrote:
> Hi,
>
> I am trying to estimate what (if any) are the implications of accumulating data in phoenix secondary index. I have a secondary index on 3 columns and would like to know if anyone has an idea of how to estimate memory footprint of secondary index (if any) based on number of entries in data table. The index is global and not local.
>
> Plus, what is the impact of setting IN_MEMORY=true when index is created?
>
> Thanks,
> Sumit
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
>
Re: Secondary index memory footprint
Posted by Sumit Nigam <su...@yahoo.com>.
Thank you Anil.
Any inputs on what does IN_MEMORY = false do at index creation time?
I get the separate table part. But not sure I understood why would memory footprint be equal to data size. Would this secondary index reside entirely in memory? I guess only parts of it would reside on block cache, right?
Thanks for such a prompt reply.Sumit
From: anil gupta <an...@gmail.com>
To: "user@phoenix.apache.org" <us...@phoenix.apache.org>; Sumit Nigam <su...@yahoo.com>
Sent: Tuesday, March 22, 2016 9:11 PM
Subject: Re: Secondary index memory footprint
Global indexes are stored in a separate hbase table. So, you can estimate memory footprint by looking at the data size of that index currently.
HTH,
Anil Gupta
On Tue, Mar 22, 2016 at 7:19 AM, Sumit Nigam <su...@yahoo.com> wrote:
Hi,
I am trying to estimate what (if any) are the implications of accumulating data in phoenix secondary index. I have a secondary index on 3 columns and would like to know if anyone has an idea of how to estimate memory footprint of secondary index (if any) based on number of entries in data table. The index is global and not local.
Plus, what is the impact of setting IN_MEMORY=true when index is created?
Thanks,Sumit
--
Thanks & Regards,
Anil Gupta
Re: Secondary index memory footprint
Posted by anil gupta <an...@gmail.com>.
Global indexes are stored in a separate hbase table. So, you can estimate
memory footprint by looking at the data size of that index currently.
HTH,
Anil Gupta
On Tue, Mar 22, 2016 at 7:19 AM, Sumit Nigam <su...@yahoo.com> wrote:
> Hi,
>
> I am trying to estimate what (if any) are the implications of accumulating
> data in phoenix secondary index. I have a secondary index on 3 columns and
> would like to know if anyone has an idea of how to estimate memory
> footprint of secondary index (if any) based on number of entries in data
> table. The index is global and not local.
>
> Plus, what is the impact of setting IN_MEMORY=true when index is created?
>
> Thanks,
> Sumit
>
--
Thanks & Regards,
Anil Gupta