You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by Sergey Malov <sm...@collective.com> on 2015/07/07 01:05:37 UTC
Re: create a view on existing production table ?
Thanks for your replies, guys.
I’m trying to understand that dynamic column feature, not sure how to apply it in my case.
Let’s look at the concrete example : suppose I have an existing Hbase table “X", which has only one column family “A”. Each record has either one or two or three columns with qualifiers “A1”,”A2”, or “A3”, I.e. some records has only one column A.A1, or A.A2, or A.A3, some just two, some all three.
How can I create a view using dynamic columns that I can query later ?
Thank you,
Sergey Malov
From: anil gupta <an...@gmail.com>>
Reply-To: "user@phoenix.apache.org<ma...@phoenix.apache.org>" <us...@phoenix.apache.org>>
Date: Friday, June 26, 2015 at 18:28
To: "user@phoenix.apache.org<ma...@phoenix.apache.org>" <us...@phoenix.apache.org>>
Subject: Re: create a view on existing production table ?
Hi Sergey,
Since you have hundreds of thousand of columns. You can query your data by using dynamic columns features of phoenix. In this way, you wont need to predefine 100's of thousands of columns.
Thanks,
Anil Gupta
On Fri, Jun 26, 2015 at 11:34 AM, James Taylor <ja...@apache.org>> wrote:
Sergey,
It is possible, but maybe in your case it's not feasible.
Thanks,
James
On Friday, June 26, 2015, Sergey Malov <sm...@collective.com>> wrote:
Hi James,
>From you reply I understand that it is NOT possible to create such a view, because each family can have different number of columns, and it could be just one column per family for one PK, and hundreds of thousands for another PK. How can I possibly accommodate it in a view specification, if I need to explicitly define column by name ?
Or I misunderstand something ?
Thank you,
Sergey
From: James Taylor <ja...@apache.org>
Reply-To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
Date: Friday, June 26, 2015 at 12:04
To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
Subject: Re: create a view on existing production table ?
Hi Sergey,
Yes, you can create a Phoenix view over this HBase table, but you have to explicitly list columns by name (i.e. column qualifier) either at view creation time or at read time (using dynamic columns). Also, the row key must conform to what Phoenix expects if there are multiple columns in your PK.
Thanks,
James
On Wednesday, June 24, 2015, Sergey Malov <sm...@collective.com> wrote:
My apologies if this info can be found somewhere, I don’t see it.
I have a production table “profile” with the following schema :
{NAME => 'edge', BLOOMFILTER => 'ROW', VERSIONS => '2', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'false', DATA_BLOCK_ENCODING => 'FAST_DIFF', COMPRESSION => 'SNAPPY', TTL => '
FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
{NAME => 'export', BLOOMFILTER => 'ROW', VERSIONS => '2', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'false', DATA_BLOCK_ENCODING => 'FAST_DIFF', COMPRESSION => 'SNAPPY', TTL =>
'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
{NAME => 'visitor', BLOOMFILTER => 'ROW', VERSIONS => '2', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'false', DATA_BLOCK_ENCODING => 'FAST_DIFF', COMPRESSION => 'SNAPPY', TTL =
> 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
Each of these families can have variable number of column for a given key, with some having hundreds of thousands columns.
Is it possible to create a Phoenix view for such a table ?
Thank you,
Sergey Malov
--
Thanks & Regards,
Anil Gupta
Re: create a view on existing production table ?
Posted by Anil Gupta <an...@gmail.com>.
Yup, that's right. Sql takes away flexibility of NoSql.
I have been battling with this tradeoff for a while. ;)
Sent from my iPhone
> On Jul 7, 2015, at 6:46 AM, Sergey Malov <sm...@collective.com> wrote:
>
> Thanks, Anil, that’s what I thought initially. I was a bit confused with what James wrote, that I can create a view, but it wouldn’t be feasible.
> Essentially, it seems, I need to impose a strict schema on Hbase table for Phoenix to work on it directly, which defeats a purpose of schema-less db imho. Not complaining, I understand that to marry SQL and non-SQL design, sacrifices should be made.
>
> Thank you,
> Sergey Malov
>
> From: anil gupta <an...@gmail.com>
> Reply-To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
> Date: Tuesday, July 7, 2015 at 02:49
> To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
> Subject: Re: create a view on existing production table ?
>
> Hi Sergey,
>
> You **can't** create a view that would work with dynamic columns. Dynamic columns are specified along with query(you can specify one, two or three columns, Phoenix will not complain) because Phoenix has no idea about existence of these columns(otherwise they wont be called Dynamic columns). Hope this clarifies further.
>
> Thanks,
> Anil Gupta
>
>> On Mon, Jul 6, 2015 at 4:05 PM, Sergey Malov <sm...@collective.com> wrote:
>> Thanks for your replies, guys.
>> I’m trying to understand that dynamic column feature, not sure how to apply it in my case.
>> Let’s look at the concrete example : suppose I have an existing Hbase table “X", which has only one column family “A”. Each record has either one or two or three columns with qualifiers “A1”,”A2”, or “A3”, I.e. some records has only one column A.A1, or A.A2, or A.A3, some just two, some all three.
>> How can I create a view using dynamic columns that I can query later ?
>>
>> Thank you,
>> Sergey Malov
>>
>> From: anil gupta <an...@gmail.com>
>> Reply-To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
>> Date: Friday, June 26, 2015 at 18:28
>>
>> To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
>> Subject: Re: create a view on existing production table ?
>>
>> Hi Sergey,
>>
>> Since you have hundreds of thousand of columns. You can query your data by using dynamic columns features of phoenix. In this way, you wont need to predefine 100's of thousands of columns.
>>
>> Thanks,
>> Anil Gupta
>>
>>> On Fri, Jun 26, 2015 at 11:34 AM, James Taylor <ja...@apache.org> wrote:
>>> Sergey,
>>> It is possible, but maybe in your case it's not feasible.
>>> Thanks,
>>> James
>>>
>>>
>>>> On Friday, June 26, 2015, Sergey Malov <sm...@collective.com> wrote:
>>>> Hi James,
>>>> From you reply I understand that it is NOT possible to create such a view, because each family can have different number of columns, and it could be just one column per family for one PK, and hundreds of thousands for another PK. How can I possibly accommodate it in a view specification, if I need to explicitly define column by name ?
>>>> Or I misunderstand something ?
>>>>
>>>> Thank you,
>>>> Sergey
>>>>
>>>> From: James Taylor <ja...@apache.org>
>>>> Reply-To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
>>>> Date: Friday, June 26, 2015 at 12:04
>>>> To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
>>>> Subject: Re: create a view on existing production table ?
>>>>
>>>> Hi Sergey,
>>>> Yes, you can create a Phoenix view over this HBase table, but you have to explicitly list columns by name (i.e. column qualifier) either at view creation time or at read time (using dynamic columns). Also, the row key must conform to what Phoenix expects if there are multiple columns in your PK.
>>>> Thanks,
>>>> James
>>>>
>>>>> On Wednesday, June 24, 2015, Sergey Malov <sm...@collective.com> wrote:
>>>>> My apologies if this info can be found somewhere, I don’t see it.
>>>>> I have a production table “profile” with the following schema :
>>>>>
>>>>> {NAME => 'edge', BLOOMFILTER => 'ROW', VERSIONS => '2', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'false', DATA_BLOCK_ENCODING => 'FAST_DIFF', COMPRESSION => 'SNAPPY', TTL => '
>>>>> FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
>>>>> {NAME => 'export', BLOOMFILTER => 'ROW', VERSIONS => '2', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'false', DATA_BLOCK_ENCODING => 'FAST_DIFF', COMPRESSION => 'SNAPPY', TTL =>
>>>>> 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
>>>>> {NAME => 'visitor', BLOOMFILTER => 'ROW', VERSIONS => '2', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'false', DATA_BLOCK_ENCODING => 'FAST_DIFF', COMPRESSION => 'SNAPPY', TTL =
>>>>> > 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
>>>>>
>>>>> Each of these families can have variable number of column for a given key, with some having hundreds of thousands columns.
>>>>> Is it possible to create a Phoenix view for such a table ?
>>>>>
>>>>> Thank you,
>>>>> Sergey Malov
>>
>>
>>
>> --
>> Thanks & Regards,
>> Anil Gupta
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
Re: create a view on existing production table ?
Posted by Sergey Malov <sm...@collective.com>.
Thanks, Anil, that’s what I thought initially. I was a bit confused with what James wrote, that I can create a view, but it wouldn’t be feasible.
Essentially, it seems, I need to impose a strict schema on Hbase table for Phoenix to work on it directly, which defeats a purpose of schema-less db imho. Not complaining, I understand that to marry SQL and non-SQL design, sacrifices should be made.
Thank you,
Sergey Malov
From: anil gupta <an...@gmail.com>>
Reply-To: "user@phoenix.apache.org<ma...@phoenix.apache.org>" <us...@phoenix.apache.org>>
Date: Tuesday, July 7, 2015 at 02:49
To: "user@phoenix.apache.org<ma...@phoenix.apache.org>" <us...@phoenix.apache.org>>
Subject: Re: create a view on existing production table ?
Hi Sergey,
You **can't** create a view that would work with dynamic columns. Dynamic columns are specified along with query(you can specify one, two or three columns, Phoenix will not complain) because Phoenix has no idea about existence of these columns(otherwise they wont be called Dynamic columns). Hope this clarifies further.
Thanks,
Anil Gupta
On Mon, Jul 6, 2015 at 4:05 PM, Sergey Malov <sm...@collective.com>> wrote:
Thanks for your replies, guys.
I’m trying to understand that dynamic column feature, not sure how to apply it in my case.
Let’s look at the concrete example : suppose I have an existing Hbase table “X", which has only one column family “A”. Each record has either one or two or three columns with qualifiers “A1”,”A2”, or “A3”, I.e. some records has only one column A.A1, or A.A2, or A.A3, some just two, some all three.
How can I create a view using dynamic columns that I can query later ?
Thank you,
Sergey Malov
From: anil gupta <an...@gmail.com>>
Reply-To: "user@phoenix.apache.org<ma...@phoenix.apache.org>" <us...@phoenix.apache.org>>
Date: Friday, June 26, 2015 at 18:28
To: "user@phoenix.apache.org<ma...@phoenix.apache.org>" <us...@phoenix.apache.org>>
Subject: Re: create a view on existing production table ?
Hi Sergey,
Since you have hundreds of thousand of columns. You can query your data by using dynamic columns features of phoenix. In this way, you wont need to predefine 100's of thousands of columns.
Thanks,
Anil Gupta
On Fri, Jun 26, 2015 at 11:34 AM, James Taylor <ja...@apache.org>> wrote:
Sergey,
It is possible, but maybe in your case it's not feasible.
Thanks,
James
On Friday, June 26, 2015, Sergey Malov <sm...@collective.com>> wrote:
Hi James,
>From you reply I understand that it is NOT possible to create such a view, because each family can have different number of columns, and it could be just one column per family for one PK, and hundreds of thousands for another PK. How can I possibly accommodate it in a view specification, if I need to explicitly define column by name ?
Or I misunderstand something ?
Thank you,
Sergey
From: James Taylor <ja...@apache.org>
Reply-To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
Date: Friday, June 26, 2015 at 12:04
To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
Subject: Re: create a view on existing production table ?
Hi Sergey,
Yes, you can create a Phoenix view over this HBase table, but you have to explicitly list columns by name (i.e. column qualifier) either at view creation time or at read time (using dynamic columns). Also, the row key must conform to what Phoenix expects if there are multiple columns in your PK.
Thanks,
James
On Wednesday, June 24, 2015, Sergey Malov <sm...@collective.com> wrote:
My apologies if this info can be found somewhere, I don’t see it.
I have a production table “profile” with the following schema :
{NAME => 'edge', BLOOMFILTER => 'ROW', VERSIONS => '2', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'false', DATA_BLOCK_ENCODING => 'FAST_DIFF', COMPRESSION => 'SNAPPY', TTL => '
FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
{NAME => 'export', BLOOMFILTER => 'ROW', VERSIONS => '2', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'false', DATA_BLOCK_ENCODING => 'FAST_DIFF', COMPRESSION => 'SNAPPY', TTL =>
'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
{NAME => 'visitor', BLOOMFILTER => 'ROW', VERSIONS => '2', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'false', DATA_BLOCK_ENCODING => 'FAST_DIFF', COMPRESSION => 'SNAPPY', TTL =
> 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '1'}
Each of these families can have variable number of column for a given key, with some having hundreds of thousands columns.
Is it possible to create a Phoenix view for such a table ?
Thank you,
Sergey Malov
--
Thanks & Regards,
Anil Gupta
--
Thanks & Regards,
Anil Gupta
Re: create a view on existing production table ?
Posted by anil gupta <an...@gmail.com>.
Hi Sergey,
You **can't** create a view that would work with dynamic columns. Dynamic
columns are specified along with query(you can specify one, two or three
columns, Phoenix will not complain) because Phoenix has no idea about
existence of these columns(otherwise they wont be called Dynamic columns).
Hope this clarifies further.
Thanks,
Anil Gupta
On Mon, Jul 6, 2015 at 4:05 PM, Sergey Malov <sm...@collective.com> wrote:
> Thanks for your replies, guys.
> I’m trying to understand that dynamic column feature, not sure how to
> apply it in my case.
> Let’s look at the concrete example : suppose I have an existing Hbase
> table “X", which has only one column family “A”. Each record has either one
> or two or three columns with qualifiers “A1”,”A2”, or “A3”, I.e. some
> records has only one column A.A1, or A.A2, or A.A3, some just two, some all
> three.
> How can I create a view using dynamic columns that I can query later ?
>
> Thank you,
> Sergey Malov
>
> From: anil gupta <an...@gmail.com>
> Reply-To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
> Date: Friday, June 26, 2015 at 18:28
>
> To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
> Subject: Re: create a view on existing production table ?
>
> Hi Sergey,
>
> Since you have hundreds of thousand of columns. You can query your data
> by using dynamic columns features of phoenix. In this way, you wont need to
> predefine 100's of thousands of columns.
>
> Thanks,
> Anil Gupta
>
> On Fri, Jun 26, 2015 at 11:34 AM, James Taylor <ja...@apache.org>
> wrote:
>
>> Sergey,
>> It is possible, but maybe in your case it's not feasible.
>> Thanks,
>> James
>>
>>
>> On Friday, June 26, 2015, Sergey Malov <sm...@collective.com> wrote:
>>
>>> Hi James,
>>> From you reply I understand that it is NOT possible to create such a
>>> view, because each family can have different number of columns, and it
>>> could be just one column per family for one PK, and hundreds of thousands
>>> for another PK. How can I possibly accommodate it in a view specification,
>>> if I need to explicitly define column by name ?
>>> Or I misunderstand something ?
>>>
>>> Thank you,
>>> Sergey
>>>
>>> From: James Taylor <ja...@apache.org>
>>> Reply-To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
>>> Date: Friday, June 26, 2015 at 12:04
>>> To: "user@phoenix.apache.org" <us...@phoenix.apache.org>
>>> Subject: Re: create a view on existing production table ?
>>>
>>> Hi Sergey,
>>> Yes, you can create a Phoenix view over this HBase table, but you have
>>> to explicitly list columns by name (i.e. column qualifier) either at view
>>> creation time or at read time (using dynamic columns). Also, the row key
>>> must conform to what Phoenix expects if there are multiple columns in your
>>> PK.
>>> Thanks,
>>> James
>>>
>>> On Wednesday, June 24, 2015, Sergey Malov <sm...@collective.com> wrote:
>>>
>>>> My apologies if this info can be found somewhere, I don’t see it.
>>>> I have a production table “profile” with the following schema :
>>>>
>>>> {NAME => 'edge', BLOOMFILTER => 'ROW', VERSIONS => '2', IN_MEMORY =>
>>>> 'false', KEEP_DELETED_CELLS => 'false', DATA_BLOCK_ENCODING => 'FAST_DIFF',
>>>> COMPRESSION => 'SNAPPY', TTL => '
>>>>
>>>> FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE =>
>>>> '65536', REPLICATION_SCOPE => '1'}
>>>>
>>>>
>>>> {NAME => 'export', BLOOMFILTER => 'ROW', VERSIONS => '2', IN_MEMORY =>
>>>> 'false', KEEP_DELETED_CELLS => 'false', DATA_BLOCK_ENCODING => 'FAST_DIFF',
>>>> COMPRESSION => 'SNAPPY', TTL =>
>>>>
>>>> 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE =>
>>>> '65536', REPLICATION_SCOPE => '1'}
>>>>
>>>>
>>>> {NAME => 'visitor', BLOOMFILTER => 'ROW', VERSIONS => '2', IN_MEMORY =>
>>>> 'false', KEEP_DELETED_CELLS => 'false', DATA_BLOCK_ENCODING => 'FAST_DIFF',
>>>> COMPRESSION => 'SNAPPY', TTL =
>>>>
>>>> > 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE =>
>>>> '65536', REPLICATION_SCOPE => '1'}
>>>>
>>>>
>>>> Each of these families can have variable number of column for a given
>>>> key, with some having hundreds of thousands columns.
>>>> Is it possible to create a Phoenix view for such a table ?
>>>>
>>>> Thank you,
>>>> Sergey Malov
>>>>
>>>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
--
Thanks & Regards,
Anil Gupta