You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by anil gupta <an...@gmail.com> on 2015/06/15 02:06:10 UTC
Re: column qualifier length
Hi Yanlin,
If you are doing reads. You can use a view.
For writes, i dont think Phoenix provides anything for aliasing column
names. It would be good if we have this feature because using short column
names is a common thing in HBase.
Thanks,
Anil Gupta
On Sun, Jun 14, 2015 at 12:19 PM, yanlin wang <wa...@gmail.com> wrote:
> Hi,
>
> Usually one method to reduce IO in hbase is to reduce the size of column
> qualifier’s length and use a dictionary to map the application column name
> to the actual column qualifier. Kiji schema project is doing exactly that.
> Is there any column name encoding support in Phoenix? if no, any plan to do
> that? Another option is that we can create phoenix table on top of kiji
> table, but kiji has its own column serializer which is based on avro and
> the Phoenix client needs to be modified to do the custom deserialization
> and mapping etc. The last resort will be putting these column name mapping
> biz into application DAO layer. Do you have any suggestion?
>
> Thx,
> Yanlin
--
Thanks & Regards,
Anil Gupta
Re: column qualifier length
Posted by Yanlin Wang <wa...@gmail.com>.
Got it. Thanks James, that's what I suspected.
We will wait for the feature :)
Thx,
Yanlin
Sent from my iPhone
> On Jun 14, 2015, at 7:30 PM, James Taylor <ja...@apache.org> wrote:
>
> Views won't help you here (and unrelated, but we only support "create view select *" but you can alter the view by removing columns after you create it).
>
> Anil - the space savings Yanlin is looking for is shorter column qualifiers while still having readable column names.
>
>> On Sunday, June 14, 2015, anil gupta <an...@gmail.com> wrote:
>> Hi James,
>>
>> Sorry, i dont understand the context of "Would be good to quantify the potential gain with and without FAST_DIFF encoding (the default) and/or Snappy compression."?
>> The feature Yanlin is requesting is more for ease of usability/readability.
>>
>>
>>> On Sun, Jun 14, 2015 at 5:59 PM, James Taylor <ja...@apache.org> wrote:
>>> Good idea. Please file a JIRA. Would be good to quantify the potential gain with and without FAST_DIFF encoding (the default) and/or Snappy compression.
>>>
>>>
>>>> On Sunday, June 14, 2015, yanlin wang <wa...@gmail.com> wrote:
>>>> Thanks for the reply Anil.
>>>>
>>>> Is this what you referring:
>>>> create view my view as
>>>> select a as appColumn1, b as appColum2 from table ?
>>>>
>>>> By reading the document, seems view will introduce limitations e.g schema update issue. It would be really good if phoenix put the auto aliasing column name support in its road map ��
>>>>
>>>>
>>>> Thx
>>>> Yanlin
>>>>
>>>>
>>>>
>>>>
>>>>> On Jun 14, 2015, at 5:06 PM, anil gupta <an...@gmail.com> wrote:
>>>>>
>>>>> Hi Yanlin,
>>>>>
>>>>> If you are doing reads. You can use a view.
>>>>> For writes, i dont think Phoenix provides anything for aliasing column names. It would be good if we have this feature because using short column names is a common thing in HBase.
>>>>>
>>>>> Thanks,
>>>>> Anil Gupta
>>>>>
>>>>>> On Sun, Jun 14, 2015 at 12:19 PM, yanlin wang <wa...@gmail.com> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Usually one method to reduce IO in hbase is to reduce the size of column qualifier��s length and use a dictionary to map the application column name to the actual column qualifier. Kiji schema project is doing exactly that. Is there any column name encoding support in Phoenix? if no, any plan to do that? Another option is that we can create phoenix table on top of kiji table, but kiji has its own column serializer which is based on avro and the Phoenix client needs to be modified to do the custom deserialization and mapping etc. The last resort will be putting these column name mapping biz into application DAO layer. Do you have any suggestion?
>>>>>>
>>>>>> Thx,
>>>>>> Yanlin
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks & Regards,
>>>>> Anil Gupta
>>
>>
>>
>> --
>> Thanks & Regards,
>> Anil Gupta
Re: column qualifier length
Posted by anil gupta <an...@gmail.com>.
Found the documentation on HBase site:
http://hbase.apache.org/book.html#keysize
On Sun, Jun 14, 2015 at 9:32 PM, anil gupta <an...@gmail.com> wrote:
> Hi James,
>
> AFAIK, this trick/practice is being recommended by HBase community. I
> cannot find the doc that says this.
>
> Thanks,
> Anil Gupta
>
> On Sun, Jun 14, 2015 at 7:30 PM, James Taylor <ja...@apache.org>
> wrote:
>
>> Views won't help you here (and unrelated, but we only support "create
>> view select *" but you can alter the view by removing columns after you
>> create it).
>>
>> Anil - the space savings Yanlin is looking for is shorter column
>> qualifiers while still having readable column names.
>>
>>
>> On Sunday, June 14, 2015, anil gupta <an...@gmail.com> wrote:
>>
>>> Hi James,
>>>
>>> Sorry, i dont understand the context of "Would be good to quantify the
>>> potential gain with and without FAST_DIFF encoding (the default) and/or
>>> Snappy compression."?
>>> The feature Yanlin is requesting is more for ease of
>>> usability/readability.
>>>
>>>
>>> On Sun, Jun 14, 2015 at 5:59 PM, James Taylor <ja...@apache.org>
>>> wrote:
>>>
>>>> Good idea. Please file a JIRA. Would be good to quantify the potential
>>>> gain with and without FAST_DIFF encoding (the default) and/or Snappy
>>>> compression.
>>>>
>>>>
>>>> On Sunday, June 14, 2015, yanlin wang <wa...@gmail.com> wrote:
>>>>
>>>>> Thanks for the reply Anil.
>>>>>
>>>>> Is this what you referring:
>>>>> create view my view as
>>>>> select a as appColumn1, b as appColum2 from table ?
>>>>>
>>>>> By reading the document, seems view will introduce limitations e.g
>>>>> schema update issue. It would be really good if phoenix put the auto
>>>>> aliasing column name support in its road map …
>>>>>
>>>>>
>>>>> Thx
>>>>> Yanlin
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Jun 14, 2015, at 5:06 PM, anil gupta <an...@gmail.com> wrote:
>>>>>
>>>>> Hi Yanlin,
>>>>>
>>>>> If you are doing reads. You can use a view.
>>>>> For writes, i dont think Phoenix provides anything for aliasing column
>>>>> names. It would be good if we have this feature because using short column
>>>>> names is a common thing in HBase.
>>>>>
>>>>> Thanks,
>>>>> Anil Gupta
>>>>>
>>>>> On Sun, Jun 14, 2015 at 12:19 PM, yanlin wang <wa...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Usually one method to reduce IO in hbase is to reduce the size of
>>>>>> column qualifier’s length and use a dictionary to map the application
>>>>>> column name to the actual column qualifier. Kiji schema project is doing
>>>>>> exactly that. Is there any column name encoding support in Phoenix? if no,
>>>>>> any plan to do that? Another option is that we can create phoenix table on
>>>>>> top of kiji table, but kiji has its own column serializer which is based on
>>>>>> avro and the Phoenix client needs to be modified to do the custom
>>>>>> deserialization and mapping etc. The last resort will be putting these
>>>>>> column name mapping biz into application DAO layer. Do you have any
>>>>>> suggestion?
>>>>>>
>>>>>> Thx,
>>>>>> Yanlin
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks & Regards,
>>>>> Anil Gupta
>>>>>
>>>>>
>>>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anil Gupta
>>>
>>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
--
Thanks & Regards,
Anil Gupta
Re: column qualifier length
Posted by anil gupta <an...@gmail.com>.
Hi James,
AFAIK, this trick/practice is being recommended by HBase community. I
cannot find the doc that says this.
Thanks,
Anil Gupta
On Sun, Jun 14, 2015 at 7:30 PM, James Taylor <ja...@apache.org>
wrote:
> Views won't help you here (and unrelated, but we only support "create view
> select *" but you can alter the view by removing columns after you create
> it).
>
> Anil - the space savings Yanlin is looking for is shorter column
> qualifiers while still having readable column names.
>
>
> On Sunday, June 14, 2015, anil gupta <an...@gmail.com> wrote:
>
>> Hi James,
>>
>> Sorry, i dont understand the context of "Would be good to quantify the
>> potential gain with and without FAST_DIFF encoding (the default) and/or
>> Snappy compression."?
>> The feature Yanlin is requesting is more for ease of
>> usability/readability.
>>
>>
>> On Sun, Jun 14, 2015 at 5:59 PM, James Taylor <ja...@apache.org>
>> wrote:
>>
>>> Good idea. Please file a JIRA. Would be good to quantify the potential
>>> gain with and without FAST_DIFF encoding (the default) and/or Snappy
>>> compression.
>>>
>>>
>>> On Sunday, June 14, 2015, yanlin wang <wa...@gmail.com> wrote:
>>>
>>>> Thanks for the reply Anil.
>>>>
>>>> Is this what you referring:
>>>> create view my view as
>>>> select a as appColumn1, b as appColum2 from table ?
>>>>
>>>> By reading the document, seems view will introduce limitations e.g
>>>> schema update issue. It would be really good if phoenix put the auto
>>>> aliasing column name support in its road map …
>>>>
>>>>
>>>> Thx
>>>> Yanlin
>>>>
>>>>
>>>>
>>>>
>>>> On Jun 14, 2015, at 5:06 PM, anil gupta <an...@gmail.com> wrote:
>>>>
>>>> Hi Yanlin,
>>>>
>>>> If you are doing reads. You can use a view.
>>>> For writes, i dont think Phoenix provides anything for aliasing column
>>>> names. It would be good if we have this feature because using short column
>>>> names is a common thing in HBase.
>>>>
>>>> Thanks,
>>>> Anil Gupta
>>>>
>>>> On Sun, Jun 14, 2015 at 12:19 PM, yanlin wang <wa...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Usually one method to reduce IO in hbase is to reduce the size of
>>>>> column qualifier’s length and use a dictionary to map the application
>>>>> column name to the actual column qualifier. Kiji schema project is doing
>>>>> exactly that. Is there any column name encoding support in Phoenix? if no,
>>>>> any plan to do that? Another option is that we can create phoenix table on
>>>>> top of kiji table, but kiji has its own column serializer which is based on
>>>>> avro and the Phoenix client needs to be modified to do the custom
>>>>> deserialization and mapping etc. The last resort will be putting these
>>>>> column name mapping biz into application DAO layer. Do you have any
>>>>> suggestion?
>>>>>
>>>>> Thx,
>>>>> Yanlin
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Anil Gupta
>>>>
>>>>
>>>>
>>
>>
>> --
>> Thanks & Regards,
>> Anil Gupta
>>
>
--
Thanks & Regards,
Anil Gupta
Re: column qualifier length
Posted by James Taylor <ja...@apache.org>.
Views won't help you here (and unrelated, but we only support "create view
select *" but you can alter the view by removing columns after you create
it).
Anil - the space savings Yanlin is looking for is shorter column qualifiers
while still having readable column names.
On Sunday, June 14, 2015, anil gupta <an...@gmail.com> wrote:
> Hi James,
>
> Sorry, i dont understand the context of "Would be good to quantify the
> potential gain with and without FAST_DIFF encoding (the default) and/or
> Snappy compression."?
> The feature Yanlin is requesting is more for ease of usability/readability.
>
>
> On Sun, Jun 14, 2015 at 5:59 PM, James Taylor <jamestaylor@apache.org
> <javascript:_e(%7B%7D,'cvml','jamestaylor@apache.org');>> wrote:
>
>> Good idea. Please file a JIRA. Would be good to quantify the potential
>> gain with and without FAST_DIFF encoding (the default) and/or Snappy
>> compression.
>>
>>
>> On Sunday, June 14, 2015, yanlin wang <wangyanlin@gmail.com
>> <javascript:_e(%7B%7D,'cvml','wangyanlin@gmail.com');>> wrote:
>>
>>> Thanks for the reply Anil.
>>>
>>> Is this what you referring:
>>> create view my view as
>>> select a as appColumn1, b as appColum2 from table ?
>>>
>>> By reading the document, seems view will introduce limitations e.g
>>> schema update issue. It would be really good if phoenix put the auto
>>> aliasing column name support in its road map …
>>>
>>>
>>> Thx
>>> Yanlin
>>>
>>>
>>>
>>>
>>> On Jun 14, 2015, at 5:06 PM, anil gupta <an...@gmail.com> wrote:
>>>
>>> Hi Yanlin,
>>>
>>> If you are doing reads. You can use a view.
>>> For writes, i dont think Phoenix provides anything for aliasing column
>>> names. It would be good if we have this feature because using short column
>>> names is a common thing in HBase.
>>>
>>> Thanks,
>>> Anil Gupta
>>>
>>> On Sun, Jun 14, 2015 at 12:19 PM, yanlin wang <wa...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Usually one method to reduce IO in hbase is to reduce the size of
>>>> column qualifier’s length and use a dictionary to map the application
>>>> column name to the actual column qualifier. Kiji schema project is doing
>>>> exactly that. Is there any column name encoding support in Phoenix? if no,
>>>> any plan to do that? Another option is that we can create phoenix table on
>>>> top of kiji table, but kiji has its own column serializer which is based on
>>>> avro and the Phoenix client needs to be modified to do the custom
>>>> deserialization and mapping etc. The last resort will be putting these
>>>> column name mapping biz into application DAO layer. Do you have any
>>>> suggestion?
>>>>
>>>> Thx,
>>>> Yanlin
>>>
>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anil Gupta
>>>
>>>
>>>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
Re: column qualifier length
Posted by anil gupta <an...@gmail.com>.
Hi James,
Sorry, i dont understand the context of "Would be good to quantify the
potential gain with and without FAST_DIFF encoding (the default) and/or
Snappy compression."?
The feature Yanlin is requesting is more for ease of usability/readability.
On Sun, Jun 14, 2015 at 5:59 PM, James Taylor <ja...@apache.org>
wrote:
> Good idea. Please file a JIRA. Would be good to quantify the potential
> gain with and without FAST_DIFF encoding (the default) and/or Snappy
> compression.
>
>
> On Sunday, June 14, 2015, yanlin wang <wa...@gmail.com> wrote:
>
>> Thanks for the reply Anil.
>>
>> Is this what you referring:
>> create view my view as
>> select a as appColumn1, b as appColum2 from table ?
>>
>> By reading the document, seems view will introduce limitations e.g schema
>> update issue. It would be really good if phoenix put the auto aliasing
>> column name support in its road map …
>>
>>
>> Thx
>> Yanlin
>>
>>
>>
>>
>> On Jun 14, 2015, at 5:06 PM, anil gupta <an...@gmail.com> wrote:
>>
>> Hi Yanlin,
>>
>> If you are doing reads. You can use a view.
>> For writes, i dont think Phoenix provides anything for aliasing column
>> names. It would be good if we have this feature because using short column
>> names is a common thing in HBase.
>>
>> Thanks,
>> Anil Gupta
>>
>> On Sun, Jun 14, 2015 at 12:19 PM, yanlin wang <wa...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Usually one method to reduce IO in hbase is to reduce the size of column
>>> qualifier’s length and use a dictionary to map the application column name
>>> to the actual column qualifier. Kiji schema project is doing exactly that.
>>> Is there any column name encoding support in Phoenix? if no, any plan to do
>>> that? Another option is that we can create phoenix table on top of kiji
>>> table, but kiji has its own column serializer which is based on avro and
>>> the Phoenix client needs to be modified to do the custom deserialization
>>> and mapping etc. The last resort will be putting these column name mapping
>>> biz into application DAO layer. Do you have any suggestion?
>>>
>>> Thx,
>>> Yanlin
>>
>>
>>
>>
>> --
>> Thanks & Regards,
>> Anil Gupta
>>
>>
>>
--
Thanks & Regards,
Anil Gupta
Re: column qualifier length
Posted by yanlin wang <wa...@gmail.com>.
Thanks James, my team mate will try to create a jira.
BTW, how to use view to archive column name aliasing? What i tried is - create view my view as select a as appColumn1, b as appColum2 from table , the ddl got rejected. Did i miss anything?
Thx,
Yanlin
> On Jun 14, 2015, at 5:59 PM, James Taylor <ja...@apache.org> wrote:
>
> Good idea. Please file a JIRA. Would be good to quantify the potential gain with and without FAST_DIFF encoding (the default) and/or Snappy compression.
>
> On Sunday, June 14, 2015, yanlin wang <wangyanlin@gmail.com <ma...@gmail.com>> wrote:
> Thanks for the reply Anil.
>
> Is this what you referring:
> create view my view as
> select a as appColumn1, b as appColum2 from table ?
>
> By reading the document, seems view will introduce limitations e.g schema update issue. It would be really good if phoenix put the auto aliasing column name support in its road map …
>
>
> Thx
> Yanlin
>
>
>
>
>> On Jun 14, 2015, at 5:06 PM, anil gupta <anilgupta84@gmail.com <javascript:_e(%7B%7D,'cvml','anilgupta84@gmail.com');>> wrote:
>>
>> Hi Yanlin,
>>
>> If you are doing reads. You can use a view.
>> For writes, i dont think Phoenix provides anything for aliasing column names. It would be good if we have this feature because using short column names is a common thing in HBase.
>>
>> Thanks,
>> Anil Gupta
>>
>> On Sun, Jun 14, 2015 at 12:19 PM, yanlin wang <wangyanlin@gmail.com <javascript:_e(%7B%7D,'cvml','wangyanlin@gmail.com');>> wrote:
>> Hi,
>>
>> Usually one method to reduce IO in hbase is to reduce the size of column qualifier’s length and use a dictionary to map the application column name to the actual column qualifier. Kiji schema project is doing exactly that. Is there any column name encoding support in Phoenix? if no, any plan to do that? Another option is that we can create phoenix table on top of kiji table, but kiji has its own column serializer which is based on avro and the Phoenix client needs to be modified to do the custom deserialization and mapping etc. The last resort will be putting these column name mapping biz into application DAO layer. Do you have any suggestion?
>>
>> Thx,
>> Yanlin
>>
>>
>>
>> --
>> Thanks & Regards,
>> Anil Gupta
>
Re: column qualifier length
Posted by James Taylor <ja...@apache.org>.
Good idea. Please file a JIRA. Would be good to quantify the potential gain
with and without FAST_DIFF encoding (the default) and/or Snappy compression.
On Sunday, June 14, 2015, yanlin wang <wa...@gmail.com> wrote:
> Thanks for the reply Anil.
>
> Is this what you referring:
> create view my view as
> select a as appColumn1, b as appColum2 from table ?
>
> By reading the document, seems view will introduce limitations e.g schema
> update issue. It would be really good if phoenix put the auto aliasing
> column name support in its road map …
>
>
> Thx
> Yanlin
>
>
>
>
> On Jun 14, 2015, at 5:06 PM, anil gupta <anilgupta84@gmail.com
> <javascript:_e(%7B%7D,'cvml','anilgupta84@gmail.com');>> wrote:
>
> Hi Yanlin,
>
> If you are doing reads. You can use a view.
> For writes, i dont think Phoenix provides anything for aliasing column
> names. It would be good if we have this feature because using short column
> names is a common thing in HBase.
>
> Thanks,
> Anil Gupta
>
> On Sun, Jun 14, 2015 at 12:19 PM, yanlin wang <wangyanlin@gmail.com
> <javascript:_e(%7B%7D,'cvml','wangyanlin@gmail.com');>> wrote:
>
>> Hi,
>>
>> Usually one method to reduce IO in hbase is to reduce the size of column
>> qualifier’s length and use a dictionary to map the application column name
>> to the actual column qualifier. Kiji schema project is doing exactly that.
>> Is there any column name encoding support in Phoenix? if no, any plan to do
>> that? Another option is that we can create phoenix table on top of kiji
>> table, but kiji has its own column serializer which is based on avro and
>> the Phoenix client needs to be modified to do the custom deserialization
>> and mapping etc. The last resort will be putting these column name mapping
>> biz into application DAO layer. Do you have any suggestion?
>>
>> Thx,
>> Yanlin
>
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
>
>
Re: column qualifier length
Posted by yanlin wang <wa...@gmail.com>.
Thanks for the reply Anil.
Is this what you referring:
create view my view as
select a as appColumn1, b as appColum2 from table ?
By reading the document, seems view will introduce limitations e.g schema update issue. It would be really good if phoenix put the auto aliasing column name support in its road map …
Thx
Yanlin
> On Jun 14, 2015, at 5:06 PM, anil gupta <an...@gmail.com> wrote:
>
> Hi Yanlin,
>
> If you are doing reads. You can use a view.
> For writes, i dont think Phoenix provides anything for aliasing column names. It would be good if we have this feature because using short column names is a common thing in HBase.
>
> Thanks,
> Anil Gupta
>
> On Sun, Jun 14, 2015 at 12:19 PM, yanlin wang <wangyanlin@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> Usually one method to reduce IO in hbase is to reduce the size of column qualifier’s length and use a dictionary to map the application column name to the actual column qualifier. Kiji schema project is doing exactly that. Is there any column name encoding support in Phoenix? if no, any plan to do that? Another option is that we can create phoenix table on top of kiji table, but kiji has its own column serializer which is based on avro and the Phoenix client needs to be modified to do the custom deserialization and mapping etc. The last resort will be putting these column name mapping biz into application DAO layer. Do you have any suggestion?
>
> Thx,
> Yanlin
>
>
>
> --
> Thanks & Regards,
> Anil Gupta