You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by Mohammad Adnan Raza <ad...@gmail.com> on 2016/03/22 09:30:35 UTC

How phoenix converts Integer to byte array under the hood

Hello Everyone,

I have created phoenix table like this

CREATE TABLE PRODUCT_DETAILS(NAME VARCHAR NOT NULL PRIMARY KEY,CF.VOLUME
INTEGER,CF.PRICE INTEGER,CF.DISCOUNT INTEGER,CF.BASELINE INTEGER,CF.UPLIFT
INTEGER,CF.FINALPRICE INTEGER,CF.SALEPRICE INTEGER);

Now the datatype Integer is 4 byte signed integer. I wonder how phoenix
converts this to hbase specific byte array.
https://phoenix.apache.org/language/datatypes.html link does talk about
conversion of other data types but not for INTEGER. For example
UNSIGNED_INT is converted as Bytes.toInt(). I don't get a proper method for
integer.
anyone knows about it?

-- 

*With Best Regards,*

*       Mohd Adnan*

Re: How phoenix converts Integer to byte array under the hood

Posted by James Taylor <ja...@apache.org>.
Our only public APIs are JDBC and our various integrations with Spark, MR,
etc. Though it's unlikely these APIs will change as Sergey mentioned, it's
possible. The actual binary format won't change, though (at least for
existing tables).

Thanks,
James

On Wed, Mar 23, 2016 at 1:05 PM, Sergey Soldatov <se...@gmail.com>
wrote:

> Mohammad,
> Honestly speaking, I'm not sure. Possible other guys have a definitive
> answer. All I can say is that this API didn't change for last 1.5
> years.
> Thanks,
> Sergey
>
> On Wed, Mar 23, 2016 at 1:07 AM, Mohammad Adnan Raza
> <ad...@gmail.com> wrote:
> > Thank you Sergey for quick response. I was exactly looking for this. This
> > saved my hours digging into phoenix code base.
> > Now I am decoding like -
> > PInteger.INSTANCE.getCodec().decodeInt(bytes, 0, SortOrder.getDefault())
> > and encoding it like
> > byte[] baselineBytes = new byte[PInteger.INSTANCE.getByteSize()];
> > PInteger.INSTANCE.getCodec().encodeInt(baseline, baselineBytes, 0);
> >
> > I just have a doubt here.. Does phoenix expose this class as an API or
> it is
> > internal class? if they do expose it as API I can use it otherwise in
> future
> > (with all JAVA9 JIGSAW feature or may be changes in API itself), it may
> > start breaking.
> >
> >
> >
> > On Tue, Mar 22, 2016 at 11:11 PM, Sergey Soldatov <
> sergeysoldatov@gmail.com>
> > wrote:
> >>
> >> Hi Mohammad,
> >> The right class to look into is PInteger. It has static class IntCodec
> >> which is using for code/decode integers.
> >>
> >> Thanks,
> >> Sergey
> >>
> >> On Tue, Mar 22, 2016 at 7:15 AM, Mohammad Adnan Raza
> >> <ad...@gmail.com> wrote:
> >> > I am changing my question a bit to be more precise...
> >> > Given a phoenix table with INTEGER column type. And if I fire upsert
> >> > statement with integer value. How phoenix converts it to byte array
> and
> >> > put
> >> > in the Hbase table.
> >> > Or if anyone can tell me which class is responsible for that
> conversion
> >> > so I
> >> > can look that code.
> >> >
> >> >
> >> >
> >> > On Tue, Mar 22, 2016 at 2:00 PM, Mohammad Adnan Raza
> >> > <ad...@gmail.com>
> >> > wrote:
> >> >>
> >> >> Hello Everyone,
> >> >>
> >> >> I have created phoenix table like this
> >> >>
> >> >> CREATE TABLE PRODUCT_DETAILS(NAME VARCHAR NOT NULL PRIMARY
> >> >> KEY,CF.VOLUME
> >> >> INTEGER,CF.PRICE INTEGER,CF.DISCOUNT INTEGER,CF.BASELINE
> >> >> INTEGER,CF.UPLIFT
> >> >> INTEGER,CF.FINALPRICE INTEGER,CF.SALEPRICE INTEGER);
> >> >>
> >> >> Now the datatype Integer is 4 byte signed integer. I wonder how
> phoenix
> >> >> converts this to hbase specific byte array.
> >> >> https://phoenix.apache.org/language/datatypes.html link does talk
> about
> >> >> conversion of other data types but not for INTEGER. For example
> >> >> UNSIGNED_INT
> >> >> is converted as Bytes.toInt(). I don't get a proper method for
> integer.
> >> >> anyone knows about it?
> >> >>
> >> >> --
> >> >>
> >> >> With Best Regards,
> >> >>
> >> >>        Mohd Adnan
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > With Best Regards,
> >> >
> >> >        Mohd Adnan
> >> >
> >> >        Feature Development Lead
> >> >
> >> >        Mobile               +91-7498194516
> >> >        Blog                   adnanfaizan.blogspot.in
> >> >
> >
> >
> >
> >
> > --
> >
> > With Best Regards,
> >
> >        Mohd Adnan
> >
> >        Feature Development Lead
> >
> >        Mobile               +91-7498194516
> >        Blog                   adnanfaizan.blogspot.in
> >
>

Re: How phoenix converts Integer to byte array under the hood

Posted by Sergey Soldatov <se...@gmail.com>.
Mohammad,
Honestly speaking, I'm not sure. Possible other guys have a definitive
answer. All I can say is that this API didn't change for last 1.5
years.
Thanks,
Sergey

On Wed, Mar 23, 2016 at 1:07 AM, Mohammad Adnan Raza
<ad...@gmail.com> wrote:
> Thank you Sergey for quick response. I was exactly looking for this. This
> saved my hours digging into phoenix code base.
> Now I am decoding like -
> PInteger.INSTANCE.getCodec().decodeInt(bytes, 0, SortOrder.getDefault())
> and encoding it like
> byte[] baselineBytes = new byte[PInteger.INSTANCE.getByteSize()];
> PInteger.INSTANCE.getCodec().encodeInt(baseline, baselineBytes, 0);
>
> I just have a doubt here.. Does phoenix expose this class as an API or it is
> internal class? if they do expose it as API I can use it otherwise in future
> (with all JAVA9 JIGSAW feature or may be changes in API itself), it may
> start breaking.
>
>
>
> On Tue, Mar 22, 2016 at 11:11 PM, Sergey Soldatov <se...@gmail.com>
> wrote:
>>
>> Hi Mohammad,
>> The right class to look into is PInteger. It has static class IntCodec
>> which is using for code/decode integers.
>>
>> Thanks,
>> Sergey
>>
>> On Tue, Mar 22, 2016 at 7:15 AM, Mohammad Adnan Raza
>> <ad...@gmail.com> wrote:
>> > I am changing my question a bit to be more precise...
>> > Given a phoenix table with INTEGER column type. And if I fire upsert
>> > statement with integer value. How phoenix converts it to byte array and
>> > put
>> > in the Hbase table.
>> > Or if anyone can tell me which class is responsible for that conversion
>> > so I
>> > can look that code.
>> >
>> >
>> >
>> > On Tue, Mar 22, 2016 at 2:00 PM, Mohammad Adnan Raza
>> > <ad...@gmail.com>
>> > wrote:
>> >>
>> >> Hello Everyone,
>> >>
>> >> I have created phoenix table like this
>> >>
>> >> CREATE TABLE PRODUCT_DETAILS(NAME VARCHAR NOT NULL PRIMARY
>> >> KEY,CF.VOLUME
>> >> INTEGER,CF.PRICE INTEGER,CF.DISCOUNT INTEGER,CF.BASELINE
>> >> INTEGER,CF.UPLIFT
>> >> INTEGER,CF.FINALPRICE INTEGER,CF.SALEPRICE INTEGER);
>> >>
>> >> Now the datatype Integer is 4 byte signed integer. I wonder how phoenix
>> >> converts this to hbase specific byte array.
>> >> https://phoenix.apache.org/language/datatypes.html link does talk about
>> >> conversion of other data types but not for INTEGER. For example
>> >> UNSIGNED_INT
>> >> is converted as Bytes.toInt(). I don't get a proper method for integer.
>> >> anyone knows about it?
>> >>
>> >> --
>> >>
>> >> With Best Regards,
>> >>
>> >>        Mohd Adnan
>> >>
>> >
>> >
>> >
>> > --
>> >
>> > With Best Regards,
>> >
>> >        Mohd Adnan
>> >
>> >        Feature Development Lead
>> >
>> >        Mobile               +91-7498194516
>> >        Blog                   adnanfaizan.blogspot.in
>> >
>
>
>
>
> --
>
> With Best Regards,
>
>        Mohd Adnan
>
>        Feature Development Lead
>
>        Mobile               +91-7498194516
>        Blog                   adnanfaizan.blogspot.in
>

Re: How phoenix converts Integer to byte array under the hood

Posted by Mohammad Adnan Raza <ad...@gmail.com>.
Thank you Sergey for quick response. I was exactly looking for this. This
saved my hours digging into phoenix code base.
Now I am decoding like -
PInteger.INSTANCE.getCodec().decodeInt(bytes, 0, SortOrder.getDefault())
and encoding it like
byte[] baselineBytes = new byte[PInteger.INSTANCE.getByteSize()];
PInteger.INSTANCE.getCodec().encodeInt(baseline, baselineBytes, 0);

I just have a doubt here.. Does phoenix expose this class as an API or it
is internal class? if they do expose it as API I can use it otherwise in
future (with all JAVA9 JIGSAW feature or may be changes in API itself), it
may start breaking.



On Tue, Mar 22, 2016 at 11:11 PM, Sergey Soldatov <se...@gmail.com>
wrote:

> Hi Mohammad,
> The right class to look into is PInteger. It has static class IntCodec
> which is using for code/decode integers.
>
> Thanks,
> Sergey
>
> On Tue, Mar 22, 2016 at 7:15 AM, Mohammad Adnan Raza
> <ad...@gmail.com> wrote:
> > I am changing my question a bit to be more precise...
> > Given a phoenix table with INTEGER column type. And if I fire upsert
> > statement with integer value. How phoenix converts it to byte array and
> put
> > in the Hbase table.
> > Or if anyone can tell me which class is responsible for that conversion
> so I
> > can look that code.
> >
> >
> >
> > On Tue, Mar 22, 2016 at 2:00 PM, Mohammad Adnan Raza <
> adnanfaizan@gmail.com>
> > wrote:
> >>
> >> Hello Everyone,
> >>
> >> I have created phoenix table like this
> >>
> >> CREATE TABLE PRODUCT_DETAILS(NAME VARCHAR NOT NULL PRIMARY KEY,CF.VOLUME
> >> INTEGER,CF.PRICE INTEGER,CF.DISCOUNT INTEGER,CF.BASELINE
> INTEGER,CF.UPLIFT
> >> INTEGER,CF.FINALPRICE INTEGER,CF.SALEPRICE INTEGER);
> >>
> >> Now the datatype Integer is 4 byte signed integer. I wonder how phoenix
> >> converts this to hbase specific byte array.
> >> https://phoenix.apache.org/language/datatypes.html link does talk about
> >> conversion of other data types but not for INTEGER. For example
> UNSIGNED_INT
> >> is converted as Bytes.toInt(). I don't get a proper method for integer.
> >> anyone knows about it?
> >>
> >> --
> >>
> >> With Best Regards,
> >>
> >>        Mohd Adnan
> >>
> >
> >
> >
> > --
> >
> > With Best Regards,
> >
> >        Mohd Adnan
> >
> >        Feature Development Lead
> >
> >        Mobile               +91-7498194516
> >        Blog                   adnanfaizan.blogspot.in
> >
>



-- 

*With Best Regards,*

*       Mohd Adnan*

*       Feature Development Lead*
       *Mobile               +91-7498194516 *


*       Blog                   adnanfaizan.blogspot.in
<http://adnanfaizan.blogspot.in>*

Re: How phoenix converts Integer to byte array under the hood

Posted by Sergey Soldatov <se...@gmail.com>.
Hi Mohammad,
The right class to look into is PInteger. It has static class IntCodec
which is using for code/decode integers.

Thanks,
Sergey

On Tue, Mar 22, 2016 at 7:15 AM, Mohammad Adnan Raza
<ad...@gmail.com> wrote:
> I am changing my question a bit to be more precise...
> Given a phoenix table with INTEGER column type. And if I fire upsert
> statement with integer value. How phoenix converts it to byte array and put
> in the Hbase table.
> Or if anyone can tell me which class is responsible for that conversion so I
> can look that code.
>
>
>
> On Tue, Mar 22, 2016 at 2:00 PM, Mohammad Adnan Raza <ad...@gmail.com>
> wrote:
>>
>> Hello Everyone,
>>
>> I have created phoenix table like this
>>
>> CREATE TABLE PRODUCT_DETAILS(NAME VARCHAR NOT NULL PRIMARY KEY,CF.VOLUME
>> INTEGER,CF.PRICE INTEGER,CF.DISCOUNT INTEGER,CF.BASELINE INTEGER,CF.UPLIFT
>> INTEGER,CF.FINALPRICE INTEGER,CF.SALEPRICE INTEGER);
>>
>> Now the datatype Integer is 4 byte signed integer. I wonder how phoenix
>> converts this to hbase specific byte array.
>> https://phoenix.apache.org/language/datatypes.html link does talk about
>> conversion of other data types but not for INTEGER. For example UNSIGNED_INT
>> is converted as Bytes.toInt(). I don't get a proper method for integer.
>> anyone knows about it?
>>
>> --
>>
>> With Best Regards,
>>
>>        Mohd Adnan
>>
>
>
>
> --
>
> With Best Regards,
>
>        Mohd Adnan
>
>        Feature Development Lead
>
>        Mobile               +91-7498194516
>        Blog                   adnanfaizan.blogspot.in
>

Re: How phoenix converts Integer to byte array under the hood

Posted by Mohammad Adnan Raza <ad...@gmail.com>.
I am changing my question a bit to be more precise...
Given a phoenix table with INTEGER column type. And if I fire upsert
statement with integer value. How phoenix converts it to byte array and put
in the Hbase table.
Or if anyone can tell me which class is responsible for that conversion so
I can look that code.



On Tue, Mar 22, 2016 at 2:00 PM, Mohammad Adnan Raza <ad...@gmail.com>
wrote:

> Hello Everyone,
>
> I have created phoenix table like this
>
> CREATE TABLE PRODUCT_DETAILS(NAME VARCHAR NOT NULL PRIMARY KEY,CF.VOLUME
> INTEGER,CF.PRICE INTEGER,CF.DISCOUNT INTEGER,CF.BASELINE INTEGER,CF.UPLIFT
> INTEGER,CF.FINALPRICE INTEGER,CF.SALEPRICE INTEGER);
>
> Now the datatype Integer is 4 byte signed integer. I wonder how phoenix
> converts this to hbase specific byte array.
> https://phoenix.apache.org/language/datatypes.html link does talk about
> conversion of other data types but not for INTEGER. For example
> UNSIGNED_INT is converted as Bytes.toInt(). I don't get a proper method for
> integer.
> anyone knows about it?
>
> --
>
> *With Best Regards,*
>
> *       Mohd Adnan*
>
>


-- 

*With Best Regards,*

*       Mohd Adnan*

*       Feature Development Lead*
       *Mobile               +91-7498194516 *


*       Blog                   adnanfaizan.blogspot.in
<http://adnanfaizan.blogspot.in>*

Re: How phoenix converts Integer to byte array under the hood

Posted by Mohammad Adnan Raza <ad...@gmail.com>.
I am changing my question a bit to be more precise...
Given a phoenix table with INTEGER column type. And if I fire upsert
statement with integer value. How phoenix converts it to byte array and put
in the Hbase table.
Or if anyone can tell me which class is responsible for that conversion so
I can look that code.



On Tue, Mar 22, 2016 at 2:00 PM, Mohammad Adnan Raza <ad...@gmail.com>
wrote:

> Hello Everyone,
>
> I have created phoenix table like this
>
> CREATE TABLE PRODUCT_DETAILS(NAME VARCHAR NOT NULL PRIMARY KEY,CF.VOLUME
> INTEGER,CF.PRICE INTEGER,CF.DISCOUNT INTEGER,CF.BASELINE INTEGER,CF.UPLIFT
> INTEGER,CF.FINALPRICE INTEGER,CF.SALEPRICE INTEGER);
>
> Now the datatype Integer is 4 byte signed integer. I wonder how phoenix
> converts this to hbase specific byte array.
> https://phoenix.apache.org/language/datatypes.html link does talk about
> conversion of other data types but not for INTEGER. For example
> UNSIGNED_INT is converted as Bytes.toInt(). I don't get a proper method for
> integer.
> anyone knows about it?
>
> --
>
> *With Best Regards,*
>
> *       Mohd Adnan*
>
>


-- 

*With Best Regards,*

*       Mohd Adnan*

*       Feature Development Lead*
       *Mobile               +91-7498194516 *


*       Blog                   adnanfaizan.blogspot.in
<http://adnanfaizan.blogspot.in>*