You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by anil gupta <an...@gmail.com> on 2014/04/23 02:12:39 UTC
Date DataType broken when Migrating from Phoenix2.0.2 to Phoenix3.0.0
Hi All,
We have written data into our HBase tables using PDataType of Phoenix2.0.2.
We have custom MR loaders that use PDataType so that we can use Phoenix for
adhoc querying
I am trying to migrate to Phoenix3.0.0, but all the Date type columns
values are not coming out correctly. These are big tables(TB's of data) and
i would like to avoid making changes in my tables due to Phoenix upgrade to
3.0.0.
Is Phoenix3.0.0 not backward compatible with Phoenix2.0.2? Is there any
easy fix for this(without rewriting Date Columns)? Also, i am curious to
know what was broken in Date DataType in Phoenix2.0.2?
--
Thanks & Regards,
Anil Gupta
Re: Date DataType broken when Migrating from Phoenix2.0.2 to Phoenix3.0.0
Posted by anil gupta <an...@gmail.com>.
Thanks for your help, James. Everything is looking good till now.
On Wed, Apr 23, 2014 at 12:02 AM, James Taylor <ja...@apache.org>wrote:
> The definitive guide is the upgrade code
> in
> org.apache.phoenix.query.ConnectionQueryServicesImpl.upgradeMetaDataTo3_0().
> You're somewhat in uncharted territory if you bypass the standard,
> documented means of upgrading. Are you upgrading by re-running your DDL?
>
> Other than the date/time change I mentioned already (not sure if you're
> using other date/time types other than DATE, as they'd need to be changed
> as well), the default column family name changed. So if you have column
> families that are implicitly named, you'll need to specify a property value
> of DEFAULT_COLUMN_FAMILY='_0' at DDL time to maintain the prior default.
> Also, if you have secondary indexes, these would need to be added again
> (and would be rebuilt from scratch).
>
> The upgrade process outlined here takes care of all the conversion
> automatically (without requiring your secondary indexes to be rebuilt):
> http://phoenix.incubator.apache.org/upgrade_from_2_2.html
>
> Thanks,
> James
>
>
> On Tue, Apr 22, 2014 at 8:05 PM, anil gupta <an...@gmail.com> wrote:
>
> > Now the dates are looking fine after following your suggestion.
> >
> > I made 2 code changes in my Java code:
> > 1. Changed the package from com.salesforce to org.apache.
> > 2. Changed all references to PDataType.DATE to PDataType.UNSIGNED_DATE
> >
> > I deleted system.catalog table and Phoenix created
> >
> > Things are looking good for now. Please let me know if i miss anything
> > other data type that i need to take care of.
> >
> > Thanks for your help!
> > Anil
> >
> >
> >
> >
> > On Tue, Apr 22, 2014 at 5:25 PM, anil gupta <an...@gmail.com>
> wrote:
> >
> > > Hi James,
> > >
> > > Thanks for the quick response. I will test my stuff with changes that
> you
> > > suggested. I was just worried about rewriting Date Columns. This seems
> to
> > > be an easier fix. :)
> > >
> > > Thanks,
> > > Anil Gupta
> > >
> > >
> > > On Tue, Apr 22, 2014 at 5:20 PM, James Taylor <jamestaylor@apache.org
> > >wrote:
> > >
> > >> Hi Anil,
> > >> Please see
> http://phoenix.incubator.apache.org/upgrade_from_2_2.htmlfor
> > >> how to upgrade from Phoenix 2.2.x to Apache Phoenix 3.0/4.0. Part of
> > what
> > >> this will do is modify your table metadata to use the UNSIGNED_DATE,
> > >> UNSIGNED_TIME, and UNSIGNED_TIMESTAMP types instead of the now signed
> > >> ones.
> > >> The reason this change was made is so that we can support negative
> time
> > >> values to increase the range of dates we can represent (PHOENIX-618).
> If
> > >> you have any explicit code references to the internal PDataType enum
> for
> > >> PDataType.DATE, PDataType.TIME, or PDataType.TIMESTAMP, these should
> be
> > >> updated as well to their UNSIGNED equivalent.
> > >> Thanks,
> > >> James
> > >>
> > >>
> > >> On Tue, Apr 22, 2014 at 5:12 PM, anil gupta <an...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi All,
> > >> >
> > >> > We have written data into our HBase tables using PDataType of
> > >> Phoenix2.0.2.
> > >> > We have custom MR loaders that use PDataType so that we can use
> > Phoenix
> > >> for
> > >> > adhoc querying
> > >> > I am trying to migrate to Phoenix3.0.0, but all the Date type
> columns
> > >> > values are not coming out correctly. These are big tables(TB's of
> > data)
> > >> and
> > >> > i would like to avoid making changes in my tables due to Phoenix
> > >> upgrade to
> > >> > 3.0.0.
> > >> > Is Phoenix3.0.0 not backward compatible with Phoenix2.0.2? Is there
> > any
> > >> > easy fix for this(without rewriting Date Columns)? Also, i am
> curious
> > to
> > >> > know what was broken in Date DataType in Phoenix2.0.2?
> > >> >
> > >> > --
> > >> > Thanks & Regards,
> > >> > Anil Gupta
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Thanks & Regards,
> > > Anil Gupta
> > >
> >
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
>
--
Thanks & Regards,
Anil Gupta
Re: Date DataType broken when Migrating from Phoenix2.0.2 to Phoenix3.0.0
Posted by James Taylor <ja...@apache.org>.
The definitive guide is the upgrade code
in org.apache.phoenix.query.ConnectionQueryServicesImpl.upgradeMetaDataTo3_0().
You're somewhat in uncharted territory if you bypass the standard,
documented means of upgrading. Are you upgrading by re-running your DDL?
Other than the date/time change I mentioned already (not sure if you're
using other date/time types other than DATE, as they'd need to be changed
as well), the default column family name changed. So if you have column
families that are implicitly named, you'll need to specify a property value
of DEFAULT_COLUMN_FAMILY='_0' at DDL time to maintain the prior default.
Also, if you have secondary indexes, these would need to be added again
(and would be rebuilt from scratch).
The upgrade process outlined here takes care of all the conversion
automatically (without requiring your secondary indexes to be rebuilt):
http://phoenix.incubator.apache.org/upgrade_from_2_2.html
Thanks,
James
On Tue, Apr 22, 2014 at 8:05 PM, anil gupta <an...@gmail.com> wrote:
> Now the dates are looking fine after following your suggestion.
>
> I made 2 code changes in my Java code:
> 1. Changed the package from com.salesforce to org.apache.
> 2. Changed all references to PDataType.DATE to PDataType.UNSIGNED_DATE
>
> I deleted system.catalog table and Phoenix created
>
> Things are looking good for now. Please let me know if i miss anything
> other data type that i need to take care of.
>
> Thanks for your help!
> Anil
>
>
>
>
> On Tue, Apr 22, 2014 at 5:25 PM, anil gupta <an...@gmail.com> wrote:
>
> > Hi James,
> >
> > Thanks for the quick response. I will test my stuff with changes that you
> > suggested. I was just worried about rewriting Date Columns. This seems to
> > be an easier fix. :)
> >
> > Thanks,
> > Anil Gupta
> >
> >
> > On Tue, Apr 22, 2014 at 5:20 PM, James Taylor <jamestaylor@apache.org
> >wrote:
> >
> >> Hi Anil,
> >> Please see http://phoenix.incubator.apache.org/upgrade_from_2_2.htmlfor
> >> how to upgrade from Phoenix 2.2.x to Apache Phoenix 3.0/4.0. Part of
> what
> >> this will do is modify your table metadata to use the UNSIGNED_DATE,
> >> UNSIGNED_TIME, and UNSIGNED_TIMESTAMP types instead of the now signed
> >> ones.
> >> The reason this change was made is so that we can support negative time
> >> values to increase the range of dates we can represent (PHOENIX-618). If
> >> you have any explicit code references to the internal PDataType enum for
> >> PDataType.DATE, PDataType.TIME, or PDataType.TIMESTAMP, these should be
> >> updated as well to their UNSIGNED equivalent.
> >> Thanks,
> >> James
> >>
> >>
> >> On Tue, Apr 22, 2014 at 5:12 PM, anil gupta <an...@gmail.com>
> >> wrote:
> >>
> >> > Hi All,
> >> >
> >> > We have written data into our HBase tables using PDataType of
> >> Phoenix2.0.2.
> >> > We have custom MR loaders that use PDataType so that we can use
> Phoenix
> >> for
> >> > adhoc querying
> >> > I am trying to migrate to Phoenix3.0.0, but all the Date type columns
> >> > values are not coming out correctly. These are big tables(TB's of
> data)
> >> and
> >> > i would like to avoid making changes in my tables due to Phoenix
> >> upgrade to
> >> > 3.0.0.
> >> > Is Phoenix3.0.0 not backward compatible with Phoenix2.0.2? Is there
> any
> >> > easy fix for this(without rewriting Date Columns)? Also, i am curious
> to
> >> > know what was broken in Date DataType in Phoenix2.0.2?
> >> >
> >> > --
> >> > Thanks & Regards,
> >> > Anil Gupta
> >> >
> >>
> >
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
Re: Date DataType broken when Migrating from Phoenix2.0.2 to Phoenix3.0.0
Posted by anil gupta <an...@gmail.com>.
Now the dates are looking fine after following your suggestion.
I made 2 code changes in my Java code:
1. Changed the package from com.salesforce to org.apache.
2. Changed all references to PDataType.DATE to PDataType.UNSIGNED_DATE
I deleted system.catalog table and Phoenix created
Things are looking good for now. Please let me know if i miss anything
other data type that i need to take care of.
Thanks for your help!
Anil
On Tue, Apr 22, 2014 at 5:25 PM, anil gupta <an...@gmail.com> wrote:
> Hi James,
>
> Thanks for the quick response. I will test my stuff with changes that you
> suggested. I was just worried about rewriting Date Columns. This seems to
> be an easier fix. :)
>
> Thanks,
> Anil Gupta
>
>
> On Tue, Apr 22, 2014 at 5:20 PM, James Taylor <ja...@apache.org>wrote:
>
>> Hi Anil,
>> Please see http://phoenix.incubator.apache.org/upgrade_from_2_2.html for
>> how to upgrade from Phoenix 2.2.x to Apache Phoenix 3.0/4.0. Part of what
>> this will do is modify your table metadata to use the UNSIGNED_DATE,
>> UNSIGNED_TIME, and UNSIGNED_TIMESTAMP types instead of the now signed
>> ones.
>> The reason this change was made is so that we can support negative time
>> values to increase the range of dates we can represent (PHOENIX-618). If
>> you have any explicit code references to the internal PDataType enum for
>> PDataType.DATE, PDataType.TIME, or PDataType.TIMESTAMP, these should be
>> updated as well to their UNSIGNED equivalent.
>> Thanks,
>> James
>>
>>
>> On Tue, Apr 22, 2014 at 5:12 PM, anil gupta <an...@gmail.com>
>> wrote:
>>
>> > Hi All,
>> >
>> > We have written data into our HBase tables using PDataType of
>> Phoenix2.0.2.
>> > We have custom MR loaders that use PDataType so that we can use Phoenix
>> for
>> > adhoc querying
>> > I am trying to migrate to Phoenix3.0.0, but all the Date type columns
>> > values are not coming out correctly. These are big tables(TB's of data)
>> and
>> > i would like to avoid making changes in my tables due to Phoenix
>> upgrade to
>> > 3.0.0.
>> > Is Phoenix3.0.0 not backward compatible with Phoenix2.0.2? Is there any
>> > easy fix for this(without rewriting Date Columns)? Also, i am curious to
>> > know what was broken in Date DataType in Phoenix2.0.2?
>> >
>> > --
>> > Thanks & Regards,
>> > Anil Gupta
>> >
>>
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
--
Thanks & Regards,
Anil Gupta
Re: Date DataType broken when Migrating from Phoenix2.0.2 to Phoenix3.0.0
Posted by anil gupta <an...@gmail.com>.
Hi James,
Thanks for the quick response. I will test my stuff with changes that you
suggested. I was just worried about rewriting Date Columns. This seems to
be an easier fix. :)
Thanks,
Anil Gupta
On Tue, Apr 22, 2014 at 5:20 PM, James Taylor <ja...@apache.org>wrote:
> Hi Anil,
> Please see http://phoenix.incubator.apache.org/upgrade_from_2_2.html for
> how to upgrade from Phoenix 2.2.x to Apache Phoenix 3.0/4.0. Part of what
> this will do is modify your table metadata to use the UNSIGNED_DATE,
> UNSIGNED_TIME, and UNSIGNED_TIMESTAMP types instead of the now signed ones.
> The reason this change was made is so that we can support negative time
> values to increase the range of dates we can represent (PHOENIX-618). If
> you have any explicit code references to the internal PDataType enum for
> PDataType.DATE, PDataType.TIME, or PDataType.TIMESTAMP, these should be
> updated as well to their UNSIGNED equivalent.
> Thanks,
> James
>
>
> On Tue, Apr 22, 2014 at 5:12 PM, anil gupta <an...@gmail.com> wrote:
>
> > Hi All,
> >
> > We have written data into our HBase tables using PDataType of
> Phoenix2.0.2.
> > We have custom MR loaders that use PDataType so that we can use Phoenix
> for
> > adhoc querying
> > I am trying to migrate to Phoenix3.0.0, but all the Date type columns
> > values are not coming out correctly. These are big tables(TB's of data)
> and
> > i would like to avoid making changes in my tables due to Phoenix upgrade
> to
> > 3.0.0.
> > Is Phoenix3.0.0 not backward compatible with Phoenix2.0.2? Is there any
> > easy fix for this(without rewriting Date Columns)? Also, i am curious to
> > know what was broken in Date DataType in Phoenix2.0.2?
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
>
--
Thanks & Regards,
Anil Gupta
Re: Date DataType broken when Migrating from Phoenix2.0.2 to Phoenix3.0.0
Posted by James Taylor <ja...@apache.org>.
Hi Anil,
Please see http://phoenix.incubator.apache.org/upgrade_from_2_2.html for
how to upgrade from Phoenix 2.2.x to Apache Phoenix 3.0/4.0. Part of what
this will do is modify your table metadata to use the UNSIGNED_DATE,
UNSIGNED_TIME, and UNSIGNED_TIMESTAMP types instead of the now signed ones.
The reason this change was made is so that we can support negative time
values to increase the range of dates we can represent (PHOENIX-618). If
you have any explicit code references to the internal PDataType enum for
PDataType.DATE, PDataType.TIME, or PDataType.TIMESTAMP, these should be
updated as well to their UNSIGNED equivalent.
Thanks,
James
On Tue, Apr 22, 2014 at 5:12 PM, anil gupta <an...@gmail.com> wrote:
> Hi All,
>
> We have written data into our HBase tables using PDataType of Phoenix2.0.2.
> We have custom MR loaders that use PDataType so that we can use Phoenix for
> adhoc querying
> I am trying to migrate to Phoenix3.0.0, but all the Date type columns
> values are not coming out correctly. These are big tables(TB's of data) and
> i would like to avoid making changes in my tables due to Phoenix upgrade to
> 3.0.0.
> Is Phoenix3.0.0 not backward compatible with Phoenix2.0.2? Is there any
> easy fix for this(without rewriting Date Columns)? Also, i am curious to
> know what was broken in Date DataType in Phoenix2.0.2?
>
> --
> Thanks & Regards,
> Anil Gupta
>