You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Namit Jain <nj...@fb.com> on 2012/05/25 03:49:31 UTC
non-string partition columns
http://svn.apache.org/viewvc?view=revision&revision=1308427
The patch above broke drop partitions if the partition happens to be non-string.
This is due to a JDO issue with non-string columns.
Is anyone using non-string partition columns ?
Should be force the partition columns to be only of type string ?
The documentation probably does not specify anything clearly.
If someone is dependent on non-string partition column, we need to revert this patch, or make a
special case for string partition columns.
Thanks,
-namit
Re: non-string partition columns
Posted by Ashutosh Chauhan <ha...@apache.org>.
FWIW.. HCatalog only allows partition columns of type string precisely
because in backend datastore type information is not recorded. In my
opinion, partition type should be restricted to type string until we fix
this problem, otherwise it gives unexpected behavior to endusers and/or
bug-reps. One possibility is to introduce config variable
hive.partition.column.type and has it value set to "string" by default.
This ensures that new users get expected behavior of string-only partition
columns. Users who already use other types can reset this config value to
"all" in their deployment when they upgrade to newer version of Hive
(assuming new version comes out without a proper fix). This extra step of
reseting default config will help them to understand the risk they are
taking by changing default value.
Thanks,
Ashutosh
On Tue, May 29, 2012 at 10:02 AM, Namit Jain <nj...@fb.com> wrote:
> OK, I will keep the support.
> Add special casing for string columns in DDLTask
>
> On 5/29/12 9:27 AM, "Edward Capriolo" <ed...@gmail.com> wrote:
>
> >We use them to we store our dates as integers like 20120130. This
> >allows us to do partition pruning with ranges.
> >
> >On Tue, May 29, 2012 at 4:10 AM, Aniket Mokashi <an...@gmail.com>
> >wrote:
> >> We are using non-string partition columns in production as well.
> >>
> >> Thanks,
> >> Aniket
> >>
> >> On Sat, May 26, 2012 at 1:20 AM, Philip Tromans
> >> <ph...@gmail.com>wrote:
> >>
> >>> We're using non-string partition columns in production. I think non
> >>>string
> >>> partition columns are a good thing to have - it allows you to do all
> >>>sorts
> >>> of date range calculations etc. AFAIK, MySQL's partition columns can
> >>>be of
> >>> any type.
> >>>
> >>> Phil.
> >>> On May 26, 2012 7:55 AM, "Namit Jain" <nj...@fb.com> wrote:
> >>>
> >>> > Should I go ahead and file a jira to disallow non-string partition
> >>> columns
> >>> > ?
> >>> > Or, someone depends on that functionality.
> >>> >
> >>> >
> >>> > On 5/25/12 10:01 AM, "Namit Jain" <nj...@fb.com> wrote:
> >>> >
> >>> > >Yes, but the meta-question is:
> >>> > >
> >>> > >Is anyone dependent on non-string partition columns ? Should we
> >>>drop the
> >>> > >support for non-string
> >>> > >partition columns ?
> >>> > >
> >>> > >
> >>> > >Thanks,
> >>> > >-namit
> >>> > >
> >>> > >On 5/24/12 11:21 PM, "Ashutosh Chauhan" <ha...@apache.org>
> >>>wrote:
> >>> > >
> >>> > >>Some discussion for this has happened on
> >>> > >>https://issues.apache.org/jira/browse/HIVE-2702 Is the underlying
> >>> > problem
> >>> > >>same as the one which I described on that jira ?
> >>> > >>
> >>> > >>Thanks,
> >>> > >>Ashutosh
> >>> > >>
> >>> > >>On Thu, May 24, 2012 at 10:59 PM, Namit Jain <nj...@fb.com> wrote:
> >>> > >>
> >>> > >>> Should we disallow non-string partition columns completely ?
> >>> > >>> Does anyone depend on that ?
> >>> > >>>
> >>> > >>>
> >>> > >>> On 5/24/12 6:49 PM, "Namit Jain" <nj...@fb.com> wrote:
> >>> > >>>
> >>> > >>> >
> >>> > >>> >http://svn.apache.org/viewvc?view=revision&revision=1308427
> >>> > >>> >
> >>> > >>> >The patch above broke drop partitions if the partition happens
> >>>to be
> >>> > >>> >non-string.
> >>> > >>> >This is due to a JDO issue with non-string columns.
> >>> > >>> >
> >>> > >>> >Is anyone using non-string partition columns ?
> >>> > >>> >Should be force the partition columns to be only of type string
> >>>?
> >>> > >>> >The documentation probably does not specify anything clearly.
> >>> > >>> >
> >>> > >>> >If someone is dependent on non-string partition column, we need
> >>>to
> >>> > >>>revert
> >>> > >>> >this patch, or make a
> >>> > >>> >special case for string partition columns.
> >>> > >>> >
> >>> > >>> >Thanks,
> >>> > >>> >-namit
> >>> > >>> >
> >>> > >>> >
> >>> > >>>
> >>> > >>>
> >>> > >
> >>> >
> >>> >
> >>>
> >>
> >>
> >>
> >> --
> >> "...:::Aniket:::... Quetzalco@tl"
>
>
Re: non-string partition columns
Posted by Namit Jain <nj...@fb.com>.
OK, I will keep the support.
Add special casing for string columns in DDLTask
On 5/29/12 9:27 AM, "Edward Capriolo" <ed...@gmail.com> wrote:
>We use them to we store our dates as integers like 20120130. This
>allows us to do partition pruning with ranges.
>
>On Tue, May 29, 2012 at 4:10 AM, Aniket Mokashi <an...@gmail.com>
>wrote:
>> We are using non-string partition columns in production as well.
>>
>> Thanks,
>> Aniket
>>
>> On Sat, May 26, 2012 at 1:20 AM, Philip Tromans
>> <ph...@gmail.com>wrote:
>>
>>> We're using non-string partition columns in production. I think non
>>>string
>>> partition columns are a good thing to have - it allows you to do all
>>>sorts
>>> of date range calculations etc. AFAIK, MySQL's partition columns can
>>>be of
>>> any type.
>>>
>>> Phil.
>>> On May 26, 2012 7:55 AM, "Namit Jain" <nj...@fb.com> wrote:
>>>
>>> > Should I go ahead and file a jira to disallow non-string partition
>>> columns
>>> > ?
>>> > Or, someone depends on that functionality.
>>> >
>>> >
>>> > On 5/25/12 10:01 AM, "Namit Jain" <nj...@fb.com> wrote:
>>> >
>>> > >Yes, but the meta-question is:
>>> > >
>>> > >Is anyone dependent on non-string partition columns ? Should we
>>>drop the
>>> > >support for non-string
>>> > >partition columns ?
>>> > >
>>> > >
>>> > >Thanks,
>>> > >-namit
>>> > >
>>> > >On 5/24/12 11:21 PM, "Ashutosh Chauhan" <ha...@apache.org>
>>>wrote:
>>> > >
>>> > >>Some discussion for this has happened on
>>> > >>https://issues.apache.org/jira/browse/HIVE-2702 Is the underlying
>>> > problem
>>> > >>same as the one which I described on that jira ?
>>> > >>
>>> > >>Thanks,
>>> > >>Ashutosh
>>> > >>
>>> > >>On Thu, May 24, 2012 at 10:59 PM, Namit Jain <nj...@fb.com> wrote:
>>> > >>
>>> > >>> Should we disallow non-string partition columns completely ?
>>> > >>> Does anyone depend on that ?
>>> > >>>
>>> > >>>
>>> > >>> On 5/24/12 6:49 PM, "Namit Jain" <nj...@fb.com> wrote:
>>> > >>>
>>> > >>> >
>>> > >>> >http://svn.apache.org/viewvc?view=revision&revision=1308427
>>> > >>> >
>>> > >>> >The patch above broke drop partitions if the partition happens
>>>to be
>>> > >>> >non-string.
>>> > >>> >This is due to a JDO issue with non-string columns.
>>> > >>> >
>>> > >>> >Is anyone using non-string partition columns ?
>>> > >>> >Should be force the partition columns to be only of type string
>>>?
>>> > >>> >The documentation probably does not specify anything clearly.
>>> > >>> >
>>> > >>> >If someone is dependent on non-string partition column, we need
>>>to
>>> > >>>revert
>>> > >>> >this patch, or make a
>>> > >>> >special case for string partition columns.
>>> > >>> >
>>> > >>> >Thanks,
>>> > >>> >-namit
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> > >>>
>>> > >
>>> >
>>> >
>>>
>>
>>
>>
>> --
>> "...:::Aniket:::... Quetzalco@tl"
Re: non-string partition columns
Posted by Edward Capriolo <ed...@gmail.com>.
We use them to we store our dates as integers like 20120130. This
allows us to do partition pruning with ranges.
On Tue, May 29, 2012 at 4:10 AM, Aniket Mokashi <an...@gmail.com> wrote:
> We are using non-string partition columns in production as well.
>
> Thanks,
> Aniket
>
> On Sat, May 26, 2012 at 1:20 AM, Philip Tromans
> <ph...@gmail.com>wrote:
>
>> We're using non-string partition columns in production. I think non string
>> partition columns are a good thing to have - it allows you to do all sorts
>> of date range calculations etc. AFAIK, MySQL's partition columns can be of
>> any type.
>>
>> Phil.
>> On May 26, 2012 7:55 AM, "Namit Jain" <nj...@fb.com> wrote:
>>
>> > Should I go ahead and file a jira to disallow non-string partition
>> columns
>> > ?
>> > Or, someone depends on that functionality.
>> >
>> >
>> > On 5/25/12 10:01 AM, "Namit Jain" <nj...@fb.com> wrote:
>> >
>> > >Yes, but the meta-question is:
>> > >
>> > >Is anyone dependent on non-string partition columns ? Should we drop the
>> > >support for non-string
>> > >partition columns ?
>> > >
>> > >
>> > >Thanks,
>> > >-namit
>> > >
>> > >On 5/24/12 11:21 PM, "Ashutosh Chauhan" <ha...@apache.org> wrote:
>> > >
>> > >>Some discussion for this has happened on
>> > >>https://issues.apache.org/jira/browse/HIVE-2702 Is the underlying
>> > problem
>> > >>same as the one which I described on that jira ?
>> > >>
>> > >>Thanks,
>> > >>Ashutosh
>> > >>
>> > >>On Thu, May 24, 2012 at 10:59 PM, Namit Jain <nj...@fb.com> wrote:
>> > >>
>> > >>> Should we disallow non-string partition columns completely ?
>> > >>> Does anyone depend on that ?
>> > >>>
>> > >>>
>> > >>> On 5/24/12 6:49 PM, "Namit Jain" <nj...@fb.com> wrote:
>> > >>>
>> > >>> >
>> > >>> >http://svn.apache.org/viewvc?view=revision&revision=1308427
>> > >>> >
>> > >>> >The patch above broke drop partitions if the partition happens to be
>> > >>> >non-string.
>> > >>> >This is due to a JDO issue with non-string columns.
>> > >>> >
>> > >>> >Is anyone using non-string partition columns ?
>> > >>> >Should be force the partition columns to be only of type string ?
>> > >>> >The documentation probably does not specify anything clearly.
>> > >>> >
>> > >>> >If someone is dependent on non-string partition column, we need to
>> > >>>revert
>> > >>> >this patch, or make a
>> > >>> >special case for string partition columns.
>> > >>> >
>> > >>> >Thanks,
>> > >>> >-namit
>> > >>> >
>> > >>> >
>> > >>>
>> > >>>
>> > >
>> >
>> >
>>
>
>
>
> --
> "...:::Aniket:::... Quetzalco@tl"
Re: non-string partition columns
Posted by Aniket Mokashi <an...@gmail.com>.
We are using non-string partition columns in production as well.
Thanks,
Aniket
On Sat, May 26, 2012 at 1:20 AM, Philip Tromans
<ph...@gmail.com>wrote:
> We're using non-string partition columns in production. I think non string
> partition columns are a good thing to have - it allows you to do all sorts
> of date range calculations etc. AFAIK, MySQL's partition columns can be of
> any type.
>
> Phil.
> On May 26, 2012 7:55 AM, "Namit Jain" <nj...@fb.com> wrote:
>
> > Should I go ahead and file a jira to disallow non-string partition
> columns
> > ?
> > Or, someone depends on that functionality.
> >
> >
> > On 5/25/12 10:01 AM, "Namit Jain" <nj...@fb.com> wrote:
> >
> > >Yes, but the meta-question is:
> > >
> > >Is anyone dependent on non-string partition columns ? Should we drop the
> > >support for non-string
> > >partition columns ?
> > >
> > >
> > >Thanks,
> > >-namit
> > >
> > >On 5/24/12 11:21 PM, "Ashutosh Chauhan" <ha...@apache.org> wrote:
> > >
> > >>Some discussion for this has happened on
> > >>https://issues.apache.org/jira/browse/HIVE-2702 Is the underlying
> > problem
> > >>same as the one which I described on that jira ?
> > >>
> > >>Thanks,
> > >>Ashutosh
> > >>
> > >>On Thu, May 24, 2012 at 10:59 PM, Namit Jain <nj...@fb.com> wrote:
> > >>
> > >>> Should we disallow non-string partition columns completely ?
> > >>> Does anyone depend on that ?
> > >>>
> > >>>
> > >>> On 5/24/12 6:49 PM, "Namit Jain" <nj...@fb.com> wrote:
> > >>>
> > >>> >
> > >>> >http://svn.apache.org/viewvc?view=revision&revision=1308427
> > >>> >
> > >>> >The patch above broke drop partitions if the partition happens to be
> > >>> >non-string.
> > >>> >This is due to a JDO issue with non-string columns.
> > >>> >
> > >>> >Is anyone using non-string partition columns ?
> > >>> >Should be force the partition columns to be only of type string ?
> > >>> >The documentation probably does not specify anything clearly.
> > >>> >
> > >>> >If someone is dependent on non-string partition column, we need to
> > >>>revert
> > >>> >this patch, or make a
> > >>> >special case for string partition columns.
> > >>> >
> > >>> >Thanks,
> > >>> >-namit
> > >>> >
> > >>> >
> > >>>
> > >>>
> > >
> >
> >
>
--
"...:::Aniket:::... Quetzalco@tl"
Re: non-string partition columns
Posted by Philip Tromans <ph...@gmail.com>.
We're using non-string partition columns in production. I think non string
partition columns are a good thing to have - it allows you to do all sorts
of date range calculations etc. AFAIK, MySQL's partition columns can be of
any type.
Phil.
On May 26, 2012 7:55 AM, "Namit Jain" <nj...@fb.com> wrote:
> Should I go ahead and file a jira to disallow non-string partition columns
> ?
> Or, someone depends on that functionality.
>
>
> On 5/25/12 10:01 AM, "Namit Jain" <nj...@fb.com> wrote:
>
> >Yes, but the meta-question is:
> >
> >Is anyone dependent on non-string partition columns ? Should we drop the
> >support for non-string
> >partition columns ?
> >
> >
> >Thanks,
> >-namit
> >
> >On 5/24/12 11:21 PM, "Ashutosh Chauhan" <ha...@apache.org> wrote:
> >
> >>Some discussion for this has happened on
> >>https://issues.apache.org/jira/browse/HIVE-2702 Is the underlying
> problem
> >>same as the one which I described on that jira ?
> >>
> >>Thanks,
> >>Ashutosh
> >>
> >>On Thu, May 24, 2012 at 10:59 PM, Namit Jain <nj...@fb.com> wrote:
> >>
> >>> Should we disallow non-string partition columns completely ?
> >>> Does anyone depend on that ?
> >>>
> >>>
> >>> On 5/24/12 6:49 PM, "Namit Jain" <nj...@fb.com> wrote:
> >>>
> >>> >
> >>> >http://svn.apache.org/viewvc?view=revision&revision=1308427
> >>> >
> >>> >The patch above broke drop partitions if the partition happens to be
> >>> >non-string.
> >>> >This is due to a JDO issue with non-string columns.
> >>> >
> >>> >Is anyone using non-string partition columns ?
> >>> >Should be force the partition columns to be only of type string ?
> >>> >The documentation probably does not specify anything clearly.
> >>> >
> >>> >If someone is dependent on non-string partition column, we need to
> >>>revert
> >>> >this patch, or make a
> >>> >special case for string partition columns.
> >>> >
> >>> >Thanks,
> >>> >-namit
> >>> >
> >>> >
> >>>
> >>>
> >
>
>
Re: non-string partition columns
Posted by Namit Jain <nj...@fb.com>.
Should I go ahead and file a jira to disallow non-string partition columns
?
Or, someone depends on that functionality.
On 5/25/12 10:01 AM, "Namit Jain" <nj...@fb.com> wrote:
>Yes, but the meta-question is:
>
>Is anyone dependent on non-string partition columns ? Should we drop the
>support for non-string
>partition columns ?
>
>
>Thanks,
>-namit
>
>On 5/24/12 11:21 PM, "Ashutosh Chauhan" <ha...@apache.org> wrote:
>
>>Some discussion for this has happened on
>>https://issues.apache.org/jira/browse/HIVE-2702 Is the underlying problem
>>same as the one which I described on that jira ?
>>
>>Thanks,
>>Ashutosh
>>
>>On Thu, May 24, 2012 at 10:59 PM, Namit Jain <nj...@fb.com> wrote:
>>
>>> Should we disallow non-string partition columns completely ?
>>> Does anyone depend on that ?
>>>
>>>
>>> On 5/24/12 6:49 PM, "Namit Jain" <nj...@fb.com> wrote:
>>>
>>> >
>>> >http://svn.apache.org/viewvc?view=revision&revision=1308427
>>> >
>>> >The patch above broke drop partitions if the partition happens to be
>>> >non-string.
>>> >This is due to a JDO issue with non-string columns.
>>> >
>>> >Is anyone using non-string partition columns ?
>>> >Should be force the partition columns to be only of type string ?
>>> >The documentation probably does not specify anything clearly.
>>> >
>>> >If someone is dependent on non-string partition column, we need to
>>>revert
>>> >this patch, or make a
>>> >special case for string partition columns.
>>> >
>>> >Thanks,
>>> >-namit
>>> >
>>> >
>>>
>>>
>
Re: non-string partition columns
Posted by Namit Jain <nj...@fb.com>.
Yes, but the meta-question is:
Is anyone dependent on non-string partition columns ? Should we drop the
support for non-string
partition columns ?
Thanks,
-namit
On 5/24/12 11:21 PM, "Ashutosh Chauhan" <ha...@apache.org> wrote:
>Some discussion for this has happened on
>https://issues.apache.org/jira/browse/HIVE-2702 Is the underlying problem
>same as the one which I described on that jira ?
>
>Thanks,
>Ashutosh
>
>On Thu, May 24, 2012 at 10:59 PM, Namit Jain <nj...@fb.com> wrote:
>
>> Should we disallow non-string partition columns completely ?
>> Does anyone depend on that ?
>>
>>
>> On 5/24/12 6:49 PM, "Namit Jain" <nj...@fb.com> wrote:
>>
>> >
>> >http://svn.apache.org/viewvc?view=revision&revision=1308427
>> >
>> >The patch above broke drop partitions if the partition happens to be
>> >non-string.
>> >This is due to a JDO issue with non-string columns.
>> >
>> >Is anyone using non-string partition columns ?
>> >Should be force the partition columns to be only of type string ?
>> >The documentation probably does not specify anything clearly.
>> >
>> >If someone is dependent on non-string partition column, we need to
>>revert
>> >this patch, or make a
>> >special case for string partition columns.
>> >
>> >Thanks,
>> >-namit
>> >
>> >
>>
>>
Re: non-string partition columns
Posted by Ashutosh Chauhan <ha...@apache.org>.
Some discussion for this has happened on
https://issues.apache.org/jira/browse/HIVE-2702 Is the underlying problem
same as the one which I described on that jira ?
Thanks,
Ashutosh
On Thu, May 24, 2012 at 10:59 PM, Namit Jain <nj...@fb.com> wrote:
> Should we disallow non-string partition columns completely ?
> Does anyone depend on that ?
>
>
> On 5/24/12 6:49 PM, "Namit Jain" <nj...@fb.com> wrote:
>
> >
> >http://svn.apache.org/viewvc?view=revision&revision=1308427
> >
> >The patch above broke drop partitions if the partition happens to be
> >non-string.
> >This is due to a JDO issue with non-string columns.
> >
> >Is anyone using non-string partition columns ?
> >Should be force the partition columns to be only of type string ?
> >The documentation probably does not specify anything clearly.
> >
> >If someone is dependent on non-string partition column, we need to revert
> >this patch, or make a
> >special case for string partition columns.
> >
> >Thanks,
> >-namit
> >
> >
>
>
Re: non-string partition columns
Posted by Namit Jain <nj...@fb.com>.
Should we disallow non-string partition columns completely ?
Does anyone depend on that ?
On 5/24/12 6:49 PM, "Namit Jain" <nj...@fb.com> wrote:
>
>http://svn.apache.org/viewvc?view=revision&revision=1308427
>
>The patch above broke drop partitions if the partition happens to be
>non-string.
>This is due to a JDO issue with non-string columns.
>
>Is anyone using non-string partition columns ?
>Should be force the partition columns to be only of type string ?
>The documentation probably does not specify anything clearly.
>
>If someone is dependent on non-string partition column, we need to revert
>this patch, or make a
>special case for string partition columns.
>
>Thanks,
>-namit
>
>