You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Kristof Vanbecelaere <kr...@gmail.com> on 2014/07/22 22:07:16 UTC
select list for dynamic partition insert
While playing with the movielens data set to learn about dynamic partitions
I ran
from u_data insert overwrite table u_data_p partition (rating) select *
This failed with
[Error 20004]: Fatal error occurred when node tried to create too many
dynamic partitions. The maximum number of dynamic partitions is controlled
by hive.exec.max.dynamic.partitions and
hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100
Apparently select * cannot be used and one should explicitly list the
fields in the select list. I guess a beter error message is required here.
Re: select list for dynamic partition insert
Posted by Kristof Vanbecelaere <kr...@gmail.com>.
I see. The last column in u_data is unixtime while I wanted to partition on
rating. I just assumed Hive would use the same-named column as the one
mentioned in the partition spec. Thanks for clarifying this, I missed that
bit in the documentation.
On Tue, Jul 22, 2014 at 10:13 PM, Prasanth Jayachandran <
pjayachandran@hortonworks.com> wrote:
> From the error msg it looks like there are too many distinct values in
> partition column. Try increasing the count
> hive.exec.max.dynamic.partitions.pernode to a number >100. If you already
> know the number of distinct values in partition column, try a value greater
> than or equal to that number.
> Hive uses the last column of select * query as the partition column.
> Projecting the list of columns in select query is not always mandatory.
>
> Thanks
> Prasanth Jayachandran
>
> On Jul 22, 2014, at 1:07 PM, Kristof Vanbecelaere <
> kristof.vanbecelaere@gmail.com> wrote:
>
> While playing with the movielens data set to learn about dynamic
> partitions I ran
>
> from u_data insert overwrite table u_data_p partition (rating) select *
>
> This failed with
>
> [Error 20004]: Fatal error occurred when node tried to create too many
> dynamic partitions. The maximum number of dynamic partitions is controlled
> by hive.exec.max.dynamic.partitions and
> hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100
>
> Apparently select * cannot be used and one should explicitly list the
> fields in the select list. I guess a beter error message is required here.
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: select list for dynamic partition insert
Posted by Prasanth Jayachandran <pj...@hortonworks.com>.
>From the error msg it looks like there are too many distinct values in partition column. Try increasing the count hive.exec.max.dynamic.partitions.pernode to a number >100. If you already know the number of distinct values in partition column, try a value greater than or equal to that number.
Hive uses the last column of select * query as the partition column. Projecting the list of columns in select query is not always mandatory.
Thanks
Prasanth Jayachandran
On Jul 22, 2014, at 1:07 PM, Kristof Vanbecelaere <kr...@gmail.com> wrote:
> While playing with the movielens data set to learn about dynamic partitions I ran
>
> from u_data insert overwrite table u_data_p partition (rating) select *
>
> This failed with
>
> [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100
>
> Apparently select * cannot be used and one should explicitly list the fields in the select list. I guess a beter error message is required here.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.