You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Pengcheng Xiong (JIRA)" <ji...@apache.org> on 2014/12/05 00:38:12 UTC
[jira] [Commented] (HIVE-6998) Select query can only support
maximum 128 distinct expressions
[ https://issues.apache.org/jira/browse/HIVE-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234794#comment-14234794 ]
Pengcheng Xiong commented on HIVE-6998:
---------------------------------------
run with 129 and also 200 distinct expressions, no problem
hive> select count(distinct c0),count(distinct c1),count(distinct c2),count(distinct c3),count(distinct c4),count(distinct c5),count(distinct c6),count(distinct c7),count(distinct c8),count(distinct c9),count(distinct c10),count(distinct c11),count(distinct c12),count(distinct c13),count(distinct c14),count(distinct c15),count(distinct c16),count(distinct c17),count(distinct c18),count(distinct c19),count(distinct c20),count(distinct c21),count(distinct c22),count(distinct c23),count(distinct c24),count(distinct c25),count(distinct c26),count(distinct c27),count(distinct c28),count(distinct c29),count(distinct c30),count(distinct c31),count(distinct c32),count(distinct c33),count(distinct c34),count(distinct c35),count(distinct c36),count(distinct c37),count(distinct c38),count(distinct c39),count(distinct c40),count(distinct c41),count(distinct c42),count(distinct c43),count(distinct c44),count(distinct c45),count(distinct c46),count(distinct c47),count(distinct c48),count(distinct c49),count(distinct c50),count(distinct c51),count(distinct c52),count(distinct c53),count(distinct c54),count(distinct c55),count(distinct c56),count(distinct c57),count(distinct c58),count(distinct c59),count(distinct c60),count(distinct c61),count(distinct c62),count(distinct c63),count(distinct c64),count(distinct c65),count(distinct c66),count(distinct c67),count(distinct c68),count(distinct c69),count(distinct c70),count(distinct c71),count(distinct c72),count(distinct c73),count(distinct c74),count(distinct c75),count(distinct c76),count(distinct c77),count(distinct c78),count(distinct c79),count(distinct c80),count(distinct c81),count(distinct c82),count(distinct c83),count(distinct c84),count(distinct c85),count(distinct c86),count(distinct c87),count(distinct c88),count(distinct c89),count(distinct c90),count(distinct c91),count(distinct c92),count(distinct c93),count(distinct c94),count(distinct c95),count(distinct c96),count(distinct c97),count(distinct c98),count(distinct c99),count(distinct c100),count(distinct c101),count(distinct c102),count(distinct c103),count(distinct c104),count(distinct c105),count(distinct c106),count(distinct c107),count(distinct c108),count(distinct c109),count(distinct c110),count(distinct c111),count(distinct c112),count(distinct c113),count(distinct c114),count(distinct c115),count(distinct c116),count(distinct c117),count(distinct c118),count(distinct c119),count(distinct c120),count(distinct c121),count(distinct c122),count(distinct c123),count(distinct c124),count(distinct c125),count(distinct c126),count(distinct c127),count(distinct c128),count(distinct c129),count(distinct c130),count(distinct c131),count(distinct c132),count(distinct c133),count(distinct c134),count(distinct c135),count(distinct c136),count(distinct c137),count(distinct c138),count(distinct c139),count(distinct c140),count(distinct c141),count(distinct c142),count(distinct c143),count(distinct c144),count(distinct c145),count(distinct c146),count(distinct c147),count(distinct c148),count(distinct c149),count(distinct c150),count(distinct c151),count(distinct c152),count(distinct c153),count(distinct c154),count(distinct c155),count(distinct c156),count(distinct c157),count(distinct c158),count(distinct c159),count(distinct c160),count(distinct c161),count(distinct c162),count(distinct c163),count(distinct c164),count(distinct c165),count(distinct c166),count(distinct c167),count(distinct c168),count(distinct c169),count(distinct c170),count(distinct c171),count(distinct c172),count(distinct c173),count(distinct c174),count(distinct c175),count(distinct c176),count(distinct c177),count(distinct c178),count(distinct c179),count(distinct c180),count(distinct c181),count(distinct c182),count(distinct c183),count(distinct c184),count(distinct c185),count(distinct c186),count(distinct c187),count(distinct c188),count(distinct c189),count(distinct c190),count(distinct c191),count(distinct c192),count(distinct c193),count(distinct c194),count(distinct c195),count(distinct c196),count(distinct c197),count(distinct c198),count(distinct c199)from tbl_200columns;
OK
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Time taken: 1.727 seconds, Fetched: 1 row(s)
[~ashutoshc], I would like to close this jira ticket.
> Select query can only support maximum 128 distinct expressions
> --------------------------------------------------------------
>
> Key: HIVE-6998
> URL: https://issues.apache.org/jira/browse/HIVE-6998
> Project: Hive
> Issue Type: Bug
> Components: Query Processor, Serializers/Deserializers
> Affects Versions: 0.14.0
> Reporter: Chaoyu Tang
>
> Select query can only support maximum 128 distinct expressions. Otherwise, you will be thrown ArrayIndexOutOfBoundsException. For a query like:
> select count(distinct c1), count(distinct c2), count(distinct c3), count(distinct c4), count(distinct c5), count(distinct c6), ...., count(distinct c128), count(distinct c129) from tbl_129columns;
> you will get error like:
> {code}
> java.lang.Exception: java.lang.RuntimeException: Hive Runtime Error while closing operators
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:354)
> Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:260)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
> at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:223)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:695)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: -128
> at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1141)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:579)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:227)
> ... 10 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: -128
> at org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1099)
> at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1138)
> ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: -128
> at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:327)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
> at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1064)
> at org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1082)
> ... 16 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -128
> at java.util.ArrayList.get(ArrayList.java:324)
> at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.serialize(BinarySortableSerDe.java:838)
> at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.serialize(BinarySortableSerDe.java:600)
> at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.toHiveKey(ReduceSinkOperator.java:401)
> at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:320)
> ... 19 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)