You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Wataru Yukawa (JIRA)" <ji...@apache.org> on 2016/03/29 03:51:25 UTC
[jira] [Created] (HIVE-13374) HiveServer2 hangs up if query to scan
too many partition is submitted
Wataru Yukawa created HIVE-13374:
------------------------------------
Summary: HiveServer2 hangs up if query to scan too many partition is submitted
Key: HIVE-13374
URL: https://issues.apache.org/jira/browse/HIVE-13374
Project: Hive
Issue Type: Bug
Components: HiveServer2
Affects Versions: 1.2.1
Environment: HDP2.4.0
JDK1.8
Reporter: Wataru Yukawa
Assignee: Vaibhav Gumashta
If query to scan too many partition(for example, more than 40,000 partitions) is submitted, hiveserver2 hangs up
hiveserver2 java heap status is the following.
https://gyazo.com/e0a35bd628a0dc904bc24f19cdb2261e
https://gyazo.com/65f12cd3875330995508ae11afd394a2
OutOfMemoryError doesn't occur. but other queries fails.
Maybe it is necessary to spill to disk...
hiveserver2.log
{code}
....
2016-03-28 13:44:28,923 INFO [HiveServer2-Handler-Pool: Thread-766923]: io.CombineHiveInputFormat (CombineHiveInputFormat.java:getSplits(517)) - Total number of paths: 47409, launching 50 threads to check non-combinable ones.
2016-03-28 13:44:29,444 INFO [HiveServer2-Handler-Pool: Thread-766923]: io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(439)) - CombineHiveInputSplit creating pool for hdfs://.../aaa=.../yyyymmdd=20150721; using filter path hdfs://.../aaa=.../yyyymmdd=20150721
....
2016-03-28 13:44:34,811 INFO [HiveServer2-Handler-Pool: Thread-766923]: io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(444)) - CombineHiveInputSplit: pool is already created for hdfs://.../aaa=.../yyyymmdd=20160326; using filter path hdfs://.../aaa=.../yyyymmdd=20160326
2016-03-28 13:44:34,811 INFO [HiveServer2-Handler-Pool: Thread-766923]: io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(444)) - CombineHiveInputSplit: pool is already created for hdfs://.../aaa=.../yyyymmdd=20160327; using filter path hdfs://.../aaa=.../yyyymmdd=20160327
2016-03-28 13:44:34,811 INFO [HiveServer2-Handler-Pool: Thread-766923]: io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(444)) - CombineHiveInputSplit: pool is already created for hdfs://.../aaa=.../yyyymmdd=20160328; using filter path hdfs://.../aaa=.../yyyymmdd=20160328
2016-03-28 13:45:00,285 INFO [org.apache.hadoop.util.JvmPauseMonitor$Monitor@6f5d0190]: util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected pause in JVM or host machine (eg GC): pause of approximately 4587ms
GC pool 'PS MarkSweep' had collection(s): count=1 time=4699ms
GC pool 'PS Scavenge' had collection(s): count=1 time=63ms
....
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)