You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Vaibhav Gumashta (JIRA)" <ji...@apache.org> on 2016/05/05 22:17:12 UTC
[jira] [Updated] (HIVE-13374) HiveServer2 hangs up if query to scan
too many partition is submitted
[ https://issues.apache.org/jira/browse/HIVE-13374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vaibhav Gumashta updated HIVE-13374:
------------------------------------
Assignee: (was: Vaibhav Gumashta)
> HiveServer2 hangs up if query to scan too many partition is submitted
> ---------------------------------------------------------------------
>
> Key: HIVE-13374
> URL: https://issues.apache.org/jira/browse/HIVE-13374
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Affects Versions: 1.2.1
> Environment: HDP2.4.0
> JDK1.8
> Reporter: Wataru Yukawa
>
> If query to scan too many partition(for example, more than 40,000 partitions) is submitted, hiveserver2 hangs up
> hiveserver2 java heap status is the following.
> https://gyazo.com/e0a35bd628a0dc904bc24f19cdb2261e
> https://gyazo.com/65f12cd3875330995508ae11afd394a2
> OutOfMemoryError doesn't occur. but other queries fails.
> Maybe it is necessary to spill to disk...
> hiveserver2.log
> {code}
> ....
> 2016-03-28 13:44:28,923 INFO [HiveServer2-Handler-Pool: Thread-766923]: io.CombineHiveInputFormat (CombineHiveInputFormat.java:getSplits(517)) - Total number of paths: 47409, launching 50 threads to check non-combinable ones.
> 2016-03-28 13:44:29,444 INFO [HiveServer2-Handler-Pool: Thread-766923]: io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(439)) - CombineHiveInputSplit creating pool for hdfs://.../aaa=.../yyyymmdd=20150721; using filter path hdfs://.../aaa=.../yyyymmdd=20150721
> ....
> 2016-03-28 13:44:34,811 INFO [HiveServer2-Handler-Pool: Thread-766923]: io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(444)) - CombineHiveInputSplit: pool is already created for hdfs://.../aaa=.../yyyymmdd=20160326; using filter path hdfs://.../aaa=.../yyyymmdd=20160326
> 2016-03-28 13:44:34,811 INFO [HiveServer2-Handler-Pool: Thread-766923]: io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(444)) - CombineHiveInputSplit: pool is already created for hdfs://.../aaa=.../yyyymmdd=20160327; using filter path hdfs://.../aaa=.../yyyymmdd=20160327
> 2016-03-28 13:44:34,811 INFO [HiveServer2-Handler-Pool: Thread-766923]: io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(444)) - CombineHiveInputSplit: pool is already created for hdfs://.../aaa=.../yyyymmdd=20160328; using filter path hdfs://.../aaa=.../yyyymmdd=20160328
> 2016-03-28 13:45:00,285 INFO [org.apache.hadoop.util.JvmPauseMonitor$Monitor@6f5d0190]: util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected pause in JVM or host machine (eg GC): pause of approximately 4587ms
> GC pool 'PS MarkSweep' had collection(s): count=1 time=4699ms
> GC pool 'PS Scavenge' had collection(s): count=1 time=63ms
> ....
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)