You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by "r7raul1984@163.com" <r7...@163.com> on 2015/08/28 03:03:59 UTC
sql mapjoin very slow
When I enable mapjoin ,I see Mapjoin task run very slow. My envrioment is hadoop 2.3.0 hive 1.1.0.
My attach is one map hive log and this map's xprof log.
In map xprof log ,I see
Compiled + native Method
92.3% 643527 + 0 java.util.HashMap.get
2.8% 19856 + 0 java.util.HashMap.put
1.2% 8623 + 0 org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper$GetAdaptor.setFromRow
0.1% 953 + 0 org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate
0.1% 576 + 0 org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject
r7raul1984@163.com
Re: Re: sql mapjoin very slow
Posted by "r7raul1984@163.com" <r7...@163.com>.
I use MR.
My mapjoin config as showed in follow picture:
r7raul1984@163.com
From: Sergey Shelukhin
Date: 2015-08-28 09:21
To: user
Subject: Re: sql mapjoin very slow
Are you using MR and Tez? You could try optimized hash table in case of Tez, although it’s supposed to improve memory, not necessarily perf.
Can you also share characteristics of the query and data? It is surprising to see so much time for HashMap.get.
From: "r7raul1984@163.com" <r7...@163.com>
Reply-To: user <us...@hive.apache.org>
Date: Thursday, August 27, 2015 at 18:03
To: user <us...@hive.apache.org>
Subject: sql mapjoin very slow
When I enable mapjoin ,I see Mapjoin task run very slow. My envrioment is hadoop 2.3.0 hive 1.1.0.
My attach is one map hive log and this map's xprof log.
In map xprof log ,I see
Compiled + native Method
92.3% 643527 + 0 java.util.HashMap.get
2.8% 19856 + 0 java.util.HashMap.put
1.2% 8623 + 0 org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper$GetAdaptor.setFromRow
0.1% 953 + 0 org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate
0.1% 576 + 0 org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject
r7raul1984@163.com
Re: sql mapjoin very slow
Posted by Sergey Shelukhin <se...@hortonworks.com>.
Are you using MR and Tez? You could try optimized hash table in case of Tez, although it’s supposed to improve memory, not necessarily perf.
Can you also share characteristics of the query and data? It is surprising to see so much time for HashMap.get.
From: "r7raul1984@163.com<ma...@163.com>" <r7...@163.com>>
Reply-To: user <us...@hive.apache.org>>
Date: Thursday, August 27, 2015 at 18:03
To: user <us...@hive.apache.org>>
Subject: sql mapjoin very slow
When I enable mapjoin ,I see Mapjoin task run very slow. My envrioment is hadoop 2.3.0 hive 1.1.0.
My attach is one map hive log and this map's xprof log.
In map xprof log ,I see
Compiled + native Method
92.3% 643527 + 0 java.util.HashMap.get
2.8% 19856 + 0 java.util.HashMap.put
1.2% 8623 + 0 org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper$GetAdaptor.setFromRow
0.1% 953 + 0 org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate
0.1% 576 + 0 org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject
________________________________
r7raul1984@163.com<ma...@163.com>