You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Ryu Kobayashi (JIRA)" <ji...@apache.org> on 2015/09/09 09:26:46 UTC
[jira] [Commented] (HIVE-11353) Map env does not reflect in the
Local Map Join
[ https://issues.apache.org/jira/browse/HIVE-11353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14736366#comment-14736366 ]
Ryu Kobayashi commented on HIVE-11353:
--------------------------------------
[~ashutoshc], [~brocknoland] Could you check please?
> Map env does not reflect in the Local Map Join
> ----------------------------------------------
>
> Key: HIVE-11353
> URL: https://issues.apache.org/jira/browse/HIVE-11353
> Project: Hive
> Issue Type: Bug
> Reporter: Ryu Kobayashi
> Assignee: Ryu Kobayashi
> Attachments: HIVE-11353.1.patch
>
>
> mapreduce.map.env is not reflected when the Local Map Join is ran. Following a sample query:
> {code}
> hive> set mapreduce.map.env=AAA=111,BBB=222,CCC=333;
> hive> select
> > reflect("java.lang.System", "getenv", "CCC") as CCC,
> > a.AAA,
> > b.BBB
> > from (
> > SELECT
> > reflect("java.lang.System", "getenv", "AAA") as AAA
> > from
> > foo
> > ) a
> > join (
> > select
> > reflect("java.lang.System", "getenv", "BBB") as BBB
> > from
> > foo
> > ) b
> > limit 1;
> Warning: Map Join MAPJOIN[10][bigTable=?] in task 'Stage-3:MAPRED' is a cross product
> Query ID = root_20150716013643_a8ca1539-68ae-4f13-b9fa-7a8b88f01f13
> Total jobs = 1
> 15/07/16 01:36:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> Execution log at: /tmp/root/root_20150716013643_a8ca1539-68ae-4f13-b9fa-7a8b88f01f13.log
> 2015-07-16 01:36:47 Starting to launch local task to process map join; maximum memory = 477102080
> 2015-07-16 01:36:48 Dump the side-table for tag: 0 with group count: 1 into file: file:/tmp/root/9b900f85-d5e4-4632-90bc-19f4bac516ff/hive_2015-07-16_01-36-43_217_8812243019719259041-1/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
> 2015-07-16 01:36:48 Uploaded 1 File to: file:/tmp/root/9b900f85-d5e4-4632-90bc-19f4bac516ff/hive_2015-07-16_01-36-43_217_8812243019719259041-1/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable (282 bytes)
> 2015-07-16 01:36:48 End of local task; Time Taken: 0.934 sec.
> Execution completed successfully
> MapredLocal task succeeded
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_1436962851556_0015, Tracking URL = http://hadoop27:8088/proxy/application_1436962851556_0015/
> Kill Command = /usr/local/hadoop/bin/hadoop job -kill job_1436962851556_0015
> Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 0
> 2015-07-16 01:36:56,488 Stage-3 map = 0%, reduce = 0%
> 2015-07-16 01:37:01,656 Stage-3 map = 100%, reduce = 0%, Cumulative CPU 1.28 sec
> MapReduce Total cumulative CPU time: 1 seconds 280 msec
> Ended Job = job_1436962851556_0015
> MapReduce Jobs Launched:
> Stage-Stage-3: Map: 1 Cumulative CPU: 1.28 sec HDFS Read: 5428 HDFS Write: 13 SUCCESS
> Total MapReduce CPU Time Spent: 1 seconds 280 msec
> OK
> 333 null 222
> Time taken: 19.562 seconds, Fetched: 1 row(s)
> {code}
> The attached patch will include those taken from Hadoop's code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)