You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Frank Singleton <b1...@gmail.com> on 2008/08/14 06:25:34 UTC
task execution environment clarification
Hi,
I have an observation about the task execution environment.
Its a little long but here we go ..........
I have read
http://hadoop.apache.org/core/docs/current/mapred_tutorial.html#Task+Execution+%26+Environment
but wanted a clarification.
I have a mapper written in python that does some processing of data.
This is working ok
using streaming and I get the expected results :-)
Now it so happens that my mapper also creates unique dirs under /tmp (I
thought)
using python's tempdir = tempfile.mkdtemp() for each line of data
streamed to it (to hold
some intermediate processing data).
It is launched as follows (as user hadoop), and completes successfully.
./hadoop-0.17.1/bin/hadoop jar
/home/hadoop/hadoop-0.17.1/contrib/streaming/hadoop-0.17.1-streaming.jar
-input "/user/hadoop/test2/small_file_list.txt" -output
/user/hadoop/mapout79 -mapper /home/hadoop/mapper.py -reducer
/home/hadoop/reducer.py
tionalConfSpec_:null
null=@@@userJobConfProps_.get(stream.shipped.hadoopstreaming
packageJobJar: [/tmp/hadoop-hadoop/hadoop-unjar50444/] []
/tmp/streamjob50445.jar tmpDir=null
08/08/13 22:23:54 INFO mapred.FileInputFormat: Total input paths to
process : 1
08/08/13 22:23:55 INFO streaming.StreamJob: getLocalDirs():
[/home/hadoop/mapred/local]
08/08/13 22:23:55 INFO streaming.StreamJob: Running job:
job_200808132053_0006
08/08/13 22:23:55 INFO streaming.StreamJob: To kill this job, run:
08/08/13 22:23:55 INFO streaming.StreamJob:
/home/hadoop/hadoop-0.17.1/bin/../bin/hadoop job
-Dmapred.job.tracker=hdfs://XXX.XXX.XXX.XXX:9001/ -kill
job_200808132053_0006
08/08/13 22:23:55 INFO streaming.StreamJob: Tracking URL:
http://XXXXXXXXXXX:50030/jobdetails.jsp?jobid=job_200808132053_0006
08/08/13 22:23:56 INFO streaming.StreamJob: map 0% reduce 0%
08/08/13 22:24:05 INFO streaming.StreamJob: map 50% reduce 0%
08/08/13 22:24:06 INFO streaming.StreamJob: map 100% reduce 0%
08/08/13 22:25:17 INFO streaming.StreamJob: map 100% reduce 17%
08/08/13 22:25:21 INFO streaming.StreamJob: map 100% reduce 100%
08/08/13 22:25:21 INFO streaming.StreamJob: Job complete:
job_200808132053_0006
08/08/13 22:25:21 INFO streaming.StreamJob: Output: /user/hadoop/mapout79
[
Anyway , using pythons logging framework, I could see the following
entry in my log thats shows the tmpdir
tmpdir =
[/home/hadoop/mapred/local/taskTracker/jobcache/job_200808132053_0006/task_200808132053_0006_m_000000_0/work/tmp/tmpFW_lIc"]
The suprise to me was that I thought the mapper was creating a tmp dir
called /tmp/tmpFW_1Ic on the underlying filesyste (/tmp on ext3
on a fedora box), but the logging messages appears to indicate
/home/hadoop/mapred/local/taskTracker/jobcache/job_200808132053_0006/task_200808132053_0006_m_000000_0/work/tmp/tmpFW_lIc
I have some questions.
1. Is this correct or did I miss something ?
2. If this is correct. what if I really wanted to write to /tmp/ or
/some_other_path on the underlying filesystem, how
would one achieve that
3. When I unittest my code outside the hadoop streaming framework (eg:
cat data | mapper.py | reducer.py ) I am sure
I saw writes to /tmp on the underlying file system (ok this not a
question, just an observation).
Perhaps this is just the consequence of the abstracted view hadoop [job]
has of /tmp etc and that i need to keep
this in mind when unittesting mappers/reducers and mappers creating tmp
resources on the "filesystem" both using hadoop
streaming framework and also outside on this framework.
Then again, its late and I probably goofed somewhere ;-)
Comments welcome :-)
Thanks / Frank
Re: task execution environment clarification
Posted by Frank Singleton <b1...@gmail.com>.
Allen Wittenauer wrote:
>
> On 8/13/08 9:25 PM, "Frank Singleton" <b1...@gmail.com> wrote:
>
>> Perhaps this is just the consequence of the abstracted view hadoop [job]
>> has of /tmp etc and that i need to keep
>> this in mind when unittesting mappers/reducers and mappers creating tmp
>> resources on the "filesystem" both using hadoop
>> streaming framework and also outside on this framework.
>>
>> Then again, its late and I probably goofed somewhere ;-)
>>
>> Comments welcome :-)
>>
>
> I suspect you're seeing the results of
> https://issues.apache.org/jira/browse/HADOOP-2735 . What is your
> mapred.child.tmp set to?
>
>
>
>
hehe!!
setting mapred.child.tmp to /tmp and not ./tmp seems to solve my problem.
Thanks / Frank
Re: task execution environment clarification
Posted by Frank Singleton <b1...@gmail.com>.
Allen Wittenauer wrote:
>
> On 8/13/08 9:25 PM, "Frank Singleton" <b1...@gmail.com> wrote:
>
>> Perhaps this is just the consequence of the abstracted view hadoop [job]
>> has of /tmp etc and that i need to keep
>> this in mind when unittesting mappers/reducers and mappers creating tmp
>> resources on the "filesystem" both using hadoop
>> streaming framework and also outside on this framework.
>>
>> Then again, its late and I probably goofed somewhere ;-)
>>
>> Comments welcome :-)
>>
>
> I suspect you're seeing the results of
> https://issues.apache.org/jira/browse/HADOOP-2735 . What is your
> mapred.child.tmp set to?
>
>
>
>
Hmm, is that ./tmp ie relative path (and not /tmp) doh!!
<property>
<name>mapred.child.tmp</name>
<value>./tmp</value>
<description> To set the value of tmp directory for map and reduce tasks.
If the value is an absolute path, it is directly assigned. Otherwise,
it is
prepended with task's working directory. The java tasks are executed with
option -Djava.io.tmpdir='the absolute path of the tmp dir'. Pipes and
streaming are set with environment variable,
TMPDIR='the absolute path of the tmp dir'
</description>
</property>
Thanks / Frank
Re: task execution environment clarification
Posted by Allen Wittenauer <aw...@yahoo-inc.com>.
On 8/13/08 9:25 PM, "Frank Singleton" <b1...@gmail.com> wrote:
> Perhaps this is just the consequence of the abstracted view hadoop [job]
> has of /tmp etc and that i need to keep
> this in mind when unittesting mappers/reducers and mappers creating tmp
> resources on the "filesystem" both using hadoop
> streaming framework and also outside on this framework.
>
> Then again, its late and I probably goofed somewhere ;-)
>
> Comments welcome :-)
I suspect you're seeing the results of
https://issues.apache.org/jira/browse/HADOOP-2735 . What is your
mapred.child.tmp set to?