unable to see anything in stdout

[Asim <li...@gmail.com>]

Hi,

I am not able to see any job output in userlogs/<task_id>/stdout. It
remains empty even though I have many println statements. Are there
any steps to debug this problem?

Regards,
Asim

Re: unable to see anything in stdout

[jason hadoop <ja...@gmail.com>]

Less work by skipping setting up the input splits, distributing the job jar
files, scheduling the map tasks on the task trackers, collecting the task
status results, then starting all the reduce tasks, collecting all the
results, sorting them, feeding them to the reduce tasks, then writing them
to hdfs.
No jvm forking etc.
The overhead of co-ordination is not small, but pays off when the jobs are
large and the number of machines are large.

local works great for jobs that really only need the resources of a single
machine and only require a single reduce task.

On Fri, May 1, 2009 at 6:09 PM, Asim <li...@gmail.com> wrote:

> Thanks Aaron. That worked! However, when i run everything as local, I
> see everything executing much faster on local as compared to a single
> node. Is there any reason for the same?
>
> -Asim
>
> On Thu, Apr 30, 2009 at 9:23 AM, Aaron Kimball <aa...@cloudera.com> wrote:
> > First thing I would do is to run the job in the local jobrunner (as a
> single
> > process on your local machine without involving the cluster):
> >
> > JobConf conf = .....
> > // set other params, mapper, etc. here
> > conf.set("mapred.job.tracker", "local"); // use localjobrunner
> > conf.set("fs.default.name", "file:///"); // read from local hard disk
> > instead of hdfs
> >
> > JobClient.runJob(conf);
> >
> >
> > This will actually print stdout, stderr, etc. to your local terminal. Try
> > this on a single input file. This will let you confirm that it does, in
> > fact, write to stdout.
> >
> > - Aaron
> >
> > On Thu, Apr 30, 2009 at 9:00 AM, Asim <li...@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> I am not able to see any job output in userlogs/<task_id>/stdout. It
> >> remains empty even though I have many println statements. Are there
> >> any steps to debug this problem?
> >>
> >> Regards,
> >> Asim
> >>
> >
>



-- 
Alpha Chapters of my book on Hadoop are available
http://www.apress.com/book/view/9781430219422

Re: unable to see anything in stdout

[Asim <li...@gmail.com>]

Thanks Aaron. That worked! However, when i run everything as local, I
see everything executing much faster on local as compared to a single
node. Is there any reason for the same?

-Asim

On Thu, Apr 30, 2009 at 9:23 AM, Aaron Kimball <aa...@cloudera.com> wrote:
> First thing I would do is to run the job in the local jobrunner (as a single
> process on your local machine without involving the cluster):
>
> JobConf conf = .....
> // set other params, mapper, etc. here
> conf.set("mapred.job.tracker", "local"); // use localjobrunner
> conf.set("fs.default.name", "file:///"); // read from local hard disk
> instead of hdfs
>
> JobClient.runJob(conf);
>
>
> This will actually print stdout, stderr, etc. to your local terminal. Try
> this on a single input file. This will let you confirm that it does, in
> fact, write to stdout.
>
> - Aaron
>
> On Thu, Apr 30, 2009 at 9:00 AM, Asim <li...@gmail.com> wrote:
>
>> Hi,
>>
>> I am not able to see any job output in userlogs/<task_id>/stdout. It
>> remains empty even though I have many println statements. Are there
>> any steps to debug this problem?
>>
>> Regards,
>> Asim
>>
>

Re: unable to see anything in stdout

[Aaron Kimball <aa...@cloudera.com>]

First thing I would do is to run the job in the local jobrunner (as a single
process on your local machine without involving the cluster):

JobConf conf = .....
// set other params, mapper, etc. here
conf.set("mapred.job.tracker", "local"); // use localjobrunner
conf.set("fs.default.name", "file:///"); // read from local hard disk
instead of hdfs

JobClient.runJob(conf);


This will actually print stdout, stderr, etc. to your local terminal. Try
this on a single input file. This will let you confirm that it does, in
fact, write to stdout.

- Aaron

On Thu, Apr 30, 2009 at 9:00 AM, Asim <li...@gmail.com> wrote:

> Hi,
>
> I am not able to see any job output in userlogs/<task_id>/stdout. It
> remains empty even though I have many println statements. Are there
> any steps to debug this problem?
>
> Regards,
> Asim
>