You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Sh...@cognizant.com on 2011/09/12 12:03:22 UTC
Issues starting TaskTracker
Hi,
I am trying to setup Hadoop on windows using Cygwin.
Facing issues while starting task tracker:
2011-09-12 15:20:16,890 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system started
2011-09-12 15:20:17,375 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2011-09-12 15:20:17,390 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2011-09-12 15:20:17,562 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2011-09-12 15:20:17,671 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2011-09-12 15:20:17,718 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2011-09-12 15:20:17,750 INFO org.apache.hadoop.mapred.TaskTracker: Starting tasktracker with owner as SYSTEM
2011-09-12 15:20:17,750 ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker because java.io.IOException: Failed to set permissions of path: /usr/local/mapredLocalDir/ttprivate to 0700
at org.apache.hadoop.fs.RawLocalFileSystem.checkReturnValue(RawLocalFileSystem.java:525)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:499)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183)
at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:635)
at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1328)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3430)
2011-09-12 15:20:17,765 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down TaskTracker
Have tried manually setting permissions to 0700 but that doesnt help... Still getting the same issue.
Regards,
Shreya
RE: Issues starting TaskTracker
Posted by Sh...@cognizant.com.
Hi Harsh
Is it possible to run my mapreduce programs in cloudera VM (VM is run
using vmware player)??
How can I copy my jar files and input data there.
Regards,
Shreya
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Monday, September 12, 2011 10:50 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Issues starting TaskTracker
Shreya,
> I was getting the message owner SYSTEM when I was using default
> I was getting the message - running as TITUN, but the same error
What user are you actually launching the TaskTracker as? The directory
owner (user) must be == process owner (user) of the TT, and things
should be fine! Can you confirm that this isn't the case? What version
of Hadoop are you using?
P.s. Am really beginning to dislike MS Exchange or your sysadmin's
mailer settings here :-)
On Mon, Sep 12, 2011 at 9:23 PM, <Sh...@cognizant.com> wrote:
> This e-mail and any files transmitted with it are for the sole use of
the intended recipient(s) and may contain confidential and privileged
information.
> If you are not the intended recipient, please contact the sender by
reply e-mail and destroy all copies of the original message.
> Any unauthorised review, use, disclosure, dissemination, forwarding,
printing or copying of this email or any action taken in reliance on
this e-mail is strictly
> prohibited and may be unlawful.
--
Harsh J
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
Any unauthorised review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly
prohibited and may be unlawful.
Re: Issues starting TaskTracker
Posted by be...@gmail.com.
Hi Shreya
I'm not well verse with the Cygwin setup in windows. Even I tried cygwin for hadoop but was running into multiple issues,So dropped it then n there. Also when you want work on multiple hadoop tools like pig,hive,hbase etc maintaining everything becomes a hassle.(That is what I think about it). My suggestion would be to try out a linux box which would be much better. Get a linux Ubuntu image follow the steps on cloudera site in setting up hadoop in pseudo distributed mode on top of the same.(You can go in for the apache url as well). Just set up hadoop first and once it is success, go in for the required sub projects. It would help you to keep things rolling till we get a 32 bit VM image released from Cloudera.
Also when you go for the VM image go for Ubuntu 10.10 or previous versions 32 Bit.(Since your host OS is 32 bit). BeTter not to go in for Ubuntu 11 cos the CDH distribution don't list it as a supporting OS for their release and may need to go in for installation from the tarball and so on. To keep installation simple and straight forward,check the list of supporting OS from Cloudera site before you go in for a VM image download.
Hope it helps.
Regards
Bejoy K S
-----Original Message-----
From: <Sh...@cognizant.com>
Date: Wed, 14 Sep 2011 18:16:21
To: <be...@gmail.com>
Subject: RE: Issues starting TaskTracker
Hi Bejoy
I am still facing the same issue with task tracker not starting
This was the mail I had sent
Hi,
I am trying to setup Hadoop on windows using Cygwin.
Facing issues while starting task tracker:
2011-09-12 15:20:16,890 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
system started
2011-09-12 15:20:17,375 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
ugi registered.
2011-09-12 15:20:17,390 WARN
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi
already exists!
2011-09-12 15:20:17,562 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2011-09-12 15:20:17,671 INFO org.apache.hadoop.http.HttpServer: Added
global filtersafety
(class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2011-09-12 15:20:17,718 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
Initializing logs' truncater with mapRetainSize=-1 and
reduceRetainSize=-1
2011-09-12 15:20:17,750 INFO org.apache.hadoop.mapred.TaskTracker:
Starting tasktracker with owner as SYSTEM
2011-09-12 15:20:17,750 ERROR org.apache.hadoop.mapred.TaskTracker: Can
not start task tracker because java.io.IOException: Failed to set
permissions of path: /usr/local/mapredLocalDir/ttprivate to 0700
at
org.apache.hadoop.fs.RawLocalFileSystem.checkReturnValue(RawLocalFileSys
tem.java:525)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem
.java:499)
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:3
18)
at
org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183)
at
org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:635)
at
org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1328)
at
org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3430)
2011-09-12 15:20:17,765 INFO org.apache.hadoop.mapred.TaskTracker:
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down TaskTracker
Have tried manually setting permissions to 0700 but that doesnt help...
Still getting the same issue.
Regards,
Shreya
The owner is showing as system whereas
I was getting the message owner SYSTEM when I was using default
locations for "mapred.system.dir" and "mapred.local.dir" in
mapred-site.xml After giving my own locations in mapred-site.xml as
below:
<property><name>mapred.local.dir</name><value>/usr/local/mapredLocalDir<
/value></property>
<property><name>mapred.system.dir</name><value>/usr/local/mapredSystemDi
r</value></property>
I was getting the message - running as TITUN, but the same error
Following is the output of ls -al c:/usr/local/mapredLocalDir $ ls -al
c:/usr/local/mapredLocalDir total 0
drwx------+ 4 Titun None 0 Sep 12 15:20 .
drwx------+ 5 Titun None 0 Sep 12 15:20 ..
drwx------+ 2 Titun None 0 Sep 12 15:20 taskTracker
drwx------+ 2 Titun None 0 Sep 12 15:20 ttprivate
Do you know what can be the issue? I am assuming owner shd be the
problem but how can I change process owner to Titun?
Any pointers would help
Regards,
Shreya
From: Bejoy KS [mailto:bejoy.hadoop@gmail.com]
Sent: Wednesday, September 14, 2011 5:53 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Issues starting TaskTracker
Shreya
You are right. Even I ran into the same hurdle recently and just
raised our concern with Cloudera team. You can join the discussion
https://groups.google.com/a/cloudera.org/group/cdh-user/browse_thread/th
read/56bc678ff473a15f#
<https://groups.google.com/a/cloudera.org/group/cdh-user/browse_thread/t
hread/56bc678ff473a15f>
It is just with the recent release, the new VM released is out of
64 bit Cent OS. Where as the previous versions were based out of 32 bit
Ubuntu OS.(I do have the 32 bit one but size is too large to be send
across by any means). You can check with the Cloudera archives. (not
sure where you can find the link to older versions, need to reach out
Cloudera team for help).
Regards
Bejoy.K.S
On Wed, Sep 14, 2011 at 9:33 AM, <Sh...@cognizant.com> wrote:
Hi
I downloaded cloudera VM
(https://ccp.cloudera.com/display/SUPPORT/Cloudera's+Hadoop+Demo+VM#Clou
dera%27sHadoopDemoVM-DemoVMWareImage
<https://ccp.cloudera.com/display/SUPPORT/Cloudera%27s+Hadoop+Demo+VM#Cl
oudera%27sHadoopDemoVM-DemoVMWareImage> ) for VMware and vmware player.
The VM is 64 bit but my OS is 32 bit.
What can be the solution?
Regards,
Shreya
From: Bejoy KS [mailto:bejoy.hadoop@gmail.com]
Sent: Tuesday, September 13, 2011 3:08 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Issues starting TaskTracker
Shreya
To add on. From cloudera website you would get images for
different VMs like VM Ware, Virtual Box etc. Choose the appropriate one
for your use as per your availabe software.
To your question, it is definitely possible to run map reduce
progarms from Cloudera VM and in fact it is the most comfortable way(at
least for me) to test my map reduce code. When you are on cloudera VM to
test your plain map reduce code in fact you don't even need to pack your
source code into jar, deploy the same and then execute it.(now you might
be doing development in windows and deployment and test in linux) To
test your code just follow the sequence of steps
* Download and install eclipse on the VM (any IDE you are on)
* Create your project with Mapper, Reducer and Driver classes (may
be a single file also would be fine as per your convenience)
* Click on the class that contains your main method, give run as
java application
* It'd do the job for you.
The few things you need to keep in mind are
* Use very minimal test data. Larger data volumes would lead to
very slow execution due to limited resources.(just use VM to test the
logic)
* Normally in our driver class we get the input and output
directory from command line when we deploy as jars and run the same, but
when you run from eclipse just alter the lines of code specifying input
and output as
For input and output directory in hdfs
FileInputFormat.addInputPath(job, new
Path("hdfs://localhost/<full path in hdfs>"));
FileOutputFormat.setOutputPath(job, new
Path(hdfs://localhost/<full path in hdfs>"));
For input and output directory in lfs
FileInputFormat.addInputPath(job, new Path("<full path in
lfs>"));
FileOutputFormat.setOutputPath(job, new Path("<full path in
lfs>"));
Hope it helps
Regards
Bejoy.K.S
On Tue, Sep 13, 2011 at 2:40 PM, Bejoy KS <be...@gmail.com>
wrote:
Hi Shreya
You can copy files from windows to the linux on VM using any
ftp tools like filezilla.
Take a terminal on your linix, type ifconfig , the value given under
'inet addr:' would be your IP address.
Use this IP address and default port (22) to connect to liux image from
Windows through filezilla. The Cloudera VM has the user name and
password as 'cloudera' itself.
Hope It helps
Regards
Bejoy.KS
On Tue, Sep 13, 2011 at 2:18 PM, <Sh...@cognizant.com> wrote:
Hi Harsh,
Version of Hadoop - hadoop-0.20.203.0
How do I make the process owner same as directory owner
Directory owner is - Titun
Regards
Shreya
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Monday, September 12, 2011 10:50 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Issues starting TaskTracker
Shreya,
> I was getting the message owner SYSTEM when I was using default
> I was getting the message - running as TITUN, but the same error
What user are you actually launching the TaskTracker as? The directory
owner (user) must be == process owner (user) of the TT, and things
should be fine! Can you confirm that this isn't the case? What version
of Hadoop are you using?
P.s. Am really beginning to dislike MS Exchange or your sysadmin's
mailer settings here :-)
On Mon, Sep 12, 2011 at 9:23 PM, <Sh...@cognizant.com> wrote:
> This e-mail and any files transmitted with it are for the sole use of
the intended recipient(s) and may contain confidential and privileged
information.
> If you are not the intended recipient, please contact the sender by
reply e-mail and destroy all copies of the original message.
> Any unauthorised review, use, disclosure, dissemination, forwarding,
printing or copying of this email or any action taken in reliance on
this e-mail is strictly
> prohibited and may be unlawful.
--
Harsh J
This e-mail and any files transmitted with it are for the sole use of
the intended recipient(s) and may contain confidential and privileged
information.
If you are not the intended recipient, please contact the sender by
reply e-mail and destroy all copies of the original message.
Any unauthorised review, use, disclosure, dissemination, forwarding,
printing or copying of this email or any action taken in reliance on
this e-mail is strictly
prohibited and may be unlawful.
This e-mail and any files transmitted with it are for the sole use of
the intended recipient(s) and may contain confidential and privileged
information.
If you are not the intended recipient, please contact the sender by
reply e-mail and destroy all copies of the original message.
Any unauthorised review, use, disclosure, dissemination, forwarding,
printing or copying of this email or any action taken in reliance on
this e-mail is strictly
prohibited and may be unlawful.
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
Any unauthorised review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly
prohibited and may be unlawful.
Re: Issues starting TaskTracker
Posted by Bejoy KS <be...@gmail.com>.
Shreya
You are right. Even I ran into the same hurdle recently and just
raised our concern with Cloudera team. You can join the discussion
https://groups.google.com/a/cloudera.org/group/cdh-user/browse_thread/thread/56bc678ff473a15f#
It is just with the recent release, the new VM released is out of 64
bit Cent OS. Where as the previous versions were based out of 32 bit Ubuntu
OS.(I do have the 32 bit one but size is too large to be send across by any
means). You can check with the Cloudera archives. (not sure where you can
find the link to older versions, need to reach out Cloudera team for
help).
Regards
Bejoy.K.S
On Wed, Sep 14, 2011 at 9:33 AM, <Sh...@cognizant.com> wrote:
> Hi****
>
> ** **
>
> I downloaded cloudera VM (
> https://ccp.cloudera.com/display/SUPPORT/Cloudera's+Hadoop+Demo+VM#Cloudera%27sHadoopDemoVM-DemoVMWareImage<https://ccp.cloudera.com/display/SUPPORT/Cloudera%27s+Hadoop+Demo+VM#Cloudera%27sHadoopDemoVM-DemoVMWareImage>)
> for VMware and vmware player.****
>
> The VM is 64 bit but my OS is 32 bit.****
>
> What can be the solution?****
>
> ** **
>
> ** **
>
> Regards,****
>
> Shreya****
>
> ** **
>
> *From:* Bejoy KS [mailto:bejoy.hadoop@gmail.com]
> *Sent:* Tuesday, September 13, 2011 3:08 PM
>
> *To:* mapreduce-user@hadoop.apache.org
> *Subject:* Re: Issues starting TaskTracker****
>
> ** **
>
> Shreya
> To add on. From cloudera website you would get images for different
> VMs like VM Ware, Virtual Box etc. Choose the appropriate one for your use
> as per your availabe software.
> To your question, it is definitely possible to run map reduce
> progarms from Cloudera VM and in fact it is the most comfortable way(at
> least for me) to test my map reduce code. When you are on cloudera VM to
> test your plain map reduce code in fact you don't even need to pack your
> source code into jar, deploy the same and then execute it.(now you might be
> doing development in windows and deployment and test in linux) To test your
> code just follow the sequence of steps****
>
> - Download and install eclipse on the VM (any IDE you are on)****
> - Create your project with Mapper, Reducer and Driver classes (may be a
> single file also would be fine as per your convenience)****
> - Click on the class that contains your main method, give run as java
> application****
> - It'd do the job for you.****
>
> The few things you need to keep in mind are****
>
> - Use very minimal test data. Larger data volumes would lead to very
> slow execution due to limited resources.(just use VM to test the logic)
> ****
> - Normally in our driver class we get the input and output directory
> from command line when we deploy as jars and run the same, but when you run
> from eclipse just alter the lines of code specifying input and output as
> ****
>
> For input and output directory in hdfs
> FileInputFormat.addInputPath(job, new
> Path("hdfs://localhost/<full path in hdfs>"));
> FileOutputFormat.setOutputPath(job, new
> Path(hdfs://localhost/<full path in hdfs>"));
>
> For input and output directory in lfs
> FileInputFormat.addInputPath(job, new Path("<full path in
> lfs>"));
> FileOutputFormat.setOutputPath(job, new Path("<full path in
> lfs>"));
>
> Hope it helps
>
> Regards
> Bejoy.K.S
>
> ****
>
> On Tue, Sep 13, 2011 at 2:40 PM, Bejoy KS <be...@gmail.com> wrote:*
> ***
>
> Hi Shreya
> You can copy files from windows to the linux on VM using any ftp
> tools like filezilla.
> Take a terminal on your linix, type ifconfig , the value given under 'inet
> addr:' would be your IP address.
> Use this IP address and default port (22) to connect to liux image from
> Windows through filezilla. The Cloudera VM has the user name and password as
> 'cloudera' itself.
>
> Hope It helps
>
> Regards
> Bejoy.KS****
>
> ** **
>
> On Tue, Sep 13, 2011 at 2:18 PM, <Sh...@cognizant.com> wrote:****
>
>
> Hi Harsh,
>
> Version of Hadoop - hadoop-0.20.203.0
> How do I make the process owner same as directory owner
> Directory owner is - Titun****
>
>
> Regards
> Shreya
>
> -----Original Message-----
> From: Harsh J [mailto:harsh@cloudera.com]
> Sent: Monday, September 12, 2011 10:50 PM
> To: mapreduce-user@hadoop.apache.org
> Subject: Re: Issues starting TaskTracker****
>
> Shreya,
>
> > I was getting the message owner SYSTEM when I was using default
> > I was getting the message - running as TITUN, but the same error
>
> What user are you actually launching the TaskTracker as? The directory
> owner (user) must be == process owner (user) of the TT, and things
> should be fine! Can you confirm that this isn't the case? What version
> of Hadoop are you using?
>
> P.s. Am really beginning to dislike MS Exchange or your sysadmin's
> mailer settings here :-)
>
> On Mon, Sep 12, 2011 at 9:23 PM, <Sh...@cognizant.com> wrote:
> > This e-mail and any files transmitted with it are for the sole use of
> the intended recipient(s) and may contain confidential and privileged
> information.
> > If you are not the intended recipient, please contact the sender by
> reply e-mail and destroy all copies of the original message.
> > Any unauthorised review, use, disclosure, dissemination, forwarding,
> printing or copying of this email or any action taken in reliance on
> this e-mail is strictly
> > prohibited and may be unlawful.
>
>
>
> --
> Harsh J
>
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information.
> If you are not the intended recipient, please contact the sender by reply
> e-mail and destroy all copies of the original message.
> Any unauthorised review, use, disclosure, dissemination, forwarding,
> printing or copying of this email or any action taken in reliance on this
> e-mail is strictly
> prohibited and may be unlawful.****
>
> ** **
>
> ** **
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information.
> If you are not the intended recipient, please contact the sender by reply
> e-mail and destroy all copies of the original message.
> Any unauthorised review, use, disclosure, dissemination, forwarding,
> printing or copying of this email or any action taken in reliance on this
> e-mail is strictly
> prohibited and may be unlawful.
>
RE: Issues starting TaskTracker
Posted by Sh...@cognizant.com.
Hi
I downloaded cloudera VM
(https://ccp.cloudera.com/display/SUPPORT/Cloudera's+Hadoop+Demo+VM#Clou
dera%27sHadoopDemoVM-DemoVMWareImage) for VMware and vmware player.
The VM is 64 bit but my OS is 32 bit.
What can be the solution?
Regards,
Shreya
From: Bejoy KS [mailto:bejoy.hadoop@gmail.com]
Sent: Tuesday, September 13, 2011 3:08 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Issues starting TaskTracker
Shreya
To add on. From cloudera website you would get images for
different VMs like VM Ware, Virtual Box etc. Choose the appropriate one
for your use as per your availabe software.
To your question, it is definitely possible to run map reduce
progarms from Cloudera VM and in fact it is the most comfortable way(at
least for me) to test my map reduce code. When you are on cloudera VM to
test your plain map reduce code in fact you don't even need to pack your
source code into jar, deploy the same and then execute it.(now you might
be doing development in windows and deployment and test in linux) To
test your code just follow the sequence of steps
* Download and install eclipse on the VM (any IDE you are on)
* Create your project with Mapper, Reducer and Driver classes (may
be a single file also would be fine as per your convenience)
* Click on the class that contains your main method, give run as
java application
* It'd do the job for you.
The few things you need to keep in mind are
* Use very minimal test data. Larger data volumes would lead to
very slow execution due to limited resources.(just use VM to test the
logic)
* Normally in our driver class we get the input and output
directory from command line when we deploy as jars and run the same, but
when you run from eclipse just alter the lines of code specifying input
and output as
For input and output directory in hdfs
FileInputFormat.addInputPath(job, new
Path("hdfs://localhost/<full path in hdfs>"));
FileOutputFormat.setOutputPath(job, new
Path(hdfs://localhost/<full path in hdfs>"));
For input and output directory in lfs
FileInputFormat.addInputPath(job, new Path("<full path in
lfs>"));
FileOutputFormat.setOutputPath(job, new Path("<full path in
lfs>"));
Hope it helps
Regards
Bejoy.K.S
On Tue, Sep 13, 2011 at 2:40 PM, Bejoy KS <be...@gmail.com>
wrote:
Hi Shreya
You can copy files from windows to the linux on VM using any
ftp tools like filezilla.
Take a terminal on your linix, type ifconfig , the value given under
'inet addr:' would be your IP address.
Use this IP address and default port (22) to connect to liux image from
Windows through filezilla. The Cloudera VM has the user name and
password as 'cloudera' itself.
Hope It helps
Regards
Bejoy.KS
On Tue, Sep 13, 2011 at 2:18 PM, <Sh...@cognizant.com> wrote:
Hi Harsh,
Version of Hadoop - hadoop-0.20.203.0
How do I make the process owner same as directory owner
Directory owner is - Titun
Regards
Shreya
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Monday, September 12, 2011 10:50 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Issues starting TaskTracker
Shreya,
> I was getting the message owner SYSTEM when I was using default
> I was getting the message - running as TITUN, but the same error
What user are you actually launching the TaskTracker as? The directory
owner (user) must be == process owner (user) of the TT, and things
should be fine! Can you confirm that this isn't the case? What version
of Hadoop are you using?
P.s. Am really beginning to dislike MS Exchange or your sysadmin's
mailer settings here :-)
On Mon, Sep 12, 2011 at 9:23 PM, <Sh...@cognizant.com> wrote:
> This e-mail and any files transmitted with it are for the sole use of
the intended recipient(s) and may contain confidential and privileged
information.
> If you are not the intended recipient, please contact the sender by
reply e-mail and destroy all copies of the original message.
> Any unauthorised review, use, disclosure, dissemination, forwarding,
printing or copying of this email or any action taken in reliance on
this e-mail is strictly
> prohibited and may be unlawful.
--
Harsh J
This e-mail and any files transmitted with it are for the sole use of
the intended recipient(s) and may contain confidential and privileged
information.
If you are not the intended recipient, please contact the sender by
reply e-mail and destroy all copies of the original message.
Any unauthorised review, use, disclosure, dissemination, forwarding,
printing or copying of this email or any action taken in reliance on
this e-mail is strictly
prohibited and may be unlawful.
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
Any unauthorised review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly
prohibited and may be unlawful.
Re: Issues starting TaskTracker
Posted by Bejoy KS <be...@gmail.com>.
Shreya
To add on. From cloudera website you would get images for different
VMs like VM Ware, Virtual Box etc. Choose the appropriate one for your use
as per your availabe software.
To your question, it is definitely possible to run map reduce progarms
from Cloudera VM and in fact it is the most comfortable way(at least for me)
to test my map reduce code. When you are on cloudera VM to test your plain
map reduce code in fact you don't even need to pack your source code into
jar, deploy the same and then execute it.(now you might be doing development
in windows and deployment and test in linux) To test your code just follow
the sequence of steps
- Download and install eclipse on the VM (any IDE you are on)
- Create your project with Mapper, Reducer and Driver classes (may be a
single file also would be fine as per your convenience)
- Click on the class that contains your main method, give run as java
application
- It'd do the job for you.
The few things you need to keep in mind are
- Use very minimal test data. Larger data volumes would lead to very slow
execution due to limited resources.(just use VM to test the logic)
- Normally in our driver class we get the input and output directory from
command line when we deploy as jars and run the same, but when you run from
eclipse just alter the lines of code specifying input and output as
For input and output directory in hdfs
FileInputFormat.addInputPath(job, new
Path("hdfs://localhost/<full path in hdfs>"));
FileOutputFormat.setOutputPath(job, new
Path(hdfs://localhost/<full path in hdfs>"));
For input and output directory in lfs
FileInputFormat.addInputPath(job, new Path("<full path in
lfs>"));
FileOutputFormat.setOutputPath(job, new Path("<full path in
lfs>"));
Hope it helps
Regards
Bejoy.K.S
On Tue, Sep 13, 2011 at 2:40 PM, Bejoy KS <be...@gmail.com> wrote:
> Hi Shreya
> You can copy files from windows to the linux on VM using any ftp
> tools like filezilla.
> Take a terminal on your linix, type ifconfig , the value given under 'inet
> addr:' would be your IP address.
> Use this IP address and default port (22) to connect to liux image from
> Windows through filezilla. The Cloudera VM has the user name and password as
> 'cloudera' itself.
>
> Hope It helps
>
> Regards
> Bejoy.KS
>
>
> On Tue, Sep 13, 2011 at 2:18 PM, <Sh...@cognizant.com> wrote:
>
>>
>> Hi Harsh,
>>
>> Version of Hadoop - hadoop-0.20.203.0
>> How do I make the process owner same as directory owner
>> Directory owner is - Titun
>>
>> Regards
>> Shreya
>>
>> -----Original Message-----
>> From: Harsh J [mailto:harsh@cloudera.com]
>> Sent: Monday, September 12, 2011 10:50 PM
>> To: mapreduce-user@hadoop.apache.org
>> Subject: Re: Issues starting TaskTracker
>>
>> Shreya,
>>
>> > I was getting the message owner SYSTEM when I was using default
>> > I was getting the message - running as TITUN, but the same error
>>
>> What user are you actually launching the TaskTracker as? The directory
>> owner (user) must be == process owner (user) of the TT, and things
>> should be fine! Can you confirm that this isn't the case? What version
>> of Hadoop are you using?
>>
>> P.s. Am really beginning to dislike MS Exchange or your sysadmin's
>> mailer settings here :-)
>>
>> On Mon, Sep 12, 2011 at 9:23 PM, <Sh...@cognizant.com> wrote:
>> > This e-mail and any files transmitted with it are for the sole use of
>> the intended recipient(s) and may contain confidential and privileged
>> information.
>> > If you are not the intended recipient, please contact the sender by
>> reply e-mail and destroy all copies of the original message.
>> > Any unauthorised review, use, disclosure, dissemination, forwarding,
>> printing or copying of this email or any action taken in reliance on
>> this e-mail is strictly
>> > prohibited and may be unlawful.
>>
>>
>>
>> --
>> Harsh J
>>
>> This e-mail and any files transmitted with it are for the sole use of the
>> intended recipient(s) and may contain confidential and privileged
>> information.
>> If you are not the intended recipient, please contact the sender by reply
>> e-mail and destroy all copies of the original message.
>> Any unauthorised review, use, disclosure, dissemination, forwarding,
>> printing or copying of this email or any action taken in reliance on this
>> e-mail is strictly
>> prohibited and may be unlawful.
>>
>
>
Re: Issues starting TaskTracker
Posted by Bejoy KS <be...@gmail.com>.
Hi Shreya
You can copy files from windows to the linux on VM using any ftp
tools like filezilla.
Take a terminal on your linix, type ifconfig , the value given under 'inet
addr:' would be your IP address.
Use this IP address and default port (22) to connect to liux image from
Windows through filezilla. The Cloudera VM has the user name and password as
'cloudera' itself.
Hope It helps
Regards
Bejoy.KS
On Tue, Sep 13, 2011 at 2:18 PM, <Sh...@cognizant.com> wrote:
>
> Hi Harsh,
>
> Version of Hadoop - hadoop-0.20.203.0
> How do I make the process owner same as directory owner
> Directory owner is - Titun
>
> Regards
> Shreya
>
> -----Original Message-----
> From: Harsh J [mailto:harsh@cloudera.com]
> Sent: Monday, September 12, 2011 10:50 PM
> To: mapreduce-user@hadoop.apache.org
> Subject: Re: Issues starting TaskTracker
>
> Shreya,
>
> > I was getting the message owner SYSTEM when I was using default
> > I was getting the message - running as TITUN, but the same error
>
> What user are you actually launching the TaskTracker as? The directory
> owner (user) must be == process owner (user) of the TT, and things
> should be fine! Can you confirm that this isn't the case? What version
> of Hadoop are you using?
>
> P.s. Am really beginning to dislike MS Exchange or your sysadmin's
> mailer settings here :-)
>
> On Mon, Sep 12, 2011 at 9:23 PM, <Sh...@cognizant.com> wrote:
> > This e-mail and any files transmitted with it are for the sole use of
> the intended recipient(s) and may contain confidential and privileged
> information.
> > If you are not the intended recipient, please contact the sender by
> reply e-mail and destroy all copies of the original message.
> > Any unauthorised review, use, disclosure, dissemination, forwarding,
> printing or copying of this email or any action taken in reliance on
> this e-mail is strictly
> > prohibited and may be unlawful.
>
>
>
> --
> Harsh J
>
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information.
> If you are not the intended recipient, please contact the sender by reply
> e-mail and destroy all copies of the original message.
> Any unauthorised review, use, disclosure, dissemination, forwarding,
> printing or copying of this email or any action taken in reliance on this
> e-mail is strictly
> prohibited and may be unlawful.
>
RE: Issues starting TaskTracker
Posted by Sh...@cognizant.com.
Hi Harsh,
Version of Hadoop - hadoop-0.20.203.0
How do I make the process owner same as directory owner
Directory owner is - Titun
Regards
Shreya
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Monday, September 12, 2011 10:50 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Issues starting TaskTracker
Shreya,
> I was getting the message owner SYSTEM when I was using default
> I was getting the message - running as TITUN, but the same error
What user are you actually launching the TaskTracker as? The directory
owner (user) must be == process owner (user) of the TT, and things
should be fine! Can you confirm that this isn't the case? What version
of Hadoop are you using?
P.s. Am really beginning to dislike MS Exchange or your sysadmin's
mailer settings here :-)
On Mon, Sep 12, 2011 at 9:23 PM, <Sh...@cognizant.com> wrote:
> This e-mail and any files transmitted with it are for the sole use of
the intended recipient(s) and may contain confidential and privileged
information.
> If you are not the intended recipient, please contact the sender by
reply e-mail and destroy all copies of the original message.
> Any unauthorised review, use, disclosure, dissemination, forwarding,
printing or copying of this email or any action taken in reliance on
this e-mail is strictly
> prohibited and may be unlawful.
--
Harsh J
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
Any unauthorised review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly
prohibited and may be unlawful.
Re: Issues starting TaskTracker
Posted by Harsh J <ha...@cloudera.com>.
Shreya,
> I was getting the message owner SYSTEM when I was using default
> I was getting the message - running as TITUN, but the same error
What user are you actually launching the TaskTracker as? The directory
owner (user) must be == process owner (user) of the TT, and things
should be fine! Can you confirm that this isn't the case? What version
of Hadoop are you using?
P.s. Am really beginning to dislike MS Exchange or your sysadmin's
mailer settings here :-)
On Mon, Sep 12, 2011 at 9:23 PM, <Sh...@cognizant.com> wrote:
> This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
> If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
> Any unauthorised review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly
> prohibited and may be unlawful.
--
Harsh J
RE: Issues starting TaskTracker
Posted by Sh...@cognizant.com.
Hi Harsh
I was getting the message owner SYSTEM when I was using default locations for "mapred.system.dir" and "mapred.local.dir" in mapred-site.xml
After giving my own locations in mapred-site.xml as below:
<property><name>mapred.local.dir</name><value>/usr/local/mapredLocalDir</value></property>
<property><name>mapred.system.dir</name><value>/usr/local/mapredSystemDir</value></property>
I was getting the message - running as TITUN, but the same error
Following is the output of ls -al c:/usr/local/mapredLocalDir
$ ls -al c:/usr/local/mapredLocalDir
total 0
drwx------+ 4 Titun None 0 Sep 12 15:20 .
drwx------+ 5 Titun None 0 Sep 12 15:20 ..
drwx------+ 2 Titun None 0 Sep 12 15:20 taskTracker
drwx------+ 2 Titun None 0 Sep 12 15:20 ttprivate
Regards
Shreya
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Mon 9/12/2011 9:17 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Issues starting TaskTracker
Shreya,
What's the output of "ls -al /usr/local/mapredLocalDir", run on Cygwin?
On Mon, Sep 12, 2011 at 9:01 PM, <Sh...@cognizant.com> wrote:
> Hi Harsh
>
> I tried changing the owner, still getting the same error.
>
> Regards
> Shreya
>
--
Harsh J
Re: Issues starting TaskTracker
Posted by Harsh J <ha...@cloudera.com>.
Shreya,
What's the output of "ls -al /usr/local/mapredLocalDir", run on Cygwin?
On Mon, Sep 12, 2011 at 9:01 PM, <Sh...@cognizant.com> wrote:
> Hi Harsh
>
> I tried changing the owner, still getting the same error.
>
> Regards
> Shreya
>
--
Harsh J
RE: Issues starting TaskTracker
Posted by Sh...@cognizant.com.
Hi Harsh
I tried changing the owner, still getting the same error.
Regards
Shreya
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Mon 9/12/2011 5:45 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Issues starting TaskTracker
Hello Shreya,
> Starting tasktracker with owner as SYSTEM
> Failed to set permissions of path: /usr/local/mapredLocalDir/ttprivate to 0700
Is this directory owned by user 'SYSTEM'? For the TaskTracker to start
properly, ensure that the mapred.local.dirs are all owned by the same
user as the one the TaskTracker would run as.
P.s. Your original email came as some form of attachment, and what the
list users actually saw was the legal mumbo jumbo footers alone. I
don't suppose that's normal.
On Mon, Sep 12, 2011 at 3:33 PM, <Sh...@cognizant.com> wrote:
> This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
> If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
> Any unauthorised review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly
> prohibited and may be unlawful.
--
Harsh J
Re: Issues starting TaskTracker
Posted by Harsh J <ha...@cloudera.com>.
Hello Shreya,
> Starting tasktracker with owner as SYSTEM
> Failed to set permissions of path: /usr/local/mapredLocalDir/ttprivate to 0700
Is this directory owned by user 'SYSTEM'? For the TaskTracker to start
properly, ensure that the mapred.local.dirs are all owned by the same
user as the one the TaskTracker would run as.
P.s. Your original email came as some form of attachment, and what the
list users actually saw was the legal mumbo jumbo footers alone. I
don't suppose that's normal.
On Mon, Sep 12, 2011 at 3:33 PM, <Sh...@cognizant.com> wrote:
> This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information.
> If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
> Any unauthorised review, use, disclosure, dissemination, forwarding, printing or copying of this email or any action taken in reliance on this e-mail is strictly
> prohibited and may be unlawful.
--
Harsh J