You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Tim Gautier <ti...@gmail.com> on 2007/11/09 01:28:02 UTC
Hadoop .15 and eclipse on windows
I do my nutch development and debugging on a Windows XP machine before
transferring my jar files to a Linux cluster for actual production
runs. This has worked fine in the past, but there seems to be a
problem now that we're using Hadoop .15. Now when I run the injector
(presumably other classes as well) from Eclipse, Hadoop makes a shell
command call to "df" and of course doesn't find it on a Windows
machine so the job fails. There has to be a way around this so I can
debug from Eclipse, anybody know what that might be? Configuration
setting or something to tell it that I'm running on Windows?
Re: Hadoop .15 and eclipse on windows
Posted by Tim Gautier <ti...@gmail.com>.
Sorry about the delay on this response, I was pulled onto other work.
I figured out the problem. I needed to add the Cygwin\bin directory
to my path so the Cygwin\bin\df.exe would execute. Apparently
something changed in Hadoop 0.14 that caused the default configuration
to execute that code path when it didn't prior to 0.14, but most
people didn't notice because they already had it in their path or were
running on Linux. It appears there was even an issue opened for it:
https://issues.apache.org/jira/browse/HADOOP-1792
On Nov 9, 2007 9:20 AM, Enis Soztutar <en...@gmail.com> wrote:
> I checked for the usages of df in Hadoop for you. DF is used by many
> parts of the system including dfs and mapred, but it will not run on a
> node that does not run TaskTracker of Datanode. Could you please check
> your configuration if you're missing smt. Can you confirm that you just
> only submit the job from the windows machine?
>
>
> Tim Gautier wrote:
> > Thanks for the reply, but I think you missed my point. I've been
> > running nutch and Hadoop through eclipse under Windows for several
> > months now. It never called Linux shell commands before, now it is.
> > Maybe it called df from some path I never hit, I don't know. What I
> > do know is that I could do an entire Nutch crawl in eclipse on Windows
> > with Hadoop 0.13 and I can't even inject with the latest version in
> > trunk which is using Hadoop 0.15 because it calls df. I realize I can
> > run it from the command line under cygwin, I just don't want to, I
> > want to use eclipse like I have been for months.
> >
> > I feel like I'm missing something simple, but I can't figure out what
> > it is. Anyone else have any ideas?
> >
> > On Nov 9, 2007 7:00 AM, Enis Soztutar <en...@gmail.com> wrote:
> >
> >> Hadoop has been running df for a long time way before 0.13. You can run
> >> hadoop under cygwin ın windows. Please refer to Hadoop's documentation.
> >>
> >>
> >> Tim Gautier wrote:
> >>
> >>> I do my nutch development and debugging on a Windows XP machine before
> >>> transferring my jar files to a Linux cluster for actual production
> >>> runs. This has worked fine in the past, but there seems to be a
> >>> problem now that we're using Hadoop .15. Now when I run the injector
> >>> (presumably other classes as well) from Eclipse, Hadoop makes a shell
> >>> command call to "df" and of course doesn't find it on a Windows
> >>> machine so the job fails. There has to be a way around this so I can
> >>> debug from Eclipse, anybody know what that might be? Configuration
> >>> setting or something to tell it that I'm running on Windows?
> >>>
> >>>
> >>>
>
Re: Hadoop .15 and eclipse on windows
Posted by Enis Soztutar <en...@gmail.com>.
I checked for the usages of df in Hadoop for you. DF is used by many
parts of the system including dfs and mapred, but it will not run on a
node that does not run TaskTracker of Datanode. Could you please check
your configuration if you're missing smt. Can you confirm that you just
only submit the job from the windows machine?
Tim Gautier wrote:
> Thanks for the reply, but I think you missed my point. I've been
> running nutch and Hadoop through eclipse under Windows for several
> months now. It never called Linux shell commands before, now it is.
> Maybe it called df from some path I never hit, I don't know. What I
> do know is that I could do an entire Nutch crawl in eclipse on Windows
> with Hadoop 0.13 and I can't even inject with the latest version in
> trunk which is using Hadoop 0.15 because it calls df. I realize I can
> run it from the command line under cygwin, I just don't want to, I
> want to use eclipse like I have been for months.
>
> I feel like I'm missing something simple, but I can't figure out what
> it is. Anyone else have any ideas?
>
> On Nov 9, 2007 7:00 AM, Enis Soztutar <en...@gmail.com> wrote:
>
>> Hadoop has been running df for a long time way before 0.13. You can run
>> hadoop under cygwin ın windows. Please refer to Hadoop's documentation.
>>
>>
>> Tim Gautier wrote:
>>
>>> I do my nutch development and debugging on a Windows XP machine before
>>> transferring my jar files to a Linux cluster for actual production
>>> runs. This has worked fine in the past, but there seems to be a
>>> problem now that we're using Hadoop .15. Now when I run the injector
>>> (presumably other classes as well) from Eclipse, Hadoop makes a shell
>>> command call to "df" and of course doesn't find it on a Windows
>>> machine so the job fails. There has to be a way around this so I can
>>> debug from Eclipse, anybody know what that might be? Configuration
>>> setting or something to tell it that I'm running on Windows?
>>>
>>>
>>>
Re: Hadoop .15 and eclipse on windows
Posted by Tim Gautier <ti...@gmail.com>.
Thanks for the reply, but I think you missed my point. I've been
running nutch and Hadoop through eclipse under Windows for several
months now. It never called Linux shell commands before, now it is.
Maybe it called df from some path I never hit, I don't know. What I
do know is that I could do an entire Nutch crawl in eclipse on Windows
with Hadoop 0.13 and I can't even inject with the latest version in
trunk which is using Hadoop 0.15 because it calls df. I realize I can
run it from the command line under cygwin, I just don't want to, I
want to use eclipse like I have been for months.
I feel like I'm missing something simple, but I can't figure out what
it is. Anyone else have any ideas?
On Nov 9, 2007 7:00 AM, Enis Soztutar <en...@gmail.com> wrote:
> Hadoop has been running df for a long time way before 0.13. You can run
> hadoop under cygwin ın windows. Please refer to Hadoop's documentation.
>
>
> Tim Gautier wrote:
> > I do my nutch development and debugging on a Windows XP machine before
> > transferring my jar files to a Linux cluster for actual production
> > runs. This has worked fine in the past, but there seems to be a
> > problem now that we're using Hadoop .15. Now when I run the injector
> > (presumably other classes as well) from Eclipse, Hadoop makes a shell
> > command call to "df" and of course doesn't find it on a Windows
> > machine so the job fails. There has to be a way around this so I can
> > debug from Eclipse, anybody know what that might be? Configuration
> > setting or something to tell it that I'm running on Windows?
> >
> >
>
Re: Hadoop .15 and eclipse on windows
Posted by Enis Soztutar <en...@gmail.com>.
Hadoop has been running df for a long time way before 0.13. You can run
hadoop under cygwin ın windows. Please refer to Hadoop's documentation.
Tim Gautier wrote:
> I do my nutch development and debugging on a Windows XP machine before
> transferring my jar files to a Linux cluster for actual production
> runs. This has worked fine in the past, but there seems to be a
> problem now that we're using Hadoop .15. Now when I run the injector
> (presumably other classes as well) from Eclipse, Hadoop makes a shell
> command call to "df" and of course doesn't find it on a Windows
> machine so the job fails. There has to be a way around this so I can
> debug from Eclipse, anybody know what that might be? Configuration
> setting or something to tell it that I'm running on Windows?
>
>