You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Jos Backus <jo...@catnook.com> on 2012/08/07 01:42:45 UTC
MRv1 mapred command no longer supports starting the jobtracker/tasktracker
Hi,
Not sure if this is a BigTop issue or if I should report this to the
MapReduce project. Pointers appreciated!
This is related to https://issues.apache.org/jira/browse/BIGTOP-460
https://svn.apache.org/viewvc?view=revision&revision=1235956 erroneously
removed the jobtracker and tasktracker commands without providing
replacement functionality. The change points to the yarn command but that
script doesn't provide MRv1 support.
The hadoop command, when invoked as 'hadoop jobtracker', says
DEPRECATED: Use of this script to execute mapred command is deprecated.
Instead use the mapred command for it.
But `mapred jobtracker' says:
Sorry, the jobtracker command is no longer supported.
you may find similar functionality with the "yarn" shell command.
Usage: mapred [--config confdir] COMMAND
where COMMAND is one of:
pipes run a Pipes job
job manipulate MapReduce jobs
queue get information regarding JobQueues
classpath prints the class path needed for running
mapreduce subcommands
groups get the groups which users belong to
historyserver run job history servers as a standalone daemon
distcp <srcurl> <desturl> copy file or directories recursively
archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop
archive
Most commands print help when invoked w/o parameters.
Thanks for any help/insight you may be able to provide.
Cheers,
Jos
--
Jos Backus
jos at catnook.com
Re: MRv1 mapred command no longer supports starting the jobtracker/tasktracker
Posted by Roman Shaposhnik <rv...@apache.org>.
Hi Jos!
On Wed, Aug 8, 2012 at 1:17 PM, Jos Backus <jo...@catnook.com> wrote:
>> To the best of my knowledge that code in the trunk/branch-2 hasn't been
>> maintained for quite some time. It seems to compile, but I can't make it
>> work even with a patched version of wrapper scripts. Is your experience
>> with that code different? Can you provide more details on what exact
>> branch you're using and how exactly you're using TT/JT from there?
>>
>
> Well, I am using the CDH4 RPMs, being under the impression that the changes
> from the open source version are minimal. Are you saying that basically
> MRv1 is deprecated/unsupported at this point as far as the Apache folks are
> concerned? In that case, I should take the issue up with Cloudera if/once
> we have a support contract :)
Different folks mean different things when they mention MRv1. For CDH4, MRv1
means the implementation of the 0.20.2+ mapreduce functionality made available
to work on top of HDFS from the 2.0.0 Hadoop code line. Take a look at this
archive: http://archive.cloudera.com/cdh4/cdh/4/mr1-latest.tar.gz (and
especially
the cloudera/patches folder) if you want to see how it is implemented.
This work, is, of course, completely different from what some folks refer to as
an upstream MRv1 codebase. To the best of my knowledge, the code available
upstream under hadoop-mapreduce-project/src/java/org/apache/hadoop/mapreduce
is no longer maintained maintained and may have bitrotted. It is certainly true,
that, for example, the scripts for launching the TT/JT are gone,
although it seems
theoretically possible to run these daemons.
So, here's my question again: which one are *you* interested in?
> That doesn't bother me as it doesn't affect what I am working on :) I care
> about someone breaking this interface for no reason. I could understand it
> if MRv1 was no longer supported by Apache or Cloudera, but that appears to
> not be that case from what I can tell. So in my mind it's just a bug.
To the best of my knowledge, at this point Apache Hadoop community does
NOT support (or develop) MRv1 functionality in trunk/branch-2.
>> I already did, I sent you an email on May 11th :) Right now I'm trying to
> get the scripts to work properly with CDH4, which is why I ran into this
> problem. Then I checked the Apache repo and saw that it was broken there,
> too, hence my email. At any rate, once I have the scripts in working order
> I will send an update.
Right. I remember that. What I was referring to is that perhaps you could
consider posting the code on the JIRA so that it can be eventually included
in the Bigtop project. It has to go via JIRA due to Apache rules.
Thanks,
Roman.
Re: MRv1 mapred command no longer supports starting the jobtracker/tasktracker
Posted by Jos Backus <jo...@catnook.com>.
Hi Roman!
On Wed, Aug 8, 2012 at 11:56 AM, Roman Shaposhnik <rv...@apache.org> wrote:
> Hi Jos!
>
> On Mon, Aug 6, 2012 at 10:54 PM, Jos Backus <jo...@catnook.com> wrote:
> > I'm aware of that. We'd like to use MRv1 instead of YARN because we are
> not
> > ready/willing to switch at this point. The code is still there, just the
> > mapred script UI was removed. Right now I'm using the hadoop command to
> > start the jobtracker and tasktracker but it seems ugly and someone may
> come
> > in and break that interface, too, as it's advertised as deprecated.
>
> To the best of my knowledge that code in the trunk/branch-2 hasn't been
> maintained for quite some time. It seems to compile, but I can't make it
> work even with a patched version of wrapper scripts. Is your experience
> with that code different? Can you provide more details on what exact
> branch you're using and how exactly you're using TT/JT from there?
>
Well, I am using the CDH4 RPMs, being under the impression that the changes
from the open source version are minimal. Are you saying that basically
MRv1 is deprecated/unsupported at this point as far as the Apache folks are
concerned? In that case, I should take the issue up with Cloudera if/once
we have a support contract :)
>
> Please also note, that MRv1 as delivered by CDH4 distribution is completely
> different from the code that exists in trunk/branch-2.
>
That doesn't bother me as it doesn't affect what I am working on :) I care
about someone breaking this interface for no reason. I could understand it
if MRv1 was no longer supported by Apache or Cloudera, but that appears to
not be that case from what I can tell. So in my mind it's just a bug.
> > So far the HDFS bits seem to work fine under daemontools-encore; I have
> > been using it for several months now without problems.
>
> Great! Any chance you can share your code?
>
> I already did, I sent you an email on May 11th :) Right now I'm trying to
get the scripts to work properly with CDH4, which is why I ran into this
problem. Then I checked the Apache repo and saw that it was broken there,
too, hence my email. At any rate, once I have the scripts in working order
I will send an update.
Thanks,
Jos
> Thanks,
> Roman.
>
--
Jos Backus
jos at catnook.com
Re: MRv1 mapred command no longer supports starting the jobtracker/tasktracker
Posted by Roman Shaposhnik <rv...@apache.org>.
Hi Jos!
On Mon, Aug 6, 2012 at 10:54 PM, Jos Backus <jo...@catnook.com> wrote:
> I'm aware of that. We'd like to use MRv1 instead of YARN because we are not
> ready/willing to switch at this point. The code is still there, just the
> mapred script UI was removed. Right now I'm using the hadoop command to
> start the jobtracker and tasktracker but it seems ugly and someone may come
> in and break that interface, too, as it's advertised as deprecated.
To the best of my knowledge that code in the trunk/branch-2 hasn't been
maintained for quite some time. It seems to compile, but I can't make it
work even with a patched version of wrapper scripts. Is your experience
with that code different? Can you provide more details on what exact
branch you're using and how exactly you're using TT/JT from there?
Please also note, that MRv1 as delivered by CDH4 distribution is completely
different from the code that exists in trunk/branch-2.
> So far the HDFS bits seem to work fine under daemontools-encore; I have
> been using it for several months now without problems.
Great! Any chance you can share your code?
Thanks,
Roman.
Re: MRv1 mapred command no longer supports starting the jobtracker/tasktracker
Posted by Jos Backus <jo...@catnook.com>.
Hi Bruno,
Thanks for responding. Please see below.
On Aug 6, 2012 9:48 PM, "Bruno Mahé" <bm...@apache.org> wrote:
> Hi Jos,
>
> I am not sure I fully understand your issue, but the jobtracker and
tasktracker do not exist anymore in Apache Hadoop 2.0.
> See
>
http://hadoop.apache.org/common/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html
>
>
> Thanks,
> Bruno
I'm aware of that. We'd like to use MRv1 instead of YARN because we are not
ready/willing to switch at this point. The code is still there, just the
mapred script UI was removed. Right now I'm using the hadoop command to
start the jobtracker and tasktracker but it seems ugly and someone may come
in and break that interface, too, as it's advertised as deprecated.
So far the HDFS bits seem to work fine under daemontools-encore; I have
been using it for several months now without problems.
Thanks,
Jos
Re: MRv1 mapred command no longer supports starting the jobtracker/tasktracker
Posted by Bruno Mahé <bm...@apache.org>.
On 08/06/2012 04:42 PM, Jos Backus wrote:
> Hi,
>
> Not sure if this is a BigTop issue or if I should report this to the
> MapReduce project. Pointers appreciated!
>
> This is related to https://issues.apache.org/jira/browse/BIGTOP-460
>
> https://svn.apache.org/viewvc?view=revision&revision=1235956 erroneously
> removed the jobtracker and tasktracker commands without providing
> replacement functionality. The change points to the yarn command but that
> script doesn't provide MRv1 support.
>
> The hadoop command, when invoked as 'hadoop jobtracker', says
>
> DEPRECATED: Use of this script to execute mapred command is deprecated.
> Instead use the mapred command for it.
>
> But `mapred jobtracker' says:
>
> Sorry, the jobtracker command is no longer supported.
> you may find similar functionality with the "yarn" shell command.
> Usage: mapred [--config confdir] COMMAND
> where COMMAND is one of:
> pipes run a Pipes job
> job manipulate MapReduce jobs
> queue get information regarding JobQueues
> classpath prints the class path needed for running
> mapreduce subcommands
> groups get the groups which users belong to
> historyserver run job history servers as a standalone daemon
> distcp <srcurl> <desturl> copy file or directories recursively
> archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop
> archive
>
> Most commands print help when invoked w/o parameters.
>
> Thanks for any help/insight you may be able to provide.
>
> Cheers,
> Jos
>
Hi Jos,
I am not sure I fully understand your issue, but the jobtracker and
tasktracker do not exist anymore in Apache Hadoop 2.0.
See
http://hadoop.apache.org/common/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html
Thanks,
Bruno