You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Gordon Linoff <gl...@gmail.com> on 2009/11/23 00:56:56 UTC
Saving Intermediate Results from the Mapper
I am starting to learn Hadoop, using the Yahoo virtual machine with version
0.18.
My question is rather simple. I would like to execute a map/reduce job. In
addition to getting the results from the reduce, I would also like to save
the intermediate results from the map in another HDFS file. Is this
possible?
--gordon
Re: Saving Intermediate Results from the Mapper
Posted by Amogh Vasekar <am...@yahoo-inc.com>.
Hi,
I'm not sure if this will apply to your case since i'm not aware of the common part of job2:mapper and job3:mapper but would like to give it a shot.
The whole process can be combined into a single mapred job. The mapper will read a record and process till the "saved data part" , then for each record will output 2 records , one each for the job2 and job3 mappers. The keys of records will be tagged ( <tag,key> ) depending on what reducer processing you want to do. In reduce() you can use this tag to make processing decision. A custom partitioner might be needed depending on the key types to ensure unique sets for reducer.
Ignore if this doesn't fit your bill :)
Amogh
On 11/25/09 9:35 AM, "Gordon Linoff" <gl...@gmail.com> wrote:
Does anyone have a pointer to code that allows the map to save data in
intermediate files, for use in a later map/reduce job? I have been looking
for an example and cannot find one.
I have investigated MultipleOutputFormat and MultipleOutputs. Because I am
using version 0.18.3, I don't have MultipleOutputs. The problem with
MultipleOutputFormat is that the data I want to save is a different format
from the data I want to pass to the Reducer. I have also tried opening a
sequence file directly from the mapper, but I am concerned that this is not
fault tolerant.
The process currently is:
Job1: Mapper: reads complicated data, saves out data structure.
Job2: Mapper: reads saved data, processes and sends data to Reducer 2.
Job3: Mapper: reads saved data, processes and sends data to Reducer 3.
I would like to combine the first two steps, so the process is:
Job1: Mapper: reads complicated data, saves out data structure, and passes
processed data to Reducer 2.
Job2: Mapper: reads saved data, processes and sends to Reducer 3.
--gordon
On Sun, Nov 22, 2009 at 9:27 PM, Jason Venner <ja...@gmail.com>wrote:
> You can manually write the map output to a new file, there are a number of
> examples of opening a sequence file and writing to it on the web or in the
> example code for various hadoop books.
>
> You can also disable the removal of intermediate data, which will result in
> potentially large amounts of data being left in the mapred.local.dir.
>
>
>
> On Sun, Nov 22, 2009 at 3:56 PM, Gordon Linoff <gl...@gmail.com> wrote:
>
>> I am starting to learn Hadoop, using the Yahoo virtual machine with
>> version
>> 0.18.
>>
>> My question is rather simple. I would like to execute a map/reduce job.
>> In
>> addition to getting the results from the reduce, I would also like to save
>> the intermediate results from the map in another HDFS file. Is this
>> possible?
>>
>> --gordon
>>
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=jewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>
Re: Saving Intermediate Results from the Mapper
Posted by Rekha Joshi <re...@yahoo-inc.com>.
https://issues.apache.org/jira/browse/HADOOP-372 has valuable information on InputFormat/MapInput/RecordReader, you may try using the pseudo code. Thanks!
On 11/25/09 9:35 AM, "Gordon Linoff" <gl...@gmail.com> wrote:
Does anyone have a pointer to code that allows the map to save data in
intermediate files, for use in a later map/reduce job? I have been looking
for an example and cannot find one.
I have investigated MultipleOutputFormat and MultipleOutputs. Because I am
using version 0.18.3, I don't have MultipleOutputs. The problem with
MultipleOutputFormat is that the data I want to save is a different format
from the data I want to pass to the Reducer. I have also tried opening a
sequence file directly from the mapper, but I am concerned that this is not
fault tolerant.
The process currently is:
Job1: Mapper: reads complicated data, saves out data structure.
Job2: Mapper: reads saved data, processes and sends data to Reducer 2.
Job3: Mapper: reads saved data, processes and sends data to Reducer 3.
I would like to combine the first two steps, so the process is:
Job1: Mapper: reads complicated data, saves out data structure, and passes
processed data to Reducer 2.
Job2: Mapper: reads saved data, processes and sends to Reducer 3.
--gordon
On Sun, Nov 22, 2009 at 9:27 PM, Jason Venner <ja...@gmail.com>wrote:
> You can manually write the map output to a new file, there are a number of
> examples of opening a sequence file and writing to it on the web or in the
> example code for various hadoop books.
>
> You can also disable the removal of intermediate data, which will result in
> potentially large amounts of data being left in the mapred.local.dir.
>
>
>
> On Sun, Nov 22, 2009 at 3:56 PM, Gordon Linoff <gl...@gmail.com> wrote:
>
>> I am starting to learn Hadoop, using the Yahoo virtual machine with
>> version
>> 0.18.
>>
>> My question is rather simple. I would like to execute a map/reduce job.
>> In
>> addition to getting the results from the reduce, I would also like to save
>> the intermediate results from the map in another HDFS file. Is this
>> possible?
>>
>> --gordon
>>
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=jewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>
Re: Saving Intermediate Results from the Mapper
Posted by Gordon Linoff <gl...@gmail.com>.
Does anyone have a pointer to code that allows the map to save data in
intermediate files, for use in a later map/reduce job? I have been looking
for an example and cannot find one.
I have investigated MultipleOutputFormat and MultipleOutputs. Because I am
using version 0.18.3, I don't have MultipleOutputs. The problem with
MultipleOutputFormat is that the data I want to save is a different format
from the data I want to pass to the Reducer. I have also tried opening a
sequence file directly from the mapper, but I am concerned that this is not
fault tolerant.
The process currently is:
Job1: Mapper: reads complicated data, saves out data structure.
Job2: Mapper: reads saved data, processes and sends data to Reducer 2.
Job3: Mapper: reads saved data, processes and sends data to Reducer 3.
I would like to combine the first two steps, so the process is:
Job1: Mapper: reads complicated data, saves out data structure, and passes
processed data to Reducer 2.
Job2: Mapper: reads saved data, processes and sends to Reducer 3.
--gordon
On Sun, Nov 22, 2009 at 9:27 PM, Jason Venner <ja...@gmail.com>wrote:
> You can manually write the map output to a new file, there are a number of
> examples of opening a sequence file and writing to it on the web or in the
> example code for various hadoop books.
>
> You can also disable the removal of intermediate data, which will result in
> potentially large amounts of data being left in the mapred.local.dir.
>
>
>
> On Sun, Nov 22, 2009 at 3:56 PM, Gordon Linoff <gl...@gmail.com> wrote:
>
>> I am starting to learn Hadoop, using the Yahoo virtual machine with
>> version
>> 0.18.
>>
>> My question is rather simple. I would like to execute a map/reduce job.
>> In
>> addition to getting the results from the reduce, I would also like to save
>> the intermediate results from the map in another HDFS file. Is this
>> possible?
>>
>> --gordon
>>
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=jewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>
Re: Saving Intermediate Results from the Mapper
Posted by Jason Venner <ja...@gmail.com>.
<property>
<name>keep.task.files.pattern</name>
<value>.*_m_123456_0</value>
<description>Keep all files from tasks whose task names match the given
regular expression. Defaults to none.</description>
</property>
On Sun, Nov 22, 2009 at 11:46 PM, Amogh Vasekar <am...@yahoo-inc.com> wrote:
> Hi,
> keep.tasks.files.pattern is what you need, as the name suggests its a
> pattern match on intermediate outputs generated.
>
> Wrt to copying map data to hdfs, your mappers close() method should help
> you achieve this, but might slow up your tasks.
>
> Amogh
>
>
> On 11/23/09 8:08 AM, "Jeff Zhang" <zj...@gmail.com> wrote:
>
> Hi Jason,
>
> which option is for setting disable the removal of intermediate data ?
>
> Thank you
>
> Jeff Zhang
>
>
> On Mon, Nov 23, 2009 at 10:27 AM, Jason Venner <jason.hadoop@gmail.com
> >wrote:
>
> > You can manually write the map output to a new file, there are a number
> of
> > examples of opening a sequence file and writing to it on the web or in
> the
> > example code for various hadoop books.
> >
> > You can also disable the removal of intermediate data, which will result
> in
> > potentially large amounts of data being left in the mapred.local.dir.
> >
> >
> > On Sun, Nov 22, 2009 at 3:56 PM, Gordon Linoff <gl...@gmail.com>
> wrote:
> >
> > > I am starting to learn Hadoop, using the Yahoo virtual machine with
> > version
> > > 0.18.
> > >
> > > My question is rather simple. I would like to execute a map/reduce
> job.
> > > In
> > > addition to getting the results from the reduce, I would also like to
> > save
> > > the intermediate results from the map in another HDFS file. Is this
> > > possible?
> > >
> > > --gordon
> > >
> >
> >
> >
> > --
> > Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> > http://www.amazon.com/dp/1430219424?tag=jewlerymall
> > www.prohadoopbook.com a community for Hadoop Professionals
> >
>
>
--
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals
Re: Saving Intermediate Results from the Mapper
Posted by Amogh Vasekar <am...@yahoo-inc.com>.
Hi,
keep.tasks.files.pattern is what you need, as the name suggests its a pattern match on intermediate outputs generated.
Wrt to copying map data to hdfs, your mappers close() method should help you achieve this, but might slow up your tasks.
Amogh
On 11/23/09 8:08 AM, "Jeff Zhang" <zj...@gmail.com> wrote:
Hi Jason,
which option is for setting disable the removal of intermediate data ?
Thank you
Jeff Zhang
On Mon, Nov 23, 2009 at 10:27 AM, Jason Venner <ja...@gmail.com>wrote:
> You can manually write the map output to a new file, there are a number of
> examples of opening a sequence file and writing to it on the web or in the
> example code for various hadoop books.
>
> You can also disable the removal of intermediate data, which will result in
> potentially large amounts of data being left in the mapred.local.dir.
>
>
> On Sun, Nov 22, 2009 at 3:56 PM, Gordon Linoff <gl...@gmail.com> wrote:
>
> > I am starting to learn Hadoop, using the Yahoo virtual machine with
> version
> > 0.18.
> >
> > My question is rather simple. I would like to execute a map/reduce job.
> > In
> > addition to getting the results from the reduce, I would also like to
> save
> > the intermediate results from the map in another HDFS file. Is this
> > possible?
> >
> > --gordon
> >
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=jewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>
Re: Saving Intermediate Results from the Mapper
Posted by Jeff Zhang <zj...@gmail.com>.
Hi Jason,
which option is for setting disable the removal of intermediate data ?
Thank you
Jeff Zhang
On Mon, Nov 23, 2009 at 10:27 AM, Jason Venner <ja...@gmail.com>wrote:
> You can manually write the map output to a new file, there are a number of
> examples of opening a sequence file and writing to it on the web or in the
> example code for various hadoop books.
>
> You can also disable the removal of intermediate data, which will result in
> potentially large amounts of data being left in the mapred.local.dir.
>
>
> On Sun, Nov 22, 2009 at 3:56 PM, Gordon Linoff <gl...@gmail.com> wrote:
>
> > I am starting to learn Hadoop, using the Yahoo virtual machine with
> version
> > 0.18.
> >
> > My question is rather simple. I would like to execute a map/reduce job.
> > In
> > addition to getting the results from the reduce, I would also like to
> save
> > the intermediate results from the map in another HDFS file. Is this
> > possible?
> >
> > --gordon
> >
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=jewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>
Re: Saving Intermediate Results from the Mapper
Posted by Jason Venner <ja...@gmail.com>.
You can manually write the map output to a new file, there are a number of
examples of opening a sequence file and writing to it on the web or in the
example code for various hadoop books.
You can also disable the removal of intermediate data, which will result in
potentially large amounts of data being left in the mapred.local.dir.
On Sun, Nov 22, 2009 at 3:56 PM, Gordon Linoff <gl...@gmail.com> wrote:
> I am starting to learn Hadoop, using the Yahoo virtual machine with version
> 0.18.
>
> My question is rather simple. I would like to execute a map/reduce job.
> In
> addition to getting the results from the reduce, I would also like to save
> the intermediate results from the map in another HDFS file. Is this
> possible?
>
> --gordon
>
--
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals
Re: Error trying to build hadoop eclipse plugin
Posted by Stephen Watt <sw...@us.ibm.com>.
Can you go into detail regarding your scenario. What exactly is failing
and how is it failing ?
Kind regards
Steve Watt
From:
Raymond Jennings III <ra...@yahoo.com>
To:
common-user@hadoop.apache.org
Date:
11/23/2009 11:17 AM
Subject:
Re: Error trying to build hadoop eclipse plugin
Hi, it's hadoop-0.20.1 and eclipse Ganymede. I did try that prebuilt
plugin over the weekend but no luck with that on Windows. Thank you.
--- On Mon, 11/23/09, Stephen Watt <sw...@us.ibm.com> wrote:
> From: Stephen Watt <sw...@us.ibm.com>
> Subject: Re: Error trying to build hadoop eclipse plugin
> To: common-user@hadoop.apache.org
> Cc: common-user@hadoop.apache.org
> Date: Monday, November 23, 2009, 12:01 PM
> You'll need to be more specific about
> which version of Hadoop and Eclipse
> you're using. There are known issues building the plugin on
> Hadoop 0.20.1.
> I am in the process of providing a patch for this, in the
> interim, you
> can try using the plugin jar attached at
> http://issues.apache.org/jira/browse/HADOOP-6360
>
> Kind regards
> Steve Watt
>
>
>
> From:
> Raymond Jennings III <ra...@yahoo.com>
> To:
> common-user@hadoop.apache.org
> Date:
> 11/23/2009 10:46 AM
> Subject:
> Error trying to build hadoop eclipse plugin
>
>
>
> I am running ant on windows from the
> src/contrib/eclipse-plugin directory
> Using jdk1.6.0_18:
>
> Anyone have any experience with this? Thank you.
>
>
>
> [ivy:retrieve] 0 artifacts copied, 2 already
> retrieved (0kB/0ms)
> No ivy:settings found for the default reference
> 'ivy.instance'. A default
> insta
> nce will be used
> DEPRECATED: 'ivy.conf.file' is deprecated, use
> 'ivy.settings.file' instead
> :: loading settings :: file =
> E:\hadoop\hadoop-0.20.1\ivy\ivysettings.xml
>
> compile:
> [echo] contrib: eclipse-plugin
> [javac] Compiling 45 source files to
> E:\hadoop\hadoop-0.20.1\build\contrib\eclipse-plugin\classes
> [javac]
>
E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:35:
>
> cannot find symbol
> [javac] symbol : class
> JavaApplicationLaunchShortcut
> [javac] location: package
> org.eclipse.jdt.internal.debug.ui.launcher
> [javac] import
>
org.eclipse.jdt.internal.debug.ui.launcher.JavaApplicationLaunchShortcut;
> [javac]
>
>
> ^
> [javac]
>
E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:49:
>
> cannot find symbol
> [javac] symbol: class
> JavaApplicationLaunchShortcut
> [javac]
> JavaApplicationLaunchShortcut {
> [javac] ^
> [javac]
>
E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:66:
>
> cannot find symbol
> [javac] symbol : variable super
> [javac] location: class
> org.apache.hadoop.eclipse.launch.HadoopApplicationLa
> unchShortcut
> [javac]
> super.findLaunchConfiguration(type,
> configType);
> [javac]
> ^
> [javac]
>
E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:60:
>
> method does not override or implement a method from a
> supertype
> [javac] @Override
> [javac] ^
> [javac] Note: Some input files use or
> override a deprecated API.
> [javac] Note: Recompile with
> -Xlint:deprecation for details.
> [javac] Note: Some input files use unchecked
> or unsafe operations.
> [javac] Note: Recompile with -Xlint:unchecked
> for details.
> [javac] 4 errors
>
> BUILD FAILED
> E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\build.xml:63:
> Compile
> failed;
> see the compiler error output for details.
>
>
>
>
>
>
Re: Error trying to build hadoop eclipse plugin
Posted by Raymond Jennings III <ra...@yahoo.com>.
>From what I understand IBM created the first mapreduce plugin (which they still make available for download.) But once it was integrated into the Hadoop the build the developers here (I also work for IBM) were no longer involved. They told me that the primary person who was maintaining it now works for Google. The original IBM plugin installs okay on Europa but has limited functionality and the DFS browse option does not work (for me with hadoop 0.20.1) I just created a new Ganymede install and used Stephen's prebuilt plugin - it seems okay now at least initially. Not sure if I had some invalid plugin data from my attempted build within eclipse or what. But I am looking forward to playing with it now. I'll post back here if I find anything of value that might help get it working properly. Thanks all.
--- On Mon, 11/23/09, Edmund Kohlwey <ek...@gmail.com> wrote:
> From: Edmund Kohlwey <ek...@gmail.com>
> Subject: Re: Error trying to build hadoop eclipse plugin
> To: common-user@hadoop.apache.org
> Date: Monday, November 23, 2009, 10:02 PM
> The eclipse plug-in doesn't build
> using Hadoop 0.20.1 and Eclipse 3.5 (Galileo). I haven't
> tried to build it on 3.4, but the build errors you're
> getting are the same as what you get for 3.5.
>
> Even if you eliminate the compiler errors, the ant script
> doesn't properly package the various plug-in resources. I'm
> not sure if the JIRA listed below takes all this into
> account or not, as I haven't tried their build, but I
> suspect it doesn't given your issues. You can, however,
> easily fix and export the plug-in by importing the source
> into your eclipse workspace, opening up any offending files,
> pressing cmd+shift+o (which will import the correct packages
> -- the classes were just moved, but the api is the same),
> and then going through the normal PDE export process (which
> you should be able to find via Google).
>
> The eclipse build issues (and maintaining the eclipse
> plugin) are complicated by the fact that there's no unit
> tests for the plug-in, and Eclipse doesn't publish
> up-to-date artifacts in any public maven or ant repository,
> so its a pain to get Hudson, or whatever build server to
> check it. I don't know weather or not this plug-in will ever
> be well maintained unless a commercial vendor starts putting
> effort into it, simply because Eclipse's APIs shift so
> frequently as to make maintaining plug-ins for it quite
> prohibitive without some sort of commercial interest.
>
>
> On 11/23/09 12:16 PM, Raymond Jennings III wrote:
> > Hi, it's hadoop-0.20.1 and eclipse Ganymede. I
> did try that prebuilt plugin over the weekend but no luck
> with that on Windows. Thank you.
> >
> >
> > --- On Mon, 11/23/09, Stephen Watt<sw...@us.ibm.com>
> wrote:
> >
> >
> >> From: Stephen Watt<sw...@us.ibm.com>
> >> Subject: Re: Error trying to build hadoop eclipse
> plugin
> >> To: common-user@hadoop.apache.org
> >> Cc: common-user@hadoop.apache.org
> >> Date: Monday, November 23, 2009, 12:01 PM
> >> You'll need to be more specific about
> >> which version of Hadoop and Eclipse
> >> you're using. There are known issues building the
> plugin on
> >> Hadoop 0.20.1.
> >> I am in the process of providing
> a patch for this, in the
> >> interim, you
> >> can try using the plugin jar attached at
> >> http://issues.apache.org/jira/browse/HADOOP-6360
> >>
> >> Kind regards
> >> Steve Watt
> >>
> >>
> >>
> >> From:
> >> Raymond Jennings III<ra...@yahoo.com>
> >> To:
> >> common-user@hadoop.apache.org
> >> Date:
> >> 11/23/2009 10:46 AM
> >> Subject:
> >> Error trying to build hadoop eclipse plugin
> >>
> >>
> >>
> >> I am running ant on windows from the
> >> src/contrib/eclipse-plugin directory
> >> Using jdk1.6.0_18:
> >>
> >> Anyone have any experience with this? Thank
> you.
> >>
> >>
> >>
> >> [ivy:retrieve] 0 artifacts copied, 2
> already
> >> retrieved (0kB/0ms)
> >> No ivy:settings found for the default reference
> >> 'ivy.instance'. A default
> >> insta
> >> nce will be used
> >> DEPRECATED: 'ivy.conf.file' is deprecated, use
> >> 'ivy.settings.file' instead
> >> :: loading settings :: file =
> >> E:\hadoop\hadoop-0.20.1\ivy\ivysettings.xml
> >>
> >> compile:
> >> [echo] contrib:
> eclipse-plugin
> >> [javac] Compiling 45 source
> files to
> >>
> E:\hadoop\hadoop-0.20.1\build\contrib\eclipse-plugin\classes
> >> [javac]
> >>
> E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:35:
> >>
> >> cannot find symbol
> >> [javac] symbol : class
> >> JavaApplicationLaunchShortcut
> >> [javac] location: package
> >> org.eclipse.jdt.internal.debug.ui.launcher
> >> [javac] import
> >>
> org.eclipse.jdt.internal.debug.ui.launcher.JavaApplicationLaunchShortcut;
> >> [javac]
>
>
> ^
> >> [javac]
> >>
> E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:49:
> >>
> >> cannot find symbol
> >> [javac] symbol: class
> >> JavaApplicationLaunchShortcut
> >> [javac]
> JavaApplicationLaunchShortcut {
> >> [javac]
> ^
> >> [javac]
> >>
> E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:66:
> >>
> >> cannot find symbol
> >> [javac] symbol :
> variable super
> >> [javac] location: class
> >>
> org.apache.hadoop.eclipse.launch.HadoopApplicationLa
> >> unchShortcut
> >> [javac]
> super.findLaunchConfiguration(type,
> >> configType);
> >> [javac]
> ^
> >> [javac]
> >>
> E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:60:
> >>
> >> method does not override or implement a method
> from a
> >> supertype
> >>
> [javac] @Override
> >> [javac] ^
> >> [javac] Note: Some input files
> use or
> >> override a deprecated API.
> >> [javac] Note: Recompile with
> >> -Xlint:deprecation for details.
> >> [javac] Note: Some input files
> use unchecked
> >> or unsafe operations.
> >> [javac] Note: Recompile with
> -Xlint:unchecked
> >> for details.
> >> [javac] 4 errors
> >>
> >> BUILD FAILED
> >>
> E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\build.xml:63:
> >> Compile
> >> failed;
> >> see the compiler error output for
> details.
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
> >
>
>
Re: Error trying to build hadoop eclipse plugin
Posted by Edmund Kohlwey <ek...@gmail.com>.
The eclipse plug-in doesn't build using Hadoop 0.20.1 and Eclipse 3.5
(Galileo). I haven't tried to build it on 3.4, but the build errors
you're getting are the same as what you get for 3.5.
Even if you eliminate the compiler errors, the ant script doesn't
properly package the various plug-in resources. I'm not sure if the JIRA
listed below takes all this into account or not, as I haven't tried
their build, but I suspect it doesn't given your issues. You can,
however, easily fix and export the plug-in by importing the source into
your eclipse workspace, opening up any offending files, pressing
cmd+shift+o (which will import the correct packages -- the classes were
just moved, but the api is the same), and then going through the normal
PDE export process (which you should be able to find via Google).
The eclipse build issues (and maintaining the eclipse plugin) are
complicated by the fact that there's no unit tests for the plug-in, and
Eclipse doesn't publish up-to-date artifacts in any public maven or ant
repository, so its a pain to get Hudson, or whatever build server to
check it. I don't know weather or not this plug-in will ever be well
maintained unless a commercial vendor starts putting effort into it,
simply because Eclipse's APIs shift so frequently as to make maintaining
plug-ins for it quite prohibitive without some sort of commercial interest.
On 11/23/09 12:16 PM, Raymond Jennings III wrote:
> Hi, it's hadoop-0.20.1 and eclipse Ganymede. I did try that prebuilt plugin over the weekend but no luck with that on Windows. Thank you.
>
>
> --- On Mon, 11/23/09, Stephen Watt<sw...@us.ibm.com> wrote:
>
>
>> From: Stephen Watt<sw...@us.ibm.com>
>> Subject: Re: Error trying to build hadoop eclipse plugin
>> To: common-user@hadoop.apache.org
>> Cc: common-user@hadoop.apache.org
>> Date: Monday, November 23, 2009, 12:01 PM
>> You'll need to be more specific about
>> which version of Hadoop and Eclipse
>> you're using. There are known issues building the plugin on
>> Hadoop 0.20.1.
>> I am in the process of providing a patch for this, in the
>> interim, you
>> can try using the plugin jar attached at
>> http://issues.apache.org/jira/browse/HADOOP-6360
>>
>> Kind regards
>> Steve Watt
>>
>>
>>
>> From:
>> Raymond Jennings III<ra...@yahoo.com>
>> To:
>> common-user@hadoop.apache.org
>> Date:
>> 11/23/2009 10:46 AM
>> Subject:
>> Error trying to build hadoop eclipse plugin
>>
>>
>>
>> I am running ant on windows from the
>> src/contrib/eclipse-plugin directory
>> Using jdk1.6.0_18:
>>
>> Anyone have any experience with this? Thank you.
>>
>>
>>
>> [ivy:retrieve] 0 artifacts copied, 2 already
>> retrieved (0kB/0ms)
>> No ivy:settings found for the default reference
>> 'ivy.instance'. A default
>> insta
>> nce will be used
>> DEPRECATED: 'ivy.conf.file' is deprecated, use
>> 'ivy.settings.file' instead
>> :: loading settings :: file =
>> E:\hadoop\hadoop-0.20.1\ivy\ivysettings.xml
>>
>> compile:
>> [echo] contrib: eclipse-plugin
>> [javac] Compiling 45 source files to
>> E:\hadoop\hadoop-0.20.1\build\contrib\eclipse-plugin\classes
>> [javac]
>> E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:35:
>>
>> cannot find symbol
>> [javac] symbol : class
>> JavaApplicationLaunchShortcut
>> [javac] location: package
>> org.eclipse.jdt.internal.debug.ui.launcher
>> [javac] import
>> org.eclipse.jdt.internal.debug.ui.launcher.JavaApplicationLaunchShortcut;
>> [javac]
>>
>>
>> ^
>> [javac]
>> E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:49:
>>
>> cannot find symbol
>> [javac] symbol: class
>> JavaApplicationLaunchShortcut
>> [javac]
>> JavaApplicationLaunchShortcut {
>> [javac] ^
>> [javac]
>> E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:66:
>>
>> cannot find symbol
>> [javac] symbol : variable super
>> [javac] location: class
>> org.apache.hadoop.eclipse.launch.HadoopApplicationLa
>> unchShortcut
>> [javac]
>> super.findLaunchConfiguration(type,
>> configType);
>> [javac]
>> ^
>> [javac]
>> E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:60:
>>
>> method does not override or implement a method from a
>> supertype
>> [javac] @Override
>> [javac] ^
>> [javac] Note: Some input files use or
>> override a deprecated API.
>> [javac] Note: Recompile with
>> -Xlint:deprecation for details.
>> [javac] Note: Some input files use unchecked
>> or unsafe operations.
>> [javac] Note: Recompile with -Xlint:unchecked
>> for details.
>> [javac] 4 errors
>>
>> BUILD FAILED
>> E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\build.xml:63:
>> Compile
>> failed;
>> see the compiler error output for details.
>>
>>
>>
>>
>>
>>
>>
>
>
>
Re: Error trying to build hadoop eclipse plugin
Posted by Raymond Jennings III <ra...@yahoo.com>.
Hi, it's hadoop-0.20.1 and eclipse Ganymede. I did try that prebuilt plugin over the weekend but no luck with that on Windows. Thank you.
--- On Mon, 11/23/09, Stephen Watt <sw...@us.ibm.com> wrote:
> From: Stephen Watt <sw...@us.ibm.com>
> Subject: Re: Error trying to build hadoop eclipse plugin
> To: common-user@hadoop.apache.org
> Cc: common-user@hadoop.apache.org
> Date: Monday, November 23, 2009, 12:01 PM
> You'll need to be more specific about
> which version of Hadoop and Eclipse
> you're using. There are known issues building the plugin on
> Hadoop 0.20.1.
> I am in the process of providing a patch for this, in the
> interim, you
> can try using the plugin jar attached at
> http://issues.apache.org/jira/browse/HADOOP-6360
>
> Kind regards
> Steve Watt
>
>
>
> From:
> Raymond Jennings III <ra...@yahoo.com>
> To:
> common-user@hadoop.apache.org
> Date:
> 11/23/2009 10:46 AM
> Subject:
> Error trying to build hadoop eclipse plugin
>
>
>
> I am running ant on windows from the
> src/contrib/eclipse-plugin directory
> Using jdk1.6.0_18:
>
> Anyone have any experience with this? Thank you.
>
>
>
> [ivy:retrieve] 0 artifacts copied, 2 already
> retrieved (0kB/0ms)
> No ivy:settings found for the default reference
> 'ivy.instance'. A default
> insta
> nce will be used
> DEPRECATED: 'ivy.conf.file' is deprecated, use
> 'ivy.settings.file' instead
> :: loading settings :: file =
> E:\hadoop\hadoop-0.20.1\ivy\ivysettings.xml
>
> compile:
> [echo] contrib: eclipse-plugin
> [javac] Compiling 45 source files to
> E:\hadoop\hadoop-0.20.1\build\contrib\eclipse-plugin\classes
> [javac]
> E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:35:
>
> cannot find symbol
> [javac] symbol : class
> JavaApplicationLaunchShortcut
> [javac] location: package
> org.eclipse.jdt.internal.debug.ui.launcher
> [javac] import
> org.eclipse.jdt.internal.debug.ui.launcher.JavaApplicationLaunchShortcut;
> [javac]
>
>
> ^
> [javac]
> E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:49:
>
> cannot find symbol
> [javac] symbol: class
> JavaApplicationLaunchShortcut
> [javac]
> JavaApplicationLaunchShortcut {
> [javac] ^
> [javac]
> E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:66:
>
> cannot find symbol
> [javac] symbol : variable super
> [javac] location: class
> org.apache.hadoop.eclipse.launch.HadoopApplicationLa
> unchShortcut
> [javac]
> super.findLaunchConfiguration(type,
> configType);
> [javac]
> ^
> [javac]
> E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:60:
>
> method does not override or implement a method from a
> supertype
> [javac] @Override
> [javac] ^
> [javac] Note: Some input files use or
> override a deprecated API.
> [javac] Note: Recompile with
> -Xlint:deprecation for details.
> [javac] Note: Some input files use unchecked
> or unsafe operations.
> [javac] Note: Recompile with -Xlint:unchecked
> for details.
> [javac] 4 errors
>
> BUILD FAILED
> E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\build.xml:63:
> Compile
> failed;
> see the compiler error output for details.
>
>
>
>
>
>
Re: Error trying to build hadoop eclipse plugin
Posted by Stephen Watt <sw...@us.ibm.com>.
You'll need to be more specific about which version of Hadoop and Eclipse
you're using. There are known issues building the plugin on Hadoop 0.20.1.
I am in the process of providing a patch for this, in the interim, you
can try using the plugin jar attached at
http://issues.apache.org/jira/browse/HADOOP-6360
Kind regards
Steve Watt
From:
Raymond Jennings III <ra...@yahoo.com>
To:
common-user@hadoop.apache.org
Date:
11/23/2009 10:46 AM
Subject:
Error trying to build hadoop eclipse plugin
I am running ant on windows from the src/contrib/eclipse-plugin directory
Using jdk1.6.0_18:
Anyone have any experience with this? Thank you.
[ivy:retrieve] 0 artifacts copied, 2 already retrieved (0kB/0ms)
No ivy:settings found for the default reference 'ivy.instance'. A default
insta
nce will be used
DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
:: loading settings :: file = E:\hadoop\hadoop-0.20.1\ivy\ivysettings.xml
compile:
[echo] contrib: eclipse-plugin
[javac] Compiling 45 source files to
E:\hadoop\hadoop-0.20.1\build\contrib\eclipse-plugin\classes
[javac]
E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:35:
cannot find symbol
[javac] symbol : class JavaApplicationLaunchShortcut
[javac] location: package org.eclipse.jdt.internal.debug.ui.launcher
[javac] import
org.eclipse.jdt.internal.debug.ui.launcher.JavaApplicationLaunchShortcut;
[javac] ^
[javac]
E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:49:
cannot find symbol
[javac] symbol: class JavaApplicationLaunchShortcut
[javac] JavaApplicationLaunchShortcut {
[javac] ^
[javac]
E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:66:
cannot find symbol
[javac] symbol : variable super
[javac] location: class
org.apache.hadoop.eclipse.launch.HadoopApplicationLa
unchShortcut
[javac] super.findLaunchConfiguration(type, configType);
[javac] ^
[javac]
E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:60:
method does not override or implement a method from a supertype
[javac] @Override
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 4 errors
BUILD FAILED
E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\build.xml:63: Compile
failed;
see the compiler error output for details.
Error trying to build hadoop eclipse plugin
Posted by Raymond Jennings III <ra...@yahoo.com>.
I am running ant on windows from the src/contrib/eclipse-plugin directory
Using jdk1.6.0_18:
Anyone have any experience with this? Thank you.
[ivy:retrieve] 0 artifacts copied, 2 already retrieved (0kB/0ms)
No ivy:settings found for the default reference 'ivy.instance'. A default insta
nce will be used
DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
:: loading settings :: file = E:\hadoop\hadoop-0.20.1\ivy\ivysettings.xml
compile:
[echo] contrib: eclipse-plugin
[javac] Compiling 45 source files to E:\hadoop\hadoop-0.20.1\build\contrib\eclipse-plugin\classes
[javac] E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:35: cannot find symbol
[javac] symbol : class JavaApplicationLaunchShortcut
[javac] location: package org.eclipse.jdt.internal.debug.ui.launcher
[javac] import org.eclipse.jdt.internal.debug.ui.launcher.JavaApplicationLaunchShortcut;
[javac] ^
[javac] E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:49: cannot find symbol
[javac] symbol: class JavaApplicationLaunchShortcut
[javac] JavaApplicationLaunchShortcut {
[javac] ^
[javac] E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:66: cannot find symbol
[javac] symbol : variable super
[javac] location: class org.apache.hadoop.eclipse.launch.HadoopApplicationLa
unchShortcut
[javac] super.findLaunchConfiguration(type, configType);
[javac] ^
[javac] E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\launch\HadoopApplicationLaunchShortcut.java:60: method does not override or implement a method from a supertype
[javac] @Override
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 4 errors
BUILD FAILED
E:\hadoop\hadoop-0.20.1\src\contrib\eclipse-plugin\build.xml:63: Compile failed;
see the compiler error output for details.