You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Mohammad Islam <mi...@yahoo.com> on 2012/11/21 21:00:31 UTC
[VOTE] Release Oozie 3.3.0 (candidate 0)
Dear Oozie community,
The release candidate 0 for for Oozie 3.3.0 is available.
Oozie 3.3.0 has the following new features:
1. Bulk Monitoring API - Consolidated view of jobs
2. Eliminate redundancies in xml through global section.
3. Add formal parameters to XML for early valiadation
4. Visualize color coded job DAG at runtime.
5. Load Hbase/HCat credentials in Job conf
6. Support direct map-reduce job submission through Oozie CLI without workflow XML
7. Add support for multiple/configurable sharelibs for each action type
In addition, it includes several improvements for performance and stability
and several bug fixes. Detail release log could be found at:
http://people.apache.org/~kamrul/oozie-3.3.0-rc0/release-log.txt
Keys used to sign the release are available at
http://www.apache.org/dist/oozie/KEYS
Please download, test, and try it out:
http://people.apache.org/~kamrul/oozie-3.3.0-rc0/
The release, md5 signature, gpg signature, and rat report can all be found
at the above URL.
Vote closes in 3 days .
Regards,
Mohammad
Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Posted by Roman Shaposhnik <rv...@apache.org>.
On Mon, Nov 26, 2012 at 8:59 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> OK,
>
> With the help of Roman I've put together a patch and tested in in Hadoop
> 2.0.2-alpha deployment.
>
> https://issues.apache.org/jira/browse/OOZIE-1089
>
> The patch introduces a workaround which has to be explicitly enabled.
>
> I think we should apply this patch to branch-3.3, then we'd be able to do a
> release that works with Hadop 2.0.2-alpha.
A strong +1 -- this seems to be a non-invasive change that is easy to
remove once we no longer need the workaround, it doesn't change the
default behavior and it was tested on Hadoop 2.0.2, Hadoop 1.1.0 and
Hadoop 1.0.X
Now if I only to make one more request this would be to RELNOTE this
workaround in the Oozie docs for the upcoming release.
Thanks,
Roman.
Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Posted by Mohammad Islam <mi...@yahoo.com>.
Thanks . Working on that.
________________________________
From: Alejandro Abdelnur <tu...@cloudera.com>
To: "dev@oozie.apache.org" <de...@oozie.apache.org>
Sent: Tuesday, November 27, 2012 12:21 PM
Subject: Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
I've just committed OOZIE-1089 to branch-3.3.
Mohammad, will you be cutting a new RC?
Thx
On Mon, Nov 26, 2012 at 8:59 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> OK,
>
> With the help of Roman I've put together a patch and tested in in Hadoop
> 2.0.2-alpha deployment.
>
> https://issues.apache.org/jira/browse/OOZIE-1089
>
> The patch introduces a workaround which has to be explicitly enabled.
>
> I think we should apply this patch to branch-3.3, then we'd be able to do
> a release that works with Hadop 2.0.2-alpha.
>
> Thx.
>
>
> On Mon, Nov 26, 2012 at 12:06 PM, Alejandro Abdelnur <tu...@gmail.com>wrote:
>
>> Sorry for the delay jumping into this issue, I spend the whole morning
>> tracing the issue, didn't want to comment until I knew for sure.
>>
>> The problem is not because of using BigTop, the problem is with Hadoop
>> 2.0.2-alpha because of MAPREDUCE-4549.
>>
>> I've filed https://issues.apache.org/jira/browse/MAPREDUCE-4820
>>
>> I don't think from Oozie we can do much about it until this is
>> addressed in Hadoop.
>>
>> Before continuing with the vote, lets see what our options are.
>>
>> Thx
>>
>> On Mon, Nov 26, 2012 at 8:26 AM, Roman Shaposhnik <rv...@apache.org> wrote:
>> > On Mon, Nov 26, 2012 at 12:34 AM, Mohammad Islam <mi...@yahoo.com>
>> wrote:
>> >> Roman,
>> >> We are not using Bigtop.
>> >
>> > Well, Bigtop is the only place where integration testing between Oozie
>> > and various pieces of Hadoop ecosystem happens at Apache -- so perhaps
>> > you guys should consider it -- otherwise it seems like you'd be relying
>> > exclusively on commercial vendors/companies to do that kind of testing.
>> >
>> >> Can you please try after removing the
>> "share/lib/sqoop/hsqldb-1.8.0.10.jar"?
>> >> Just to make sure that is the root cause.
>> >
>> > If that gets removed -- the Oozie sqoop launcher job fails way earlier.
>> >
>> >> One more thing, we need to make sure whether this version (Oozie 3.3)
>> will
>> >> support Hadoop 2.x.
>> >
>> > Personally I think that any release that doesn't support Hadoop 2.X
>> would be
>> > at a pretty significant disadvantage at this point. The choice is
>> > yours, of course,
>> > but at the very least you'd have to RELNOTE this limitation and/or
>> disable
>> > the 2.X profile.
>> >
>> > Thanks,
>> > Roman.
>>
>
>
>
> --
> Alejandro
>
--
Alejandro
Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
I've just committed OOZIE-1089 to branch-3.3.
Mohammad, will you be cutting a new RC?
Thx
On Mon, Nov 26, 2012 at 8:59 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> OK,
>
> With the help of Roman I've put together a patch and tested in in Hadoop
> 2.0.2-alpha deployment.
>
> https://issues.apache.org/jira/browse/OOZIE-1089
>
> The patch introduces a workaround which has to be explicitly enabled.
>
> I think we should apply this patch to branch-3.3, then we'd be able to do
> a release that works with Hadop 2.0.2-alpha.
>
> Thx.
>
>
> On Mon, Nov 26, 2012 at 12:06 PM, Alejandro Abdelnur <tu...@gmail.com>wrote:
>
>> Sorry for the delay jumping into this issue, I spend the whole morning
>> tracing the issue, didn't want to comment until I knew for sure.
>>
>> The problem is not because of using BigTop, the problem is with Hadoop
>> 2.0.2-alpha because of MAPREDUCE-4549.
>>
>> I've filed https://issues.apache.org/jira/browse/MAPREDUCE-4820
>>
>> I don't think from Oozie we can do much about it until this is
>> addressed in Hadoop.
>>
>> Before continuing with the vote, lets see what our options are.
>>
>> Thx
>>
>> On Mon, Nov 26, 2012 at 8:26 AM, Roman Shaposhnik <rv...@apache.org> wrote:
>> > On Mon, Nov 26, 2012 at 12:34 AM, Mohammad Islam <mi...@yahoo.com>
>> wrote:
>> >> Roman,
>> >> We are not using Bigtop.
>> >
>> > Well, Bigtop is the only place where integration testing between Oozie
>> > and various pieces of Hadoop ecosystem happens at Apache -- so perhaps
>> > you guys should consider it -- otherwise it seems like you'd be relying
>> > exclusively on commercial vendors/companies to do that kind of testing.
>> >
>> >> Can you please try after removing the
>> "share/lib/sqoop/hsqldb-1.8.0.10.jar"?
>> >> Just to make sure that is the root cause.
>> >
>> > If that gets removed -- the Oozie sqoop launcher job fails way earlier.
>> >
>> >> One more thing, we need to make sure whether this version (Oozie 3.3)
>> will
>> >> support Hadoop 2.x.
>> >
>> > Personally I think that any release that doesn't support Hadoop 2.X
>> would be
>> > at a pretty significant disadvantage at this point. The choice is
>> > yours, of course,
>> > but at the very least you'd have to RELNOTE this limitation and/or
>> disable
>> > the 2.X profile.
>> >
>> > Thanks,
>> > Roman.
>>
>
>
>
> --
> Alejandro
>
--
Alejandro
Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
OK,
With the help of Roman I've put together a patch and tested in in Hadoop
2.0.2-alpha deployment.
https://issues.apache.org/jira/browse/OOZIE-1089
The patch introduces a workaround which has to be explicitly enabled.
I think we should apply this patch to branch-3.3, then we'd be able to do a
release that works with Hadop 2.0.2-alpha.
Thx.
On Mon, Nov 26, 2012 at 12:06 PM, Alejandro Abdelnur <tu...@gmail.com>wrote:
> Sorry for the delay jumping into this issue, I spend the whole morning
> tracing the issue, didn't want to comment until I knew for sure.
>
> The problem is not because of using BigTop, the problem is with Hadoop
> 2.0.2-alpha because of MAPREDUCE-4549.
>
> I've filed https://issues.apache.org/jira/browse/MAPREDUCE-4820
>
> I don't think from Oozie we can do much about it until this is
> addressed in Hadoop.
>
> Before continuing with the vote, lets see what our options are.
>
> Thx
>
> On Mon, Nov 26, 2012 at 8:26 AM, Roman Shaposhnik <rv...@apache.org> wrote:
> > On Mon, Nov 26, 2012 at 12:34 AM, Mohammad Islam <mi...@yahoo.com>
> wrote:
> >> Roman,
> >> We are not using Bigtop.
> >
> > Well, Bigtop is the only place where integration testing between Oozie
> > and various pieces of Hadoop ecosystem happens at Apache -- so perhaps
> > you guys should consider it -- otherwise it seems like you'd be relying
> > exclusively on commercial vendors/companies to do that kind of testing.
> >
> >> Can you please try after removing the
> "share/lib/sqoop/hsqldb-1.8.0.10.jar"?
> >> Just to make sure that is the root cause.
> >
> > If that gets removed -- the Oozie sqoop launcher job fails way earlier.
> >
> >> One more thing, we need to make sure whether this version (Oozie 3.3)
> will
> >> support Hadoop 2.x.
> >
> > Personally I think that any release that doesn't support Hadoop 2.X
> would be
> > at a pretty significant disadvantage at this point. The choice is
> > yours, of course,
> > but at the very least you'd have to RELNOTE this limitation and/or
> disable
> > the 2.X profile.
> >
> > Thanks,
> > Roman.
>
--
Alejandro
Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Posted by Alejandro Abdelnur <tu...@gmail.com>.
Sorry for the delay jumping into this issue, I spend the whole morning
tracing the issue, didn't want to comment until I knew for sure.
The problem is not because of using BigTop, the problem is with Hadoop
2.0.2-alpha because of MAPREDUCE-4549.
I've filed https://issues.apache.org/jira/browse/MAPREDUCE-4820
I don't think from Oozie we can do much about it until this is
addressed in Hadoop.
Before continuing with the vote, lets see what our options are.
Thx
On Mon, Nov 26, 2012 at 8:26 AM, Roman Shaposhnik <rv...@apache.org> wrote:
> On Mon, Nov 26, 2012 at 12:34 AM, Mohammad Islam <mi...@yahoo.com> wrote:
>> Roman,
>> We are not using Bigtop.
>
> Well, Bigtop is the only place where integration testing between Oozie
> and various pieces of Hadoop ecosystem happens at Apache -- so perhaps
> you guys should consider it -- otherwise it seems like you'd be relying
> exclusively on commercial vendors/companies to do that kind of testing.
>
>> Can you please try after removing the "share/lib/sqoop/hsqldb-1.8.0.10.jar"?
>> Just to make sure that is the root cause.
>
> If that gets removed -- the Oozie sqoop launcher job fails way earlier.
>
>> One more thing, we need to make sure whether this version (Oozie 3.3) will
>> support Hadoop 2.x.
>
> Personally I think that any release that doesn't support Hadoop 2.X would be
> at a pretty significant disadvantage at this point. The choice is
> yours, of course,
> but at the very least you'd have to RELNOTE this limitation and/or disable
> the 2.X profile.
>
> Thanks,
> Roman.
Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Posted by Roman Shaposhnik <rv...@apache.org>.
On Mon, Nov 26, 2012 at 12:34 AM, Mohammad Islam <mi...@yahoo.com> wrote:
> Roman,
> We are not using Bigtop.
Well, Bigtop is the only place where integration testing between Oozie
and various pieces of Hadoop ecosystem happens at Apache -- so perhaps
you guys should consider it -- otherwise it seems like you'd be relying
exclusively on commercial vendors/companies to do that kind of testing.
> Can you please try after removing the "share/lib/sqoop/hsqldb-1.8.0.10.jar"?
> Just to make sure that is the root cause.
If that gets removed -- the Oozie sqoop launcher job fails way earlier.
> One more thing, we need to make sure whether this version (Oozie 3.3) will
> support Hadoop 2.x.
Personally I think that any release that doesn't support Hadoop 2.X would be
at a pretty significant disadvantage at this point. The choice is
yours, of course,
but at the very least you'd have to RELNOTE this limitation and/or disable
the 2.X profile.
Thanks,
Roman.
Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Hi Jinwei Zhu,
That will happen soon after a proposed RC passes the vote.
A vote on an RC typically runs for 3 days. And I'd qualify 'soon after' as
more day until bits are pushed to the dist servers.
Thanks.
On Mon, Nov 26, 2012 at 9:19 PM, jinwei zhu <zh...@gmail.com> wrote:
> Hi, Mohammad,
> You are saying the schedule of oozie-3.3.0-rc0 , right? But
> actually, I want to know the release date of the* finall release* of
> oozie3.3.0 . Could you tell me probably date of the * finall release ?
>
>
> *Best regards!*
>
> *
>
--
Alejandro
Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Posted by Roman Shaposhnik <rv...@apache.org>.
On Mon, Nov 26, 2012 at 9:19 PM, jinwei zhu <zh...@gmail.com> wrote:
> Hi, Mohammad,
> You are saying the schedule of oozie-3.3.0-rc0 , right? But
> actually, I want to know the release date of the* finall release* of
> oozie3.3.0 . Could you tell me probably date of the * finall release ?
I don't think anybody can predict that -- we may be lucky and get it
out quickly or we may be less lucky find a lot of bugs and spend time
fixing them.
The best way to help get it out sooner is to pitch in with testing the
RC in the env. that you're using!
Thanks,
Roman.
Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Posted by jinwei zhu <zh...@gmail.com>.
Hi, Mohammad,
You are saying the schedule of oozie-3.3.0-rc0 , right? But
actually, I want to know the release date of the* finall release* of
oozie3.3.0 . Could you tell me probably date of the * finall release ?
*Best regards!*
*
Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Posted by Mohammad Islam <mi...@yahoo.com>.
Hi Jinwei,
As you can see, we need to decide if we will support hadoop 2.x as well with Oozie 3.3.
I expect this decision to be made in a day or two. After that, we need to follow the process of voting.
Regards,
Mohammad
________________________________
From: jinwei zhu <zh...@gmail.com>
To: user@oozie.apache.org; Mohammad Islam <mi...@yahoo.com>
Sent: Monday, November 26, 2012 5:01 PM
Subject: Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Hi, Mohammad,
Will you release the final oozie3.3.0 recently? If yes, When will
you release it?
Best Regards！
2012/11/26 Mohammad Islam <mi...@yahoo.com>
> Adding to Rohini.
> I also created a Jira for the same few months back:
> https://issues.apache.org/jira/browse/MAPREDUCE-4568
>
>
> Roman,
> We are not using Bigtop. Can you please try after removing the
> "share/lib/sqoop/hsqldb-1.8.0.10.jar"?
> Just to make sure that is the root cause.
>
> One more thing, we need to make sure whether this version (Oozie 3.3) will
> support Hadoop 2.x. I'm sure without fixing MAPREDUCE-4568, it will be hard
> to support that newer hadoop version. As the JIRA discussions show that we
> are not close to agreement yet on MR fix.
> Need more input from others about this issue.
>
> Regards,
> Mohammad
>
>
>
>
> ________________________________
> From: Rohini Palaniswamy <ro...@gmail.com>
> To: dev@oozie.apache.org
> Cc: Mohammad Islam <mi...@yahoo.com>; "user@oozie.apache.org" <
> user@oozie.apache.org>; dev@bigtop.apache.org
> Sent: Sunday, November 25, 2012 8:21 PM
> Subject: Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
>
> Context on that issue is in
> https://issues.apache.org/jira/browse/MAPREDUCE-4549. Failing with that
> error in hadoop has been reverted in 0.23.x line but not in 2.0.x line.
>
> Regards,
> Rohini
>
> On Wed, Nov 21, 2012 at 6:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:
>
> > -1 based on Bigtop testing against Hadoop 2.0.2 and Hadoop 1.1.0 Sqoop
> > and Hive wf fail to execute.
> >
> > Here's how to reproduce:
> >
> > 1. On your linux box hook up 2 Bigtop repos to your system:
> >
> >
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/
> > http://bigtop01.cloudera.org:8080/job/Bigtop-git/
> > e.g. on Ubuntu Lucid you'd run:
> > # curl
> >
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=lucid/lastSuccessfulBuild/artifact/output/bigtop.list
> > > /etc/apt/sources.list.d/bigtop1.list
> > # curl
> >
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=lucid/lastSuccessfulBuild/artifact/repo/bigtop.list
> > > /etc/apt/sources.list.d/bigtop2.list
> > # apt-get update
> > on CentOS 5 you'd run:
> > # curl
> >
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=centos5/lastSuccessfulBuild/artifact/output/bigtop.repo
> > > /etc/yum.repos.d/bigtop1.repo
> > # curl
> >
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=centos5/lastSuccessfulBuild/artifact/repo/bigtop.repo
> > > /etc/yum.repos.d/bigtop2.repo
> >
> > 2. Install Hadoop, Sqoop and Oozie in pseudo distributed mode:
> > ubuntu# apt-get install -y hadoop-conf-pseudo sqoop oozie
> > RedHat# yum install -y sqoop hadoop-conf-pseudo oozie
> >
> > 3. Init and start the services
> > # service hadoop-hdfs-namenode init
> > # service oozie init
> > # sudo -u hdfs hadoop fs -chmod -R 777 /
> > # for i in hadoop-hdfs-namenode hadoop-hdfs-datanode
> > hadoop-yarn-resourcemanager hadoop-yarn-nodemanager
> > hadoop-mapreduce-historyserver ; do service $i start ; done
> > # service oozie restart
> >
> > 4. Make sure that Oozie is up and running
> > # oozie admin -version -oozie http://localhost:11000/oozie
> >
> > 5. Install examples and try running a workflow
> > # cd /tmp
> > # tar xzvf /usr/share/doc/oozie*/oozie-examples.tar.gz
> > # hadoop fs -mkdir -p /user/oozie/share/lib/sqoop
> > # hadoop fs -mkdir -p /user/root
> > # hadoop fs -put examples /user/root/examples
> > # hadoop fs -put /usr/lib/sqoop/*.jar /usr/lib/sqoop/lib/*.jar
> > /user/oozie/share/lib/sqoop
> > # oozie job -DnameNode=hdfs://localhost:8020
> > -DjobTracker=localhost:8032 -config examples/apps/sqoop/job.properties
> > -run -oozie http://localhost:11000/oozie
> >
> > At this point the workflow would fail with the following:
> >
> > eption: cache file (mapreduce.job.cache.files) scheme: "hdfs", host:
> > "localhost", port: 8020, file:
> > "/user/oozie/share/lib/sqoop/hsqldb-1.8.0.10.jar", conflicts with
> > cache file (mapreduce.job.cache.files)
> >
> >
> hdfs://localhost:8020/tmp/hadoop-yarn/staging/root/.staging/job_1353549235835_0004/libjars/hsqldb-1.8.0.10.jar
> > at
> >
> org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:338)
> > at
> >
> org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:273)
> > at
> >
> org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:419)
> > at
> > org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:288)
> > at
> >
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:391)
> > at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> > at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
> > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
> > at
> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1236)
> > at
> > org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:141)
> > at
> >
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:202)
> > at
> > org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:465)
> > at
> > org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
> > at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
> > at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> > at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> > at
> > org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:205)
> > at
> org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:174)
> > at
> > org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
> > at
> org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:47)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:473)
> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> > at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:400)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335)
> > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
> > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
> >
> > Intercepting System.exit(1)
> >
> > <<< Invocation of Main class completed <<<
> >
> > Thanks,
> > Roman.
> >
> > On Wed, Nov 21, 2012 at 12:00 PM, Mohammad Islam <mi...@yahoo.com>
> > wrote:
> > > Dear Oozie community,
> > >
> > > The release candidate 0 for for Oozie 3.3.0 is available.
> > >
> > >
> > > Oozie 3.3.0 has the following new features:
> > > 1. Bulk Monitoring API - Consolidated view of jobs
> > > 2. Eliminate redundancies in xml through global section.
> > > 3. Add formal parameters to XML for early valiadation
> > >
> > > 4. Visualize color coded job DAG at runtime.
> > >
> > > 5. Load Hbase/HCat credentials in Job conf
> > > 6. Support direct map-reduce job submission through Oozie CLI
> without
> > workflow XML
> > > 7. Add support for multiple/configurable sharelibs for each action type
> > >
> > >
> > >
> > > In addition, it includes several improvements for performance and
> > stability
> > > and several bug fixes. Detail release log could be found at:
> > > http://people.apache.org/~kamrul/oozie-3.3.0-rc0/release-log.txt
> > >
> > >
> > > Keys used to sign the release are available at
> > > http://www.apache.org/dist/oozie/KEYS
> > >
> > > Please download, test, and try it out:
> > > http://people.apache.org/~kamrul/oozie-3.3.0-rc0/
> > >
> > > The release, md5 signature, gpg signature, and rat report can all be
> > found
> > > at the above URL.
> > >
> > > Vote closes in 3 days .
> > >
> > > Regards,
> > >
> > > Mohammad
> >
>
Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Posted by jinwei zhu <zh...@gmail.com>.
Hi, Mohammad,
Will you release the final oozie3.3.0 recently? If yes, When will
you release it?
Best Regards！
2012/11/26 Mohammad Islam <mi...@yahoo.com>
> Adding to Rohini.
> I also created a Jira for the same few months back:
> https://issues.apache.org/jira/browse/MAPREDUCE-4568
>
>
> Roman,
> We are not using Bigtop. Can you please try after removing the
> "share/lib/sqoop/hsqldb-1.8.0.10.jar"?
> Just to make sure that is the root cause.
>
> One more thing, we need to make sure whether this version (Oozie 3.3) will
> support Hadoop 2.x. I'm sure without fixing MAPREDUCE-4568, it will be hard
> to support that newer hadoop version. As the JIRA discussions show that we
> are not close to agreement yet on MR fix.
> Need more input from others about this issue.
>
> Regards,
> Mohammad
>
>
>
>
> ________________________________
> From: Rohini Palaniswamy <ro...@gmail.com>
> To: dev@oozie.apache.org
> Cc: Mohammad Islam <mi...@yahoo.com>; "user@oozie.apache.org" <
> user@oozie.apache.org>; dev@bigtop.apache.org
> Sent: Sunday, November 25, 2012 8:21 PM
> Subject: Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
>
> Context on that issue is in
> https://issues.apache.org/jira/browse/MAPREDUCE-4549. Failing with that
> error in hadoop has been reverted in 0.23.x line but not in 2.0.x line.
>
> Regards,
> Rohini
>
> On Wed, Nov 21, 2012 at 6:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:
>
> > -1 based on Bigtop testing against Hadoop 2.0.2 and Hadoop 1.1.0 Sqoop
> > and Hive wf fail to execute.
> >
> > Here's how to reproduce:
> >
> > 1. On your linux box hook up 2 Bigtop repos to your system:
> >
> >
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/
> > http://bigtop01.cloudera.org:8080/job/Bigtop-git/
> > e.g. on Ubuntu Lucid you'd run:
> > # curl
> >
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=lucid/lastSuccessfulBuild/artifact/output/bigtop.list
> > > /etc/apt/sources.list.d/bigtop1.list
> > # curl
> >
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=lucid/lastSuccessfulBuild/artifact/repo/bigtop.list
> > > /etc/apt/sources.list.d/bigtop2.list
> > # apt-get update
> > on CentOS 5 you'd run:
> > # curl
> >
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=centos5/lastSuccessfulBuild/artifact/output/bigtop.repo
> > > /etc/yum.repos.d/bigtop1.repo
> > # curl
> >
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=centos5/lastSuccessfulBuild/artifact/repo/bigtop.repo
> > > /etc/yum.repos.d/bigtop2.repo
> >
> > 2. Install Hadoop, Sqoop and Oozie in pseudo distributed mode:
> > ubuntu# apt-get install -y hadoop-conf-pseudo sqoop oozie
> > RedHat# yum install -y sqoop hadoop-conf-pseudo oozie
> >
> > 3. Init and start the services
> > # service hadoop-hdfs-namenode init
> > # service oozie init
> > # sudo -u hdfs hadoop fs -chmod -R 777 /
> > # for i in hadoop-hdfs-namenode hadoop-hdfs-datanode
> > hadoop-yarn-resourcemanager hadoop-yarn-nodemanager
> > hadoop-mapreduce-historyserver ; do service $i start ; done
> > # service oozie restart
> >
> > 4. Make sure that Oozie is up and running
> > # oozie admin -version -oozie http://localhost:11000/oozie
> >
> > 5. Install examples and try running a workflow
> > # cd /tmp
> > # tar xzvf /usr/share/doc/oozie*/oozie-examples.tar.gz
> > # hadoop fs -mkdir -p /user/oozie/share/lib/sqoop
> > # hadoop fs -mkdir -p /user/root
> > # hadoop fs -put examples /user/root/examples
> > # hadoop fs -put /usr/lib/sqoop/*.jar /usr/lib/sqoop/lib/*.jar
> > /user/oozie/share/lib/sqoop
> > # oozie job -DnameNode=hdfs://localhost:8020
> > -DjobTracker=localhost:8032 -config examples/apps/sqoop/job.properties
> > -run -oozie http://localhost:11000/oozie
> >
> > At this point the workflow would fail with the following:
> >
> > eption: cache file (mapreduce.job.cache.files) scheme: "hdfs", host:
> > "localhost", port: 8020, file:
> > "/user/oozie/share/lib/sqoop/hsqldb-1.8.0.10.jar", conflicts with
> > cache file (mapreduce.job.cache.files)
> >
> >
> hdfs://localhost:8020/tmp/hadoop-yarn/staging/root/.staging/job_1353549235835_0004/libjars/hsqldb-1.8.0.10.jar
> > at
> >
> org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:338)
> > at
> >
> org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:273)
> > at
> >
> org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:419)
> > at
> > org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:288)
> > at
> >
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:391)
> > at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> > at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
> > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
> > at
> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1236)
> > at
> > org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:141)
> > at
> >
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:202)
> > at
> > org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:465)
> > at
> > org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
> > at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
> > at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> > at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> > at
> > org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:205)
> > at
> org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:174)
> > at
> > org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
> > at
> org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:47)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:473)
> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> > at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:400)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335)
> > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
> > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
> >
> > Intercepting System.exit(1)
> >
> > <<< Invocation of Main class completed <<<
> >
> > Thanks,
> > Roman.
> >
> > On Wed, Nov 21, 2012 at 12:00 PM, Mohammad Islam <mi...@yahoo.com>
> > wrote:
> > > Dear Oozie community,
> > >
> > > The release candidate 0 for for Oozie 3.3.0 is available.
> > >
> > >
> > > Oozie 3.3.0 has the following new features:
> > > 1. Bulk Monitoring API - Consolidated view of jobs
> > > 2. Eliminate redundancies in xml through global section.
> > > 3. Add formal parameters to XML for early valiadation
> > >
> > > 4. Visualize color coded job DAG at runtime.
> > >
> > > 5. Load Hbase/HCat credentials in Job conf
> > > 6. Support direct map-reduce job submission through Oozie CLI
> without
> > workflow XML
> > > 7. Add support for multiple/configurable sharelibs for each action type
> > >
> > >
> > >
> > > In addition, it includes several improvements for performance and
> > stability
> > > and several bug fixes. Detail release log could be found at:
> > > http://people.apache.org/~kamrul/oozie-3.3.0-rc0/release-log.txt
> > >
> > >
> > > Keys used to sign the release are available at
> > > http://www.apache.org/dist/oozie/KEYS
> > >
> > > Please download, test, and try it out:
> > > http://people.apache.org/~kamrul/oozie-3.3.0-rc0/
> > >
> > > The release, md5 signature, gpg signature, and rat report can all be
> > found
> > > at the above URL.
> > >
> > > Vote closes in 3 days .
> > >
> > > Regards,
> > >
> > > Mohammad
> >
>
Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Posted by Mohammad Islam <mi...@yahoo.com>.
Adding to Rohini.
I also created a Jira for the same few months back:
https://issues.apache.org/jira/browse/MAPREDUCE-4568
Roman,
We are not using Bigtop. Can you please try after removing the "share/lib/sqoop/hsqldb-1.8.0.10.jar"?
Just to make sure that is the root cause.
One more thing, we need to make sure whether this version (Oozie 3.3) will support Hadoop 2.x. I'm sure without fixing MAPREDUCE-4568, it will be hard to support that newer hadoop version. As the JIRA discussions show that we are not close to agreement yet on MR fix.
Need more input from others about this issue.
Regards,
Mohammad
________________________________
From: Rohini Palaniswamy <ro...@gmail.com>
To: dev@oozie.apache.org
Cc: Mohammad Islam <mi...@yahoo.com>; "user@oozie.apache.org" <us...@oozie.apache.org>; dev@bigtop.apache.org
Sent: Sunday, November 25, 2012 8:21 PM
Subject: Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Context on that issue is in
https://issues.apache.org/jira/browse/MAPREDUCE-4549. Failing with that
error in hadoop has been reverted in 0.23.x line but not in 2.0.x line.
Regards,
Rohini
On Wed, Nov 21, 2012 at 6:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> -1 based on Bigtop testing against Hadoop 2.0.2 and Hadoop 1.1.0 Sqoop
> and Hive wf fail to execute.
>
> Here's how to reproduce:
>
> 1. On your linux box hook up 2 Bigtop repos to your system:
>
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/
> e.g. on Ubuntu Lucid you'd run:
> # curl
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=lucid/lastSuccessfulBuild/artifact/output/bigtop.list
> > /etc/apt/sources.list.d/bigtop1.list
> # curl
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=lucid/lastSuccessfulBuild/artifact/repo/bigtop.list
> > /etc/apt/sources.list.d/bigtop2.list
> # apt-get update
> on CentOS 5 you'd run:
> # curl
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=centos5/lastSuccessfulBuild/artifact/output/bigtop.repo
> > /etc/yum.repos.d/bigtop1.repo
> # curl
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=centos5/lastSuccessfulBuild/artifact/repo/bigtop.repo
> > /etc/yum.repos.d/bigtop2.repo
>
> 2. Install Hadoop, Sqoop and Oozie in pseudo distributed mode:
> ubuntu# apt-get install -y hadoop-conf-pseudo sqoop oozie
> RedHat# yum install -y sqoop hadoop-conf-pseudo oozie
>
> 3. Init and start the services
> # service hadoop-hdfs-namenode init
> # service oozie init
> # sudo -u hdfs hadoop fs -chmod -R 777 /
> # for i in hadoop-hdfs-namenode hadoop-hdfs-datanode
> hadoop-yarn-resourcemanager hadoop-yarn-nodemanager
> hadoop-mapreduce-historyserver ; do service $i start ; done
> # service oozie restart
>
> 4. Make sure that Oozie is up and running
> # oozie admin -version -oozie http://localhost:11000/oozie
>
> 5. Install examples and try running a workflow
> # cd /tmp
> # tar xzvf /usr/share/doc/oozie*/oozie-examples.tar.gz
> # hadoop fs -mkdir -p /user/oozie/share/lib/sqoop
> # hadoop fs -mkdir -p /user/root
> # hadoop fs -put examples /user/root/examples
> # hadoop fs -put /usr/lib/sqoop/*.jar /usr/lib/sqoop/lib/*.jar
> /user/oozie/share/lib/sqoop
> # oozie job -DnameNode=hdfs://localhost:8020
> -DjobTracker=localhost:8032 -config examples/apps/sqoop/job.properties
> -run -oozie http://localhost:11000/oozie
>
> At this point the workflow would fail with the following:
>
> eption: cache file (mapreduce.job.cache.files) scheme: "hdfs", host:
> "localhost", port: 8020, file:
> "/user/oozie/share/lib/sqoop/hsqldb-1.8.0.10.jar", conflicts with
> cache file (mapreduce.job.cache.files)
>
> hdfs://localhost:8020/tmp/hadoop-yarn/staging/root/.staging/job_1353549235835_0004/libjars/hsqldb-1.8.0.10.jar
> at
> org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:338)
> at
> org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:273)
> at
> org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:419)
> at
> org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:288)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:391)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1236)
> at
> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:141)
> at
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:202)
> at
> org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:465)
> at
> org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> at
> org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:205)
> at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:174)
> at
> org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
> at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:47)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:473)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:400)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
>
> Intercepting System.exit(1)
>
> <<< Invocation of Main class completed <<<
>
> Thanks,
> Roman.
>
> On Wed, Nov 21, 2012 at 12:00 PM, Mohammad Islam <mi...@yahoo.com>
> wrote:
> > Dear Oozie community,
> >
> > The release candidate 0 for for Oozie 3.3.0 is available.
> >
> >
> > Oozie 3.3.0 has the following new features:
> > 1. Bulk Monitoring API - Consolidated view of jobs
> > 2. Eliminate redundancies in xml through global section.
> > 3. Add formal parameters to XML for early valiadation
> >
> > 4. Visualize color coded job DAG at runtime.
> >
> > 5. Load Hbase/HCat credentials in Job conf
> > 6. Support direct map-reduce job submission through Oozie CLI without
> workflow XML
> > 7. Add support for multiple/configurable sharelibs for each action type
> >
> >
> >
> > In addition, it includes several improvements for performance and
> stability
> > and several bug fixes. Detail release log could be found at:
> > http://people.apache.org/~kamrul/oozie-3.3.0-rc0/release-log.txt
> >
> >
> > Keys used to sign the release are available at
> > http://www.apache.org/dist/oozie/KEYS
> >
> > Please download, test, and try it out:
> > http://people.apache.org/~kamrul/oozie-3.3.0-rc0/
> >
> > The release, md5 signature, gpg signature, and rat report can all be
> found
> > at the above URL.
> >
> > Vote closes in 3 days .
> >
> > Regards,
> >
> > Mohammad
>
Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Posted by Mohammad Islam <mi...@yahoo.com>.
Adding to Rohini.
I also created a Jira for the same few months back:
https://issues.apache.org/jira/browse/MAPREDUCE-4568
Roman,
We are not using Bigtop. Can you please try after removing the "share/lib/sqoop/hsqldb-1.8.0.10.jar"?
Just to make sure that is the root cause.
One more thing, we need to make sure whether this version (Oozie 3.3) will support Hadoop 2.x. I'm sure without fixing MAPREDUCE-4568, it will be hard to support that newer hadoop version. As the JIRA discussions show that we are not close to agreement yet on MR fix.
Need more input from others about this issue.
Regards,
Mohammad
________________________________
From: Rohini Palaniswamy <ro...@gmail.com>
To: dev@oozie.apache.org
Cc: Mohammad Islam <mi...@yahoo.com>; "user@oozie.apache.org" <us...@oozie.apache.org>; dev@bigtop.apache.org
Sent: Sunday, November 25, 2012 8:21 PM
Subject: Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Context on that issue is in
https://issues.apache.org/jira/browse/MAPREDUCE-4549. Failing with that
error in hadoop has been reverted in 0.23.x line but not in 2.0.x line.
Regards,
Rohini
On Wed, Nov 21, 2012 at 6:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> -1 based on Bigtop testing against Hadoop 2.0.2 and Hadoop 1.1.0 Sqoop
> and Hive wf fail to execute.
>
> Here's how to reproduce:
>
> 1. On your linux box hook up 2 Bigtop repos to your system:
>
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/
> e.g. on Ubuntu Lucid you'd run:
> # curl
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=lucid/lastSuccessfulBuild/artifact/output/bigtop.list
> > /etc/apt/sources.list.d/bigtop1.list
> # curl
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=lucid/lastSuccessfulBuild/artifact/repo/bigtop.list
> > /etc/apt/sources.list.d/bigtop2.list
> # apt-get update
> on CentOS 5 you'd run:
> # curl
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=centos5/lastSuccessfulBuild/artifact/output/bigtop.repo
> > /etc/yum.repos.d/bigtop1.repo
> # curl
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=centos5/lastSuccessfulBuild/artifact/repo/bigtop.repo
> > /etc/yum.repos.d/bigtop2.repo
>
> 2. Install Hadoop, Sqoop and Oozie in pseudo distributed mode:
> ubuntu# apt-get install -y hadoop-conf-pseudo sqoop oozie
> RedHat# yum install -y sqoop hadoop-conf-pseudo oozie
>
> 3. Init and start the services
> # service hadoop-hdfs-namenode init
> # service oozie init
> # sudo -u hdfs hadoop fs -chmod -R 777 /
> # for i in hadoop-hdfs-namenode hadoop-hdfs-datanode
> hadoop-yarn-resourcemanager hadoop-yarn-nodemanager
> hadoop-mapreduce-historyserver ; do service $i start ; done
> # service oozie restart
>
> 4. Make sure that Oozie is up and running
> # oozie admin -version -oozie http://localhost:11000/oozie
>
> 5. Install examples and try running a workflow
> # cd /tmp
> # tar xzvf /usr/share/doc/oozie*/oozie-examples.tar.gz
> # hadoop fs -mkdir -p /user/oozie/share/lib/sqoop
> # hadoop fs -mkdir -p /user/root
> # hadoop fs -put examples /user/root/examples
> # hadoop fs -put /usr/lib/sqoop/*.jar /usr/lib/sqoop/lib/*.jar
> /user/oozie/share/lib/sqoop
> # oozie job -DnameNode=hdfs://localhost:8020
> -DjobTracker=localhost:8032 -config examples/apps/sqoop/job.properties
> -run -oozie http://localhost:11000/oozie
>
> At this point the workflow would fail with the following:
>
> eption: cache file (mapreduce.job.cache.files) scheme: "hdfs", host:
> "localhost", port: 8020, file:
> "/user/oozie/share/lib/sqoop/hsqldb-1.8.0.10.jar", conflicts with
> cache file (mapreduce.job.cache.files)
>
> hdfs://localhost:8020/tmp/hadoop-yarn/staging/root/.staging/job_1353549235835_0004/libjars/hsqldb-1.8.0.10.jar
> at
> org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:338)
> at
> org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:273)
> at
> org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:419)
> at
> org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:288)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:391)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1236)
> at
> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:141)
> at
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:202)
> at
> org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:465)
> at
> org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> at
> org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:205)
> at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:174)
> at
> org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
> at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:47)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:473)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:400)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
>
> Intercepting System.exit(1)
>
> <<< Invocation of Main class completed <<<
>
> Thanks,
> Roman.
>
> On Wed, Nov 21, 2012 at 12:00 PM, Mohammad Islam <mi...@yahoo.com>
> wrote:
> > Dear Oozie community,
> >
> > The release candidate 0 for for Oozie 3.3.0 is available.
> >
> >
> > Oozie 3.3.0 has the following new features:
> > 1. Bulk Monitoring API - Consolidated view of jobs
> > 2. Eliminate redundancies in xml through global section.
> > 3. Add formal parameters to XML for early valiadation
> >
> > 4. Visualize color coded job DAG at runtime.
> >
> > 5. Load Hbase/HCat credentials in Job conf
> > 6. Support direct map-reduce job submission through Oozie CLI without
> workflow XML
> > 7. Add support for multiple/configurable sharelibs for each action type
> >
> >
> >
> > In addition, it includes several improvements for performance and
> stability
> > and several bug fixes. Detail release log could be found at:
> > http://people.apache.org/~kamrul/oozie-3.3.0-rc0/release-log.txt
> >
> >
> > Keys used to sign the release are available at
> > http://www.apache.org/dist/oozie/KEYS
> >
> > Please download, test, and try it out:
> > http://people.apache.org/~kamrul/oozie-3.3.0-rc0/
> >
> > The release, md5 signature, gpg signature, and rat report can all be
> found
> > at the above URL.
> >
> > Vote closes in 3 days .
> >
> > Regards,
> >
> > Mohammad
>
Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Posted by Mohammad Islam <mi...@yahoo.com>.
Adding to Rohini.
I also created a Jira for the same few months back:
https://issues.apache.org/jira/browse/MAPREDUCE-4568
Roman,
We are not using Bigtop. Can you please try after removing the "share/lib/sqoop/hsqldb-1.8.0.10.jar"?
Just to make sure that is the root cause.
One more thing, we need to make sure whether this version (Oozie 3.3) will support Hadoop 2.x. I'm sure without fixing MAPREDUCE-4568, it will be hard to support that newer hadoop version. As the JIRA discussions show that we are not close to agreement yet on MR fix.
Need more input from others about this issue.
Regards,
Mohammad
________________________________
From: Rohini Palaniswamy <ro...@gmail.com>
To: dev@oozie.apache.org
Cc: Mohammad Islam <mi...@yahoo.com>; "user@oozie.apache.org" <us...@oozie.apache.org>; dev@bigtop.apache.org
Sent: Sunday, November 25, 2012 8:21 PM
Subject: Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Context on that issue is in
https://issues.apache.org/jira/browse/MAPREDUCE-4549. Failing with that
error in hadoop has been reverted in 0.23.x line but not in 2.0.x line.
Regards,
Rohini
On Wed, Nov 21, 2012 at 6:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> -1 based on Bigtop testing against Hadoop 2.0.2 and Hadoop 1.1.0 Sqoop
> and Hive wf fail to execute.
>
> Here's how to reproduce:
>
> 1. On your linux box hook up 2 Bigtop repos to your system:
>
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/
> e.g. on Ubuntu Lucid you'd run:
> # curl
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=lucid/lastSuccessfulBuild/artifact/output/bigtop.list
> > /etc/apt/sources.list.d/bigtop1.list
> # curl
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=lucid/lastSuccessfulBuild/artifact/repo/bigtop.list
> > /etc/apt/sources.list.d/bigtop2.list
> # apt-get update
> on CentOS 5 you'd run:
> # curl
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=centos5/lastSuccessfulBuild/artifact/output/bigtop.repo
> > /etc/yum.repos.d/bigtop1.repo
> # curl
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=centos5/lastSuccessfulBuild/artifact/repo/bigtop.repo
> > /etc/yum.repos.d/bigtop2.repo
>
> 2. Install Hadoop, Sqoop and Oozie in pseudo distributed mode:
> ubuntu# apt-get install -y hadoop-conf-pseudo sqoop oozie
> RedHat# yum install -y sqoop hadoop-conf-pseudo oozie
>
> 3. Init and start the services
> # service hadoop-hdfs-namenode init
> # service oozie init
> # sudo -u hdfs hadoop fs -chmod -R 777 /
> # for i in hadoop-hdfs-namenode hadoop-hdfs-datanode
> hadoop-yarn-resourcemanager hadoop-yarn-nodemanager
> hadoop-mapreduce-historyserver ; do service $i start ; done
> # service oozie restart
>
> 4. Make sure that Oozie is up and running
> # oozie admin -version -oozie http://localhost:11000/oozie
>
> 5. Install examples and try running a workflow
> # cd /tmp
> # tar xzvf /usr/share/doc/oozie*/oozie-examples.tar.gz
> # hadoop fs -mkdir -p /user/oozie/share/lib/sqoop
> # hadoop fs -mkdir -p /user/root
> # hadoop fs -put examples /user/root/examples
> # hadoop fs -put /usr/lib/sqoop/*.jar /usr/lib/sqoop/lib/*.jar
> /user/oozie/share/lib/sqoop
> # oozie job -DnameNode=hdfs://localhost:8020
> -DjobTracker=localhost:8032 -config examples/apps/sqoop/job.properties
> -run -oozie http://localhost:11000/oozie
>
> At this point the workflow would fail with the following:
>
> eption: cache file (mapreduce.job.cache.files) scheme: "hdfs", host:
> "localhost", port: 8020, file:
> "/user/oozie/share/lib/sqoop/hsqldb-1.8.0.10.jar", conflicts with
> cache file (mapreduce.job.cache.files)
>
> hdfs://localhost:8020/tmp/hadoop-yarn/staging/root/.staging/job_1353549235835_0004/libjars/hsqldb-1.8.0.10.jar
> at
> org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:338)
> at
> org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:273)
> at
> org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:419)
> at
> org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:288)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:391)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1236)
> at
> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:141)
> at
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:202)
> at
> org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:465)
> at
> org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> at
> org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:205)
> at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:174)
> at
> org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
> at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:47)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:473)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:400)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
>
> Intercepting System.exit(1)
>
> <<< Invocation of Main class completed <<<
>
> Thanks,
> Roman.
>
> On Wed, Nov 21, 2012 at 12:00 PM, Mohammad Islam <mi...@yahoo.com>
> wrote:
> > Dear Oozie community,
> >
> > The release candidate 0 for for Oozie 3.3.0 is available.
> >
> >
> > Oozie 3.3.0 has the following new features:
> > 1. Bulk Monitoring API - Consolidated view of jobs
> > 2. Eliminate redundancies in xml through global section.
> > 3. Add formal parameters to XML for early valiadation
> >
> > 4. Visualize color coded job DAG at runtime.
> >
> > 5. Load Hbase/HCat credentials in Job conf
> > 6. Support direct map-reduce job submission through Oozie CLI without
> workflow XML
> > 7. Add support for multiple/configurable sharelibs for each action type
> >
> >
> >
> > In addition, it includes several improvements for performance and
> stability
> > and several bug fixes. Detail release log could be found at:
> > http://people.apache.org/~kamrul/oozie-3.3.0-rc0/release-log.txt
> >
> >
> > Keys used to sign the release are available at
> > http://www.apache.org/dist/oozie/KEYS
> >
> > Please download, test, and try it out:
> > http://people.apache.org/~kamrul/oozie-3.3.0-rc0/
> >
> > The release, md5 signature, gpg signature, and rat report can all be
> found
> > at the above URL.
> >
> > Vote closes in 3 days .
> >
> > Regards,
> >
> > Mohammad
>
Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Posted by Rohini Palaniswamy <ro...@gmail.com>.
Context on that issue is in
https://issues.apache.org/jira/browse/MAPREDUCE-4549. Failing with that
error in hadoop has been reverted in 0.23.x line but not in 2.0.x line.
Regards,
Rohini
On Wed, Nov 21, 2012 at 6:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> -1 based on Bigtop testing against Hadoop 2.0.2 and Hadoop 1.1.0 Sqoop
> and Hive wf fail to execute.
>
> Here's how to reproduce:
>
> 1. On your linux box hook up 2 Bigtop repos to your system:
>
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/
> e.g. on Ubuntu Lucid you'd run:
> # curl
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=lucid/lastSuccessfulBuild/artifact/output/bigtop.list
> > /etc/apt/sources.list.d/bigtop1.list
> # curl
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=lucid/lastSuccessfulBuild/artifact/repo/bigtop.list
> > /etc/apt/sources.list.d/bigtop2.list
> # apt-get update
> on CentOS 5 you'd run:
> # curl
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=centos5/lastSuccessfulBuild/artifact/output/bigtop.repo
> > /etc/yum.repos.d/bigtop1.repo
> # curl
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=centos5/lastSuccessfulBuild/artifact/repo/bigtop.repo
> > /etc/yum.repos.d/bigtop2.repo
>
> 2. Install Hadoop, Sqoop and Oozie in pseudo distributed mode:
> ubuntu# apt-get install -y hadoop-conf-pseudo sqoop oozie
> RedHat# yum install -y sqoop hadoop-conf-pseudo oozie
>
> 3. Init and start the services
> # service hadoop-hdfs-namenode init
> # service oozie init
> # sudo -u hdfs hadoop fs -chmod -R 777 /
> # for i in hadoop-hdfs-namenode hadoop-hdfs-datanode
> hadoop-yarn-resourcemanager hadoop-yarn-nodemanager
> hadoop-mapreduce-historyserver ; do service $i start ; done
> # service oozie restart
>
> 4. Make sure that Oozie is up and running
> # oozie admin -version -oozie http://localhost:11000/oozie
>
> 5. Install examples and try running a workflow
> # cd /tmp
> # tar xzvf /usr/share/doc/oozie*/oozie-examples.tar.gz
> # hadoop fs -mkdir -p /user/oozie/share/lib/sqoop
> # hadoop fs -mkdir -p /user/root
> # hadoop fs -put examples /user/root/examples
> # hadoop fs -put /usr/lib/sqoop/*.jar /usr/lib/sqoop/lib/*.jar
> /user/oozie/share/lib/sqoop
> # oozie job -DnameNode=hdfs://localhost:8020
> -DjobTracker=localhost:8032 -config examples/apps/sqoop/job.properties
> -run -oozie http://localhost:11000/oozie
>
> At this point the workflow would fail with the following:
>
> eption: cache file (mapreduce.job.cache.files) scheme: "hdfs", host:
> "localhost", port: 8020, file:
> "/user/oozie/share/lib/sqoop/hsqldb-1.8.0.10.jar", conflicts with
> cache file (mapreduce.job.cache.files)
>
> hdfs://localhost:8020/tmp/hadoop-yarn/staging/root/.staging/job_1353549235835_0004/libjars/hsqldb-1.8.0.10.jar
> at
> org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:338)
> at
> org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:273)
> at
> org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:419)
> at
> org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:288)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:391)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1236)
> at
> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:141)
> at
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:202)
> at
> org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:465)
> at
> org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> at
> org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:205)
> at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:174)
> at
> org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
> at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:47)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:473)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:400)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
>
> Intercepting System.exit(1)
>
> <<< Invocation of Main class completed <<<
>
> Thanks,
> Roman.
>
> On Wed, Nov 21, 2012 at 12:00 PM, Mohammad Islam <mi...@yahoo.com>
> wrote:
> > Dear Oozie community,
> >
> > The release candidate 0 for for Oozie 3.3.0 is available.
> >
> >
> > Oozie 3.3.0 has the following new features:
> > 1. Bulk Monitoring API - Consolidated view of jobs
> > 2. Eliminate redundancies in xml through global section.
> > 3. Add formal parameters to XML for early valiadation
> >
> > 4. Visualize color coded job DAG at runtime.
> >
> > 5. Load Hbase/HCat credentials in Job conf
> > 6. Support direct map-reduce job submission through Oozie CLI without
> workflow XML
> > 7. Add support for multiple/configurable sharelibs for each action type
> >
> >
> >
> > In addition, it includes several improvements for performance and
> stability
> > and several bug fixes. Detail release log could be found at:
> > http://people.apache.org/~kamrul/oozie-3.3.0-rc0/release-log.txt
> >
> >
> > Keys used to sign the release are available at
> > http://www.apache.org/dist/oozie/KEYS
> >
> > Please download, test, and try it out:
> > http://people.apache.org/~kamrul/oozie-3.3.0-rc0/
> >
> > The release, md5 signature, gpg signature, and rat report can all be
> found
> > at the above URL.
> >
> > Vote closes in 3 days .
> >
> > Regards,
> >
> > Mohammad
>
Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Posted by Rohini Palaniswamy <ro...@gmail.com>.
Context on that issue is in
https://issues.apache.org/jira/browse/MAPREDUCE-4549. Failing with that
error in hadoop has been reverted in 0.23.x line but not in 2.0.x line.
Regards,
Rohini
On Wed, Nov 21, 2012 at 6:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> -1 based on Bigtop testing against Hadoop 2.0.2 and Hadoop 1.1.0 Sqoop
> and Hive wf fail to execute.
>
> Here's how to reproduce:
>
> 1. On your linux box hook up 2 Bigtop repos to your system:
>
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/
> e.g. on Ubuntu Lucid you'd run:
> # curl
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=lucid/lastSuccessfulBuild/artifact/output/bigtop.list
> > /etc/apt/sources.list.d/bigtop1.list
> # curl
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=lucid/lastSuccessfulBuild/artifact/repo/bigtop.list
> > /etc/apt/sources.list.d/bigtop2.list
> # apt-get update
> on CentOS 5 you'd run:
> # curl
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=centos5/lastSuccessfulBuild/artifact/output/bigtop.repo
> > /etc/yum.repos.d/bigtop1.repo
> # curl
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=centos5/lastSuccessfulBuild/artifact/repo/bigtop.repo
> > /etc/yum.repos.d/bigtop2.repo
>
> 2. Install Hadoop, Sqoop and Oozie in pseudo distributed mode:
> ubuntu# apt-get install -y hadoop-conf-pseudo sqoop oozie
> RedHat# yum install -y sqoop hadoop-conf-pseudo oozie
>
> 3. Init and start the services
> # service hadoop-hdfs-namenode init
> # service oozie init
> # sudo -u hdfs hadoop fs -chmod -R 777 /
> # for i in hadoop-hdfs-namenode hadoop-hdfs-datanode
> hadoop-yarn-resourcemanager hadoop-yarn-nodemanager
> hadoop-mapreduce-historyserver ; do service $i start ; done
> # service oozie restart
>
> 4. Make sure that Oozie is up and running
> # oozie admin -version -oozie http://localhost:11000/oozie
>
> 5. Install examples and try running a workflow
> # cd /tmp
> # tar xzvf /usr/share/doc/oozie*/oozie-examples.tar.gz
> # hadoop fs -mkdir -p /user/oozie/share/lib/sqoop
> # hadoop fs -mkdir -p /user/root
> # hadoop fs -put examples /user/root/examples
> # hadoop fs -put /usr/lib/sqoop/*.jar /usr/lib/sqoop/lib/*.jar
> /user/oozie/share/lib/sqoop
> # oozie job -DnameNode=hdfs://localhost:8020
> -DjobTracker=localhost:8032 -config examples/apps/sqoop/job.properties
> -run -oozie http://localhost:11000/oozie
>
> At this point the workflow would fail with the following:
>
> eption: cache file (mapreduce.job.cache.files) scheme: "hdfs", host:
> "localhost", port: 8020, file:
> "/user/oozie/share/lib/sqoop/hsqldb-1.8.0.10.jar", conflicts with
> cache file (mapreduce.job.cache.files)
>
> hdfs://localhost:8020/tmp/hadoop-yarn/staging/root/.staging/job_1353549235835_0004/libjars/hsqldb-1.8.0.10.jar
> at
> org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:338)
> at
> org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:273)
> at
> org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:419)
> at
> org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:288)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:391)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1236)
> at
> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:141)
> at
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:202)
> at
> org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:465)
> at
> org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> at
> org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:205)
> at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:174)
> at
> org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
> at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:47)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:473)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:400)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
>
> Intercepting System.exit(1)
>
> <<< Invocation of Main class completed <<<
>
> Thanks,
> Roman.
>
> On Wed, Nov 21, 2012 at 12:00 PM, Mohammad Islam <mi...@yahoo.com>
> wrote:
> > Dear Oozie community,
> >
> > The release candidate 0 for for Oozie 3.3.0 is available.
> >
> >
> > Oozie 3.3.0 has the following new features:
> > 1. Bulk Monitoring API - Consolidated view of jobs
> > 2. Eliminate redundancies in xml through global section.
> > 3. Add formal parameters to XML for early valiadation
> >
> > 4. Visualize color coded job DAG at runtime.
> >
> > 5. Load Hbase/HCat credentials in Job conf
> > 6. Support direct map-reduce job submission through Oozie CLI without
> workflow XML
> > 7. Add support for multiple/configurable sharelibs for each action type
> >
> >
> >
> > In addition, it includes several improvements for performance and
> stability
> > and several bug fixes. Detail release log could be found at:
> > http://people.apache.org/~kamrul/oozie-3.3.0-rc0/release-log.txt
> >
> >
> > Keys used to sign the release are available at
> > http://www.apache.org/dist/oozie/KEYS
> >
> > Please download, test, and try it out:
> > http://people.apache.org/~kamrul/oozie-3.3.0-rc0/
> >
> > The release, md5 signature, gpg signature, and rat report can all be
> found
> > at the above URL.
> >
> > Vote closes in 3 days .
> >
> > Regards,
> >
> > Mohammad
>
Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Posted by Rohini Palaniswamy <ro...@gmail.com>.
Context on that issue is in
https://issues.apache.org/jira/browse/MAPREDUCE-4549. Failing with that
error in hadoop has been reverted in 0.23.x line but not in 2.0.x line.
Regards,
Rohini
On Wed, Nov 21, 2012 at 6:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> -1 based on Bigtop testing against Hadoop 2.0.2 and Hadoop 1.1.0 Sqoop
> and Hive wf fail to execute.
>
> Here's how to reproduce:
>
> 1. On your linux box hook up 2 Bigtop repos to your system:
>
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/
> e.g. on Ubuntu Lucid you'd run:
> # curl
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=lucid/lastSuccessfulBuild/artifact/output/bigtop.list
> > /etc/apt/sources.list.d/bigtop1.list
> # curl
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=lucid/lastSuccessfulBuild/artifact/repo/bigtop.list
> > /etc/apt/sources.list.d/bigtop2.list
> # apt-get update
> on CentOS 5 you'd run:
> # curl
> http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=centos5/lastSuccessfulBuild/artifact/output/bigtop.repo
> > /etc/yum.repos.d/bigtop1.repo
> # curl
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=centos5/lastSuccessfulBuild/artifact/repo/bigtop.repo
> > /etc/yum.repos.d/bigtop2.repo
>
> 2. Install Hadoop, Sqoop and Oozie in pseudo distributed mode:
> ubuntu# apt-get install -y hadoop-conf-pseudo sqoop oozie
> RedHat# yum install -y sqoop hadoop-conf-pseudo oozie
>
> 3. Init and start the services
> # service hadoop-hdfs-namenode init
> # service oozie init
> # sudo -u hdfs hadoop fs -chmod -R 777 /
> # for i in hadoop-hdfs-namenode hadoop-hdfs-datanode
> hadoop-yarn-resourcemanager hadoop-yarn-nodemanager
> hadoop-mapreduce-historyserver ; do service $i start ; done
> # service oozie restart
>
> 4. Make sure that Oozie is up and running
> # oozie admin -version -oozie http://localhost:11000/oozie
>
> 5. Install examples and try running a workflow
> # cd /tmp
> # tar xzvf /usr/share/doc/oozie*/oozie-examples.tar.gz
> # hadoop fs -mkdir -p /user/oozie/share/lib/sqoop
> # hadoop fs -mkdir -p /user/root
> # hadoop fs -put examples /user/root/examples
> # hadoop fs -put /usr/lib/sqoop/*.jar /usr/lib/sqoop/lib/*.jar
> /user/oozie/share/lib/sqoop
> # oozie job -DnameNode=hdfs://localhost:8020
> -DjobTracker=localhost:8032 -config examples/apps/sqoop/job.properties
> -run -oozie http://localhost:11000/oozie
>
> At this point the workflow would fail with the following:
>
> eption: cache file (mapreduce.job.cache.files) scheme: "hdfs", host:
> "localhost", port: 8020, file:
> "/user/oozie/share/lib/sqoop/hsqldb-1.8.0.10.jar", conflicts with
> cache file (mapreduce.job.cache.files)
>
> hdfs://localhost:8020/tmp/hadoop-yarn/staging/root/.staging/job_1353549235835_0004/libjars/hsqldb-1.8.0.10.jar
> at
> org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:338)
> at
> org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:273)
> at
> org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:419)
> at
> org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:288)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:391)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1236)
> at
> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:141)
> at
> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:202)
> at
> org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:465)
> at
> org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
> at
> org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:205)
> at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:174)
> at
> org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
> at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:47)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:473)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:400)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
>
> Intercepting System.exit(1)
>
> <<< Invocation of Main class completed <<<
>
> Thanks,
> Roman.
>
> On Wed, Nov 21, 2012 at 12:00 PM, Mohammad Islam <mi...@yahoo.com>
> wrote:
> > Dear Oozie community,
> >
> > The release candidate 0 for for Oozie 3.3.0 is available.
> >
> >
> > Oozie 3.3.0 has the following new features:
> > 1. Bulk Monitoring API - Consolidated view of jobs
> > 2. Eliminate redundancies in xml through global section.
> > 3. Add formal parameters to XML for early valiadation
> >
> > 4. Visualize color coded job DAG at runtime.
> >
> > 5. Load Hbase/HCat credentials in Job conf
> > 6. Support direct map-reduce job submission through Oozie CLI without
> workflow XML
> > 7. Add support for multiple/configurable sharelibs for each action type
> >
> >
> >
> > In addition, it includes several improvements for performance and
> stability
> > and several bug fixes. Detail release log could be found at:
> > http://people.apache.org/~kamrul/oozie-3.3.0-rc0/release-log.txt
> >
> >
> > Keys used to sign the release are available at
> > http://www.apache.org/dist/oozie/KEYS
> >
> > Please download, test, and try it out:
> > http://people.apache.org/~kamrul/oozie-3.3.0-rc0/
> >
> > The release, md5 signature, gpg signature, and rat report can all be
> found
> > at the above URL.
> >
> > Vote closes in 3 days .
> >
> > Regards,
> >
> > Mohammad
>
Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Posted by Roman Shaposhnik <rv...@apache.org>.
-1 based on Bigtop testing against Hadoop 2.0.2 and Hadoop 1.1.0 Sqoop
and Hive wf fail to execute.
Here's how to reproduce:
1. On your linux box hook up 2 Bigtop repos to your system:
http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/
http://bigtop01.cloudera.org:8080/job/Bigtop-git/
e.g. on Ubuntu Lucid you'd run:
# curl http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=lucid/lastSuccessfulBuild/artifact/output/bigtop.list
> /etc/apt/sources.list.d/bigtop1.list
# curl http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=lucid/lastSuccessfulBuild/artifact/repo/bigtop.list
> /etc/apt/sources.list.d/bigtop2.list
# apt-get update
on CentOS 5 you'd run:
# curl http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=centos5/lastSuccessfulBuild/artifact/output/bigtop.repo
> /etc/yum.repos.d/bigtop1.repo
# curl http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=centos5/lastSuccessfulBuild/artifact/repo/bigtop.repo
> /etc/yum.repos.d/bigtop2.repo
2. Install Hadoop, Sqoop and Oozie in pseudo distributed mode:
ubuntu# apt-get install -y hadoop-conf-pseudo sqoop oozie
RedHat# yum install -y sqoop hadoop-conf-pseudo oozie
3. Init and start the services
# service hadoop-hdfs-namenode init
# service oozie init
# sudo -u hdfs hadoop fs -chmod -R 777 /
# for i in hadoop-hdfs-namenode hadoop-hdfs-datanode
hadoop-yarn-resourcemanager hadoop-yarn-nodemanager
hadoop-mapreduce-historyserver ; do service $i start ; done
# service oozie restart
4. Make sure that Oozie is up and running
# oozie admin -version -oozie http://localhost:11000/oozie
5. Install examples and try running a workflow
# cd /tmp
# tar xzvf /usr/share/doc/oozie*/oozie-examples.tar.gz
# hadoop fs -mkdir -p /user/oozie/share/lib/sqoop
# hadoop fs -mkdir -p /user/root
# hadoop fs -put examples /user/root/examples
# hadoop fs -put /usr/lib/sqoop/*.jar /usr/lib/sqoop/lib/*.jar
/user/oozie/share/lib/sqoop
# oozie job -DnameNode=hdfs://localhost:8020
-DjobTracker=localhost:8032 -config examples/apps/sqoop/job.properties
-run -oozie http://localhost:11000/oozie
At this point the workflow would fail with the following:
eption: cache file (mapreduce.job.cache.files) scheme: "hdfs", host:
"localhost", port: 8020, file:
"/user/oozie/share/lib/sqoop/hsqldb-1.8.0.10.jar", conflicts with
cache file (mapreduce.job.cache.files)
hdfs://localhost:8020/tmp/hadoop-yarn/staging/root/.staging/job_1353549235835_0004/libjars/hsqldb-1.8.0.10.jar
at org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:338)
at org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:273)
at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:419)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:288)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:391)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1236)
at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:141)
at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:202)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:465)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:205)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:174)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:473)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:400)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
Intercepting System.exit(1)
<<< Invocation of Main class completed <<<
Thanks,
Roman.
On Wed, Nov 21, 2012 at 12:00 PM, Mohammad Islam <mi...@yahoo.com> wrote:
> Dear Oozie community,
>
> The release candidate 0 for for Oozie 3.3.0 is available.
>
>
> Oozie 3.3.0 has the following new features:
> 1. Bulk Monitoring API - Consolidated view of jobs
> 2. Eliminate redundancies in xml through global section.
> 3. Add formal parameters to XML for early valiadation
>
> 4. Visualize color coded job DAG at runtime.
>
> 5. Load Hbase/HCat credentials in Job conf
> 6. Support direct map-reduce job submission through Oozie CLI without workflow XML
> 7. Add support for multiple/configurable sharelibs for each action type
>
>
>
> In addition, it includes several improvements for performance and stability
> and several bug fixes. Detail release log could be found at:
> http://people.apache.org/~kamrul/oozie-3.3.0-rc0/release-log.txt
>
>
> Keys used to sign the release are available at
> http://www.apache.org/dist/oozie/KEYS
>
> Please download, test, and try it out:
> http://people.apache.org/~kamrul/oozie-3.3.0-rc0/
>
> The release, md5 signature, gpg signature, and rat report can all be found
> at the above URL.
>
> Vote closes in 3 days .
>
> Regards,
>
> Mohammad
Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Posted by Roman Shaposhnik <rv...@apache.org>.
-1 based on Bigtop testing against Hadoop 2.0.2 and Hadoop 1.1.0 Sqoop
and Hive wf fail to execute.
Here's how to reproduce:
1. On your linux box hook up 2 Bigtop repos to your system:
http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/
http://bigtop01.cloudera.org:8080/job/Bigtop-git/
e.g. on Ubuntu Lucid you'd run:
# curl http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=lucid/lastSuccessfulBuild/artifact/output/bigtop.list
> /etc/apt/sources.list.d/bigtop1.list
# curl http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=lucid/lastSuccessfulBuild/artifact/repo/bigtop.list
> /etc/apt/sources.list.d/bigtop2.list
# apt-get update
on CentOS 5 you'd run:
# curl http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=centos5/lastSuccessfulBuild/artifact/output/bigtop.repo
> /etc/yum.repos.d/bigtop1.repo
# curl http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=centos5/lastSuccessfulBuild/artifact/repo/bigtop.repo
> /etc/yum.repos.d/bigtop2.repo
2. Install Hadoop, Sqoop and Oozie in pseudo distributed mode:
ubuntu# apt-get install -y hadoop-conf-pseudo sqoop oozie
RedHat# yum install -y sqoop hadoop-conf-pseudo oozie
3. Init and start the services
# service hadoop-hdfs-namenode init
# service oozie init
# sudo -u hdfs hadoop fs -chmod -R 777 /
# for i in hadoop-hdfs-namenode hadoop-hdfs-datanode
hadoop-yarn-resourcemanager hadoop-yarn-nodemanager
hadoop-mapreduce-historyserver ; do service $i start ; done
# service oozie restart
4. Make sure that Oozie is up and running
# oozie admin -version -oozie http://localhost:11000/oozie
5. Install examples and try running a workflow
# cd /tmp
# tar xzvf /usr/share/doc/oozie*/oozie-examples.tar.gz
# hadoop fs -mkdir -p /user/oozie/share/lib/sqoop
# hadoop fs -mkdir -p /user/root
# hadoop fs -put examples /user/root/examples
# hadoop fs -put /usr/lib/sqoop/*.jar /usr/lib/sqoop/lib/*.jar
/user/oozie/share/lib/sqoop
# oozie job -DnameNode=hdfs://localhost:8020
-DjobTracker=localhost:8032 -config examples/apps/sqoop/job.properties
-run -oozie http://localhost:11000/oozie
At this point the workflow would fail with the following:
eption: cache file (mapreduce.job.cache.files) scheme: "hdfs", host:
"localhost", port: 8020, file:
"/user/oozie/share/lib/sqoop/hsqldb-1.8.0.10.jar", conflicts with
cache file (mapreduce.job.cache.files)
hdfs://localhost:8020/tmp/hadoop-yarn/staging/root/.staging/job_1353549235835_0004/libjars/hsqldb-1.8.0.10.jar
at org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:338)
at org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:273)
at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:419)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:288)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:391)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1236)
at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:141)
at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:202)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:465)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:205)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:174)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:473)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:400)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
Intercepting System.exit(1)
<<< Invocation of Main class completed <<<
Thanks,
Roman.
On Wed, Nov 21, 2012 at 12:00 PM, Mohammad Islam <mi...@yahoo.com> wrote:
> Dear Oozie community,
>
> The release candidate 0 for for Oozie 3.3.0 is available.
>
>
> Oozie 3.3.0 has the following new features:
> 1. Bulk Monitoring API - Consolidated view of jobs
> 2. Eliminate redundancies in xml through global section.
> 3. Add formal parameters to XML for early valiadation
>
> 4. Visualize color coded job DAG at runtime.
>
> 5. Load Hbase/HCat credentials in Job conf
> 6. Support direct map-reduce job submission through Oozie CLI without workflow XML
> 7. Add support for multiple/configurable sharelibs for each action type
>
>
>
> In addition, it includes several improvements for performance and stability
> and several bug fixes. Detail release log could be found at:
> http://people.apache.org/~kamrul/oozie-3.3.0-rc0/release-log.txt
>
>
> Keys used to sign the release are available at
> http://www.apache.org/dist/oozie/KEYS
>
> Please download, test, and try it out:
> http://people.apache.org/~kamrul/oozie-3.3.0-rc0/
>
> The release, md5 signature, gpg signature, and rat report can all be found
> at the above URL.
>
> Vote closes in 3 days .
>
> Regards,
>
> Mohammad
Re: [VOTE] Release Oozie 3.3.0 (candidate 0)
Posted by Roman Shaposhnik <rv...@apache.org>.
-1 based on Bigtop testing against Hadoop 2.0.2 and Hadoop 1.1.0 Sqoop
and Hive wf fail to execute.
Here's how to reproduce:
1. On your linux box hook up 2 Bigtop repos to your system:
http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/
http://bigtop01.cloudera.org:8080/job/Bigtop-git/
e.g. on Ubuntu Lucid you'd run:
# curl http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=lucid/lastSuccessfulBuild/artifact/output/bigtop.list
> /etc/apt/sources.list.d/bigtop1.list
# curl http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=lucid/lastSuccessfulBuild/artifact/repo/bigtop.list
> /etc/apt/sources.list.d/bigtop2.list
# apt-get update
on CentOS 5 you'd run:
# curl http://bigtop01.cloudera.org:8080/job/Bigtop-git/label=centos5/lastSuccessfulBuild/artifact/output/bigtop.repo
> /etc/yum.repos.d/bigtop1.repo
# curl http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Repository/label=centos5/lastSuccessfulBuild/artifact/repo/bigtop.repo
> /etc/yum.repos.d/bigtop2.repo
2. Install Hadoop, Sqoop and Oozie in pseudo distributed mode:
ubuntu# apt-get install -y hadoop-conf-pseudo sqoop oozie
RedHat# yum install -y sqoop hadoop-conf-pseudo oozie
3. Init and start the services
# service hadoop-hdfs-namenode init
# service oozie init
# sudo -u hdfs hadoop fs -chmod -R 777 /
# for i in hadoop-hdfs-namenode hadoop-hdfs-datanode
hadoop-yarn-resourcemanager hadoop-yarn-nodemanager
hadoop-mapreduce-historyserver ; do service $i start ; done
# service oozie restart
4. Make sure that Oozie is up and running
# oozie admin -version -oozie http://localhost:11000/oozie
5. Install examples and try running a workflow
# cd /tmp
# tar xzvf /usr/share/doc/oozie*/oozie-examples.tar.gz
# hadoop fs -mkdir -p /user/oozie/share/lib/sqoop
# hadoop fs -mkdir -p /user/root
# hadoop fs -put examples /user/root/examples
# hadoop fs -put /usr/lib/sqoop/*.jar /usr/lib/sqoop/lib/*.jar
/user/oozie/share/lib/sqoop
# oozie job -DnameNode=hdfs://localhost:8020
-DjobTracker=localhost:8032 -config examples/apps/sqoop/job.properties
-run -oozie http://localhost:11000/oozie
At this point the workflow would fail with the following:
eption: cache file (mapreduce.job.cache.files) scheme: "hdfs", host:
"localhost", port: 8020, file:
"/user/oozie/share/lib/sqoop/hsqldb-1.8.0.10.jar", conflicts with
cache file (mapreduce.job.cache.files)
hdfs://localhost:8020/tmp/hadoop-yarn/staging/root/.staging/job_1353549235835_0004/libjars/hsqldb-1.8.0.10.jar
at org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:338)
at org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:273)
at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:419)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:288)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:391)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1236)
at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:141)
at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:202)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:465)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:205)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:174)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:473)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:400)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
Intercepting System.exit(1)
<<< Invocation of Main class completed <<<
Thanks,
Roman.
On Wed, Nov 21, 2012 at 12:00 PM, Mohammad Islam <mi...@yahoo.com> wrote:
> Dear Oozie community,
>
> The release candidate 0 for for Oozie 3.3.0 is available.
>
>
> Oozie 3.3.0 has the following new features:
> 1. Bulk Monitoring API - Consolidated view of jobs
> 2. Eliminate redundancies in xml through global section.
> 3. Add formal parameters to XML for early valiadation
>
> 4. Visualize color coded job DAG at runtime.
>
> 5. Load Hbase/HCat credentials in Job conf
> 6. Support direct map-reduce job submission through Oozie CLI without workflow XML
> 7. Add support for multiple/configurable sharelibs for each action type
>
>
>
> In addition, it includes several improvements for performance and stability
> and several bug fixes. Detail release log could be found at:
> http://people.apache.org/~kamrul/oozie-3.3.0-rc0/release-log.txt
>
>
> Keys used to sign the release are available at
> http://www.apache.org/dist/oozie/KEYS
>
> Please download, test, and try it out:
> http://people.apache.org/~kamrul/oozie-3.3.0-rc0/
>
> The release, md5 signature, gpg signature, and rat report can all be found
> at the above URL.
>
> Vote closes in 3 days .
>
> Regards,
>
> Mohammad