You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Matt Goeke <go...@gmail.com> on 2012/12/14 00:36:14 UTC
Installation and Configuration of Oozie 3.3
All,
I am falling back to Oozie 3.2 for now but can someone possibly explain how
Oozie 3.3 is supposed to be configured? I was hoping to just follow the
quick start guide but it seems like the packaging does not match up at all.
Trying to work through it I ended up downloading maven and running a 'mvn
install' on the folder which built some of the hadooplibs but I am still
missing all of the bin scripts.
--
Matt
Re: Installation and Configuration of Oozie 3.3
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Matt
Are you getting any error?
Regards,
Mohammad Tariq
On Fri, Dec 14, 2012 at 5:06 AM, Matt Goeke <go...@gmail.com> wrote:
> All,
>
> I am falling back to Oozie 3.2 for now but can someone possibly explain how
> Oozie 3.3 is supposed to be configured? I was hoping to just follow the
> quick start guide but it seems like the packaging does not match up at all.
>
> Trying to work through it I ended up downloading maven and running a 'mvn
> install' on the folder which built some of the hadooplibs but I am still
> missing all of the bin scripts.
>
> --
> Matt
>
Re: Installation and Configuration of Oozie 3.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Glad to help and sorry for missing a t in Matt in my prev email.
On Thu, Dec 20, 2012 at 9:54 AM, Matt Goeke <go...@gmail.com> wrote:
> Alejandro,
>
> I am pretty sure you just saved me a couple hours of digging :)
>
> Most of these issues stem from our use of VPC but it looks like the
> hostname that it was retrieving using the 'hostname -f' in that script was
> in fact unresolvable from the subnet my EMR instance is on. A simple
> override of that property with the IP of the Oozie EC2 box and everything
> is running perfectly.
>
> As always thank you for the kind advice.
>
> --
> Matt
>
> On Thu, Dec 20, 2012 at 3:39 AM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > Hi Mat,
> >
> > Your 11min delay seems like Oozie is using its fail safe polling
> mechanism
> > to detect that a job completed in the cluster.
> >
> > You may have to tweak the following VAR in the oozie-env.sh script:
> >
> > # The base URL for callback URLs to Oozie
> > #
> > # export OOZIE_BASE_URL="http://
> > ${OOZIE_HTTP_HOSTNAME}:${OOZIE_HTTP_PORT}/oozie"
> >
> > Oozie sets callback URLs in the jobconfs using this base URL, then the JT
> > calls back oozie as soon as a job completes. If the hostname is not
> correct
> > (see the rest of the script to find out how the hostname is resolved),
> the
> > call from the JT will never arrive and Oozie will do a check every 10
> mins
> > for each action.
> >
> > Hope this solves your prob.
> >
> > Regarding the wiki, yes, Oozie wiki,
> > https://cwiki.apache.org/confluence/display/OOZIE , which seems down at
> > the
> > moment.
> >
> > Thx
> >
> >
> >
> > On Thu, Dec 20, 2012 at 9:25 AM, Matt Goeke <go...@gmail.com>
> > wrote:
> >
> > > All,
> > >
> > > I have gotten over all of the blocker configuration hurdles between EC2
> > and
> > > EMR and I am able to submit one of my jobs with success.
> Unfortunately, I
> > > am running into a weird issue with my actions where each one will take
> > > exactly 11 mins from start to finish even though the delved MR job is
> no
> > > where near that long (1 action is exactly 34 seconds and the other is 5
> > > mins 21 seconds). I cannot guarantee that this is not an issue with
> > overall
> > > resources on the node I am running the oozie instance on, I highly
> doubt
> > it
> > > since this is on an m1.large spec, so I am curious if there are any
> > changes
> > > to the site that might be able to flesh out what is causing this issue.
> > >
> > >
> > >
> > > Alejandro: The actual setup and configuration of this is fairly
> straight
> > > forward so I am happy to write up a wiki on this if you guys have a
> > > specific wiki in mind. I am not sure many people are keen on using EMR
> > as a
> > > persistent cluster (I assume most persistant clusters are setup across
> > EC2
> > > nodes) but I am actually very pleased with it so far since it greatly
> > > reduces the amount of initial setup required to spin up a cluster.
> > >
> > > --
> > > Matt
> > >
> > >
> > > On Tue, Dec 18, 2012 at 5:48 PM, Robert Kanter <rk...@cloudera.com>
> > > wrote:
> > >
> > > > Hi Matt,
> > > >
> > > > The oozie.service.ProxyUserService.proxyuser.hadoop.hosts and
> > > > oozie.service.ProxyUserService.proxyuser.hadoop.groups
> > > > properties are part of Oozie's configuration and would go in
> > > > oozie-site.xml. This lets you impersonate users on the Oozie side of
> > > > things. See
> > > >
> > > >
> > >
> >
> http://oozie.apache.org/docs/3.3.0/AG_Install.html#User_ProxyUser_Configurationfor
> > > > more info.
> > > >
> > > > There's two similar properties for Hadoop that go into core-site.xml:
> > > > hadoop.proxyuser.oozie.hosts
> > > > and hadoop.proxyuser.oozie.groups
> > > > I think this is what you need to fix your error. See
> > > > http://hadoop.apache.org/docs/stable/Secure_Impersonation.html for
> > more
> > > > info.
> > > >
> > > > - Robert
> > > >
> > > >
> > > >
> > > > On Tue, Dec 18, 2012 at 3:38 PM, Matt Goeke <goeke.matthew@gmail.com
> >
> > > > wrote:
> > > >
> > > > > All,
> > > > >
> > > > > Still working on getting Oozie 3.3 integrated with EMR with most of
> > my
> > > > time
> > > > > so far spent resolving the security group config needed for VPC.
> The
> > > EC2
> > > > > configuration was pretty simple but the main blocker right now is
> > > getting
> > > > > past the error below:
> > > > >
> > > > > Caused by: org.apache.hadoop.ipc.RemoteException: User: hadoop is
> not
> > > > > allowed to impersonate hadoop
> > > > > at org.apache.hadoop.ipc.Client.call(Client.java:1070)
> > > > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> > > > > at $Proxy24.getProtocolVersion(Unknown Source)
> > > > > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
> > > > > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
> > > > > at
> > > > >
> > org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
> > > > > at
> > org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
> > > > > at
> > org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
> > > > > at
> > > > >
> > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
> > > > > at
> > > org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
> > > > > at
> > > > org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
> > > > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:411)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:409)
> > > > > at java.security.AccessController.doPrivileged(Native
> Method)
> > > > > at javax.security.auth.Subject.doAs(Subject.java:415)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:409)
> > > > > ... 26 more
> > > > >
> > > > > I know this is usually related to not having the correct proxy
> > configs
> > > in
> > > > > the core site but my current core site proxy configs are below
> (and I
> > > > have
> > > > > bounced both the NN and the JT since applying them):
> > > > >
> > > > >
> <property><name>dfs.permissions</name><value>false</value></property>
> > > > >
> > > > >
> > > >
> > >
> >
> <property><name>oozie.service.ProxyUserService.proxyuser.hadoop.hosts</name><value>*</value></property>
> > > > >
> > > > >
> > > >
> > >
> >
> <property><name>oozie.service.ProxyUserService.proxyuser.hadoop.groups</name><value>*</value></property>
> > > > >
> > > > > If I recall correctly this authorization is only checked at the the
> > > JT/NN
> > > > > level and therefore shouldn't need to be pushed to the core site on
> > the
> > > > > slave machines right? Also, would there be any reason the wildcard
> > > would
> > > > be
> > > > > incompatible across hadoop distros (we are currently using 1.0.3
> from
> > > > EMR)?
> > > > > Lastly, just for the sake of clarity is the proxy hosts config
> based
> > on
> > > > the
> > > > > box submitting the oozie request (edge node) or based on the boxes
> > > > actually
> > > > > running the jobs (data/task nodes)?
> > > > >
> > > > > --
> > > > > Matt
> > > > >
> > > > >
> > > > > On Thu, Dec 13, 2012 at 6:19 PM, Alejandro Abdelnur <
> > tucu@cloudera.com
> > > > > >wrote:
> > > > >
> > > > > > Matt,
> > > > > >
> > > > > > It is not matter of bundling native code or not. Officially we
> > > suppose
> > > > to
> > > > > > do source releases only. As convenience we could do binaries, but
> > > there
> > > > > are
> > > > > > discussions about that, if the could be signed or not.
> > > > > >
> > > > > > Regarding installing/running oozie in EC2. I never done it. Would
> > you
> > > > > mind
> > > > > > writing up a wiki on it once you figure it out?
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > >
> > > > > > On Thu, Dec 13, 2012 at 4:02 PM, Matt Goeke <
> > goeke.matthew@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thank you both for the follow-up.
> > > > > > >
> > > > > > > 2 other questions that pertain to this:
> > > > > > > 1) I don't remember any natives being required for Oozie so is
> > > there
> > > > a
> > > > > > > reason why we don't release with a -bin like most other apache
> > > > > projects?
> > > > > > > 2) Are there any issues I might expect to run into when trying
> to
> > > run
> > > > > > this
> > > > > > > on EC2 backed by EMR?
> > > > > > >
> > > > > > > --
> > > > > > > Matt
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Dec 13, 2012 at 5:48 PM, Alejandro Abdelnur <
> > > > tucu@cloudera.com
> > > > > > > >wrote:
> > > > > > >
> > > > > > > > Matt,
> > > > > > > >
> > > > > > > > Apache Oozie release artifacts are sources only. The easiest
> > way
> > > to
> > > > > > build
> > > > > > > > the TARBALL is:
> > > > > > > >
> > > > > > > > * install Maven
> > > > > > > > * run bin/mkdistro.sh -DskipTests
> > > > > > > >
> > > > > > > > Then follow the Quick Start instructions.
> > > > > > > >
> > > > > > > > I'll open a JIRA to add this to the docs.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Dec 13, 2012 at 3:36 PM, Matt Goeke <
> > > > goeke.matthew@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > All,
> > > > > > > > >
> > > > > > > > > I am falling back to Oozie 3.2 for now but can someone
> > possibly
> > > > > > explain
> > > > > > > > how
> > > > > > > > > Oozie 3.3 is supposed to be configured? I was hoping to
> just
> > > > follow
> > > > > > the
> > > > > > > > > quick start guide but it seems like the packaging does not
> > > match
> > > > up
> > > > > > at
> > > > > > > > all.
> > > > > > > > >
> > > > > > > > > Trying to work through it I ended up downloading maven and
> > > > running
> > > > > a
> > > > > > > 'mvn
> > > > > > > > > install' on the folder which built some of the hadooplibs
> > but I
> > > > am
> > > > > > > still
> > > > > > > > > missing all of the bin scripts.
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Matt
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Alejandro
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Alejandro
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Alejandro
> >
>
--
Alejandro
Re: Installation and Configuration of Oozie 3.3
Posted by Matt Goeke <go...@gmail.com>.
Alejandro,
I am pretty sure you just saved me a couple hours of digging :)
Most of these issues stem from our use of VPC but it looks like the
hostname that it was retrieving using the 'hostname -f' in that script was
in fact unresolvable from the subnet my EMR instance is on. A simple
override of that property with the IP of the Oozie EC2 box and everything
is running perfectly.
As always thank you for the kind advice.
--
Matt
On Thu, Dec 20, 2012 at 3:39 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Hi Mat,
>
> Your 11min delay seems like Oozie is using its fail safe polling mechanism
> to detect that a job completed in the cluster.
>
> You may have to tweak the following VAR in the oozie-env.sh script:
>
> # The base URL for callback URLs to Oozie
> #
> # export OOZIE_BASE_URL="http://
> ${OOZIE_HTTP_HOSTNAME}:${OOZIE_HTTP_PORT}/oozie"
>
> Oozie sets callback URLs in the jobconfs using this base URL, then the JT
> calls back oozie as soon as a job completes. If the hostname is not correct
> (see the rest of the script to find out how the hostname is resolved), the
> call from the JT will never arrive and Oozie will do a check every 10 mins
> for each action.
>
> Hope this solves your prob.
>
> Regarding the wiki, yes, Oozie wiki,
> https://cwiki.apache.org/confluence/display/OOZIE , which seems down at
> the
> moment.
>
> Thx
>
>
>
> On Thu, Dec 20, 2012 at 9:25 AM, Matt Goeke <go...@gmail.com>
> wrote:
>
> > All,
> >
> > I have gotten over all of the blocker configuration hurdles between EC2
> and
> > EMR and I am able to submit one of my jobs with success. Unfortunately, I
> > am running into a weird issue with my actions where each one will take
> > exactly 11 mins from start to finish even though the delved MR job is no
> > where near that long (1 action is exactly 34 seconds and the other is 5
> > mins 21 seconds). I cannot guarantee that this is not an issue with
> overall
> > resources on the node I am running the oozie instance on, I highly doubt
> it
> > since this is on an m1.large spec, so I am curious if there are any
> changes
> > to the site that might be able to flesh out what is causing this issue.
> >
> >
> >
> > Alejandro: The actual setup and configuration of this is fairly straight
> > forward so I am happy to write up a wiki on this if you guys have a
> > specific wiki in mind. I am not sure many people are keen on using EMR
> as a
> > persistent cluster (I assume most persistant clusters are setup across
> EC2
> > nodes) but I am actually very pleased with it so far since it greatly
> > reduces the amount of initial setup required to spin up a cluster.
> >
> > --
> > Matt
> >
> >
> > On Tue, Dec 18, 2012 at 5:48 PM, Robert Kanter <rk...@cloudera.com>
> > wrote:
> >
> > > Hi Matt,
> > >
> > > The oozie.service.ProxyUserService.proxyuser.hadoop.hosts and
> > > oozie.service.ProxyUserService.proxyuser.hadoop.groups
> > > properties are part of Oozie's configuration and would go in
> > > oozie-site.xml. This lets you impersonate users on the Oozie side of
> > > things. See
> > >
> > >
> >
> http://oozie.apache.org/docs/3.3.0/AG_Install.html#User_ProxyUser_Configurationfor
> > > more info.
> > >
> > > There's two similar properties for Hadoop that go into core-site.xml:
> > > hadoop.proxyuser.oozie.hosts
> > > and hadoop.proxyuser.oozie.groups
> > > I think this is what you need to fix your error. See
> > > http://hadoop.apache.org/docs/stable/Secure_Impersonation.html for
> more
> > > info.
> > >
> > > - Robert
> > >
> > >
> > >
> > > On Tue, Dec 18, 2012 at 3:38 PM, Matt Goeke <go...@gmail.com>
> > > wrote:
> > >
> > > > All,
> > > >
> > > > Still working on getting Oozie 3.3 integrated with EMR with most of
> my
> > > time
> > > > so far spent resolving the security group config needed for VPC. The
> > EC2
> > > > configuration was pretty simple but the main blocker right now is
> > getting
> > > > past the error below:
> > > >
> > > > Caused by: org.apache.hadoop.ipc.RemoteException: User: hadoop is not
> > > > allowed to impersonate hadoop
> > > > at org.apache.hadoop.ipc.Client.call(Client.java:1070)
> > > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> > > > at $Proxy24.getProtocolVersion(Unknown Source)
> > > > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
> > > > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
> > > > at
> > > >
> org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
> > > > at
> org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
> > > > at
> org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
> > > > at
> > > >
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
> > > > at
> > org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
> > > > at
> > > org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
> > > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:411)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:409)
> > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > at javax.security.auth.Subject.doAs(Subject.java:415)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:409)
> > > > ... 26 more
> > > >
> > > > I know this is usually related to not having the correct proxy
> configs
> > in
> > > > the core site but my current core site proxy configs are below (and I
> > > have
> > > > bounced both the NN and the JT since applying them):
> > > >
> > > > <property><name>dfs.permissions</name><value>false</value></property>
> > > >
> > > >
> > >
> >
> <property><name>oozie.service.ProxyUserService.proxyuser.hadoop.hosts</name><value>*</value></property>
> > > >
> > > >
> > >
> >
> <property><name>oozie.service.ProxyUserService.proxyuser.hadoop.groups</name><value>*</value></property>
> > > >
> > > > If I recall correctly this authorization is only checked at the the
> > JT/NN
> > > > level and therefore shouldn't need to be pushed to the core site on
> the
> > > > slave machines right? Also, would there be any reason the wildcard
> > would
> > > be
> > > > incompatible across hadoop distros (we are currently using 1.0.3 from
> > > EMR)?
> > > > Lastly, just for the sake of clarity is the proxy hosts config based
> on
> > > the
> > > > box submitting the oozie request (edge node) or based on the boxes
> > > actually
> > > > running the jobs (data/task nodes)?
> > > >
> > > > --
> > > > Matt
> > > >
> > > >
> > > > On Thu, Dec 13, 2012 at 6:19 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> > > > >wrote:
> > > >
> > > > > Matt,
> > > > >
> > > > > It is not matter of bundling native code or not. Officially we
> > suppose
> > > to
> > > > > do source releases only. As convenience we could do binaries, but
> > there
> > > > are
> > > > > discussions about that, if the could be signed or not.
> > > > >
> > > > > Regarding installing/running oozie in EC2. I never done it. Would
> you
> > > > mind
> > > > > writing up a wiki on it once you figure it out?
> > > > >
> > > > > Cheers
> > > > >
> > > > >
> > > > > On Thu, Dec 13, 2012 at 4:02 PM, Matt Goeke <
> goeke.matthew@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Thank you both for the follow-up.
> > > > > >
> > > > > > 2 other questions that pertain to this:
> > > > > > 1) I don't remember any natives being required for Oozie so is
> > there
> > > a
> > > > > > reason why we don't release with a -bin like most other apache
> > > > projects?
> > > > > > 2) Are there any issues I might expect to run into when trying to
> > run
> > > > > this
> > > > > > on EC2 backed by EMR?
> > > > > >
> > > > > > --
> > > > > > Matt
> > > > > >
> > > > > >
> > > > > > On Thu, Dec 13, 2012 at 5:48 PM, Alejandro Abdelnur <
> > > tucu@cloudera.com
> > > > > > >wrote:
> > > > > >
> > > > > > > Matt,
> > > > > > >
> > > > > > > Apache Oozie release artifacts are sources only. The easiest
> way
> > to
> > > > > build
> > > > > > > the TARBALL is:
> > > > > > >
> > > > > > > * install Maven
> > > > > > > * run bin/mkdistro.sh -DskipTests
> > > > > > >
> > > > > > > Then follow the Quick Start instructions.
> > > > > > >
> > > > > > > I'll open a JIRA to add this to the docs.
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Dec 13, 2012 at 3:36 PM, Matt Goeke <
> > > goeke.matthew@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > All,
> > > > > > > >
> > > > > > > > I am falling back to Oozie 3.2 for now but can someone
> possibly
> > > > > explain
> > > > > > > how
> > > > > > > > Oozie 3.3 is supposed to be configured? I was hoping to just
> > > follow
> > > > > the
> > > > > > > > quick start guide but it seems like the packaging does not
> > match
> > > up
> > > > > at
> > > > > > > all.
> > > > > > > >
> > > > > > > > Trying to work through it I ended up downloading maven and
> > > running
> > > > a
> > > > > > 'mvn
> > > > > > > > install' on the folder which built some of the hadooplibs
> but I
> > > am
> > > > > > still
> > > > > > > > missing all of the bin scripts.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Matt
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Alejandro
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Alejandro
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Alejandro
>
Re: Installation and Configuration of Oozie 3.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Hi Mat,
Your 11min delay seems like Oozie is using its fail safe polling mechanism
to detect that a job completed in the cluster.
You may have to tweak the following VAR in the oozie-env.sh script:
# The base URL for callback URLs to Oozie
#
# export OOZIE_BASE_URL="http://
${OOZIE_HTTP_HOSTNAME}:${OOZIE_HTTP_PORT}/oozie"
Oozie sets callback URLs in the jobconfs using this base URL, then the JT
calls back oozie as soon as a job completes. If the hostname is not correct
(see the rest of the script to find out how the hostname is resolved), the
call from the JT will never arrive and Oozie will do a check every 10 mins
for each action.
Hope this solves your prob.
Regarding the wiki, yes, Oozie wiki,
https://cwiki.apache.org/confluence/display/OOZIE , which seems down at the
moment.
Thx
On Thu, Dec 20, 2012 at 9:25 AM, Matt Goeke <go...@gmail.com> wrote:
> All,
>
> I have gotten over all of the blocker configuration hurdles between EC2 and
> EMR and I am able to submit one of my jobs with success. Unfortunately, I
> am running into a weird issue with my actions where each one will take
> exactly 11 mins from start to finish even though the delved MR job is no
> where near that long (1 action is exactly 34 seconds and the other is 5
> mins 21 seconds). I cannot guarantee that this is not an issue with overall
> resources on the node I am running the oozie instance on, I highly doubt it
> since this is on an m1.large spec, so I am curious if there are any changes
> to the site that might be able to flesh out what is causing this issue.
>
>
>
> Alejandro: The actual setup and configuration of this is fairly straight
> forward so I am happy to write up a wiki on this if you guys have a
> specific wiki in mind. I am not sure many people are keen on using EMR as a
> persistent cluster (I assume most persistant clusters are setup across EC2
> nodes) but I am actually very pleased with it so far since it greatly
> reduces the amount of initial setup required to spin up a cluster.
>
> --
> Matt
>
>
> On Tue, Dec 18, 2012 at 5:48 PM, Robert Kanter <rk...@cloudera.com>
> wrote:
>
> > Hi Matt,
> >
> > The oozie.service.ProxyUserService.proxyuser.hadoop.hosts and
> > oozie.service.ProxyUserService.proxyuser.hadoop.groups
> > properties are part of Oozie's configuration and would go in
> > oozie-site.xml. This lets you impersonate users on the Oozie side of
> > things. See
> >
> >
> http://oozie.apache.org/docs/3.3.0/AG_Install.html#User_ProxyUser_Configurationfor
> > more info.
> >
> > There's two similar properties for Hadoop that go into core-site.xml:
> > hadoop.proxyuser.oozie.hosts
> > and hadoop.proxyuser.oozie.groups
> > I think this is what you need to fix your error. See
> > http://hadoop.apache.org/docs/stable/Secure_Impersonation.html for more
> > info.
> >
> > - Robert
> >
> >
> >
> > On Tue, Dec 18, 2012 at 3:38 PM, Matt Goeke <go...@gmail.com>
> > wrote:
> >
> > > All,
> > >
> > > Still working on getting Oozie 3.3 integrated with EMR with most of my
> > time
> > > so far spent resolving the security group config needed for VPC. The
> EC2
> > > configuration was pretty simple but the main blocker right now is
> getting
> > > past the error below:
> > >
> > > Caused by: org.apache.hadoop.ipc.RemoteException: User: hadoop is not
> > > allowed to impersonate hadoop
> > > at org.apache.hadoop.ipc.Client.call(Client.java:1070)
> > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> > > at $Proxy24.getProtocolVersion(Unknown Source)
> > > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
> > > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
> > > at
> > > org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
> > > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
> > > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
> > > at
> > > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
> > > at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
> > > at
> > org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
> > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
> > > at
> > >
> > >
> >
> org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:411)
> > > at
> > >
> > >
> >
> org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:409)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at javax.security.auth.Subject.doAs(Subject.java:415)
> > > at
> > >
> > >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> > > at
> > >
> > >
> >
> org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:409)
> > > ... 26 more
> > >
> > > I know this is usually related to not having the correct proxy configs
> in
> > > the core site but my current core site proxy configs are below (and I
> > have
> > > bounced both the NN and the JT since applying them):
> > >
> > > <property><name>dfs.permissions</name><value>false</value></property>
> > >
> > >
> >
> <property><name>oozie.service.ProxyUserService.proxyuser.hadoop.hosts</name><value>*</value></property>
> > >
> > >
> >
> <property><name>oozie.service.ProxyUserService.proxyuser.hadoop.groups</name><value>*</value></property>
> > >
> > > If I recall correctly this authorization is only checked at the the
> JT/NN
> > > level and therefore shouldn't need to be pushed to the core site on the
> > > slave machines right? Also, would there be any reason the wildcard
> would
> > be
> > > incompatible across hadoop distros (we are currently using 1.0.3 from
> > EMR)?
> > > Lastly, just for the sake of clarity is the proxy hosts config based on
> > the
> > > box submitting the oozie request (edge node) or based on the boxes
> > actually
> > > running the jobs (data/task nodes)?
> > >
> > > --
> > > Matt
> > >
> > >
> > > On Thu, Dec 13, 2012 at 6:19 PM, Alejandro Abdelnur <tucu@cloudera.com
> > > >wrote:
> > >
> > > > Matt,
> > > >
> > > > It is not matter of bundling native code or not. Officially we
> suppose
> > to
> > > > do source releases only. As convenience we could do binaries, but
> there
> > > are
> > > > discussions about that, if the could be signed or not.
> > > >
> > > > Regarding installing/running oozie in EC2. I never done it. Would you
> > > mind
> > > > writing up a wiki on it once you figure it out?
> > > >
> > > > Cheers
> > > >
> > > >
> > > > On Thu, Dec 13, 2012 at 4:02 PM, Matt Goeke <goeke.matthew@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Thank you both for the follow-up.
> > > > >
> > > > > 2 other questions that pertain to this:
> > > > > 1) I don't remember any natives being required for Oozie so is
> there
> > a
> > > > > reason why we don't release with a -bin like most other apache
> > > projects?
> > > > > 2) Are there any issues I might expect to run into when trying to
> run
> > > > this
> > > > > on EC2 backed by EMR?
> > > > >
> > > > > --
> > > > > Matt
> > > > >
> > > > >
> > > > > On Thu, Dec 13, 2012 at 5:48 PM, Alejandro Abdelnur <
> > tucu@cloudera.com
> > > > > >wrote:
> > > > >
> > > > > > Matt,
> > > > > >
> > > > > > Apache Oozie release artifacts are sources only. The easiest way
> to
> > > > build
> > > > > > the TARBALL is:
> > > > > >
> > > > > > * install Maven
> > > > > > * run bin/mkdistro.sh -DskipTests
> > > > > >
> > > > > > Then follow the Quick Start instructions.
> > > > > >
> > > > > > I'll open a JIRA to add this to the docs.
> > > > > >
> > > > > >
> > > > > > On Thu, Dec 13, 2012 at 3:36 PM, Matt Goeke <
> > goeke.matthew@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > All,
> > > > > > >
> > > > > > > I am falling back to Oozie 3.2 for now but can someone possibly
> > > > explain
> > > > > > how
> > > > > > > Oozie 3.3 is supposed to be configured? I was hoping to just
> > follow
> > > > the
> > > > > > > quick start guide but it seems like the packaging does not
> match
> > up
> > > > at
> > > > > > all.
> > > > > > >
> > > > > > > Trying to work through it I ended up downloading maven and
> > running
> > > a
> > > > > 'mvn
> > > > > > > install' on the folder which built some of the hadooplibs but I
> > am
> > > > > still
> > > > > > > missing all of the bin scripts.
> > > > > > >
> > > > > > > --
> > > > > > > Matt
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Alejandro
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Alejandro
> > > >
> > >
> >
>
--
Alejandro
Re: Installation and Configuration of Oozie 3.3
Posted by Matt Goeke <go...@gmail.com>.
All,
I have gotten over all of the blocker configuration hurdles between EC2 and
EMR and I am able to submit one of my jobs with success. Unfortunately, I
am running into a weird issue with my actions where each one will take
exactly 11 mins from start to finish even though the delved MR job is no
where near that long (1 action is exactly 34 seconds and the other is 5
mins 21 seconds). I cannot guarantee that this is not an issue with overall
resources on the node I am running the oozie instance on, I highly doubt it
since this is on an m1.large spec, so I am curious if there are any changes
to the site that might be able to flesh out what is causing this issue.
Alejandro: The actual setup and configuration of this is fairly straight
forward so I am happy to write up a wiki on this if you guys have a
specific wiki in mind. I am not sure many people are keen on using EMR as a
persistent cluster (I assume most persistant clusters are setup across EC2
nodes) but I am actually very pleased with it so far since it greatly
reduces the amount of initial setup required to spin up a cluster.
--
Matt
On Tue, Dec 18, 2012 at 5:48 PM, Robert Kanter <rk...@cloudera.com> wrote:
> Hi Matt,
>
> The oozie.service.ProxyUserService.proxyuser.hadoop.hosts and
> oozie.service.ProxyUserService.proxyuser.hadoop.groups
> properties are part of Oozie's configuration and would go in
> oozie-site.xml. This lets you impersonate users on the Oozie side of
> things. See
>
> http://oozie.apache.org/docs/3.3.0/AG_Install.html#User_ProxyUser_Configurationfor
> more info.
>
> There's two similar properties for Hadoop that go into core-site.xml:
> hadoop.proxyuser.oozie.hosts
> and hadoop.proxyuser.oozie.groups
> I think this is what you need to fix your error. See
> http://hadoop.apache.org/docs/stable/Secure_Impersonation.html for more
> info.
>
> - Robert
>
>
>
> On Tue, Dec 18, 2012 at 3:38 PM, Matt Goeke <go...@gmail.com>
> wrote:
>
> > All,
> >
> > Still working on getting Oozie 3.3 integrated with EMR with most of my
> time
> > so far spent resolving the security group config needed for VPC. The EC2
> > configuration was pretty simple but the main blocker right now is getting
> > past the error below:
> >
> > Caused by: org.apache.hadoop.ipc.RemoteException: User: hadoop is not
> > allowed to impersonate hadoop
> > at org.apache.hadoop.ipc.Client.call(Client.java:1070)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> > at $Proxy24.getProtocolVersion(Unknown Source)
> > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
> > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
> > at
> > org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
> > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
> > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
> > at
> >
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
> > at
> > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
> > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
> > at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
> > at
> >
> >
> org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:411)
> > at
> >
> >
> org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:409)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
> >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> > at
> >
> >
> org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:409)
> > ... 26 more
> >
> > I know this is usually related to not having the correct proxy configs in
> > the core site but my current core site proxy configs are below (and I
> have
> > bounced both the NN and the JT since applying them):
> >
> > <property><name>dfs.permissions</name><value>false</value></property>
> >
> >
> <property><name>oozie.service.ProxyUserService.proxyuser.hadoop.hosts</name><value>*</value></property>
> >
> >
> <property><name>oozie.service.ProxyUserService.proxyuser.hadoop.groups</name><value>*</value></property>
> >
> > If I recall correctly this authorization is only checked at the the JT/NN
> > level and therefore shouldn't need to be pushed to the core site on the
> > slave machines right? Also, would there be any reason the wildcard would
> be
> > incompatible across hadoop distros (we are currently using 1.0.3 from
> EMR)?
> > Lastly, just for the sake of clarity is the proxy hosts config based on
> the
> > box submitting the oozie request (edge node) or based on the boxes
> actually
> > running the jobs (data/task nodes)?
> >
> > --
> > Matt
> >
> >
> > On Thu, Dec 13, 2012 at 6:19 PM, Alejandro Abdelnur <tucu@cloudera.com
> > >wrote:
> >
> > > Matt,
> > >
> > > It is not matter of bundling native code or not. Officially we suppose
> to
> > > do source releases only. As convenience we could do binaries, but there
> > are
> > > discussions about that, if the could be signed or not.
> > >
> > > Regarding installing/running oozie in EC2. I never done it. Would you
> > mind
> > > writing up a wiki on it once you figure it out?
> > >
> > > Cheers
> > >
> > >
> > > On Thu, Dec 13, 2012 at 4:02 PM, Matt Goeke <go...@gmail.com>
> > > wrote:
> > >
> > > > Thank you both for the follow-up.
> > > >
> > > > 2 other questions that pertain to this:
> > > > 1) I don't remember any natives being required for Oozie so is there
> a
> > > > reason why we don't release with a -bin like most other apache
> > projects?
> > > > 2) Are there any issues I might expect to run into when trying to run
> > > this
> > > > on EC2 backed by EMR?
> > > >
> > > > --
> > > > Matt
> > > >
> > > >
> > > > On Thu, Dec 13, 2012 at 5:48 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> > > > >wrote:
> > > >
> > > > > Matt,
> > > > >
> > > > > Apache Oozie release artifacts are sources only. The easiest way to
> > > build
> > > > > the TARBALL is:
> > > > >
> > > > > * install Maven
> > > > > * run bin/mkdistro.sh -DskipTests
> > > > >
> > > > > Then follow the Quick Start instructions.
> > > > >
> > > > > I'll open a JIRA to add this to the docs.
> > > > >
> > > > >
> > > > > On Thu, Dec 13, 2012 at 3:36 PM, Matt Goeke <
> goeke.matthew@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > All,
> > > > > >
> > > > > > I am falling back to Oozie 3.2 for now but can someone possibly
> > > explain
> > > > > how
> > > > > > Oozie 3.3 is supposed to be configured? I was hoping to just
> follow
> > > the
> > > > > > quick start guide but it seems like the packaging does not match
> up
> > > at
> > > > > all.
> > > > > >
> > > > > > Trying to work through it I ended up downloading maven and
> running
> > a
> > > > 'mvn
> > > > > > install' on the folder which built some of the hadooplibs but I
> am
> > > > still
> > > > > > missing all of the bin scripts.
> > > > > >
> > > > > > --
> > > > > > Matt
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Alejandro
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Alejandro
> > >
> >
>
Re: Installation and Configuration of Oozie 3.3
Posted by Robert Kanter <rk...@cloudera.com>.
Hi Matt,
The oozie.service.ProxyUserService.proxyuser.hadoop.hosts and
oozie.service.ProxyUserService.proxyuser.hadoop.groups
properties are part of Oozie's configuration and would go in
oozie-site.xml. This lets you impersonate users on the Oozie side of
things. See
http://oozie.apache.org/docs/3.3.0/AG_Install.html#User_ProxyUser_Configurationfor
more info.
There's two similar properties for Hadoop that go into core-site.xml:
hadoop.proxyuser.oozie.hosts
and hadoop.proxyuser.oozie.groups
I think this is what you need to fix your error. See
http://hadoop.apache.org/docs/stable/Secure_Impersonation.html for more
info.
- Robert
On Tue, Dec 18, 2012 at 3:38 PM, Matt Goeke <go...@gmail.com> wrote:
> All,
>
> Still working on getting Oozie 3.3 integrated with EMR with most of my time
> so far spent resolving the security group config needed for VPC. The EC2
> configuration was pretty simple but the main blocker right now is getting
> past the error below:
>
> Caused by: org.apache.hadoop.ipc.RemoteException: User: hadoop is not
> allowed to impersonate hadoop
> at org.apache.hadoop.ipc.Client.call(Client.java:1070)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> at $Proxy24.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
> at
> org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
> at
>
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
> at
>
> org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:411)
> at
>
> org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:409)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at
>
> org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:409)
> ... 26 more
>
> I know this is usually related to not having the correct proxy configs in
> the core site but my current core site proxy configs are below (and I have
> bounced both the NN and the JT since applying them):
>
> <property><name>dfs.permissions</name><value>false</value></property>
>
> <property><name>oozie.service.ProxyUserService.proxyuser.hadoop.hosts</name><value>*</value></property>
>
> <property><name>oozie.service.ProxyUserService.proxyuser.hadoop.groups</name><value>*</value></property>
>
> If I recall correctly this authorization is only checked at the the JT/NN
> level and therefore shouldn't need to be pushed to the core site on the
> slave machines right? Also, would there be any reason the wildcard would be
> incompatible across hadoop distros (we are currently using 1.0.3 from EMR)?
> Lastly, just for the sake of clarity is the proxy hosts config based on the
> box submitting the oozie request (edge node) or based on the boxes actually
> running the jobs (data/task nodes)?
>
> --
> Matt
>
>
> On Thu, Dec 13, 2012 at 6:19 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > Matt,
> >
> > It is not matter of bundling native code or not. Officially we suppose to
> > do source releases only. As convenience we could do binaries, but there
> are
> > discussions about that, if the could be signed or not.
> >
> > Regarding installing/running oozie in EC2. I never done it. Would you
> mind
> > writing up a wiki on it once you figure it out?
> >
> > Cheers
> >
> >
> > On Thu, Dec 13, 2012 at 4:02 PM, Matt Goeke <go...@gmail.com>
> > wrote:
> >
> > > Thank you both for the follow-up.
> > >
> > > 2 other questions that pertain to this:
> > > 1) I don't remember any natives being required for Oozie so is there a
> > > reason why we don't release with a -bin like most other apache
> projects?
> > > 2) Are there any issues I might expect to run into when trying to run
> > this
> > > on EC2 backed by EMR?
> > >
> > > --
> > > Matt
> > >
> > >
> > > On Thu, Dec 13, 2012 at 5:48 PM, Alejandro Abdelnur <tucu@cloudera.com
> > > >wrote:
> > >
> > > > Matt,
> > > >
> > > > Apache Oozie release artifacts are sources only. The easiest way to
> > build
> > > > the TARBALL is:
> > > >
> > > > * install Maven
> > > > * run bin/mkdistro.sh -DskipTests
> > > >
> > > > Then follow the Quick Start instructions.
> > > >
> > > > I'll open a JIRA to add this to the docs.
> > > >
> > > >
> > > > On Thu, Dec 13, 2012 at 3:36 PM, Matt Goeke <goeke.matthew@gmail.com
> >
> > > > wrote:
> > > >
> > > > > All,
> > > > >
> > > > > I am falling back to Oozie 3.2 for now but can someone possibly
> > explain
> > > > how
> > > > > Oozie 3.3 is supposed to be configured? I was hoping to just follow
> > the
> > > > > quick start guide but it seems like the packaging does not match up
> > at
> > > > all.
> > > > >
> > > > > Trying to work through it I ended up downloading maven and running
> a
> > > 'mvn
> > > > > install' on the folder which built some of the hadooplibs but I am
> > > still
> > > > > missing all of the bin scripts.
> > > > >
> > > > > --
> > > > > Matt
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Alejandro
> > > >
> > >
> >
> >
> >
> > --
> > Alejandro
> >
>
Re: Installation and Configuration of Oozie 3.3
Posted by Matt Goeke <go...@gmail.com>.
All,
Still working on getting Oozie 3.3 integrated with EMR with most of my time
so far spent resolving the security group config needed for VPC. The EC2
configuration was pretty simple but the main blocker right now is getting
past the error below:
Caused by: org.apache.hadoop.ipc.RemoteException: User: hadoop is not
allowed to impersonate hadoop
at org.apache.hadoop.ipc.Client.call(Client.java:1070)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at $Proxy24.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at
org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
at
org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:411)
at
org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:409)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at
org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:409)
... 26 more
I know this is usually related to not having the correct proxy configs in
the core site but my current core site proxy configs are below (and I have
bounced both the NN and the JT since applying them):
<property><name>dfs.permissions</name><value>false</value></property>
<property><name>oozie.service.ProxyUserService.proxyuser.hadoop.hosts</name><value>*</value></property>
<property><name>oozie.service.ProxyUserService.proxyuser.hadoop.groups</name><value>*</value></property>
If I recall correctly this authorization is only checked at the the JT/NN
level and therefore shouldn't need to be pushed to the core site on the
slave machines right? Also, would there be any reason the wildcard would be
incompatible across hadoop distros (we are currently using 1.0.3 from EMR)?
Lastly, just for the sake of clarity is the proxy hosts config based on the
box submitting the oozie request (edge node) or based on the boxes actually
running the jobs (data/task nodes)?
--
Matt
On Thu, Dec 13, 2012 at 6:19 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Matt,
>
> It is not matter of bundling native code or not. Officially we suppose to
> do source releases only. As convenience we could do binaries, but there are
> discussions about that, if the could be signed or not.
>
> Regarding installing/running oozie in EC2. I never done it. Would you mind
> writing up a wiki on it once you figure it out?
>
> Cheers
>
>
> On Thu, Dec 13, 2012 at 4:02 PM, Matt Goeke <go...@gmail.com>
> wrote:
>
> > Thank you both for the follow-up.
> >
> > 2 other questions that pertain to this:
> > 1) I don't remember any natives being required for Oozie so is there a
> > reason why we don't release with a -bin like most other apache projects?
> > 2) Are there any issues I might expect to run into when trying to run
> this
> > on EC2 backed by EMR?
> >
> > --
> > Matt
> >
> >
> > On Thu, Dec 13, 2012 at 5:48 PM, Alejandro Abdelnur <tucu@cloudera.com
> > >wrote:
> >
> > > Matt,
> > >
> > > Apache Oozie release artifacts are sources only. The easiest way to
> build
> > > the TARBALL is:
> > >
> > > * install Maven
> > > * run bin/mkdistro.sh -DskipTests
> > >
> > > Then follow the Quick Start instructions.
> > >
> > > I'll open a JIRA to add this to the docs.
> > >
> > >
> > > On Thu, Dec 13, 2012 at 3:36 PM, Matt Goeke <go...@gmail.com>
> > > wrote:
> > >
> > > > All,
> > > >
> > > > I am falling back to Oozie 3.2 for now but can someone possibly
> explain
> > > how
> > > > Oozie 3.3 is supposed to be configured? I was hoping to just follow
> the
> > > > quick start guide but it seems like the packaging does not match up
> at
> > > all.
> > > >
> > > > Trying to work through it I ended up downloading maven and running a
> > 'mvn
> > > > install' on the folder which built some of the hadooplibs but I am
> > still
> > > > missing all of the bin scripts.
> > > >
> > > > --
> > > > Matt
> > > >
> > >
> > >
> > >
> > > --
> > > Alejandro
> > >
> >
>
>
>
> --
> Alejandro
>
Re: Installation and Configuration of Oozie 3.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Matt,
It is not matter of bundling native code or not. Officially we suppose to
do source releases only. As convenience we could do binaries, but there are
discussions about that, if the could be signed or not.
Regarding installing/running oozie in EC2. I never done it. Would you mind
writing up a wiki on it once you figure it out?
Cheers
On Thu, Dec 13, 2012 at 4:02 PM, Matt Goeke <go...@gmail.com> wrote:
> Thank you both for the follow-up.
>
> 2 other questions that pertain to this:
> 1) I don't remember any natives being required for Oozie so is there a
> reason why we don't release with a -bin like most other apache projects?
> 2) Are there any issues I might expect to run into when trying to run this
> on EC2 backed by EMR?
>
> --
> Matt
>
>
> On Thu, Dec 13, 2012 at 5:48 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > Matt,
> >
> > Apache Oozie release artifacts are sources only. The easiest way to build
> > the TARBALL is:
> >
> > * install Maven
> > * run bin/mkdistro.sh -DskipTests
> >
> > Then follow the Quick Start instructions.
> >
> > I'll open a JIRA to add this to the docs.
> >
> >
> > On Thu, Dec 13, 2012 at 3:36 PM, Matt Goeke <go...@gmail.com>
> > wrote:
> >
> > > All,
> > >
> > > I am falling back to Oozie 3.2 for now but can someone possibly explain
> > how
> > > Oozie 3.3 is supposed to be configured? I was hoping to just follow the
> > > quick start guide but it seems like the packaging does not match up at
> > all.
> > >
> > > Trying to work through it I ended up downloading maven and running a
> 'mvn
> > > install' on the folder which built some of the hadooplibs but I am
> still
> > > missing all of the bin scripts.
> > >
> > > --
> > > Matt
> > >
> >
> >
> >
> > --
> > Alejandro
> >
>
--
Alejandro
Re: Installation and Configuration of Oozie 3.3
Posted by Matt Goeke <go...@gmail.com>.
Thank you both for the follow-up.
2 other questions that pertain to this:
1) I don't remember any natives being required for Oozie so is there a
reason why we don't release with a -bin like most other apache projects?
2) Are there any issues I might expect to run into when trying to run this
on EC2 backed by EMR?
--
Matt
On Thu, Dec 13, 2012 at 5:48 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Matt,
>
> Apache Oozie release artifacts are sources only. The easiest way to build
> the TARBALL is:
>
> * install Maven
> * run bin/mkdistro.sh -DskipTests
>
> Then follow the Quick Start instructions.
>
> I'll open a JIRA to add this to the docs.
>
>
> On Thu, Dec 13, 2012 at 3:36 PM, Matt Goeke <go...@gmail.com>
> wrote:
>
> > All,
> >
> > I am falling back to Oozie 3.2 for now but can someone possibly explain
> how
> > Oozie 3.3 is supposed to be configured? I was hoping to just follow the
> > quick start guide but it seems like the packaging does not match up at
> all.
> >
> > Trying to work through it I ended up downloading maven and running a 'mvn
> > install' on the folder which built some of the hadooplibs but I am still
> > missing all of the bin scripts.
> >
> > --
> > Matt
> >
>
>
>
> --
> Alejandro
>
Re: Installation and Configuration of Oozie 3.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Matt,
Apache Oozie release artifacts are sources only. The easiest way to build
the TARBALL is:
* install Maven
* run bin/mkdistro.sh -DskipTests
Then follow the Quick Start instructions.
I'll open a JIRA to add this to the docs.
On Thu, Dec 13, 2012 at 3:36 PM, Matt Goeke <go...@gmail.com> wrote:
> All,
>
> I am falling back to Oozie 3.2 for now but can someone possibly explain how
> Oozie 3.3 is supposed to be configured? I was hoping to just follow the
> quick start guide but it seems like the packaging does not match up at all.
>
> Trying to work through it I ended up downloading maven and running a 'mvn
> install' on the folder which built some of the hadooplibs but I am still
> missing all of the bin scripts.
>
> --
> Matt
>
--
Alejandro