You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by Siddharth Tiwari <si...@live.com> on 2015/08/07 09:13:52 UTC
Installation procedure for Faclon for Apache hadoop
Hi Team,is there any documentation available for installing and configuring falcon for stock vanila Apache hadoop ? It fairly easier fro HDP using Ambari, but the procedure is not very clear for installing it with Stock Apache hadoop. Any help in this regard will be highly appreciated. Thank you in advanced.
*------------------------*
Cheers !!!
Siddharth Tiwari
Have a refreshing day !!!
"Every duty is holy, and devotion to duty is the highest form of worship of God.”
"Maybe other people will try to limit me but I don't limit myself"
Re: Installation procedure for Faclon for Apache hadoop
Posted by Siddharth Tiwari <si...@live.com>.
Thanks Pallavi, i still had the issue, will look into the jira and patch up.
Sent from my iPhone
> On Aug 19, 2015, at 11:53 PM, Pallavi Rao <pa...@inmobi.com> wrote:
>
> Siddharth,
> Not sure if you were able to resolve the issue, we realized there was issue with EL expression resolution on Apache Oozie installation. Hence, the issue you noticed. Here is the JIRA with details and patch.
>
> Hope this helps,
> Pallavi
>
>> On Thu, Aug 13, 2015 at 9:12 AM, Pallavi Rao <pa...@inmobi.com> wrote:
>> Siddharth,
>> The only reason that error could occur is when the EL extension of Falcon is not getting picked up by Oozie. Can you double check the following:
>> 1. oozie-site.xml has the extensions. You must have the following lines:
>> dataIn=org.apache.oozie.extensions.OozieELExtensions#ph1_dataIn_echo and
>>
>> dataIn=org.apache.oozie.extensions.OozieELExtensions#ph3_dataIn,
>>
>> 2. You must have the following file in the oozie installation path :
>>
>> oozie-server/webapps/oozie/WEB-INF/lib/falcon-oozie-el-extension-0.7-SNAPSHOT.jar. It must have read all permission.
>>
>> 3. Hope you have restarted Oozie.
>>
>> If you are seeing errors even after ensuring the above. Please mail across the Oozie logs, please.
>>
>> Regards,
>>
>> Pallavi
>>
>>
>>> On Sun, Aug 9, 2015 at 10:52 AM, Siddharth Tiwari <si...@live.com> wrote:
>>> Hi Pallavi
>>> I made the changes you suggested, I am still getting the following error
>>>
>>> Caused by: org.apache.falcon.FalconException: E1004 : E1004: Expression language evaluation error, Unable to evaluate :${dataIn('input', 'null')}:
>>> at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:223)
>>> at org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:171)
>>> at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:93)
>>> ... 60 more
>>> Caused by: E1004 : E1004: Expression language evaluation error, Unable to evaluate :${dataIn('input', 'null')}:
>>> at org.apache.oozie.client.OozieClient.handleError(OozieClient.java:542)
>>> at org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:625)
>>> at org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:595)
>>> at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:514)
>>> at org.apache.oozie.client.OozieClient.dryrun(OozieClient.java:735)
>>> at org.apache.oozie.client.ProxyOozieClient.access$401(ProxyOozieClient.java:48)
>>> at org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:177)
>>> at org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:174)
>>> at org.apache.oozie.client.OozieClient.doAs(OozieClient.java:198)
>>> at org.apache.oozie.client.ProxyOozieClient.dryrun(ProxyOozieClient.java:174)
>>> at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:221)
>>>
>>> The process I am trying to schedule is the one below:-
>>>
>>>
>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <!--
>>> Hourly process, cleanses raw data
>>> -->
>>> <process name="cleanseEmailProcess" xmlns="uri:falcon:process:0.1">
>>>
>>> <tags>pipeline=churnAnalysisDataPipeline,owner=ETLGroup</tags>
>>>
>>>
>>> <clusters>
>>> <cluster name="primaryCluster">
>>> <validity start="2014-02-28T00:00Z" end="2016-03-31T00:00Z"/>
>>> </cluster>
>>> </clusters>
>>>
>>> <parallel>1</parallel>
>>> <order>FIFO</order>
>>> <frequency>hours(1)</frequency>
>>>
>>> <inputs>
>>> <input name="input" feed="rawEmailFeed" start="now(0,0)" end="now(0,0)" />
>>> </inputs>
>>>
>>> <outputs>
>>> <output name="output" feed="cleansedEmailFeed" instance="now(0,0)" />
>>> </outputs>
>>>
>>> <workflow name="emailCleanseWorkflow" version="5.0"
>>> engine="pig" path="/user/ambari-qa/falcon/demo/apps/pig/id.pig" />
>>>
>>> <retry policy="periodic" delay="minutes(15)" attempts="3" />
>>> <!-- <late-process policy="exp-backoff" delay="hours(1)">
>>> <late-input input="input" workflow-path="/apps/clickstream/late" />
>>> </late-process> -->
>>>
>>> </process>
>>>
>>>
>>>
>>>
>>> *------------------------*
>>> Cheers !!!
>>> Siddharth Tiwari
>>> Have a refreshing day !!!
>>> "Every duty is holy, and devotion to duty is the highest form of worship of God.”
>>> "Maybe other people will try to limit me but I don't limit myself"
>>>
>>>
>>> > Date: Fri, 7 Aug 2015 14:03:50 +0530
>>> > Subject: Re: Installation procedure for Faclon for Apache hadoop
>>> > From: pallavi.rao@inmobi.com
>>> > To: dev@falcon.apache.org
>>> > CC: sriksun@hotmail.com
>>> >
>>> > Hey Siddharth,
>>> > If you are not using Oozie distro that is packaged during falcon build,
>>> > ensure the following is Oozie changes are done before you start Falcon
>>> > server.
>>> >
>>> > 1.
>>> >
>>> > Copy <falcon base
>>> > dir>/oozie/libext/falcon-oozie-el-extension-0.7-incubating-SNAPSHOT.jar
>>> > to
>>> > <oozie base dir>/libext.
>>> > 2.
>>> >
>>> > Modify oozie site xml. Copy all EL related properties from <falcon
>>> > install dir>oozie/conf/oozie-site.xml to <oozie base
>>> > dir>/conf/oozie-site.xml
>>> > 3.
>>> >
>>> > Restart oozie :
>>> >
>>> >
>>> > -
>>> >
>>> > sudo -u oozie ./bin/oozie-start.sh
>>> > -
>>> >
>>> > sudo -u oozie bin/oozie-setup.sh prepare-war
>>> > -
>>> >
>>> > sudo -u oozie ./bin/oozie-start.sh
>>> >
>>> > Just realized we have missed this information in our installation docs.
>>> > Have opened JIRA <https://issues.apache.org/jira/browse/FALCON-1378> for
>>>
>>> > the same.
>>> >
>>> > I assume that you have information on how to setup up a standalone Oozie
>>> > instance. If not, let me know, I can pass along the same to you.
>>> >
>>> > Regards,
>>> > Pallavi
>>> >
>>> > On Fri, Aug 7, 2015 at 1:33 PM, Siddharth Tiwari <si...@live.com>
>>> > wrote:
>>> >
>>> > > Hi Srikanth
>>> > > I am able to build it successfully in embedded mode, the issue is
>>> > > primarily interacting with oozie and setting up oozie to interact with
>>> > > Falcon, are there any particular steps to setup oozie for Falcon ? Secondly
>>> > > is the version packed with HDP different from the one available in the git
>>> > > ? Because I cannot see the xml forms setup on the gui.
>>> > >
>>> > > Sent from my iPhone
>>> > >
>>> > > > On Aug 7, 2015, at 12:41 AM, Srikanth Sundarrajan <sr...@hotmail.com>
>>> > > wrote:
>>> > > >
>>> > > > HI Siddharth,
>>> > > > It should be fairly straight forward to use Falcon with hadoop 2.5.0 and
>>> > > above. Standard installation instructions should suffice. Is there a
>>> > > particular step where you are stuck in getting it setup ?
>>> > > >
>>> > > > Regards
>>> > > > Srikanth Sundarrajan
>>> > > >
>>> > > >> From: siddharth.tiwari@live.com
>>> > > >> To: dev@falcon.apache.org
>>> > > >> Subject: Installation procedure for Faclon for Apache hadoop
>>> > > >> Date: Fri, 7 Aug 2015 07:13:52 +0000
>>> > > >>
>>> > > >> Hi Team,is there any documentation available for installing and
>>> > > configuring falcon for stock vanila Apache hadoop ? It fairly easier fro
>>> > > HDP using Ambari, but the procedure is not very clear for installing it
>>> > > with Stock Apache hadoop. Any help in this regard will be highly
>>> > > appreciated. Thank you in advanced.
>>> > > >>
>>> > > >> *------------------------*
>>> > > >>
>>> > > >> Cheers !!!
>>> > > >>
>>> > > >> Siddharth Tiwari
>>> > > >>
>>> > > >> Have a refreshing day !!!
>>> > > >> "Every duty is holy, and devotion to duty is the highest form of
>>> > > worship of God.”
>>> > > >>
>>> > > >> "Maybe other people will try to limit me but I don't limit myself"
>>> > > >
>>> > >
>>> >
>>> > --
>>> > _____________________________________________________________
>>> > The information contained in this communication is intended solely for the
>>> > use of the individual or entity to whom it is addressed and others
>>> > authorized to receive it. It may contain confidential or legally privileged
>>> > information. If you are not the intended recipient you are hereby notified
>>> > that any disclosure, copying, distribution or taking any action in reliance
>>> > on the contents of this information is strictly prohibited and may be
>>> > unlawful. If you have received this communication in error, please notify
>>> > us immediately by responding to this email and then delete it from your
>>> > system. The firm is neither liable for the proper and complete transmission
>>> > of the information contained in this communication nor for any delay in its
>>> > receipt.
>
>
> _____________________________________________________________
> The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
Re: Installation procedure for Faclon for Apache hadoop
Posted by Pallavi Rao <pa...@inmobi.com>.
Siddharth,
Not sure if you were able to resolve the issue, we realized there was issue
with EL expression resolution on Apache Oozie installation. Hence, the
issue you noticed. Here is the JIRA
<https://issues.apache.org/jira/browse/FALCON-1412> with details and patch.
Hope this helps,
Pallavi
On Thu, Aug 13, 2015 at 9:12 AM, Pallavi Rao <pa...@inmobi.com> wrote:
> Siddharth,
> The only reason that error could occur is when the EL extension of Falcon
> is not getting picked up by Oozie. Can you double check the following:
> 1. oozie-site.xml has the extensions. You must have the following lines:
>
> dataIn=org.apache.oozie.extensions.OozieELExtensions#ph1_dataIn_echo and
>
> dataIn=org.apache.oozie.extensions.OozieELExtensions#ph3_dataIn,
>
> 2. You must have the following file in the oozie installation path :
>
> oozie-server/webapps/oozie/WEB-INF/lib/falcon-oozie-el-extension-0.7-SNAPSHOT.jar.
> It must have read all permission.
>
> 3. Hope you have restarted Oozie.
>
> If you are seeing errors even after ensuring the above. Please mail across
> the Oozie logs, please.
>
> Regards,
>
> Pallavi
>
> On Sun, Aug 9, 2015 at 10:52 AM, Siddharth Tiwari <
> siddharth.tiwari@live.com> wrote:
>
>> Hi Pallavi
>> I made the changes you suggested, I am still getting the following error
>>
>> Caused by: org.apache.falcon.FalconException: E1004 : E1004: Expression
>> language evaluation error, Unable to evaluate :${dataIn('input', 'null')}:
>> at
>> org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:223)
>> at
>> org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:171)
>> at
>> org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:93)
>> ... 60 more
>> Caused by: E1004 : E1004: Expression language evaluation error, Unable to
>> evaluate :${dataIn('input', 'null')}:
>> at
>> org.apache.oozie.client.OozieClient.handleError(OozieClient.java:542)
>> at
>> org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:625)
>> at
>> org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:595)
>> at
>> org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:514)
>> at
>> org.apache.oozie.client.OozieClient.dryrun(OozieClient.java:735)
>> at
>> org.apache.oozie.client.ProxyOozieClient.access$401(ProxyOozieClient.java:48)
>> at
>> org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:177)
>> at
>> org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:174)
>> at org.apache.oozie.client.OozieClient.doAs(OozieClient.java:198)
>> at
>> org.apache.oozie.client.ProxyOozieClient.dryrun(ProxyOozieClient.java:174)
>> at
>> org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:221)
>>
>> The process I am trying to schedule is the one below:-
>>
>>
>> <?xml version="1.0" encoding="UTF-8"?>
>> <!--
>> Hourly process, cleanses raw data
>> -->
>> <process name="cleanseEmailProcess" xmlns="uri:falcon:process:0.1">
>>
>> <tags>pipeline=churnAnalysisDataPipeline,owner=ETLGroup</tags>
>>
>>
>> <clusters>
>> <cluster name="primaryCluster">
>> <validity start="2014-02-28T00:00Z" end="2016-03-31T00:00Z"/>
>> </cluster>
>> </clusters>
>>
>> <parallel>1</parallel>
>> <order>FIFO</order>
>> <frequency>hours(1)</frequency>
>>
>> <inputs>
>> <input name="input" feed="rawEmailFeed" start="now(0,0)"
>> end="now(0,0)" />
>> </inputs>
>>
>> <outputs>
>> <output name="output" feed="cleansedEmailFeed"
>> instance="now(0,0)" />
>> </outputs>
>>
>> <workflow name="emailCleanseWorkflow" version="5.0"
>> engine="pig" path="/user/ambari-qa/falcon/demo/apps/pig/id.pig" />
>>
>> <retry policy="periodic" delay="minutes(15)" attempts="3" />
>> <!-- <late-process policy="exp-backoff" delay="hours(1)">
>> <late-input input="input" workflow-path="/apps/clickstream/late" />
>> </late-process> -->
>>
>> </process>
>>
>>
>>
>>
>> **------------------------**
>> *Cheers !!!*
>> *Siddharth Tiwari*
>> Have a refreshing day !!!
>> *"Every duty is holy, and devotion to duty is the highest form of worship
>> of God.” *
>> *"Maybe other people will try to limit me but I don't limit myself"*
>>
>>
>> > Date: Fri, 7 Aug 2015 14:03:50 +0530
>> > Subject: Re: Installation procedure for Faclon for Apache hadoop
>> > From: pallavi.rao@inmobi.com
>> > To: dev@falcon.apache.org
>> > CC: sriksun@hotmail.com
>> >
>> > Hey Siddharth,
>> > If you are not using Oozie distro that is packaged during falcon build,
>> > ensure the following is Oozie changes are done before you start Falcon
>> > server.
>> >
>> > 1.
>> >
>> > Copy <falcon base
>> > dir>/oozie/libext/falcon-oozie-el-extension-0.7-incubating-SNAPSHOT.jar
>> > to
>> > <oozie base dir>/libext.
>> > 2.
>> >
>> > Modify oozie site xml. Copy all EL related properties from <falcon
>> > install dir>oozie/conf/oozie-site.xml to <oozie base
>> > dir>/conf/oozie-site.xml
>> > 3.
>> >
>> > Restart oozie :
>> >
>> >
>> > -
>> >
>> > sudo -u oozie ./bin/oozie-start.sh
>> > -
>> >
>> > sudo -u oozie bin/oozie-setup.sh prepare-war
>> > -
>> >
>> > sudo -u oozie ./bin/oozie-start.sh
>> >
>> > Just realized we have missed this information in our installation docs.
>> > Have opened JIRA <https://issues.apache.org/jira/browse/FALCON-1378>
>> for
>>
>> > the same.
>> >
>> > I assume that you have information on how to setup up a standalone Oozie
>> > instance. If not, let me know, I can pass along the same to you.
>> >
>> > Regards,
>> > Pallavi
>> >
>> > On Fri, Aug 7, 2015 at 1:33 PM, Siddharth Tiwari <
>> siddharth.tiwari@live.com>
>> > wrote:
>> >
>> > > Hi Srikanth
>> > > I am able to build it successfully in embedded mode, the issue is
>> > > primarily interacting with oozie and setting up oozie to interact with
>> > > Falcon, are there any particular steps to setup oozie for Falcon ?
>> Secondly
>> > > is the version packed with HDP different from the one available in
>> the git
>> > > ? Because I cannot see the xml forms setup on the gui.
>> > >
>> > > Sent from my iPhone
>> > >
>> > > > On Aug 7, 2015, at 12:41 AM, Srikanth Sundarrajan <
>> sriksun@hotmail.com>
>> > > wrote:
>> > > >
>> > > > HI Siddharth,
>> > > > It should be fairly straight forward to use Falcon with hadoop
>> 2.5.0 and
>> > > above. Standard installation instructions should suffice. Is there a
>> > > particular step where you are stuck in getting it setup ?
>> > > >
>> > > > Regards
>> > > > Srikanth Sundarrajan
>> > > >
>> > > >> From: siddharth.tiwari@live.com
>> > > >> To: dev@falcon.apache.org
>> > > >> Subject: Installation procedure for Faclon for Apache hadoop
>> > > >> Date: Fri, 7 Aug 2015 07:13:52 +0000
>> > > >>
>> > > >> Hi Team,is there any documentation available for installing and
>> > > configuring falcon for stock vanila Apache hadoop ? It fairly easier
>> fro
>> > > HDP using Ambari, but the procedure is not very clear for installing
>> it
>> > > with Stock Apache hadoop. Any help in this regard will be highly
>> > > appreciated. Thank you in advanced.
>> > > >>
>> > > >> *------------------------*
>> > > >>
>> > > >> Cheers !!!
>> > > >>
>> > > >> Siddharth Tiwari
>> > > >>
>> > > >> Have a refreshing day !!!
>> > > >> "Every duty is holy, and devotion to duty is the highest form of
>> > > worship of God.”
>> > > >>
>> > > >> "Maybe other people will try to limit me but I don't limit myself"
>> > > >
>> > >
>> >
>> > --
>> > _____________________________________________________________
>> > The information contained in this communication is intended solely for
>> the
>> > use of the individual or entity to whom it is addressed and others
>> > authorized to receive it. It may contain confidential or legally
>> privileged
>> > information. If you are not the intended recipient you are hereby
>> notified
>> > that any disclosure, copying, distribution or taking any action in
>> reliance
>> > on the contents of this information is strictly prohibited and may be
>> > unlawful. If you have received this communication in error, please
>> notify
>> > us immediately by responding to this email and then delete it from your
>> > system. The firm is neither liable for the proper and complete
>> transmission
>> > of the information contained in this communication nor for any delay in
>> its
>> > receipt.
>>
>
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
RE: Installation procedure for Faclon for Apache hadoop
Posted by Siddharth Tiwari <si...@live.com>.
Hi PallaviI made the changes you suggested, I am still getting the following error
Caused by: org.apache.falcon.FalconException: E1004 : E1004: Expression language evaluation error, Unable to evaluate :${dataIn('input', 'null')}: at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:223) at org.apache.falcon.workflow.engine.OozieWorkflowEngine.schedule(OozieWorkflowEngine.java:171) at org.apache.falcon.resource.AbstractSchedulableEntityManager.scheduleInternal(AbstractSchedulableEntityManager.java:93) ... 60 moreCaused by: E1004 : E1004: Expression language evaluation error, Unable to evaluate :${dataIn('input', 'null')}: at org.apache.oozie.client.OozieClient.handleError(OozieClient.java:542) at org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:625) at org.apache.oozie.client.OozieClient$JobSubmit.call(OozieClient.java:595) at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:514) at org.apache.oozie.client.OozieClient.dryrun(OozieClient.java:735) at org.apache.oozie.client.ProxyOozieClient.access$401(ProxyOozieClient.java:48) at org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:177) at org.apache.oozie.client.ProxyOozieClient$4.call(ProxyOozieClient.java:174) at org.apache.oozie.client.OozieClient.doAs(OozieClient.java:198) at org.apache.oozie.client.ProxyOozieClient.dryrun(ProxyOozieClient.java:174) at org.apache.falcon.workflow.engine.OozieWorkflowEngine.dryRunInternal(OozieWorkflowEngine.java:221)
The process I am trying to schedule is the one below:-
<?xml version="1.0" encoding="UTF-8"?><!-- Hourly process, cleanses raw data --><process name="cleanseEmailProcess" xmlns="uri:falcon:process:0.1">
<tags>pipeline=churnAnalysisDataPipeline,owner=ETLGroup</tags>
<clusters> <cluster name="primaryCluster"> <validity start="2014-02-28T00:00Z" end="2016-03-31T00:00Z"/> </cluster> </clusters>
<parallel>1</parallel> <order>FIFO</order> <frequency>hours(1)</frequency>
<inputs> <input name="input" feed="rawEmailFeed" start="now(0,0)" end="now(0,0)" /> </inputs>
<outputs> <output name="output" feed="cleansedEmailFeed" instance="now(0,0)" /> </outputs>
<workflow name="emailCleanseWorkflow" version="5.0" engine="pig" path="/user/ambari-qa/falcon/demo/apps/pig/id.pig" />
<retry policy="periodic" delay="minutes(15)" attempts="3" /> <!-- <late-process policy="exp-backoff" delay="hours(1)"> <late-input input="input" workflow-path="/apps/clickstream/late" /> </late-process> -->
</process>
*------------------------*
Cheers !!!
Siddharth Tiwari
Have a refreshing day !!!
"Every duty is holy, and devotion to duty is the highest form of worship of God.”
"Maybe other people will try to limit me but I don't limit myself"
> Date: Fri, 7 Aug 2015 14:03:50 +0530
> Subject: Re: Installation procedure for Faclon for Apache hadoop
> From: pallavi.rao@inmobi.com
> To: dev@falcon.apache.org
> CC: sriksun@hotmail.com
>
> Hey Siddharth,
> If you are not using Oozie distro that is packaged during falcon build,
> ensure the following is Oozie changes are done before you start Falcon
> server.
>
> 1.
>
> Copy <falcon base
> dir>/oozie/libext/falcon-oozie-el-extension-0.7-incubating-SNAPSHOT.jar
> to
> <oozie base dir>/libext.
> 2.
>
> Modify oozie site xml. Copy all EL related properties from <falcon
> install dir>oozie/conf/oozie-site.xml to <oozie base
> dir>/conf/oozie-site.xml
> 3.
>
> Restart oozie :
>
>
> -
>
> sudo -u oozie ./bin/oozie-start.sh
> -
>
> sudo -u oozie bin/oozie-setup.sh prepare-war
> -
>
> sudo -u oozie ./bin/oozie-start.sh
>
> Just realized we have missed this information in our installation docs.
> Have opened JIRA <https://issues.apache.org/jira/browse/FALCON-1378> for
> the same.
>
> I assume that you have information on how to setup up a standalone Oozie
> instance. If not, let me know, I can pass along the same to you.
>
> Regards,
> Pallavi
>
> On Fri, Aug 7, 2015 at 1:33 PM, Siddharth Tiwari <si...@live.com>
> wrote:
>
> > Hi Srikanth
> > I am able to build it successfully in embedded mode, the issue is
> > primarily interacting with oozie and setting up oozie to interact with
> > Falcon, are there any particular steps to setup oozie for Falcon ? Secondly
> > is the version packed with HDP different from the one available in the git
> > ? Because I cannot see the xml forms setup on the gui.
> >
> > Sent from my iPhone
> >
> > > On Aug 7, 2015, at 12:41 AM, Srikanth Sundarrajan <sr...@hotmail.com>
> > wrote:
> > >
> > > HI Siddharth,
> > > It should be fairly straight forward to use Falcon with hadoop 2.5.0 and
> > above. Standard installation instructions should suffice. Is there a
> > particular step where you are stuck in getting it setup ?
> > >
> > > Regards
> > > Srikanth Sundarrajan
> > >
> > >> From: siddharth.tiwari@live.com
> > >> To: dev@falcon.apache.org
> > >> Subject: Installation procedure for Faclon for Apache hadoop
> > >> Date: Fri, 7 Aug 2015 07:13:52 +0000
> > >>
> > >> Hi Team,is there any documentation available for installing and
> > configuring falcon for stock vanila Apache hadoop ? It fairly easier fro
> > HDP using Ambari, but the procedure is not very clear for installing it
> > with Stock Apache hadoop. Any help in this regard will be highly
> > appreciated. Thank you in advanced.
> > >>
> > >> *------------------------*
> > >>
> > >> Cheers !!!
> > >>
> > >> Siddharth Tiwari
> > >>
> > >> Have a refreshing day !!!
> > >> "Every duty is holy, and devotion to duty is the highest form of
> > worship of God.”
> > >>
> > >> "Maybe other people will try to limit me but I don't limit myself"
> > >
> >
>
> --
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
Re: Installation procedure for Faclon for Apache hadoop
Posted by Pallavi Rao <pa...@inmobi.com>.
Siddharth,
Falcon UI is undergoing transformation. There is an old simple UI (which
you see when you hit https://localhost:16443) and a new one (it is really
an alpha release). The new one is probably what is enabled by default in
the HDP release.
As far as I know, HDP bundles a "released" version of Falcon (0.6.1 in this
case). The github code is usually ahead of the released version and less
stable too, as active development will be going on.
Hortonworks folks will be in a better position to clarify.
Regards,
Pallavi
On Fri, Aug 7, 2015 at 2:07 PM, Siddharth Tiwari <si...@live.com>
wrote:
> Hi Pallavi,
> This is great. Thank you so much for prompt response. One more question
> Pallavi, is the version on github differebt from the one being packed with
> HDP 2.3 ? I see a lot of difference between the gui and also the stability.
>
> Sent from my iPhone
>
> > On Aug 7, 2015, at 1:34 AM, Pallavi Rao <pa...@inmobi.com> wrote:
> >
> > Hey Siddharth,
> > If you are not using Oozie distro that is packaged during falcon build,
> > ensure the following is Oozie changes are done before you start Falcon
> > server.
> >
> > 1.
> >
> > Copy <falcon base
> >
> dir>/oozie/libext/falcon-oozie-el-extension-0.7-incubating-SNAPSHOT.jar
> > to
> > <oozie base dir>/libext.
> > 2.
> >
> > Modify oozie site xml. Copy all EL related properties from <falcon
> > install dir>oozie/conf/oozie-site.xml to <oozie base
> > dir>/conf/oozie-site.xml
> > 3.
> >
> > Restart oozie :
> >
> >
> > -
> >
> > sudo -u oozie ./bin/oozie-start.sh
> > -
> >
> > sudo -u oozie bin/oozie-setup.sh prepare-war
> > -
> >
> > sudo -u oozie ./bin/oozie-start.sh
> >
> > Just realized we have missed this information in our installation docs.
> > Have opened JIRA <https://issues.apache.org/jira/browse/FALCON-1378> for
> > the same.
> >
> > I assume that you have information on how to setup up a standalone Oozie
> > instance. If not, let me know, I can pass along the same to you.
> >
> > Regards,
> > Pallavi
> >
> > On Fri, Aug 7, 2015 at 1:33 PM, Siddharth Tiwari <
> siddharth.tiwari@live.com>
> > wrote:
> >
> >> Hi Srikanth
> >> I am able to build it successfully in embedded mode, the issue is
> >> primarily interacting with oozie and setting up oozie to interact with
> >> Falcon, are there any particular steps to setup oozie for Falcon ?
> Secondly
> >> is the version packed with HDP different from the one available in the
> git
> >> ? Because I cannot see the xml forms setup on the gui.
> >>
> >> Sent from my iPhone
> >>
> >>>> On Aug 7, 2015, at 12:41 AM, Srikanth Sundarrajan <
> sriksun@hotmail.com>
> >>> wrote:
> >>>
> >>> HI Siddharth,
> >>> It should be fairly straight forward to use Falcon with hadoop 2.5.0
> and
> >> above. Standard installation instructions should suffice. Is there a
> >> particular step where you are stuck in getting it setup ?
> >>>
> >>> Regards
> >>> Srikanth Sundarrajan
> >>>
> >>>> From: siddharth.tiwari@live.com
> >>>> To: dev@falcon.apache.org
> >>>> Subject: Installation procedure for Faclon for Apache hadoop
> >>>> Date: Fri, 7 Aug 2015 07:13:52 +0000
> >>>>
> >>>> Hi Team,is there any documentation available for installing and
> >> configuring falcon for stock vanila Apache hadoop ? It fairly easier fro
> >> HDP using Ambari, but the procedure is not very clear for installing it
> >> with Stock Apache hadoop. Any help in this regard will be highly
> >> appreciated. Thank you in advanced.
> >>>>
> >>>> *------------------------*
> >>>>
> >>>> Cheers !!!
> >>>>
> >>>> Siddharth Tiwari
> >>>>
> >>>> Have a refreshing day !!!
> >>>> "Every duty is holy, and devotion to duty is the highest form of
> >> worship of God.”
> >>>>
> >>>> "Maybe other people will try to limit me but I don't limit myself"
> >
> > --
> > _____________________________________________________________
> > The information contained in this communication is intended solely for
> the
> > use of the individual or entity to whom it is addressed and others
> > authorized to receive it. It may contain confidential or legally
> privileged
> > information. If you are not the intended recipient you are hereby
> notified
> > that any disclosure, copying, distribution or taking any action in
> reliance
> > on the contents of this information is strictly prohibited and may be
> > unlawful. If you have received this communication in error, please notify
> > us immediately by responding to this email and then delete it from your
> > system. The firm is neither liable for the proper and complete
> transmission
> > of the information contained in this communication nor for any delay in
> its
> > receipt.
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: Installation procedure for Faclon for Apache hadoop
Posted by Siddharth Tiwari <si...@live.com>.
Hi Pallavi,
This is great. Thank you so much for prompt response. One more question Pallavi, is the version on github differebt from the one being packed with HDP 2.3 ? I see a lot of difference between the gui and also the stability.
Sent from my iPhone
> On Aug 7, 2015, at 1:34 AM, Pallavi Rao <pa...@inmobi.com> wrote:
>
> Hey Siddharth,
> If you are not using Oozie distro that is packaged during falcon build,
> ensure the following is Oozie changes are done before you start Falcon
> server.
>
> 1.
>
> Copy <falcon base
> dir>/oozie/libext/falcon-oozie-el-extension-0.7-incubating-SNAPSHOT.jar
> to
> <oozie base dir>/libext.
> 2.
>
> Modify oozie site xml. Copy all EL related properties from <falcon
> install dir>oozie/conf/oozie-site.xml to <oozie base
> dir>/conf/oozie-site.xml
> 3.
>
> Restart oozie :
>
>
> -
>
> sudo -u oozie ./bin/oozie-start.sh
> -
>
> sudo -u oozie bin/oozie-setup.sh prepare-war
> -
>
> sudo -u oozie ./bin/oozie-start.sh
>
> Just realized we have missed this information in our installation docs.
> Have opened JIRA <https://issues.apache.org/jira/browse/FALCON-1378> for
> the same.
>
> I assume that you have information on how to setup up a standalone Oozie
> instance. If not, let me know, I can pass along the same to you.
>
> Regards,
> Pallavi
>
> On Fri, Aug 7, 2015 at 1:33 PM, Siddharth Tiwari <si...@live.com>
> wrote:
>
>> Hi Srikanth
>> I am able to build it successfully in embedded mode, the issue is
>> primarily interacting with oozie and setting up oozie to interact with
>> Falcon, are there any particular steps to setup oozie for Falcon ? Secondly
>> is the version packed with HDP different from the one available in the git
>> ? Because I cannot see the xml forms setup on the gui.
>>
>> Sent from my iPhone
>>
>>>> On Aug 7, 2015, at 12:41 AM, Srikanth Sundarrajan <sr...@hotmail.com>
>>> wrote:
>>>
>>> HI Siddharth,
>>> It should be fairly straight forward to use Falcon with hadoop 2.5.0 and
>> above. Standard installation instructions should suffice. Is there a
>> particular step where you are stuck in getting it setup ?
>>>
>>> Regards
>>> Srikanth Sundarrajan
>>>
>>>> From: siddharth.tiwari@live.com
>>>> To: dev@falcon.apache.org
>>>> Subject: Installation procedure for Faclon for Apache hadoop
>>>> Date: Fri, 7 Aug 2015 07:13:52 +0000
>>>>
>>>> Hi Team,is there any documentation available for installing and
>> configuring falcon for stock vanila Apache hadoop ? It fairly easier fro
>> HDP using Ambari, but the procedure is not very clear for installing it
>> with Stock Apache hadoop. Any help in this regard will be highly
>> appreciated. Thank you in advanced.
>>>>
>>>> *------------------------*
>>>>
>>>> Cheers !!!
>>>>
>>>> Siddharth Tiwari
>>>>
>>>> Have a refreshing day !!!
>>>> "Every duty is holy, and devotion to duty is the highest form of
>> worship of God.”
>>>>
>>>> "Maybe other people will try to limit me but I don't limit myself"
>
> --
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
Re: Installation procedure for Faclon for Apache hadoop
Posted by Pallavi Rao <pa...@inmobi.com>.
Hey Siddharth,
If you are not using Oozie distro that is packaged during falcon build,
ensure the following is Oozie changes are done before you start Falcon
server.
1.
Copy <falcon base
dir>/oozie/libext/falcon-oozie-el-extension-0.7-incubating-SNAPSHOT.jar
to
<oozie base dir>/libext.
2.
Modify oozie site xml. Copy all EL related properties from <falcon
install dir>oozie/conf/oozie-site.xml to <oozie base
dir>/conf/oozie-site.xml
3.
Restart oozie :
-
sudo -u oozie ./bin/oozie-start.sh
-
sudo -u oozie bin/oozie-setup.sh prepare-war
-
sudo -u oozie ./bin/oozie-start.sh
Just realized we have missed this information in our installation docs.
Have opened JIRA <https://issues.apache.org/jira/browse/FALCON-1378> for
the same.
I assume that you have information on how to setup up a standalone Oozie
instance. If not, let me know, I can pass along the same to you.
Regards,
Pallavi
On Fri, Aug 7, 2015 at 1:33 PM, Siddharth Tiwari <si...@live.com>
wrote:
> Hi Srikanth
> I am able to build it successfully in embedded mode, the issue is
> primarily interacting with oozie and setting up oozie to interact with
> Falcon, are there any particular steps to setup oozie for Falcon ? Secondly
> is the version packed with HDP different from the one available in the git
> ? Because I cannot see the xml forms setup on the gui.
>
> Sent from my iPhone
>
> > On Aug 7, 2015, at 12:41 AM, Srikanth Sundarrajan <sr...@hotmail.com>
> wrote:
> >
> > HI Siddharth,
> > It should be fairly straight forward to use Falcon with hadoop 2.5.0 and
> above. Standard installation instructions should suffice. Is there a
> particular step where you are stuck in getting it setup ?
> >
> > Regards
> > Srikanth Sundarrajan
> >
> >> From: siddharth.tiwari@live.com
> >> To: dev@falcon.apache.org
> >> Subject: Installation procedure for Faclon for Apache hadoop
> >> Date: Fri, 7 Aug 2015 07:13:52 +0000
> >>
> >> Hi Team,is there any documentation available for installing and
> configuring falcon for stock vanila Apache hadoop ? It fairly easier fro
> HDP using Ambari, but the procedure is not very clear for installing it
> with Stock Apache hadoop. Any help in this regard will be highly
> appreciated. Thank you in advanced.
> >>
> >> *------------------------*
> >>
> >> Cheers !!!
> >>
> >> Siddharth Tiwari
> >>
> >> Have a refreshing day !!!
> >> "Every duty is holy, and devotion to duty is the highest form of
> worship of God.”
> >>
> >> "Maybe other people will try to limit me but I don't limit myself"
> >
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: Installation procedure for Faclon for Apache hadoop
Posted by Siddharth Tiwari <si...@live.com>.
Hi Srikanth
I am able to build it successfully in embedded mode, the issue is primarily interacting with oozie and setting up oozie to interact with Falcon, are there any particular steps to setup oozie for Falcon ? Secondly is the version packed with HDP different from the one available in the git ? Because I cannot see the xml forms setup on the gui.
Sent from my iPhone
> On Aug 7, 2015, at 12:41 AM, Srikanth Sundarrajan <sr...@hotmail.com> wrote:
>
> HI Siddharth,
> It should be fairly straight forward to use Falcon with hadoop 2.5.0 and above. Standard installation instructions should suffice. Is there a particular step where you are stuck in getting it setup ?
>
> Regards
> Srikanth Sundarrajan
>
>> From: siddharth.tiwari@live.com
>> To: dev@falcon.apache.org
>> Subject: Installation procedure for Faclon for Apache hadoop
>> Date: Fri, 7 Aug 2015 07:13:52 +0000
>>
>> Hi Team,is there any documentation available for installing and configuring falcon for stock vanila Apache hadoop ? It fairly easier fro HDP using Ambari, but the procedure is not very clear for installing it with Stock Apache hadoop. Any help in this regard will be highly appreciated. Thank you in advanced.
>>
>> *------------------------*
>>
>> Cheers !!!
>>
>> Siddharth Tiwari
>>
>> Have a refreshing day !!!
>> "Every duty is holy, and devotion to duty is the highest form of worship of God.”
>>
>> "Maybe other people will try to limit me but I don't limit myself"
>
RE: Installation procedure for Faclon for Apache hadoop
Posted by Srikanth Sundarrajan <sr...@hotmail.com>.
HI Siddharth,
It should be fairly straight forward to use Falcon with hadoop 2.5.0 and above. Standard installation instructions should suffice. Is there a particular step where you are stuck in getting it setup ?
Regards
Srikanth Sundarrajan
> From: siddharth.tiwari@live.com
> To: dev@falcon.apache.org
> Subject: Installation procedure for Faclon for Apache hadoop
> Date: Fri, 7 Aug 2015 07:13:52 +0000
>
> Hi Team,is there any documentation available for installing and configuring falcon for stock vanila Apache hadoop ? It fairly easier fro HDP using Ambari, but the procedure is not very clear for installing it with Stock Apache hadoop. Any help in this regard will be highly appreciated. Thank you in advanced.
>
> *------------------------*
>
> Cheers !!!
>
> Siddharth Tiwari
>
> Have a refreshing day !!!
> "Every duty is holy, and devotion to duty is the highest form of worship of God.”
>
> "Maybe other people will try to limit me but I don't limit myself"
>