You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by Jagat Singh <ja...@gmail.com> on 2014/04/03 00:28:55 UTC
XMLs in the falcon
Hi,
I was looking at the falcon basic userguide [1] and the recent blog post of
same by Hortonworks [2]
I was just wondering if there is some proposal to reduce the amount of XML
code needed to ingest any new feed or process into the system.
Can we have some properties globally defined in the system.
Cluster A
Cluster B etc
Cluster A temp dir
Cluster B temp dir
Cluster A hive parent dir
Cluster B hive parent dir
And for any new feed we just need to write something similar to what we do
in cascading or pig script. 3-4 declarative steps what has to be done with
that data.
Write 3-4 lines of code and its all done.
We can generate XMLs in the background if needed to make it working , but
writing XMLs for ingesting every need feed is the most scary thing for me
at this moment to use it in Production. Imagine we have 500 feeds and how
many XMLs it will be needed to support
What are your thoughts on this.
Thanks,
Jagat Singh
[1] http://falcon.incubator.apache.org/docs/EntitySpecification.html
[2]
http://hortonworks.com/hadoop-tutorial/defining-processing-data-end-end-data-pipeline-apache-falcon/
RE: XMLs in the falcon
Posted by Srikanth Sundarrajan <sr...@hotmail.com>.
Hi Jagat, There are a lot of functionality that Falcon provides which would require this structured metadata (in xmls). I do understand your pain in having to create these xml especially when the number of feeds you would want to onboard is large. Since you seem to need very few customisations, perhaps we can add a feature to do expand on standard cluster & feed definition templates and overlay custom data from a property file or any other input to bootstrap the xml creation. This will reduce the burden in creating these xmls, while still providing the expressibility. Do you see this adequately addressing your concern?
Patch for the same is welcome too.
RegardsSrikanth Sundarrajan
> Date: Thu, 3 Apr 2014 09:28:55 +1100
> Subject: XMLs in the falcon
> From: jagatsingh@gmail.com
> To: dev@falcon.incubator.apache.org
>
> Hi,
>
> I was looking at the falcon basic userguide [1] and the recent blog post of
> same by Hortonworks [2]
>
> I was just wondering if there is some proposal to reduce the amount of XML
> code needed to ingest any new feed or process into the system.
>
> Can we have some properties globally defined in the system.
>
> Cluster A
> Cluster B etc
> Cluster A temp dir
> Cluster B temp dir
> Cluster A hive parent dir
> Cluster B hive parent dir
>
> And for any new feed we just need to write something similar to what we do
> in cascading or pig script. 3-4 declarative steps what has to be done with
> that data.
>
> Write 3-4 lines of code and its all done.
>
> We can generate XMLs in the background if needed to make it working , but
> writing XMLs for ingesting every need feed is the most scary thing for me
> at this moment to use it in Production. Imagine we have 500 feeds and how
> many XMLs it will be needed to support
>
> What are your thoughts on this.
>
> Thanks,
>
> Jagat Singh
>
>
>
>
>
>
>
>
> [1] http://falcon.incubator.apache.org/docs/EntitySpecification.html
> [2]
> http://hortonworks.com/hadoop-tutorial/defining-processing-data-end-end-data-pipeline-apache-falcon/
Re: XMLs in the falcon
Posted by Samarth Gupta <sa...@gmail.com>.
falcon QA team already use JAXB to generate various test xmls as runtime to
test different cases. We have a feed / process template which we modify at
runtime before submitting via api for testing.
some code might be reusable ....
below is the repo link :
https://github.com/samarthgupta437/falcon-regression
extended class with customized functions:
https://github.com/samarthgupta437/falcon-regression/blob/master/merlin-core/src/main/java/org/apache/falcon/regression/Entities/FeedMerlin.java
Thanks:
Samarth
On Thu, Apr 3, 2014 at 11:29 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>wrote:
> Hi,
>
> Even using the REST API, you need to provide the "init/descriptor" XML.
>
> Maybe you can create your own tooling (leverage falcon-client where JAXB
> descriptors are located).
> I started to create a tooling in Karaf to easily create XML:
>
> karaf@root()> falcon:cluster-create
> Write interface location:
> ...
>
> Especially the falcon:feed-create accepts a -r option to recursively
> browse a filesystem and create all feeds (for now, it's pretty basic).
>
> I plan to discuss with Srikanth and the team to "merge/donate" this
> tooling in Falcon (I tried to ping him yesterday but he was busy).
>
> My $0.02
>
> Regards
> JB
>
>
> On 04/03/2014 12:28 AM, Jagat Singh wrote:
>
>> Hi,
>>
>> I was looking at the falcon basic userguide [1] and the recent blog post
>> of
>> same by Hortonworks [2]
>>
>> I was just wondering if there is some proposal to reduce the amount of XML
>> code needed to ingest any new feed or process into the system.
>>
>> Can we have some properties globally defined in the system.
>>
>> Cluster A
>> Cluster B etc
>> Cluster A temp dir
>> Cluster B temp dir
>> Cluster A hive parent dir
>> Cluster B hive parent dir
>>
>> And for any new feed we just need to write something similar to what we do
>> in cascading or pig script. 3-4 declarative steps what has to be done with
>> that data.
>>
>> Write 3-4 lines of code and its all done.
>>
>> We can generate XMLs in the background if needed to make it working , but
>> writing XMLs for ingesting every need feed is the most scary thing for me
>> at this moment to use it in Production. Imagine we have 500 feeds and how
>> many XMLs it will be needed to support
>>
>> What are your thoughts on this.
>>
>> Thanks,
>>
>> Jagat Singh
>>
>>
>>
>>
>>
>>
>>
>>
>> [1] http://falcon.incubator.apache.org/docs/EntitySpecification.html
>> [2]
>> http://hortonworks.com/hadoop-tutorial/defining-processing-
>> data-end-end-data-pipeline-apache-falcon/
>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: XMLs in the falcon
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
Even using the REST API, you need to provide the "init/descriptor" XML.
Maybe you can create your own tooling (leverage falcon-client where JAXB
descriptors are located).
I started to create a tooling in Karaf to easily create XML:
karaf@root()> falcon:cluster-create
Write interface location:
...
Especially the falcon:feed-create accepts a -r option to recursively
browse a filesystem and create all feeds (for now, it's pretty basic).
I plan to discuss with Srikanth and the team to "merge/donate" this
tooling in Falcon (I tried to ping him yesterday but he was busy).
My $0.02
Regards
JB
On 04/03/2014 12:28 AM, Jagat Singh wrote:
> Hi,
>
> I was looking at the falcon basic userguide [1] and the recent blog post of
> same by Hortonworks [2]
>
> I was just wondering if there is some proposal to reduce the amount of XML
> code needed to ingest any new feed or process into the system.
>
> Can we have some properties globally defined in the system.
>
> Cluster A
> Cluster B etc
> Cluster A temp dir
> Cluster B temp dir
> Cluster A hive parent dir
> Cluster B hive parent dir
>
> And for any new feed we just need to write something similar to what we do
> in cascading or pig script. 3-4 declarative steps what has to be done with
> that data.
>
> Write 3-4 lines of code and its all done.
>
> We can generate XMLs in the background if needed to make it working , but
> writing XMLs for ingesting every need feed is the most scary thing for me
> at this moment to use it in Production. Imagine we have 500 feeds and how
> many XMLs it will be needed to support
>
> What are your thoughts on this.
>
> Thanks,
>
> Jagat Singh
>
>
>
>
>
>
>
>
> [1] http://falcon.incubator.apache.org/docs/EntitySpecification.html
> [2]
> http://hortonworks.com/hadoop-tutorial/defining-processing-data-end-end-data-pipeline-apache-falcon/
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com