You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by "Shaik Idris Ali (JIRA)" <ji...@apache.org> on 2014/03/05 16:21:45 UTC
[jira] [Comment Edited] (FALCON-287) Record lineage information in
post processing
[ https://issues.apache.org/jira/browse/FALCON-287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13920953#comment-13920953 ]
Shaik Idris Ali edited comment on FALCON-287 at 3/5/14 3:20 PM:
----------------------------------------------------------------
May be I missed the actual use-case of lineage, why do we need to persist it and who are the consumers of this information.
Secondly, instead of FalconPostProcessing storing this data on HDFS, which might further slowdown user workflow.
A service can subscribe to FALCON.ENTITY.TOPIC and record the details in HDFS or anywhere.
I think we discussed about this doing in ProcessSubscriberService.
{quote}
There seems to be redundant work in the current form.
>This is the beginning and as I had said previously, I'll add more data for each feed.
{quote}
>> May be I got the intent of storing this, but what all additional data we require for each feed.
+1 for moving from ARG to JSON.
Can we discuss this over a chat? 11 PM IST.
was (Author: shaik.idris):
May be I missed the actual use-case of lineage, why do we need to persist it and who are the consumers of this information.
Secondly, instead of FalconPostProcessing storing this data on HDFS, which might further slowdown user workflow.
A service can subscribe to FALCON.ENTITY.TOPIC and record the details in HDFS or anywhere.
I think we discussed about this doing in ProcessSubscriberService.
Can we discuss this over a chat? 11 PM IST.
> Record lineage information in post processing
> ---------------------------------------------
>
> Key: FALCON-287
> URL: https://issues.apache.org/jira/browse/FALCON-287
> Project: Falcon
> Issue Type: Sub-task
> Affects Versions: 0.5
> Reporter: Venkatesh Seetharam
> Assignee: Venkatesh Seetharam
> Labels: lineage
> Attachments: FALCON-287-v1.patch, FALCON-287-v2.patch, FALCON-287.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)