You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Richard Miskin <r....@gmail.com> on 2016/01/31 09:30:39 UTC
Provenance reporting
Hi,
Based on the changes in NIFI-1275 I’ve been looking at creating a ReportingTask to send provenance events to ElasticSearch.
I can see that is is possible to get all events from a specific Id by using: getProvenanceRepository().getEvents(firstRecordId, maxRecords)
Is there a standard mechanism for recording the last record id that my successfully task handled?
It seems like a common requirement for anything that is going to use that getEvents() mechanism so it seems a shame to have to create something specific for my task.
Cheers,
Richard
Re: Provenance reporting
Posted by Aldrin Piri <al...@gmail.com>.
Richard,
Yes, we are tracking and actively working toward that. The community is
working toward closing out this (NIFI-259) and the associated issues
scheduled for this release to get an RC out for the voting process.
On Sun, Jan 31, 2016 at 11:41 AM, Richard Miskin <r....@gmail.com>
wrote:
> Hi Aldrin,
>
> That State Management piece looks to cover my use case nicely, so it’d be
> good to use that rather than rolling my own solution.
>
> There was a NiFi roadmap sent out a while ago which I think had the 0.5.0
> release as 5th Feb - is that still the target? Is it looking like the State
> Management branch will be merged into master before then?
>
> Cheers,
> Richard
>
> > On 31 Jan 2016, at 15:08, Aldrin Piri <al...@gmail.com> wrote:
> >
> > Richard,
> >
> > In terms of capturing and recording these items about are things that
> make
> > the State Management [1] coming up in the 0.5.0 release for interfacing
> > with a StateManager to record these items in a framework provided
> > mechanism. This would be the preferred approach moving forward.
> >
> > Until state management, the way this has typically been accomplished is
> via
> > the DistributedMapCache [2]. You can evaluate the approach in components
> > such as ListHDFS or GetHBase in which both items are making use of a very
> > similar semantic as provided by State Management.
> >
> > [1] https://cwiki.apache.org/confluence/display/NIFI/State+Management
> > [2]
> >
> http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.distributed.cache.server.map.DistributedMapCacheServer/index.html
> >
> > On Sun, Jan 31, 2016 at 2:30 AM, Richard Miskin <r....@gmail.com>
> > wrote:
> >
> >> Hi,
> >>
> >> Based on the changes in NIFI-1275 I’ve been looking at creating a
> >> ReportingTask to send provenance events to ElasticSearch.
> >>
> >> I can see that is is possible to get all events from a specific Id by
> >> using: getProvenanceRepository().getEvents(firstRecordId, maxRecords)
> >>
> >> Is there a standard mechanism for recording the last record id that my
> >> successfully task handled?
> >>
> >> It seems like a common requirement for anything that is going to use
> that
> >> getEvents() mechanism so it seems a shame to have to create something
> >> specific for my task.
> >>
> >> Cheers,
> >> Richard
>
>
Re: Provenance reporting
Posted by Richard Miskin <r....@gmail.com>.
Hi Aldrin,
That State Management piece looks to cover my use case nicely, so it’d be good to use that rather than rolling my own solution.
There was a NiFi roadmap sent out a while ago which I think had the 0.5.0 release as 5th Feb - is that still the target? Is it looking like the State Management branch will be merged into master before then?
Cheers,
Richard
> On 31 Jan 2016, at 15:08, Aldrin Piri <al...@gmail.com> wrote:
>
> Richard,
>
> In terms of capturing and recording these items about are things that make
> the State Management [1] coming up in the 0.5.0 release for interfacing
> with a StateManager to record these items in a framework provided
> mechanism. This would be the preferred approach moving forward.
>
> Until state management, the way this has typically been accomplished is via
> the DistributedMapCache [2]. You can evaluate the approach in components
> such as ListHDFS or GetHBase in which both items are making use of a very
> similar semantic as provided by State Management.
>
> [1] https://cwiki.apache.org/confluence/display/NIFI/State+Management
> [2]
> http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.distributed.cache.server.map.DistributedMapCacheServer/index.html
>
> On Sun, Jan 31, 2016 at 2:30 AM, Richard Miskin <r....@gmail.com>
> wrote:
>
>> Hi,
>>
>> Based on the changes in NIFI-1275 I’ve been looking at creating a
>> ReportingTask to send provenance events to ElasticSearch.
>>
>> I can see that is is possible to get all events from a specific Id by
>> using: getProvenanceRepository().getEvents(firstRecordId, maxRecords)
>>
>> Is there a standard mechanism for recording the last record id that my
>> successfully task handled?
>>
>> It seems like a common requirement for anything that is going to use that
>> getEvents() mechanism so it seems a shame to have to create something
>> specific for my task.
>>
>> Cheers,
>> Richard
Re: Provenance reporting
Posted by Aldrin Piri <al...@gmail.com>.
Richard,
In terms of capturing and recording these items about are things that make
the State Management [1] coming up in the 0.5.0 release for interfacing
with a StateManager to record these items in a framework provided
mechanism. This would be the preferred approach moving forward.
Until state management, the way this has typically been accomplished is via
the DistributedMapCache [2]. You can evaluate the approach in components
such as ListHDFS or GetHBase in which both items are making use of a very
similar semantic as provided by State Management.
[1] https://cwiki.apache.org/confluence/display/NIFI/State+Management
[2]
http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.distributed.cache.server.map.DistributedMapCacheServer/index.html
On Sun, Jan 31, 2016 at 2:30 AM, Richard Miskin <r....@gmail.com>
wrote:
> Hi,
>
> Based on the changes in NIFI-1275 I’ve been looking at creating a
> ReportingTask to send provenance events to ElasticSearch.
>
> I can see that is is possible to get all events from a specific Id by
> using: getProvenanceRepository().getEvents(firstRecordId, maxRecords)
>
> Is there a standard mechanism for recording the last record id that my
> successfully task handled?
>
> It seems like a common requirement for anything that is going to use that
> getEvents() mechanism so it seems a shame to have to create something
> specific for my task.
>
> Cheers,
> Richard