You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Joe Gresock <jg...@gmail.com> on 2017/04/05 12:40:28 UTC
Reporting task for capturing status history?
I haven't completely kept up to date on the new reporting tasks since
1.1.1, so I thought I'd ask here.
I'd like to be able to store the status history metrics for processors and
queues in something like Elasticsearch. I see the StandardGangliaReporter,
which looks to capture what I want, but I'm not using Ganglia. Is there
anything that captures the metrics by producing flow files or something?
Thanks,
Joe
--
I know what it is to be in need, and I know what it is to have plenty. I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want. I can do
all this through him who gives me strength. *-Philippians 4:12-13*
Re: Reporting task for capturing status history?
Posted by Joe Witt <jo...@gmail.com>.
thanks Joe. Definitely trying to zero in on the release. Right now
'new' JIRAs should ideally be only about fixing important bugs or
closing out existing feature/improvement work. If someone is
available to help review it though i think it would be ok.
Thanks
On Fri, Apr 7, 2017 at 12:47 PM, Joe Gresock <jg...@gmail.com> wrote:
> I was able to implement the SiteToSiteStatusReportingTask, and it actually
> works pretty well. I submitted a patch for [1], but I didn't set a fix
> version in case you guys didn't want to add any new processors prior to
> cutting 1.2.0.
>
> [1] https://issues.apache.org/jira/browse/NIFI-3674
>
> On Wed, Apr 5, 2017 at 3:52 PM, Joe Gresock <jg...@gmail.com> wrote:
>
>> Thanks Mark! I've created [1] to capture this request. I started writing
>> a SiteToSiteStatusReportingTask by modifying a copy of the
>> SiteToSiteProvenanceReportingTask, but I got diverted with other tasks
>> and probably won't get a chance to work on it for a while. If anyone is
>> interested in taking on this task, I'd be happy to provide what I have so
>> far.
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-3674
>>
>> On Wed, Apr 5, 2017 at 1:13 PM, Mark Payne <ma...@hotmail.com> wrote:
>>
>>> Joe,
>>>
>>> None that I know of, but I like the idea! A Reporting Task isn't able to
>>> create FlowFiles,
>>> but one thing that you may want to look into is serializing the data as
>>> JSON or whatever
>>> makes sense and then pushing it to NiFi via Site-to-Site. The
>>> SiteToSiteProvenanceReportingTask
>>> does this for Provenance data, and it is quite nice. It gives you the
>>> ability to transfer attributes
>>> along with the 'message' if it makes sense to do so, and integrates very
>>> cleanly. You could then use
>>> whatever Processors make sense in your flow to push the data wherever you
>>> would like it to be.
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>>
>>> > On Apr 5, 2017, at 8:40 AM, Joe Gresock <jg...@gmail.com> wrote:
>>> >
>>> > I haven't completely kept up to date on the new reporting tasks since
>>> > 1.1.1, so I thought I'd ask here.
>>> >
>>> > I'd like to be able to store the status history metrics for processors
>>> and
>>> > queues in something like Elasticsearch. I see the
>>> StandardGangliaReporter,
>>> > which looks to capture what I want, but I'm not using Ganglia. Is there
>>> > anything that captures the metrics by producing flow files or something?
>>> >
>>> > Thanks,
>>> > Joe
>>> >
>>> > --
>>> > I know what it is to be in need, and I know what it is to have plenty.
>>> I
>>> > have learned the secret of being content in any and every situation,
>>> > whether well fed or hungry, whether living in plenty or in want. I can
>>> do
>>> > all this through him who gives me strength. *-Philippians 4:12-13*
>>>
>>>
>>
>>
>> --
>> I know what it is to be in need, and I know what it is to have plenty. I
>> have learned the secret of being content in any and every situation,
>> whether well fed or hungry, whether living in plenty or in want. I can
>> do all this through him who gives me strength. *-Philippians 4:12-13*
>>
>
>
>
> --
> I know what it is to be in need, and I know what it is to have plenty. I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want. I can do
> all this through him who gives me strength. *-Philippians 4:12-13*
Re: Reporting task for capturing status history?
Posted by Joe Gresock <jg...@gmail.com>.
I was able to implement the SiteToSiteStatusReportingTask, and it actually
works pretty well. I submitted a patch for [1], but I didn't set a fix
version in case you guys didn't want to add any new processors prior to
cutting 1.2.0.
[1] https://issues.apache.org/jira/browse/NIFI-3674
On Wed, Apr 5, 2017 at 3:52 PM, Joe Gresock <jg...@gmail.com> wrote:
> Thanks Mark! I've created [1] to capture this request. I started writing
> a SiteToSiteStatusReportingTask by modifying a copy of the
> SiteToSiteProvenanceReportingTask, but I got diverted with other tasks
> and probably won't get a chance to work on it for a while. If anyone is
> interested in taking on this task, I'd be happy to provide what I have so
> far.
>
> [1] https://issues.apache.org/jira/browse/NIFI-3674
>
> On Wed, Apr 5, 2017 at 1:13 PM, Mark Payne <ma...@hotmail.com> wrote:
>
>> Joe,
>>
>> None that I know of, but I like the idea! A Reporting Task isn't able to
>> create FlowFiles,
>> but one thing that you may want to look into is serializing the data as
>> JSON or whatever
>> makes sense and then pushing it to NiFi via Site-to-Site. The
>> SiteToSiteProvenanceReportingTask
>> does this for Provenance data, and it is quite nice. It gives you the
>> ability to transfer attributes
>> along with the 'message' if it makes sense to do so, and integrates very
>> cleanly. You could then use
>> whatever Processors make sense in your flow to push the data wherever you
>> would like it to be.
>>
>> Thanks
>> -Mark
>>
>>
>>
>> > On Apr 5, 2017, at 8:40 AM, Joe Gresock <jg...@gmail.com> wrote:
>> >
>> > I haven't completely kept up to date on the new reporting tasks since
>> > 1.1.1, so I thought I'd ask here.
>> >
>> > I'd like to be able to store the status history metrics for processors
>> and
>> > queues in something like Elasticsearch. I see the
>> StandardGangliaReporter,
>> > which looks to capture what I want, but I'm not using Ganglia. Is there
>> > anything that captures the metrics by producing flow files or something?
>> >
>> > Thanks,
>> > Joe
>> >
>> > --
>> > I know what it is to be in need, and I know what it is to have plenty.
>> I
>> > have learned the secret of being content in any and every situation,
>> > whether well fed or hungry, whether living in plenty or in want. I can
>> do
>> > all this through him who gives me strength. *-Philippians 4:12-13*
>>
>>
>
>
> --
> I know what it is to be in need, and I know what it is to have plenty. I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want. I can
> do all this through him who gives me strength. *-Philippians 4:12-13*
>
--
I know what it is to be in need, and I know what it is to have plenty. I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want. I can do
all this through him who gives me strength. *-Philippians 4:12-13*
Re: Reporting task for capturing status history?
Posted by Joe Gresock <jg...@gmail.com>.
Thanks Mark! I've created [1] to capture this request. I started writing
a SiteToSiteStatusReportingTask by modifying a copy of the
SiteToSiteProvenanceReportingTask, but I got diverted with other tasks and
probably won't get a chance to work on it for a while. If anyone is
interested in taking on this task, I'd be happy to provide what I have so
far.
[1] https://issues.apache.org/jira/browse/NIFI-3674
On Wed, Apr 5, 2017 at 1:13 PM, Mark Payne <ma...@hotmail.com> wrote:
> Joe,
>
> None that I know of, but I like the idea! A Reporting Task isn't able to
> create FlowFiles,
> but one thing that you may want to look into is serializing the data as
> JSON or whatever
> makes sense and then pushing it to NiFi via Site-to-Site. The
> SiteToSiteProvenanceReportingTask
> does this for Provenance data, and it is quite nice. It gives you the
> ability to transfer attributes
> along with the 'message' if it makes sense to do so, and integrates very
> cleanly. You could then use
> whatever Processors make sense in your flow to push the data wherever you
> would like it to be.
>
> Thanks
> -Mark
>
>
>
> > On Apr 5, 2017, at 8:40 AM, Joe Gresock <jg...@gmail.com> wrote:
> >
> > I haven't completely kept up to date on the new reporting tasks since
> > 1.1.1, so I thought I'd ask here.
> >
> > I'd like to be able to store the status history metrics for processors
> and
> > queues in something like Elasticsearch. I see the
> StandardGangliaReporter,
> > which looks to capture what I want, but I'm not using Ganglia. Is there
> > anything that captures the metrics by producing flow files or something?
> >
> > Thanks,
> > Joe
> >
> > --
> > I know what it is to be in need, and I know what it is to have plenty. I
> > have learned the secret of being content in any and every situation,
> > whether well fed or hungry, whether living in plenty or in want. I can
> do
> > all this through him who gives me strength. *-Philippians 4:12-13*
>
>
--
I know what it is to be in need, and I know what it is to have plenty. I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want. I can do
all this through him who gives me strength. *-Philippians 4:12-13*
Re: Reporting task for capturing status history?
Posted by Mark Payne <ma...@hotmail.com>.
Joe,
None that I know of, but I like the idea! A Reporting Task isn't able to create FlowFiles,
but one thing that you may want to look into is serializing the data as JSON or whatever
makes sense and then pushing it to NiFi via Site-to-Site. The SiteToSiteProvenanceReportingTask
does this for Provenance data, and it is quite nice. It gives you the ability to transfer attributes
along with the 'message' if it makes sense to do so, and integrates very cleanly. You could then use
whatever Processors make sense in your flow to push the data wherever you would like it to be.
Thanks
-Mark
> On Apr 5, 2017, at 8:40 AM, Joe Gresock <jg...@gmail.com> wrote:
>
> I haven't completely kept up to date on the new reporting tasks since
> 1.1.1, so I thought I'd ask here.
>
> I'd like to be able to store the status history metrics for processors and
> queues in something like Elasticsearch. I see the StandardGangliaReporter,
> which looks to capture what I want, but I'm not using Ganglia. Is there
> anything that captures the metrics by producing flow files or something?
>
> Thanks,
> Joe
>
> --
> I know what it is to be in need, and I know what it is to have plenty. I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want. I can do
> all this through him who gives me strength. *-Philippians 4:12-13*