You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by James McMahon <js...@gmail.com> on 2017/02/16 11:48:36 UTC

Means to access StandardHttpContextMap for monitoring purposes

Hello. I have a suite of applications that use Http POSTs to push content
to NiFi "on [their] demand". My HandleHttpRequest processor works for the
most part, employing an SSL Context Service and a StandardHttpContextMap.

Yesterday apps were unable to post. We are in development and test, and so
I had been sending the "success" output from HandleHttpRequest not only to
my main workflow stream, but also to a temporary holding queue feeding into
a MonitorActivity processor. In other words I was accumulating all the
input I received so that I could review it using list queue if required.

Bottom line: I started to get errors because HandleHttpRequest could not
accept requests. I suspect it was due to reaching the maximum threshold for
my StandardHttpContextMap. With all those incoming requests queued up in
that temporary queue, I believe I was not releasing these Http contexts
from the map.

Is there a means to monitor the capacity usage for StandardHttpContextMap?
I'd like to devise some means to throw a log alert if and when
StandardHttpContextMap gets to within 75% of my Maximum Outstanding
Requests associated with the context map.

Thank you in advance for your thoughts. -Jim

Re: Means to access StandardHttpContextMap for monitoring purposes

Posted by Andy LoPresto <al...@apache.org>.
Jim,

Thanks for being proactive. You can submit a Jira here [1]. Be sure the Project is “Apache NiFi” and then fill out the remaining fields as indicated. In general, we ask that you don’t provide a “Fix Version” or assign it to anyone other than yourself without coordinating with them. Populating detailed expectations, behavior, environment, versions affected, labels, etc. is very helpful for the community at large — devs deciding what tasks to tackle, reviewers knowing how to evaluate success of a patch, etc.

Anyone is welcome to submit tickets at any time. You do not have to be a PMC/committer/dev/user/consistent user of NiFi. Any constructive feedback that helps the project improve is always appreciated.

[1] https://issues.apache.org/jira/secure/CreateIssue!default.jspa <https://issues.apache.org/jira/secure/CreateIssue!default.jspa>

Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Feb 17, 2017, at 3:46 AM, James McMahon <js...@gmail.com> wrote:
> 
> Thank you Joe. Can I offer to create a ticket? I've not yet done that for Apache NiFi but would be happy to do that if you would like. If so, can you tell me the right process for submitting one for consideration, the link to visit to do that, and whether I as a member of the users group have the privilege to submit a suggested ticket?
> 
> Cheers,
> 
> Jim
> 
> On Thu, Feb 16, 2017 at 11:24 PM, Joe Witt <joe.witt@gmail.com <ma...@gmail.com>> wrote:
> Jim,
> 
> This is a great question and good point.  For controller services it
> would be really valuable to have a monitoring function exposed in some
> standard manner via their REST API.  I don't believe we have any
> generic support for this yet nor do I think the StandardHttpContentMap
> does this.  It would be a great thing to create a JIRA for.
> 
> Thanks
> Joe
> 
> On Thu, Feb 16, 2017 at 6:48 AM, James McMahon <jsmcmahon3@gmail.com <ma...@gmail.com>> wrote:
> > Hello. I have a suite of applications that use Http POSTs to push content to
> > NiFi "on [their] demand". My HandleHttpRequest processor works for the most
> > part, employing an SSL Context Service and a StandardHttpContextMap.
> >
> > Yesterday apps were unable to post. We are in development and test, and so I
> > had been sending the "success" output from HandleHttpRequest not only to my
> > main workflow stream, but also to a temporary holding queue feeding into a
> > MonitorActivity processor. In other words I was accumulating all the input I
> > received so that I could review it using list queue if required.
> >
> > Bottom line: I started to get errors because HandleHttpRequest could not
> > accept requests. I suspect it was due to reaching the maximum threshold for
> > my StandardHttpContextMap. With all those incoming requests queued up in
> > that temporary queue, I believe I was not releasing these Http contexts from
> > the map.
> >
> > Is there a means to monitor the capacity usage for StandardHttpContextMap?
> > I'd like to devise some means to throw a log alert if and when
> > StandardHttpContextMap gets to within 75% of my Maximum Outstanding Requests
> > associated with the context map.
> >
> > Thank you in advance for your thoughts. -Jim
>

Re: Means to access StandardHttpContextMap for monitoring purposes

Posted by James McMahon <js...@gmail.com>.
Thank you Joe. Can I offer to create a ticket? I've not yet done that for
Apache NiFi but would be happy to do that if you would like. If so, can you
tell me the right process for submitting one for consideration, the link to
visit to do that, and whether I as a member of the users group have the
privilege to submit a suggested ticket?

Cheers,

Jim

On Thu, Feb 16, 2017 at 11:24 PM, Joe Witt <jo...@gmail.com> wrote:

> Jim,
>
> This is a great question and good point.  For controller services it
> would be really valuable to have a monitoring function exposed in some
> standard manner via their REST API.  I don't believe we have any
> generic support for this yet nor do I think the StandardHttpContentMap
> does this.  It would be a great thing to create a JIRA for.
>
> Thanks
> Joe
>
> On Thu, Feb 16, 2017 at 6:48 AM, James McMahon <js...@gmail.com>
> wrote:
> > Hello. I have a suite of applications that use Http POSTs to push
> content to
> > NiFi "on [their] demand". My HandleHttpRequest processor works for the
> most
> > part, employing an SSL Context Service and a StandardHttpContextMap.
> >
> > Yesterday apps were unable to post. We are in development and test, and
> so I
> > had been sending the "success" output from HandleHttpRequest not only to
> my
> > main workflow stream, but also to a temporary holding queue feeding into
> a
> > MonitorActivity processor. In other words I was accumulating all the
> input I
> > received so that I could review it using list queue if required.
> >
> > Bottom line: I started to get errors because HandleHttpRequest could not
> > accept requests. I suspect it was due to reaching the maximum threshold
> for
> > my StandardHttpContextMap. With all those incoming requests queued up in
> > that temporary queue, I believe I was not releasing these Http contexts
> from
> > the map.
> >
> > Is there a means to monitor the capacity usage for
> StandardHttpContextMap?
> > I'd like to devise some means to throw a log alert if and when
> > StandardHttpContextMap gets to within 75% of my Maximum Outstanding
> Requests
> > associated with the context map.
> >
> > Thank you in advance for your thoughts. -Jim
>

Re: Means to access StandardHttpContextMap for monitoring purposes

Posted by Joe Witt <jo...@gmail.com>.
Jim,

This is a great question and good point.  For controller services it
would be really valuable to have a monitoring function exposed in some
standard manner via their REST API.  I don't believe we have any
generic support for this yet nor do I think the StandardHttpContentMap
does this.  It would be a great thing to create a JIRA for.

Thanks
Joe

On Thu, Feb 16, 2017 at 6:48 AM, James McMahon <js...@gmail.com> wrote:
> Hello. I have a suite of applications that use Http POSTs to push content to
> NiFi "on [their] demand". My HandleHttpRequest processor works for the most
> part, employing an SSL Context Service and a StandardHttpContextMap.
>
> Yesterday apps were unable to post. We are in development and test, and so I
> had been sending the "success" output from HandleHttpRequest not only to my
> main workflow stream, but also to a temporary holding queue feeding into a
> MonitorActivity processor. In other words I was accumulating all the input I
> received so that I could review it using list queue if required.
>
> Bottom line: I started to get errors because HandleHttpRequest could not
> accept requests. I suspect it was due to reaching the maximum threshold for
> my StandardHttpContextMap. With all those incoming requests queued up in
> that temporary queue, I believe I was not releasing these Http contexts from
> the map.
>
> Is there a means to monitor the capacity usage for StandardHttpContextMap?
> I'd like to devise some means to throw a log alert if and when
> StandardHttpContextMap gets to within 75% of my Maximum Outstanding Requests
> associated with the context map.
>
> Thank you in advance for your thoughts. -Jim