You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by sanjeet rath <ra...@gmail.com> on 2021/08/10 13:33:07 UTC
How to restrict custom processor execution time
Hi ,
I am building a custom processor and there is restriction i want to put for
the processor that it should not schedule to run 2 times in 1 hour time
period.
I can acheive this by passing "run schedule" 60 mins.
Is there any other way i can do in my custom processor code, So that it
won't allow the user to select "run schedule " time less than 60
mins.basically similar to we can restrict the procesor to execute on
prinary node.
Any other thought is really helpfull.
Thanks,
Sanjeet
Re: How to restrict custom processor execution time
Posted by sanjeet rath <ra...@gmail.com>.
Thanks Chris & Joe for the wonderfull ideas.
Now i am clear how to acheive this functionality.
Regards,
Sanjeet
On Thu, 12 Aug 2021, 11:06 pm Chris Sampson,
<ch...@naimuri.com.invalid> wrote:
> Consider using a static class member variable maybe (i.e. use a java-based
> solution instead of trying to find a nifi specific one). Bear in mind that
> "last execution" would then be reset if the node on which the processor is
> scheduled gets restarted. If you're in a clustered environment then the
> processor scheduling would move to another node automatically, but that may
> trigger another execution within the same hour.
>
> You could start using State in the processor (local or cluster) if you
> wanted a nifi-based solution, although you'll have to consider what happens
> when the state is reset, e.g. by a user in the UI or if the node on which
> the processor is scheduled changes within the cluster (and you're using
> local state) - would this be OK or cause a problem, etc.? Using
> remote/cluster state would allow the same "last execution" to be stored for
> all nodes across the cluster, so even if the processor scheduling is moved
> between nodes, it should be able to determine whether to run.
>
> There are @TriggerSerially and @PrimaryNodeOnly annotations for processors
> to ensure they can only be scheduled on a single node (current "primary")
> within a cluster and to only run one instance at any one time.
>
> @DefaultSchedule annotation can be used to configure the default processor
> schedule (e.g. every "1 hour"), but a user could still change that in the
> UI.
>
> Last bit of advice from me would be to call "context.yield" in "onTrigger"
> if the processor detects it should skip the current execution because it's
> within the same hour as the "last execution" - this allows nifi to schedule
> other processors to run instead of immediately retrying this processor and
> taking up one of the available threads in the pool (if someone accidentally
> configures it to run continually).
>
>
> Cheers,
>
> Chris Sampson
>
> On Thu, 12 Aug 2021, 18:05 sanjeet rath, <ra...@gmail.com> wrote:
>
> > Hi Joe,
> >
> > Keeping track of last execution & comparing with current time looks good
> > solution for my use case.There is no additional complication also.
> >
> > I am thinking of storing the "last execution time" in a processor
> > parameter.after every succesfull execution i will populate the value .so
> > Unless "current time" of execution > last execution + 60 min . The
> > processor won't do it function.
> >
> > Is there any way can i hide this parameter in UI.
> >
> > Regards,
> > Sanjeet
> >
> > On Thu, 12 Aug 2021, 10:18 pm Joe Witt, <jo...@gmail.com> wrote:
> >
> > > Sanjeet
> > >
> > > What about keeping track of 'last execution' and 'current time' to
> > > give you enough to decide if it has been at least 60 mins? Are there
> > > additional complications to consider?
> > >
> > > Thanks
> > >
> > > On Thu, Aug 12, 2021 at 9:43 AM sanjeet rath <ra...@gmail.com>
> > > wrote:
> > > >
> > > > Thanks , Joe for the quick reply.
> > > > I was wondering how can i figure out inside the processor's Ontrigger
> > > > method when it was run previously.
> > > > Basicaly want to implement ur suggestion of restricting the execution
> > > > inside the processor's code .
> > > >
> > > > Regards,
> > > > Sanjeet
> > > >
> > > > On Thu, 12 Aug 2021, 9:43 pm Joe Witt, <jo...@gmail.com> wrote:
> > > >
> > > > > Sanjeet
> > > > >
> > > > > We dont presently allow the processor developer to put such a
> > > > > restriction into the code. Your best bet for now would be to
> > document
> > > > > that the processor even if scheduled more than once in a one hour
> > > > > period will not execute its function more than once. Then in your
> > > > > code you can protect/ensure it only truly does the execution once
> and
> > > > > hour.
> > > > >
> > > > > Thanks
> > > > > Joe
> > > > >
> > > > > On Thu, Aug 12, 2021 at 9:10 AM sanjeet rath <
> rath.sanjeet@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > Hi ,
> > > > > >
> > > > > > I am building a custom processor and there is restriction i want
> to
> > > put
> > > > > for
> > > > > > the processor that it should not schedule to run 2 times in 1
> hour
> > > time
> > > > > > period.
> > > > > >
> > > > > > I can acheive this by passing "run schedule" 60 mins.
> > > > > >
> > > > > > Is there any other way i can do in my custom processor code, So
> > that
> > > it
> > > > > > won't allow the user to select "run schedule " time less than 60
> > > > > > mins.basically similar to we can restrict the procesor to execute
> > on
> > > > > > prinary node.
> > > > > >
> > > > > > Any other thought is really helpfull.
> > > > > >
> > > > > > Thanks,
> > > > > > Sanjeet
> > > > >
> > >
> >
>
Re: How to restrict custom processor execution time
Posted by Chris Sampson <ch...@naimuri.com.INVALID>.
Consider using a static class member variable maybe (i.e. use a java-based
solution instead of trying to find a nifi specific one). Bear in mind that
"last execution" would then be reset if the node on which the processor is
scheduled gets restarted. If you're in a clustered environment then the
processor scheduling would move to another node automatically, but that may
trigger another execution within the same hour.
You could start using State in the processor (local or cluster) if you
wanted a nifi-based solution, although you'll have to consider what happens
when the state is reset, e.g. by a user in the UI or if the node on which
the processor is scheduled changes within the cluster (and you're using
local state) - would this be OK or cause a problem, etc.? Using
remote/cluster state would allow the same "last execution" to be stored for
all nodes across the cluster, so even if the processor scheduling is moved
between nodes, it should be able to determine whether to run.
There are @TriggerSerially and @PrimaryNodeOnly annotations for processors
to ensure they can only be scheduled on a single node (current "primary")
within a cluster and to only run one instance at any one time.
@DefaultSchedule annotation can be used to configure the default processor
schedule (e.g. every "1 hour"), but a user could still change that in the
UI.
Last bit of advice from me would be to call "context.yield" in "onTrigger"
if the processor detects it should skip the current execution because it's
within the same hour as the "last execution" - this allows nifi to schedule
other processors to run instead of immediately retrying this processor and
taking up one of the available threads in the pool (if someone accidentally
configures it to run continually).
Cheers,
Chris Sampson
On Thu, 12 Aug 2021, 18:05 sanjeet rath, <ra...@gmail.com> wrote:
> Hi Joe,
>
> Keeping track of last execution & comparing with current time looks good
> solution for my use case.There is no additional complication also.
>
> I am thinking of storing the "last execution time" in a processor
> parameter.after every succesfull execution i will populate the value .so
> Unless "current time" of execution > last execution + 60 min . The
> processor won't do it function.
>
> Is there any way can i hide this parameter in UI.
>
> Regards,
> Sanjeet
>
> On Thu, 12 Aug 2021, 10:18 pm Joe Witt, <jo...@gmail.com> wrote:
>
> > Sanjeet
> >
> > What about keeping track of 'last execution' and 'current time' to
> > give you enough to decide if it has been at least 60 mins? Are there
> > additional complications to consider?
> >
> > Thanks
> >
> > On Thu, Aug 12, 2021 at 9:43 AM sanjeet rath <ra...@gmail.com>
> > wrote:
> > >
> > > Thanks , Joe for the quick reply.
> > > I was wondering how can i figure out inside the processor's Ontrigger
> > > method when it was run previously.
> > > Basicaly want to implement ur suggestion of restricting the execution
> > > inside the processor's code .
> > >
> > > Regards,
> > > Sanjeet
> > >
> > > On Thu, 12 Aug 2021, 9:43 pm Joe Witt, <jo...@gmail.com> wrote:
> > >
> > > > Sanjeet
> > > >
> > > > We dont presently allow the processor developer to put such a
> > > > restriction into the code. Your best bet for now would be to
> document
> > > > that the processor even if scheduled more than once in a one hour
> > > > period will not execute its function more than once. Then in your
> > > > code you can protect/ensure it only truly does the execution once and
> > > > hour.
> > > >
> > > > Thanks
> > > > Joe
> > > >
> > > > On Thu, Aug 12, 2021 at 9:10 AM sanjeet rath <rath.sanjeet@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > Hi ,
> > > > >
> > > > > I am building a custom processor and there is restriction i want to
> > put
> > > > for
> > > > > the processor that it should not schedule to run 2 times in 1 hour
> > time
> > > > > period.
> > > > >
> > > > > I can acheive this by passing "run schedule" 60 mins.
> > > > >
> > > > > Is there any other way i can do in my custom processor code, So
> that
> > it
> > > > > won't allow the user to select "run schedule " time less than 60
> > > > > mins.basically similar to we can restrict the procesor to execute
> on
> > > > > prinary node.
> > > > >
> > > > > Any other thought is really helpfull.
> > > > >
> > > > > Thanks,
> > > > > Sanjeet
> > > >
> >
>
Re: How to restrict custom processor execution time
Posted by sanjeet rath <ra...@gmail.com>.
Hi Joe,
Keeping track of last execution & comparing with current time looks good
solution for my use case.There is no additional complication also.
I am thinking of storing the "last execution time" in a processor
parameter.after every succesfull execution i will populate the value .so
Unless "current time" of execution > last execution + 60 min . The
processor won't do it function.
Is there any way can i hide this parameter in UI.
Regards,
Sanjeet
On Thu, 12 Aug 2021, 10:18 pm Joe Witt, <jo...@gmail.com> wrote:
> Sanjeet
>
> What about keeping track of 'last execution' and 'current time' to
> give you enough to decide if it has been at least 60 mins? Are there
> additional complications to consider?
>
> Thanks
>
> On Thu, Aug 12, 2021 at 9:43 AM sanjeet rath <ra...@gmail.com>
> wrote:
> >
> > Thanks , Joe for the quick reply.
> > I was wondering how can i figure out inside the processor's Ontrigger
> > method when it was run previously.
> > Basicaly want to implement ur suggestion of restricting the execution
> > inside the processor's code .
> >
> > Regards,
> > Sanjeet
> >
> > On Thu, 12 Aug 2021, 9:43 pm Joe Witt, <jo...@gmail.com> wrote:
> >
> > > Sanjeet
> > >
> > > We dont presently allow the processor developer to put such a
> > > restriction into the code. Your best bet for now would be to document
> > > that the processor even if scheduled more than once in a one hour
> > > period will not execute its function more than once. Then in your
> > > code you can protect/ensure it only truly does the execution once and
> > > hour.
> > >
> > > Thanks
> > > Joe
> > >
> > > On Thu, Aug 12, 2021 at 9:10 AM sanjeet rath <ra...@gmail.com>
> > > wrote:
> > > >
> > > > Hi ,
> > > >
> > > > I am building a custom processor and there is restriction i want to
> put
> > > for
> > > > the processor that it should not schedule to run 2 times in 1 hour
> time
> > > > period.
> > > >
> > > > I can acheive this by passing "run schedule" 60 mins.
> > > >
> > > > Is there any other way i can do in my custom processor code, So that
> it
> > > > won't allow the user to select "run schedule " time less than 60
> > > > mins.basically similar to we can restrict the procesor to execute on
> > > > prinary node.
> > > >
> > > > Any other thought is really helpfull.
> > > >
> > > > Thanks,
> > > > Sanjeet
> > >
>
Re: How to restrict custom processor execution time
Posted by Joe Witt <jo...@gmail.com>.
Sanjeet
What about keeping track of 'last execution' and 'current time' to
give you enough to decide if it has been at least 60 mins? Are there
additional complications to consider?
Thanks
On Thu, Aug 12, 2021 at 9:43 AM sanjeet rath <ra...@gmail.com> wrote:
>
> Thanks , Joe for the quick reply.
> I was wondering how can i figure out inside the processor's Ontrigger
> method when it was run previously.
> Basicaly want to implement ur suggestion of restricting the execution
> inside the processor's code .
>
> Regards,
> Sanjeet
>
> On Thu, 12 Aug 2021, 9:43 pm Joe Witt, <jo...@gmail.com> wrote:
>
> > Sanjeet
> >
> > We dont presently allow the processor developer to put such a
> > restriction into the code. Your best bet for now would be to document
> > that the processor even if scheduled more than once in a one hour
> > period will not execute its function more than once. Then in your
> > code you can protect/ensure it only truly does the execution once and
> > hour.
> >
> > Thanks
> > Joe
> >
> > On Thu, Aug 12, 2021 at 9:10 AM sanjeet rath <ra...@gmail.com>
> > wrote:
> > >
> > > Hi ,
> > >
> > > I am building a custom processor and there is restriction i want to put
> > for
> > > the processor that it should not schedule to run 2 times in 1 hour time
> > > period.
> > >
> > > I can acheive this by passing "run schedule" 60 mins.
> > >
> > > Is there any other way i can do in my custom processor code, So that it
> > > won't allow the user to select "run schedule " time less than 60
> > > mins.basically similar to we can restrict the procesor to execute on
> > > prinary node.
> > >
> > > Any other thought is really helpfull.
> > >
> > > Thanks,
> > > Sanjeet
> >
Re: How to restrict custom processor execution time
Posted by sanjeet rath <ra...@gmail.com>.
Thanks , Joe for the quick reply.
I was wondering how can i figure out inside the processor's Ontrigger
method when it was run previously.
Basicaly want to implement ur suggestion of restricting the execution
inside the processor's code .
Regards,
Sanjeet
On Thu, 12 Aug 2021, 9:43 pm Joe Witt, <jo...@gmail.com> wrote:
> Sanjeet
>
> We dont presently allow the processor developer to put such a
> restriction into the code. Your best bet for now would be to document
> that the processor even if scheduled more than once in a one hour
> period will not execute its function more than once. Then in your
> code you can protect/ensure it only truly does the execution once and
> hour.
>
> Thanks
> Joe
>
> On Thu, Aug 12, 2021 at 9:10 AM sanjeet rath <ra...@gmail.com>
> wrote:
> >
> > Hi ,
> >
> > I am building a custom processor and there is restriction i want to put
> for
> > the processor that it should not schedule to run 2 times in 1 hour time
> > period.
> >
> > I can acheive this by passing "run schedule" 60 mins.
> >
> > Is there any other way i can do in my custom processor code, So that it
> > won't allow the user to select "run schedule " time less than 60
> > mins.basically similar to we can restrict the procesor to execute on
> > prinary node.
> >
> > Any other thought is really helpfull.
> >
> > Thanks,
> > Sanjeet
>
Re: How to restrict custom processor execution time
Posted by Joe Witt <jo...@gmail.com>.
Sanjeet
We dont presently allow the processor developer to put such a
restriction into the code. Your best bet for now would be to document
that the processor even if scheduled more than once in a one hour
period will not execute its function more than once. Then in your
code you can protect/ensure it only truly does the execution once and
hour.
Thanks
Joe
On Thu, Aug 12, 2021 at 9:10 AM sanjeet rath <ra...@gmail.com> wrote:
>
> Hi ,
>
> I am building a custom processor and there is restriction i want to put for
> the processor that it should not schedule to run 2 times in 1 hour time
> period.
>
> I can acheive this by passing "run schedule" 60 mins.
>
> Is there any other way i can do in my custom processor code, So that it
> won't allow the user to select "run schedule " time less than 60
> mins.basically similar to we can restrict the procesor to execute on
> prinary node.
>
> Any other thought is really helpfull.
>
> Thanks,
> Sanjeet
How to restrict custom processor execution time
Posted by sanjeet rath <ra...@gmail.com>.
Hi ,
I am building a custom processor and there is restriction i want to put for
the processor that it should not schedule to run 2 times in 1 hour time
period.
I can acheive this by passing "run schedule" 60 mins.
Is there any other way i can do in my custom processor code, So that it
won't allow the user to select "run schedule " time less than 60
mins.basically similar to we can restrict the procesor to execute on
prinary node.
Any other thought is really helpfull.
Thanks,
Sanjeet
Re: How to restrict custom processor execution time
Posted by Russell Bateman <ru...@windofkeltia.com>.
Sanjeet,
It occurred to me that you may not be getting replies because you report
"building a custom processor" and that's the subject of the Apache NiFi
Developers' forum. Try posting there.
Best regards,
Russ
On 8/10/21 7:33 AM, sanjeet rath wrote:
> Hi ,
>
> I am building a custom processor and there is restriction i want to
> put for the processor that it should not schedule to run 2 times in 1
> hour time period.
>
> I can acheive this by passing "run schedule" 60 mins.
>
> Is there any other way i can do in my custom processor code, So that
> it won't allow the user to select "run schedule " time less than 60
> mins.basically similar to we can restrict the procesor to execute on
> prinary node.
>
> Any other thought is really helpfull.
>
> Thanks,
> Sanjeet