How to start a flow with attributes from the state store?

[Mark Rachelski <ma...@acommerce.asia>]

I have a scenario where I need to make an HTTP request to an API but taking
context into account from previous invocations. Specifically, one query
string parameter is a time where the API returns all records from that time
or later. Every day, I would issue a new request using the previous time
requested.

I have worked out that I can store the last time requested in the state
store. And use an UpdateAttributes process to retrieve it or initialize it
on the first run. I can then feed that into the InvokeHTTP processor and
build a dynamic URL from that attribute.

But my main problem is that I don't know what beginning processor to use in
this flow. UpdateAttribute needs an inbound connection. And there are no
obvious 'dummy' beginning processors that I can find in the vast array. The
only thing I need from the beginning processor is the schedule tab.

Any ideas on what my first processor in this flow should be?

Thank you in advance for any help,
Mark.

Re: How to start a flow with attributes from the state store?

[Bryan Bende <bb...@gmail.com>]

Mark,

The user guide is not necessarily going to mention every processor.

The official list is available down the side of the docs page [1].

I don't believe there is currently a way to find all source
processors, it is mentioned in the docs of each processor under the
"Input Requirement".

Might be a nice addition to automatically create a tag for processors
with input requirement of forbidden.

[1] https://nifi.apache.org/docs.html

On Thu, Oct 19, 2017 at 11:41 PM, Mark Rachelski <ma...@acommerce.asia> wrote:
> Thank you Bryan,
>
> That should fit my purposes well.
>
> BTW: That processor is not in the User Guide.
>
> As a follow on question, is there an easy way to ask NiFi for all processors
> that can be used at the beginning of a flow? There are a lot of other
> tagging that is done. But I spent a few hours last night googling for an
> answer before posting this question into the mail group.
>
> Mark.
>
> On Fri, Oct 20, 2017 at 9:33 AM Bryan Bende <bb...@gmail.com> wrote:
>>
>> Hi Mark,
>>
>> You can use GeneratFlowFile as the initial processor to trigger your flow.
>>
>> Make sure to change the run schedule appropriately otherwise you will get
>> a lot of flow files generated.
>>
>> -Bryan
>>
>> On Thu, Oct 19, 2017 at 10:08 PM Mark Rachelski <ma...@acommerce.asia>
>> wrote:
>>>
>>> I have a scenario where I need to make an HTTP request to an API but
>>> taking context into account from previous invocations. Specifically, one
>>> query string parameter is a time where the API returns all records from that
>>> time or later. Every day, I would issue a new request using the previous
>>> time requested.
>>>
>>> I have worked out that I can store the last time requested in the state
>>> store. And use an UpdateAttributes process to retrieve it or initialize it
>>> on the first run. I can then feed that into the InvokeHTTP processor and
>>> build a dynamic URL from that attribute.
>>>
>>> But my main problem is that I don't know what beginning processor to use
>>> in this flow. UpdateAttribute needs an inbound connection. And there are no
>>> obvious 'dummy' beginning processors that I can find in the vast array. The
>>> only thing I need from the beginning processor is the schedule tab.
>>>
>>> Any ideas on what my first processor in this flow should be?
>>>
>>> Thank you in advance for any help,
>>> Mark.
>>
>> --
>> Sent from Gmail Mobile

Re: How to start a flow with attributes from the state store?

[dan young <da...@gmail.com>]

Same here, works great!

On Fri, Oct 20, 2017 at 7:27 AM Mike Thomsen <mi...@gmail.com> wrote:

> I can vouch for this method. I have two flows for a client that use
> GenerateFlowFile to build a JSON DSL query for ElasticSearch and are
> executed on a timer. Works quite well with InvokeHttp.
>
> On Thu, Oct 19, 2017 at 11:41 PM, Mark Rachelski <ma...@acommerce.asia>
> wrote:
>
>> Thank you Bryan,
>>
>> That should fit my purposes well.
>>
>> BTW: That processor is not in the User Guide.
>>
>> As a follow on question, is there an easy way to ask NiFi for all
>> processors that can be used at the beginning of a flow? There are a lot of
>> other tagging that is done. But I spent a few hours last night googling for
>> an answer before posting this question into the mail group.
>>
>> Mark.
>>
>> On Fri, Oct 20, 2017 at 9:33 AM Bryan Bende <bb...@gmail.com> wrote:
>>
>>> Hi Mark,
>>>
>>> You can use GeneratFlowFile as the initial processor to trigger your
>>> flow.
>>>
>>> Make sure to change the run schedule appropriately otherwise you will
>>> get a lot of flow files generated.
>>>
>>> -Bryan
>>>
>>> On Thu, Oct 19, 2017 at 10:08 PM Mark Rachelski <ma...@acommerce.asia>
>>> wrote:
>>>
>>>> I have a scenario where I need to make an HTTP request to an API but
>>>> taking context into account from previous invocations. Specifically, one
>>>> query string parameter is a time where the API returns all records from
>>>> that time or later. Every day, I would issue a new request using the
>>>> previous time requested.
>>>>
>>>> I have worked out that I can store the last time requested in the state
>>>> store. And use an UpdateAttributes process to retrieve it or initialize it
>>>> on the first run. I can then feed that into the InvokeHTTP processor and
>>>> build a dynamic URL from that attribute.
>>>>
>>>> But my main problem is that I don't know what beginning processor to
>>>> use in this flow. UpdateAttribute needs an inbound connection. And there
>>>> are no obvious 'dummy' beginning processors that I can find in the vast
>>>> array. The only thing I need from the beginning processor is the schedule
>>>> tab.
>>>>
>>>> Any ideas on what my first processor in this flow should be?
>>>>
>>>> Thank you in advance for any help,
>>>> Mark.
>>>>
>>> --
>>> Sent from Gmail Mobile
>>>
>>

Re: How to start a flow with attributes from the state store?

[Mike Thomsen <mi...@gmail.com>]

I can vouch for this method. I have two flows for a client that use
GenerateFlowFile to build a JSON DSL query for ElasticSearch and are
executed on a timer. Works quite well with InvokeHttp.

On Thu, Oct 19, 2017 at 11:41 PM, Mark Rachelski <ma...@acommerce.asia>
wrote:

> Thank you Bryan,
>
> That should fit my purposes well.
>
> BTW: That processor is not in the User Guide.
>
> As a follow on question, is there an easy way to ask NiFi for all
> processors that can be used at the beginning of a flow? There are a lot of
> other tagging that is done. But I spent a few hours last night googling for
> an answer before posting this question into the mail group.
>
> Mark.
>
> On Fri, Oct 20, 2017 at 9:33 AM Bryan Bende <bb...@gmail.com> wrote:
>
>> Hi Mark,
>>
>> You can use GeneratFlowFile as the initial processor to trigger your flow.
>>
>> Make sure to change the run schedule appropriately otherwise you will get
>> a lot of flow files generated.
>>
>> -Bryan
>>
>> On Thu, Oct 19, 2017 at 10:08 PM Mark Rachelski <ma...@acommerce.asia>
>> wrote:
>>
>>> I have a scenario where I need to make an HTTP request to an API but
>>> taking context into account from previous invocations. Specifically, one
>>> query string parameter is a time where the API returns all records from
>>> that time or later. Every day, I would issue a new request using the
>>> previous time requested.
>>>
>>> I have worked out that I can store the last time requested in the state
>>> store. And use an UpdateAttributes process to retrieve it or initialize it
>>> on the first run. I can then feed that into the InvokeHTTP processor and
>>> build a dynamic URL from that attribute.
>>>
>>> But my main problem is that I don't know what beginning processor to use
>>> in this flow. UpdateAttribute needs an inbound connection. And there are no
>>> obvious 'dummy' beginning processors that I can find in the vast array. The
>>> only thing I need from the beginning processor is the schedule tab.
>>>
>>> Any ideas on what my first processor in this flow should be?
>>>
>>> Thank you in advance for any help,
>>> Mark.
>>>
>> --
>> Sent from Gmail Mobile
>>
>

Re: How to start a flow with attributes from the state store?

[Mark Rachelski <ma...@acommerce.asia>]

Thank you Bryan,

That should fit my purposes well.

BTW: That processor is not in the User Guide.

As a follow on question, is there an easy way to ask NiFi for all
processors that can be used at the beginning of a flow? There are a lot of
other tagging that is done. But I spent a few hours last night googling for
an answer before posting this question into the mail group.

Mark.

On Fri, Oct 20, 2017 at 9:33 AM Bryan Bende <bb...@gmail.com> wrote:

> Hi Mark,
>
> You can use GeneratFlowFile as the initial processor to trigger your flow.
>
> Make sure to change the run schedule appropriately otherwise you will get
> a lot of flow files generated.
>
> -Bryan
>
> On Thu, Oct 19, 2017 at 10:08 PM Mark Rachelski <ma...@acommerce.asia>
> wrote:
>
>> I have a scenario where I need to make an HTTP request to an API but
>> taking context into account from previous invocations. Specifically, one
>> query string parameter is a time where the API returns all records from
>> that time or later. Every day, I would issue a new request using the
>> previous time requested.
>>
>> I have worked out that I can store the last time requested in the state
>> store. And use an UpdateAttributes process to retrieve it or initialize it
>> on the first run. I can then feed that into the InvokeHTTP processor and
>> build a dynamic URL from that attribute.
>>
>> But my main problem is that I don't know what beginning processor to use
>> in this flow. UpdateAttribute needs an inbound connection. And there are no
>> obvious 'dummy' beginning processors that I can find in the vast array. The
>> only thing I need from the beginning processor is the schedule tab.
>>
>> Any ideas on what my first processor in this flow should be?
>>
>> Thank you in advance for any help,
>> Mark.
>>
> --
> Sent from Gmail Mobile
>

Re: How to start a flow with attributes from the state store?

[Bryan Bende <bb...@gmail.com>]

Hi Mark,

You can use GeneratFlowFile as the initial processor to trigger your flow.

Make sure to change the run schedule appropriately otherwise you will get a
lot of flow files generated.

-Bryan

On Thu, Oct 19, 2017 at 10:08 PM Mark Rachelski <ma...@acommerce.asia> wrote:

> I have a scenario where I need to make an HTTP request to an API but
> taking context into account from previous invocations. Specifically, one
> query string parameter is a time where the API returns all records from
> that time or later. Every day, I would issue a new request using the
> previous time requested.
>
> I have worked out that I can store the last time requested in the state
> store. And use an UpdateAttributes process to retrieve it or initialize it
> on the first run. I can then feed that into the InvokeHTTP processor and
> build a dynamic URL from that attribute.
>
> But my main problem is that I don't know what beginning processor to use
> in this flow. UpdateAttribute needs an inbound connection. And there are no
> obvious 'dummy' beginning processors that I can find in the vast array. The
> only thing I need from the beginning processor is the schedule tab.
>
> Any ideas on what my first processor in this flow should be?
>
> Thank you in advance for any help,
> Mark.
>
-- 
Sent from Gmail Mobile