You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Andy Christianson <ai...@protonmail.com> on 2018/01/04 18:27:53 UTC
MINIFICPP-357 minifi-cpp Design decision: Flow Configuration v3 yml vs properties
All,
Currently in the middle of MINIFICPP-357.
v3 includes core/repository configuration in the yml file, while traditionally in minificpp this has been set with the .properties files in config/.
E.g.
Core Properties:
flow controller graceful shutdown period: 10 sec
flow service write delay interval: 500 ms
administrative yield duration: 30 sec
bored yield duration: 10 millis
max concurrent threads: 1
variable registry properties: ''
FlowFile Repository:
partitions: 256
checkpoint interval: 2 mins
always sync: false
Swap:
threshold: 20000
in period: 5 sec
in threads: 1
out period: 5 sec
out threads: 4
Content Repository:
content claim max appendable size: 10 MB
content claim max flow files: 100
always sync: false
How should we handle this in minificpp with v3 support? Do we want to keep the convention of using the .properties file, or move all that into the yml?
Regards,
Andy I.C.
Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted email.
Re: MINIFICPP-357 minifi-cpp Design decision: Flow Configuration v3 yml vs properties
Posted by Andy Christianson <ai...@protonmail.com>.
Thanks for the feedback. It looks like we have consensus on keeping instance configuration in the properties files and keeping the yml focused on flow.
I'll run some tests to make sure the instance config properties, if present in the yml, do not interfere with the instance startup/operations.
Regards,
Andy I.C.
Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted email.
> -------- Original Message --------
> Subject: Re: MINIFICPP-357 minifi-cpp Design decision: Flow Configuration v3 yml vs properties
> Local Time: January 4, 2018 4:22 PM
> UTC Time: January 4, 2018 9:22 PM
> From: kdoran.apache@gmail.com
> To: dev@nifi.apache.org
>
> Hi Andy,
>
> My preference would also be to keep the .yml scoped to flows, and keep instance config in a .properties file.
>
> When we get further along the C2 roadmap, both of these should be updatable via C2 interactions.
>
> Thanks,
> Kevin
>
> On 1/4/18, 16:04, "Joe Witt" joe.witt@gmail.com wrote:
>
> Hello
>
> My preference is that we follow similar logic to how NiFi has
> 'instance configuration' details in nifi.properties and 'flow
> configuration' details in flow.xml.gz.
>
> What we do in minificpp will hopefully be aligned with what we do in
> minifi-java and hopefully minifi-java just becomes an assembly of
> nifi.
>
> Thanks
> Joe
>
> On Thu, Jan 4, 2018 at 3:59 PM, Aldrin Piri aldrinpiri@gmail.com wrote:
>
>> For the time being, I kind of view the C++ variant's configuration to be a
>
>> subset of that which is available for Java. Many of these items currently
>
>> have no backing, so don't know that we need to provide explicit support for
>
>> them at this juncture but ensure that their presence does not preclude
>
>> proper operation of a C++ instance.
>
>>
>
>> Looking at things from a general handling standpoint, I see these as
>
>> considerations of flow vs instance configuration. My general inclination
>
>> is to provide such separation at some point and is at the heart of
>
>> MINIFI-66 [1]. I think what is important regardless of how this carries
>
>> out, and in view of the first release of Registry, is to think how we can
>
>> make use of what facilities Registry provides both now in terms of flows as
>
>> well as future resource types. Some combination of the command and control
>
>> work (both server and the initial C2 API in C++) with Registry should allow
>
>> us to manage both sets of configuration that map to and enable the most
>
>> common workflows users would expect. I suspect the C++ approach is more
>
>> aligned with typical usage, properties generally bootstrapped in
>
>> .properties file(s), with some C2 API type interactions that may allow
>
>> adjusting these items in a separate context outside of the flow itself.
>
>>
>
>> [1] https://issues.apache.org/jira/browse/MINIFI-66
>
>>
>
>> On Thu, Jan 4, 2018 at 1:27 PM, Andy Christianson <ai...@protonmail.com>
>
>> wrote:
>
>>
>
>>> All,
>
>>>
>
>>> Currently in the middle of MINIFICPP-357.
>
>>>
>
>>> v3 includes core/repository configuration in the yml file, while
>
>>> traditionally in minificpp this has been set with the .properties files in
>
>>> config/.
>
>>>
>
>>> E.g.
>
>>>
>
>>> Core Properties:
>
>>>
>
>>> flow controller graceful shutdown period: 10 sec
>
>>>
>
>>> flow service write delay interval: 500 ms
>
>>>
>
>>> administrative yield duration: 30 sec
>
>>>
>
>>> bored yield duration: 10 millis
>
>>>
>
>>> max concurrent threads: 1
>
>>>
>
>>> variable registry properties: ''
>
>>>
>
>>> FlowFile Repository:
>
>>>
>
>>> partitions: 256
>
>>>
>
>>> checkpoint interval: 2 mins
>
>>>
>
>>> always sync: false
>
>>>
>
>>> Swap:
>
>>>
>
>>> threshold: 20000
>
>>>
>
>>> in period: 5 sec
>
>>>
>
>>> in threads: 1
>
>>>
>
>>> out period: 5 sec
>
>>>
>
>>> out threads: 4
>
>>>
>
>>> Content Repository:
>
>>>
>
>>> content claim max appendable size: 10 MB
>
>>>
>
>>> content claim max flow files: 100
>
>>>
>
>>> always sync: false
>
>>>
>
>>> How should we handle this in minificpp with v3 support? Do we want to keep
>
>>> the convention of using the .properties file, or move all that into the yml?
>
>>>
>
>>> Regards,
>
>>>
>
>>> Andy I.C.
>
>>>
>
>>> Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted
>
>>> email.
Re: MINIFICPP-357 minifi-cpp Design decision: Flow Configuration v3
yml vs properties
Posted by Kevin Doran <kd...@gmail.com>.
Hi Andy,
My preference would also be to keep the .yml scoped to flows, and keep instance config in a .properties file.
When we get further along the C2 roadmap, both of these should be updatable via C2 interactions.
Thanks,
Kevin
On 1/4/18, 16:04, "Joe Witt" <jo...@gmail.com> wrote:
Hello
My preference is that we follow similar logic to how NiFi has
'instance configuration' details in nifi.properties and 'flow
configuration' details in flow.xml.gz.
What we do in minificpp will hopefully be aligned with what we do in
minifi-java and hopefully minifi-java just becomes an assembly of
nifi.
Thanks
Joe
On Thu, Jan 4, 2018 at 3:59 PM, Aldrin Piri <al...@gmail.com> wrote:
> For the time being, I kind of view the C++ variant's configuration to be a
> subset of that which is available for Java. Many of these items currently
> have no backing, so don't know that we need to provide explicit support for
> them at this juncture but ensure that their presence does not preclude
> proper operation of a C++ instance.
>
> Looking at things from a general handling standpoint, I see these as
> considerations of flow vs instance configuration. My general inclination
> is to provide such separation at some point and is at the heart of
> MINIFI-66 [1]. I think what is important regardless of how this carries
> out, and in view of the first release of Registry, is to think how we can
> make use of what facilities Registry provides both now in terms of flows as
> well as future resource types. Some combination of the command and control
> work (both server and the initial C2 API in C++) with Registry should allow
> us to manage both sets of configuration that map to and enable the most
> common workflows users would expect. I suspect the C++ approach is more
> aligned with typical usage, properties generally bootstrapped in
> .properties file(s), with some C2 API type interactions that may allow
> adjusting these items in a separate context outside of the flow itself.
>
> [1] https://issues.apache.org/jira/browse/MINIFI-66
>
> On Thu, Jan 4, 2018 at 1:27 PM, Andy Christianson <ai...@protonmail.com>
> wrote:
>
>> All,
>>
>> Currently in the middle of MINIFICPP-357.
>>
>> v3 includes core/repository configuration in the yml file, while
>> traditionally in minificpp this has been set with the .properties files in
>> config/.
>>
>> E.g.
>>
>> Core Properties:
>>
>> flow controller graceful shutdown period: 10 sec
>>
>> flow service write delay interval: 500 ms
>>
>> administrative yield duration: 30 sec
>>
>> bored yield duration: 10 millis
>>
>> max concurrent threads: 1
>>
>> variable registry properties: ''
>>
>> FlowFile Repository:
>>
>> partitions: 256
>>
>> checkpoint interval: 2 mins
>>
>> always sync: false
>>
>> Swap:
>>
>> threshold: 20000
>>
>> in period: 5 sec
>>
>> in threads: 1
>>
>> out period: 5 sec
>>
>> out threads: 4
>>
>> Content Repository:
>>
>> content claim max appendable size: 10 MB
>>
>> content claim max flow files: 100
>>
>> always sync: false
>>
>> How should we handle this in minificpp with v3 support? Do we want to keep
>> the convention of using the .properties file, or move all that into the yml?
>>
>> Regards,
>>
>> Andy I.C.
>>
>> Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted
>> email.
Re: MINIFICPP-357 minifi-cpp Design decision: Flow Configuration v3
yml vs properties
Posted by Joe Witt <jo...@gmail.com>.
Hello
My preference is that we follow similar logic to how NiFi has
'instance configuration' details in nifi.properties and 'flow
configuration' details in flow.xml.gz.
What we do in minificpp will hopefully be aligned with what we do in
minifi-java and hopefully minifi-java just becomes an assembly of
nifi.
Thanks
Joe
On Thu, Jan 4, 2018 at 3:59 PM, Aldrin Piri <al...@gmail.com> wrote:
> For the time being, I kind of view the C++ variant's configuration to be a
> subset of that which is available for Java. Many of these items currently
> have no backing, so don't know that we need to provide explicit support for
> them at this juncture but ensure that their presence does not preclude
> proper operation of a C++ instance.
>
> Looking at things from a general handling standpoint, I see these as
> considerations of flow vs instance configuration. My general inclination
> is to provide such separation at some point and is at the heart of
> MINIFI-66 [1]. I think what is important regardless of how this carries
> out, and in view of the first release of Registry, is to think how we can
> make use of what facilities Registry provides both now in terms of flows as
> well as future resource types. Some combination of the command and control
> work (both server and the initial C2 API in C++) with Registry should allow
> us to manage both sets of configuration that map to and enable the most
> common workflows users would expect. I suspect the C++ approach is more
> aligned with typical usage, properties generally bootstrapped in
> .properties file(s), with some C2 API type interactions that may allow
> adjusting these items in a separate context outside of the flow itself.
>
> [1] https://issues.apache.org/jira/browse/MINIFI-66
>
> On Thu, Jan 4, 2018 at 1:27 PM, Andy Christianson <ai...@protonmail.com>
> wrote:
>
>> All,
>>
>> Currently in the middle of MINIFICPP-357.
>>
>> v3 includes core/repository configuration in the yml file, while
>> traditionally in minificpp this has been set with the .properties files in
>> config/.
>>
>> E.g.
>>
>> Core Properties:
>>
>> flow controller graceful shutdown period: 10 sec
>>
>> flow service write delay interval: 500 ms
>>
>> administrative yield duration: 30 sec
>>
>> bored yield duration: 10 millis
>>
>> max concurrent threads: 1
>>
>> variable registry properties: ''
>>
>> FlowFile Repository:
>>
>> partitions: 256
>>
>> checkpoint interval: 2 mins
>>
>> always sync: false
>>
>> Swap:
>>
>> threshold: 20000
>>
>> in period: 5 sec
>>
>> in threads: 1
>>
>> out period: 5 sec
>>
>> out threads: 4
>>
>> Content Repository:
>>
>> content claim max appendable size: 10 MB
>>
>> content claim max flow files: 100
>>
>> always sync: false
>>
>> How should we handle this in minificpp with v3 support? Do we want to keep
>> the convention of using the .properties file, or move all that into the yml?
>>
>> Regards,
>>
>> Andy I.C.
>>
>> Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted
>> email.
Re: MINIFICPP-357 minifi-cpp Design decision: Flow Configuration v3
yml vs properties
Posted by Aldrin Piri <al...@gmail.com>.
For the time being, I kind of view the C++ variant's configuration to be a
subset of that which is available for Java. Many of these items currently
have no backing, so don't know that we need to provide explicit support for
them at this juncture but ensure that their presence does not preclude
proper operation of a C++ instance.
Looking at things from a general handling standpoint, I see these as
considerations of flow vs instance configuration. My general inclination
is to provide such separation at some point and is at the heart of
MINIFI-66 [1]. I think what is important regardless of how this carries
out, and in view of the first release of Registry, is to think how we can
make use of what facilities Registry provides both now in terms of flows as
well as future resource types. Some combination of the command and control
work (both server and the initial C2 API in C++) with Registry should allow
us to manage both sets of configuration that map to and enable the most
common workflows users would expect. I suspect the C++ approach is more
aligned with typical usage, properties generally bootstrapped in
.properties file(s), with some C2 API type interactions that may allow
adjusting these items in a separate context outside of the flow itself.
[1] https://issues.apache.org/jira/browse/MINIFI-66
On Thu, Jan 4, 2018 at 1:27 PM, Andy Christianson <ai...@protonmail.com>
wrote:
> All,
>
> Currently in the middle of MINIFICPP-357.
>
> v3 includes core/repository configuration in the yml file, while
> traditionally in minificpp this has been set with the .properties files in
> config/.
>
> E.g.
>
> Core Properties:
>
> flow controller graceful shutdown period: 10 sec
>
> flow service write delay interval: 500 ms
>
> administrative yield duration: 30 sec
>
> bored yield duration: 10 millis
>
> max concurrent threads: 1
>
> variable registry properties: ''
>
> FlowFile Repository:
>
> partitions: 256
>
> checkpoint interval: 2 mins
>
> always sync: false
>
> Swap:
>
> threshold: 20000
>
> in period: 5 sec
>
> in threads: 1
>
> out period: 5 sec
>
> out threads: 4
>
> Content Repository:
>
> content claim max appendable size: 10 MB
>
> content claim max flow files: 100
>
> always sync: false
>
> How should we handle this in minificpp with v3 support? Do we want to keep
> the convention of using the .properties file, or move all that into the yml?
>
> Regards,
>
> Andy I.C.
>
> Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted
> email.