You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Andre F de Miranda <af...@fucs.org> on 2017/04/25 15:03:59 UTC
Making NiFi upgrade smoother
dev,
What do you think about the idea of allowing an user to configure overrides
for the "master" configuration files in order to smooth upgrades?
In my opinion candidates would be_ bootstrap.conf_ and _nifi.properties_
as they contain the links to all other files (logback.xml excluded).
Upon starting bootstrap and nifi would load first the standard files as it
does today, but once the load is done, the process would proceed to load
the local overrides (reaching its final configuration).
This way that whenever a user decides to upgrade, it would simply copy or
link the override file to the NiFi folder and voilà!
I realise this is not particularly a new idea, (YARN uses the "*-site.xml"
to configure local properties and so does a number of other JAVA based
applications I have dealt with) so I am keen to hear your thoughts
Cheers
Re: Making NiFi upgrade smoother
Posted by Andy LoPresto <al...@apache.org>.
Andre,
Aldrin and I discussed something similar in the comments of NIFI-2974 last year but didn’t follow up on it. I think you’ve gotten sufficient support here to file a Jira and scope it out. I know as someone who is constantly rebuilding and redeploying the application, I have shortcuts to copy over pre-configured files, but making the process smoother and more robust for all users would be a huge gain.
[1] https://issues.apache.org/jira/browse/NIFI-2974 <https://issues.apache.org/jira/browse/NIFI-2974>
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Apr 25, 2017, at 9:27 AM, Yolanda Davis <yo...@gmail.com> wrote:
>
> Hi Andre,
>
> Realized I responded on the wrong thread...
>
> Just wanted to add my thoughts on this. One of the things I have been
> thinking and working through in creating utilities to automate upgrades
> (see https://issues.apache.org/jira/browse/NIFI-3663) is creating the
> ability to migrate to new configurations/settings, especially in the
> instance that bootstrap.conf or a nifi.properties have been changed. Even
> master bootstrap and nifi.properties may be subject to upgrade. I think a
> large pain point is that users have needed to address those changes
> manually, so automating the migration might help in making an upgrade
> better.
>
> Also when thinking about smoother upgrades, I think there are some
> additional considerations that may need to be accounted for, such as
> whether or not certain repos should stay in place or be relocated (e.g. if
> a repository lives under the conf folder in the older installation, and not
> external, should that be moved to the newer one?). Or whether an existing
> flow template or state information (for zookeeper) needs to be migrated
> into the new installation as well. And how to easily manage a need to
> rollback if upgrade did not work for whatever reason (e.g. a flow is
> impacted by upgraded libraries). My hope is that upgrade tool will have a
> comprehensive approach for these things and also lead us down the road
> towards supporting rolling upgrades in a cluster in the future.
>
> -yolanda
>
> On Tue, Apr 25, 2017 at 11:03 AM, Andre F de Miranda <af...@fucs.org> wrote:
>
>> dev,
>>
>> What do you think about the idea of allowing an user to configure overrides
>> for the "master" configuration files in order to smooth upgrades?
>>
>> In my opinion candidates would be_ bootstrap.conf_ and _nifi.properties_
>>
>> as they contain the links to all other files (logback.xml excluded).
>>
>> Upon starting bootstrap and nifi would load first the standard files as it
>> does today, but once the load is done, the process would proceed to load
>> the local overrides (reaching its final configuration).
>>
>> This way that whenever a user decides to upgrade, it would simply copy or
>> link the override file to the NiFi folder and voilà!
>>
>> I realise this is not particularly a new idea, (YARN uses the "*-site.xml"
>> to configure local properties and so does a number of other JAVA based
>> applications I have dealt with) so I am keen to hear your thoughts
>>
>> Cheers
>>
>
>
>
> --
> --
> yolanda.m.davis@gmail.com
> @YolandaMDavis
Re: Making NiFi upgrade smoother
Posted by Yolanda Davis <yo...@gmail.com>.
Hi Andre,
Realized I responded on the wrong thread...
Just wanted to add my thoughts on this. One of the things I have been
thinking and working through in creating utilities to automate upgrades
(see https://issues.apache.org/jira/browse/NIFI-3663) is creating the
ability to migrate to new configurations/settings, especially in the
instance that bootstrap.conf or a nifi.properties have been changed. Even
master bootstrap and nifi.properties may be subject to upgrade. I think a
large pain point is that users have needed to address those changes
manually, so automating the migration might help in making an upgrade
better.
Also when thinking about smoother upgrades, I think there are some
additional considerations that may need to be accounted for, such as
whether or not certain repos should stay in place or be relocated (e.g. if
a repository lives under the conf folder in the older installation, and not
external, should that be moved to the newer one?). Or whether an existing
flow template or state information (for zookeeper) needs to be migrated
into the new installation as well. And how to easily manage a need to
rollback if upgrade did not work for whatever reason (e.g. a flow is
impacted by upgraded libraries). My hope is that upgrade tool will have a
comprehensive approach for these things and also lead us down the road
towards supporting rolling upgrades in a cluster in the future.
-yolanda
On Tue, Apr 25, 2017 at 11:03 AM, Andre F de Miranda <af...@fucs.org> wrote:
> dev,
>
> What do you think about the idea of allowing an user to configure overrides
> for the "master" configuration files in order to smooth upgrades?
>
> In my opinion candidates would be_ bootstrap.conf_ and _nifi.properties_
>
> as they contain the links to all other files (logback.xml excluded).
>
> Upon starting bootstrap and nifi would load first the standard files as it
> does today, but once the load is done, the process would proceed to load
> the local overrides (reaching its final configuration).
>
> This way that whenever a user decides to upgrade, it would simply copy or
> link the override file to the NiFi folder and voilà!
>
> I realise this is not particularly a new idea, (YARN uses the "*-site.xml"
> to configure local properties and so does a number of other JAVA based
> applications I have dealt with) so I am keen to hear your thoughts
>
> Cheers
>
--
--
yolanda.m.davis@gmail.com
@YolandaMDavis
Re: Making NiFi upgrade smoother
Posted by James Wing <jv...@gmail.com>.
I think this is a great idea. It would also provide a standard integration
point for the tls-toolkit command and similar utilities.
On Tue, Apr 25, 2017 at 8:03 AM, Andre F de Miranda <af...@fucs.org> wrote:
> dev,
>
> What do you think about the idea of allowing an user to configure overrides
> for the "master" configuration files in order to smooth upgrades?
>
> In my opinion candidates would be_ bootstrap.conf_ and _nifi.properties_
>
> as they contain the links to all other files (logback.xml excluded).
>
> Upon starting bootstrap and nifi would load first the standard files as it
> does today, but once the load is done, the process would proceed to load
> the local overrides (reaching its final configuration).
>
> This way that whenever a user decides to upgrade, it would simply copy or
> link the override file to the NiFi folder and voilà!
>
> I realise this is not particularly a new idea, (YARN uses the "*-site.xml"
> to configure local properties and so does a number of other JAVA based
> applications I have dealt with) so I am keen to hear your thoughts
>
> Cheers
>
Re: Making NiFi upgrade smoother
Posted by "Uwe@Moosheimer.com" <Uw...@Moosheimer.com>.
This is a great idea!
Mit freundlichen Grüßen / best regards
Kay-Uwe Moosheimer
> Am 25.04.2017 um 17:03 schrieb Andre F de Miranda <af...@fucs.org>:
>
> dev,
>
> What do you think about the idea of allowing an user to configure overrides
> for the "master" configuration files in order to smooth upgrades?
>
> In my opinion candidates would be_ bootstrap.conf_ and _nifi.properties_
>
> as they contain the links to all other files (logback.xml excluded).
>
> Upon starting bootstrap and nifi would load first the standard files as it
> does today, but once the load is done, the process would proceed to load
> the local overrides (reaching its final configuration).
>
> This way that whenever a user decides to upgrade, it would simply copy or
> link the override file to the NiFi folder and voilà!
>
> I realise this is not particularly a new idea, (YARN uses the "*-site.xml"
> to configure local properties and so does a number of other JAVA based
> applications I have dealt with) so I am keen to hear your thoughts
>
> Cheers