Re: Best way to replace a workflow

[Joe Witt <jo...@gmail.com>]

Hello

I've had this thread flagged for a while now.  The sorts of concerns
raised in it have been fairly common patterns of requests from various
groups.  I believe they are fairly well accounted for in the present
list of feature proposals as seen here [1].

Please take a look and advise if you think something is missing.

Thanks
Joe

[1] https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals

On Wed, Apr 29, 2015 at 9:46 AM, Esteban Aliverti
<es...@gmail.com> wrote:
> Thanks Adam,
>
> I guess I was using templates because I didn't know about the
> existence of flow.xml.gz
> :).
> Now, if I have 2 different instances of NiFi, and I want to override the
> workflow in one of them with the workflow of the other, do I need to do
> anything else than just overwrite flow.xml.gz?
>
> Regards,
>
>
>
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
>
> Esteban Aliverti
> - Blog @ http://ilesteban.wordpress.com
>
> On Wed, Apr 29, 2015 at 2:04 PM, Adam Taft <ad...@adamtaft.com> wrote:
>
>> You can get a lot of mileage using the following suggestions:
>>
>> -  There's nothing preventing you from storing the flow.xml.gz file into
>> version control.  While this file is binary and you won't see any diffs on
>> it, your VCS (git, svn, etc.) won't have a problem storing it.  If storing
>> the flow.xml.gz as binary is a problem, then you could simply uncompress it
>> to flow.xml first and version control that.
>>
>> -  You might likewise want to version control your entire conf directory.
>>
>> -  For any processor properties which can be configured using NIFI's
>> expression language, you can reference System properties and/or environment
>> variables.  This allows you to have variant configurations based on the
>> deployment context. Things like hostnames or database connection properties
>> could in theory be configured externally.
>>
>> -  Your custom controller services can likewise be configured to utilize
>> system properties or environment vars.  Most standard controller services
>> don't have this capability yet, but work is being done that might help in
>> this area as well.
>>
>> I personally don't like using templates to solve your problem.  Having flow
>> snippets that need to be stored and applied between dev/int/prod doesn't
>> make a lot of sense to me.  Better in my mind is to have a single
>> flow.xml.gz that you can carry between those tiers.  NIFI should allow you
>> to get most of the way there today, but if you run into problems, I'm sure
>> you could affect additional changes to help with this issue.
>>
>> Adam
>>
>>
>>
>> On Tue, Apr 28, 2015 at 9:58 AM, Esteban Aliverti <
>> esteban.aliverti@gmail.com> wrote:
>>
>> > Joe,
>> > thanks for your prompt response.
>> >
>> > Let me give you more background about what I'm currently doing:
>> > We are currently a group of 3 developers working on a NiFi workflow. My
>> two
>> > colleagues are in charge of building the individual processors required
>> for
>> > the workflow we are trying to create. My role, even if I'm also a
>> Processor
>> > developer myself, is kind of the "integrator" of these processors. I
>> have a
>> > local NiFi running and I'm in charge of the "workflow" design. I use this
>> > local instance as a playground for develop and initial testing.
>> > Once I've updated my local workflow, I usually "export" it into another
>> > instance that is currently being used by QA. This QA instance is only
>> > modified after I validate that my local instance is stable enough. Given
>> > that there are a number of other developers using this QA instance as
>> > clients, the update process is not so easy. Before I introduce any change
>> > in QA, I need to coordinate with the QA department first. This is one of
>> > the biggest reasons I keep a local NiFi instance for development. I can
>> do
>> > whatever I want whenever I want to in my local instance but I need to be
>> > very careful when I modify the QA instance.
>> >
>> > I have asked similar questions in this forum in the past and one of your
>> > recommendations was to use the same workflow for both QA and DEV. You
>> > suggested me to use an attribute in the FlowFiles as some kind of "flag"
>> so
>> > it can be routed to the DEV branch of the workflow. I see 2 problems with
>> > this approach (maybe because of my own development workflow):
>> > 1.- Sometimes I don't have connectivity to the server where the QA NiFi
>> > instance is. I would like to be able to continue my development tasks
>> even
>> > if I'm offline.
>> > 2.- My workflow is not stateless. I have a bunch of Controller Services
>> > that are used through the workflow that intentionally keep state. If I
>> use
>> > the same NiFi instance for both QA and DEV, I will not only have to add a
>> > new "flag" attribute in my workflows, but I will also has to implement
>> some
>> > kind of mechanism in my Controller Services as well to separate DEV and
>> QA
>> > data. I don't think this is a really good idea.
>> >
>> > Regards,
>> >
>> >
>> > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
>> >
>> > Esteban Aliverti
>> > - Blog @ http://ilesteban.wordpress.com
>> >
>> > On Tue, Apr 28, 2015 at 3:34 PM, Joe Witt <jo...@gmail.com> wrote:
>> >
>> > > Esteban,
>> > >
>> > > Thank you for providing so much detail to what you're trying to do.  I
>> > > want to walk through this further with you to better understand the
>> > > perspective.  But first I will say that the pain you refer to with
>> > > removing a flow is something we will sort out.   You should be able to
>> > > simply right-click and delete so we will improve that until it is
>> > > true.
>> > >
>> > > Can you explain why you want to build on one instance, then make a
>> > > template, then go to another instance?  I am wondering if this is a
>> > > sort of dev/integration/production workflow.
>> > >
>> > > I think our general philosophy has been to enable developers to easily
>> > > build new extensions which are effectively unit tested and then put
>> > > those out on the instance where the data is flowing.  We then optimize
>> > > extensively around the operations/administrator experience.  We've
>> > > seen a consistent theme of need though to clarify how the developer
>> > > experience can/should work.  So happy to take more feedback on what
>> > > you're trying to do and also 'why' you're trying to do it so we can
>> > > help guide and so we can learn how to adjust as needed.
>> > >
>> > > Thanks
>> > > Joe
>> > >
>> > > On Tue, Apr 28, 2015 at 8:49 AM, Esteban Aliverti
>> > > <es...@gmail.com> wrote:
>> > > > Hi guys,
>> > > > I was wondering what is the best way to replace a complete workflow
>> in
>> > > > NiFi.
>> > > > My scenario is that I have multiple instances of NiFi running the
>> same
>> > > > workflow. At some point, when changes are introduced into one of the
>> > > > instances (let's say a new processor is introduced) I would like to
>> > > > replicate the changes to the other instances. For small changes,
>> things
>> > > can
>> > > > be easily handled, but for big changes - introduction of a whole new
>> > > > branch, for example - things are not so easy.
>> > > > Given that the workflows I'm currently using are in dev stage, I
>> don't
>> > > mind
>> > > > to lose any FlowFile that already exists in the NiFi instance I'm
>> > > updating.
>> > > >
>> > > > The way I'm currently handling this is:
>> > > > 1.- I create a new template from the NiFi instance where the changes
>> > were
>> > > > introduced.
>> > > > 2.- I export the template as an xml file.
>> > > > 3.- I import the template in the NiFi instance where I want the
>> changes
>> > > to
>> > > > be applied.
>> > > > 4.- I manually remove any processor from the NiFi instance where I
>> want
>> > > the
>> > > > changes to be applied. (for complex workflows, this step is a real
>> > > pita!).
>> > > > 5.- I use the template I've previously imported in the NiFi instance
>> > > where
>> > > > I want the changes to be applied.
>> > > >
>> > > > My question is: what is the best way to achieve this?
>> > > > Is there any way to completely delete the workflow that NiFi is
>> > currently
>> > > > executing?
>> > > >
>> > > > Regards,
>> > > >
>> > > > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
>> > > >
>> > > > Esteban Aliverti
>> > > > - Blog @ http://ilesteban.wordpress.com
>> > >
>> >
>>