You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Esteban Aliverti <es...@gmail.com> on 2015/04/28 14:49:01 UTC
Best way to replace a workflow
Hi guys,
I was wondering what is the best way to replace a complete workflow in
NiFi.
My scenario is that I have multiple instances of NiFi running the same
workflow. At some point, when changes are introduced into one of the
instances (let's say a new processor is introduced) I would like to
replicate the changes to the other instances. For small changes, things can
be easily handled, but for big changes - introduction of a whole new
branch, for example - things are not so easy.
Given that the workflows I'm currently using are in dev stage, I don't mind
to lose any FlowFile that already exists in the NiFi instance I'm updating.
The way I'm currently handling this is:
1.- I create a new template from the NiFi instance where the changes were
introduced.
2.- I export the template as an xml file.
3.- I import the template in the NiFi instance where I want the changes to
be applied.
4.- I manually remove any processor from the NiFi instance where I want the
changes to be applied. (for complex workflows, this step is a real pita!).
5.- I use the template I've previously imported in the NiFi instance where
I want the changes to be applied.
My question is: what is the best way to achieve this?
Is there any way to completely delete the workflow that NiFi is currently
executing?
Regards,
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Esteban Aliverti
- Blog @ http://ilesteban.wordpress.com
Re: Best way to replace a workflow
Posted by Edgardo Vega <ed...@gmail.com>.
Joe,
The workflow I am interested in is the dev/int/prod workflow with multiple
devs and continuous integration.
Cheers,
Edgardo
On Tuesday, April 28, 2015, Joe Witt <jo...@gmail.com> wrote:
> Esteban,
>
> Thank you for providing so much detail to what you're trying to do. I
> want to walk through this further with you to better understand the
> perspective. But first I will say that the pain you refer to with
> removing a flow is something we will sort out. You should be able to
> simply right-click and delete so we will improve that until it is
> true.
>
> Can you explain why you want to build on one instance, then make a
> template, then go to another instance? I am wondering if this is a
> sort of dev/integration/production workflow.
>
> I think our general philosophy has been to enable developers to easily
> build new extensions which are effectively unit tested and then put
> those out on the instance where the data is flowing. We then optimize
> extensively around the operations/administrator experience. We've
> seen a consistent theme of need though to clarify how the developer
> experience can/should work. So happy to take more feedback on what
> you're trying to do and also 'why' you're trying to do it so we can
> help guide and so we can learn how to adjust as needed.
>
> Thanks
> Joe
>
> On Tue, Apr 28, 2015 at 8:49 AM, Esteban Aliverti
> <esteban.aliverti@gmail.com <javascript:;>> wrote:
> > Hi guys,
> > I was wondering what is the best way to replace a complete workflow in
> > NiFi.
> > My scenario is that I have multiple instances of NiFi running the same
> > workflow. At some point, when changes are introduced into one of the
> > instances (let's say a new processor is introduced) I would like to
> > replicate the changes to the other instances. For small changes, things
> can
> > be easily handled, but for big changes - introduction of a whole new
> > branch, for example - things are not so easy.
> > Given that the workflows I'm currently using are in dev stage, I don't
> mind
> > to lose any FlowFile that already exists in the NiFi instance I'm
> updating.
> >
> > The way I'm currently handling this is:
> > 1.- I create a new template from the NiFi instance where the changes were
> > introduced.
> > 2.- I export the template as an xml file.
> > 3.- I import the template in the NiFi instance where I want the changes
> to
> > be applied.
> > 4.- I manually remove any processor from the NiFi instance where I want
> the
> > changes to be applied. (for complex workflows, this step is a real
> pita!).
> > 5.- I use the template I've previously imported in the NiFi instance
> where
> > I want the changes to be applied.
> >
> > My question is: what is the best way to achieve this?
> > Is there any way to completely delete the workflow that NiFi is currently
> > executing?
> >
> > Regards,
> >
> > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> >
> > Esteban Aliverti
> > - Blog @ http://ilesteban.wordpress.com
>
--
Cheers,
Edgardo
Sent from Gmail Mobile
Re: Best way to replace a workflow
Posted by Joe Witt <jo...@gmail.com>.
Hello
I've had this thread flagged for a while now. The sorts of concerns
raised in it have been fairly common patterns of requests from various
groups. I believe they are fairly well accounted for in the present
list of feature proposals as seen here [1].
Please take a look and advise if you think something is missing.
Thanks
Joe
[1] https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
On Wed, Apr 29, 2015 at 9:46 AM, Esteban Aliverti
<es...@gmail.com> wrote:
> Thanks Adam,
>
> I guess I was using templates because I didn't know about the
> existence of flow.xml.gz
> :).
> Now, if I have 2 different instances of NiFi, and I want to override the
> workflow in one of them with the workflow of the other, do I need to do
> anything else than just overwrite flow.xml.gz?
>
> Regards,
>
>
>
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
>
> Esteban Aliverti
> - Blog @ http://ilesteban.wordpress.com
>
> On Wed, Apr 29, 2015 at 2:04 PM, Adam Taft <ad...@adamtaft.com> wrote:
>
>> You can get a lot of mileage using the following suggestions:
>>
>> - There's nothing preventing you from storing the flow.xml.gz file into
>> version control. While this file is binary and you won't see any diffs on
>> it, your VCS (git, svn, etc.) won't have a problem storing it. If storing
>> the flow.xml.gz as binary is a problem, then you could simply uncompress it
>> to flow.xml first and version control that.
>>
>> - You might likewise want to version control your entire conf directory.
>>
>> - For any processor properties which can be configured using NIFI's
>> expression language, you can reference System properties and/or environment
>> variables. This allows you to have variant configurations based on the
>> deployment context. Things like hostnames or database connection properties
>> could in theory be configured externally.
>>
>> - Your custom controller services can likewise be configured to utilize
>> system properties or environment vars. Most standard controller services
>> don't have this capability yet, but work is being done that might help in
>> this area as well.
>>
>> I personally don't like using templates to solve your problem. Having flow
>> snippets that need to be stored and applied between dev/int/prod doesn't
>> make a lot of sense to me. Better in my mind is to have a single
>> flow.xml.gz that you can carry between those tiers. NIFI should allow you
>> to get most of the way there today, but if you run into problems, I'm sure
>> you could affect additional changes to help with this issue.
>>
>> Adam
>>
>>
>>
>> On Tue, Apr 28, 2015 at 9:58 AM, Esteban Aliverti <
>> esteban.aliverti@gmail.com> wrote:
>>
>> > Joe,
>> > thanks for your prompt response.
>> >
>> > Let me give you more background about what I'm currently doing:
>> > We are currently a group of 3 developers working on a NiFi workflow. My
>> two
>> > colleagues are in charge of building the individual processors required
>> for
>> > the workflow we are trying to create. My role, even if I'm also a
>> Processor
>> > developer myself, is kind of the "integrator" of these processors. I
>> have a
>> > local NiFi running and I'm in charge of the "workflow" design. I use this
>> > local instance as a playground for develop and initial testing.
>> > Once I've updated my local workflow, I usually "export" it into another
>> > instance that is currently being used by QA. This QA instance is only
>> > modified after I validate that my local instance is stable enough. Given
>> > that there are a number of other developers using this QA instance as
>> > clients, the update process is not so easy. Before I introduce any change
>> > in QA, I need to coordinate with the QA department first. This is one of
>> > the biggest reasons I keep a local NiFi instance for development. I can
>> do
>> > whatever I want whenever I want to in my local instance but I need to be
>> > very careful when I modify the QA instance.
>> >
>> > I have asked similar questions in this forum in the past and one of your
>> > recommendations was to use the same workflow for both QA and DEV. You
>> > suggested me to use an attribute in the FlowFiles as some kind of "flag"
>> so
>> > it can be routed to the DEV branch of the workflow. I see 2 problems with
>> > this approach (maybe because of my own development workflow):
>> > 1.- Sometimes I don't have connectivity to the server where the QA NiFi
>> > instance is. I would like to be able to continue my development tasks
>> even
>> > if I'm offline.
>> > 2.- My workflow is not stateless. I have a bunch of Controller Services
>> > that are used through the workflow that intentionally keep state. If I
>> use
>> > the same NiFi instance for both QA and DEV, I will not only have to add a
>> > new "flag" attribute in my workflows, but I will also has to implement
>> some
>> > kind of mechanism in my Controller Services as well to separate DEV and
>> QA
>> > data. I don't think this is a really good idea.
>> >
>> > Regards,
>> >
>> >
>> > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
>> >
>> > Esteban Aliverti
>> > - Blog @ http://ilesteban.wordpress.com
>> >
>> > On Tue, Apr 28, 2015 at 3:34 PM, Joe Witt <jo...@gmail.com> wrote:
>> >
>> > > Esteban,
>> > >
>> > > Thank you for providing so much detail to what you're trying to do. I
>> > > want to walk through this further with you to better understand the
>> > > perspective. But first I will say that the pain you refer to with
>> > > removing a flow is something we will sort out. You should be able to
>> > > simply right-click and delete so we will improve that until it is
>> > > true.
>> > >
>> > > Can you explain why you want to build on one instance, then make a
>> > > template, then go to another instance? I am wondering if this is a
>> > > sort of dev/integration/production workflow.
>> > >
>> > > I think our general philosophy has been to enable developers to easily
>> > > build new extensions which are effectively unit tested and then put
>> > > those out on the instance where the data is flowing. We then optimize
>> > > extensively around the operations/administrator experience. We've
>> > > seen a consistent theme of need though to clarify how the developer
>> > > experience can/should work. So happy to take more feedback on what
>> > > you're trying to do and also 'why' you're trying to do it so we can
>> > > help guide and so we can learn how to adjust as needed.
>> > >
>> > > Thanks
>> > > Joe
>> > >
>> > > On Tue, Apr 28, 2015 at 8:49 AM, Esteban Aliverti
>> > > <es...@gmail.com> wrote:
>> > > > Hi guys,
>> > > > I was wondering what is the best way to replace a complete workflow
>> in
>> > > > NiFi.
>> > > > My scenario is that I have multiple instances of NiFi running the
>> same
>> > > > workflow. At some point, when changes are introduced into one of the
>> > > > instances (let's say a new processor is introduced) I would like to
>> > > > replicate the changes to the other instances. For small changes,
>> things
>> > > can
>> > > > be easily handled, but for big changes - introduction of a whole new
>> > > > branch, for example - things are not so easy.
>> > > > Given that the workflows I'm currently using are in dev stage, I
>> don't
>> > > mind
>> > > > to lose any FlowFile that already exists in the NiFi instance I'm
>> > > updating.
>> > > >
>> > > > The way I'm currently handling this is:
>> > > > 1.- I create a new template from the NiFi instance where the changes
>> > were
>> > > > introduced.
>> > > > 2.- I export the template as an xml file.
>> > > > 3.- I import the template in the NiFi instance where I want the
>> changes
>> > > to
>> > > > be applied.
>> > > > 4.- I manually remove any processor from the NiFi instance where I
>> want
>> > > the
>> > > > changes to be applied. (for complex workflows, this step is a real
>> > > pita!).
>> > > > 5.- I use the template I've previously imported in the NiFi instance
>> > > where
>> > > > I want the changes to be applied.
>> > > >
>> > > > My question is: what is the best way to achieve this?
>> > > > Is there any way to completely delete the workflow that NiFi is
>> > currently
>> > > > executing?
>> > > >
>> > > > Regards,
>> > > >
>> > > > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
>> > > >
>> > > > Esteban Aliverti
>> > > > - Blog @ http://ilesteban.wordpress.com
>> > >
>> >
>>
Re: Best way to replace a workflow
Posted by Esteban Aliverti <es...@gmail.com>.
Thanks Adam,
I guess I was using templates because I didn't know about the
existence of flow.xml.gz
:).
Now, if I have 2 different instances of NiFi, and I want to override the
workflow in one of them with the workflow of the other, do I need to do
anything else than just overwrite flow.xml.gz?
Regards,
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Esteban Aliverti
- Blog @ http://ilesteban.wordpress.com
On Wed, Apr 29, 2015 at 2:04 PM, Adam Taft <ad...@adamtaft.com> wrote:
> You can get a lot of mileage using the following suggestions:
>
> - There's nothing preventing you from storing the flow.xml.gz file into
> version control. While this file is binary and you won't see any diffs on
> it, your VCS (git, svn, etc.) won't have a problem storing it. If storing
> the flow.xml.gz as binary is a problem, then you could simply uncompress it
> to flow.xml first and version control that.
>
> - You might likewise want to version control your entire conf directory.
>
> - For any processor properties which can be configured using NIFI's
> expression language, you can reference System properties and/or environment
> variables. This allows you to have variant configurations based on the
> deployment context. Things like hostnames or database connection properties
> could in theory be configured externally.
>
> - Your custom controller services can likewise be configured to utilize
> system properties or environment vars. Most standard controller services
> don't have this capability yet, but work is being done that might help in
> this area as well.
>
> I personally don't like using templates to solve your problem. Having flow
> snippets that need to be stored and applied between dev/int/prod doesn't
> make a lot of sense to me. Better in my mind is to have a single
> flow.xml.gz that you can carry between those tiers. NIFI should allow you
> to get most of the way there today, but if you run into problems, I'm sure
> you could affect additional changes to help with this issue.
>
> Adam
>
>
>
> On Tue, Apr 28, 2015 at 9:58 AM, Esteban Aliverti <
> esteban.aliverti@gmail.com> wrote:
>
> > Joe,
> > thanks for your prompt response.
> >
> > Let me give you more background about what I'm currently doing:
> > We are currently a group of 3 developers working on a NiFi workflow. My
> two
> > colleagues are in charge of building the individual processors required
> for
> > the workflow we are trying to create. My role, even if I'm also a
> Processor
> > developer myself, is kind of the "integrator" of these processors. I
> have a
> > local NiFi running and I'm in charge of the "workflow" design. I use this
> > local instance as a playground for develop and initial testing.
> > Once I've updated my local workflow, I usually "export" it into another
> > instance that is currently being used by QA. This QA instance is only
> > modified after I validate that my local instance is stable enough. Given
> > that there are a number of other developers using this QA instance as
> > clients, the update process is not so easy. Before I introduce any change
> > in QA, I need to coordinate with the QA department first. This is one of
> > the biggest reasons I keep a local NiFi instance for development. I can
> do
> > whatever I want whenever I want to in my local instance but I need to be
> > very careful when I modify the QA instance.
> >
> > I have asked similar questions in this forum in the past and one of your
> > recommendations was to use the same workflow for both QA and DEV. You
> > suggested me to use an attribute in the FlowFiles as some kind of "flag"
> so
> > it can be routed to the DEV branch of the workflow. I see 2 problems with
> > this approach (maybe because of my own development workflow):
> > 1.- Sometimes I don't have connectivity to the server where the QA NiFi
> > instance is. I would like to be able to continue my development tasks
> even
> > if I'm offline.
> > 2.- My workflow is not stateless. I have a bunch of Controller Services
> > that are used through the workflow that intentionally keep state. If I
> use
> > the same NiFi instance for both QA and DEV, I will not only have to add a
> > new "flag" attribute in my workflows, but I will also has to implement
> some
> > kind of mechanism in my Controller Services as well to separate DEV and
> QA
> > data. I don't think this is a really good idea.
> >
> > Regards,
> >
> >
> > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> >
> > Esteban Aliverti
> > - Blog @ http://ilesteban.wordpress.com
> >
> > On Tue, Apr 28, 2015 at 3:34 PM, Joe Witt <jo...@gmail.com> wrote:
> >
> > > Esteban,
> > >
> > > Thank you for providing so much detail to what you're trying to do. I
> > > want to walk through this further with you to better understand the
> > > perspective. But first I will say that the pain you refer to with
> > > removing a flow is something we will sort out. You should be able to
> > > simply right-click and delete so we will improve that until it is
> > > true.
> > >
> > > Can you explain why you want to build on one instance, then make a
> > > template, then go to another instance? I am wondering if this is a
> > > sort of dev/integration/production workflow.
> > >
> > > I think our general philosophy has been to enable developers to easily
> > > build new extensions which are effectively unit tested and then put
> > > those out on the instance where the data is flowing. We then optimize
> > > extensively around the operations/administrator experience. We've
> > > seen a consistent theme of need though to clarify how the developer
> > > experience can/should work. So happy to take more feedback on what
> > > you're trying to do and also 'why' you're trying to do it so we can
> > > help guide and so we can learn how to adjust as needed.
> > >
> > > Thanks
> > > Joe
> > >
> > > On Tue, Apr 28, 2015 at 8:49 AM, Esteban Aliverti
> > > <es...@gmail.com> wrote:
> > > > Hi guys,
> > > > I was wondering what is the best way to replace a complete workflow
> in
> > > > NiFi.
> > > > My scenario is that I have multiple instances of NiFi running the
> same
> > > > workflow. At some point, when changes are introduced into one of the
> > > > instances (let's say a new processor is introduced) I would like to
> > > > replicate the changes to the other instances. For small changes,
> things
> > > can
> > > > be easily handled, but for big changes - introduction of a whole new
> > > > branch, for example - things are not so easy.
> > > > Given that the workflows I'm currently using are in dev stage, I
> don't
> > > mind
> > > > to lose any FlowFile that already exists in the NiFi instance I'm
> > > updating.
> > > >
> > > > The way I'm currently handling this is:
> > > > 1.- I create a new template from the NiFi instance where the changes
> > were
> > > > introduced.
> > > > 2.- I export the template as an xml file.
> > > > 3.- I import the template in the NiFi instance where I want the
> changes
> > > to
> > > > be applied.
> > > > 4.- I manually remove any processor from the NiFi instance where I
> want
> > > the
> > > > changes to be applied. (for complex workflows, this step is a real
> > > pita!).
> > > > 5.- I use the template I've previously imported in the NiFi instance
> > > where
> > > > I want the changes to be applied.
> > > >
> > > > My question is: what is the best way to achieve this?
> > > > Is there any way to completely delete the workflow that NiFi is
> > currently
> > > > executing?
> > > >
> > > > Regards,
> > > >
> > > > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> > > >
> > > > Esteban Aliverti
> > > > - Blog @ http://ilesteban.wordpress.com
> > >
> >
>
Re: Best way to replace a workflow
Posted by Adam Taft <ad...@adamtaft.com>.
You can get a lot of mileage using the following suggestions:
- There's nothing preventing you from storing the flow.xml.gz file into
version control. While this file is binary and you won't see any diffs on
it, your VCS (git, svn, etc.) won't have a problem storing it. If storing
the flow.xml.gz as binary is a problem, then you could simply uncompress it
to flow.xml first and version control that.
- You might likewise want to version control your entire conf directory.
- For any processor properties which can be configured using NIFI's
expression language, you can reference System properties and/or environment
variables. This allows you to have variant configurations based on the
deployment context. Things like hostnames or database connection properties
could in theory be configured externally.
- Your custom controller services can likewise be configured to utilize
system properties or environment vars. Most standard controller services
don't have this capability yet, but work is being done that might help in
this area as well.
I personally don't like using templates to solve your problem. Having flow
snippets that need to be stored and applied between dev/int/prod doesn't
make a lot of sense to me. Better in my mind is to have a single
flow.xml.gz that you can carry between those tiers. NIFI should allow you
to get most of the way there today, but if you run into problems, I'm sure
you could affect additional changes to help with this issue.
Adam
On Tue, Apr 28, 2015 at 9:58 AM, Esteban Aliverti <
esteban.aliverti@gmail.com> wrote:
> Joe,
> thanks for your prompt response.
>
> Let me give you more background about what I'm currently doing:
> We are currently a group of 3 developers working on a NiFi workflow. My two
> colleagues are in charge of building the individual processors required for
> the workflow we are trying to create. My role, even if I'm also a Processor
> developer myself, is kind of the "integrator" of these processors. I have a
> local NiFi running and I'm in charge of the "workflow" design. I use this
> local instance as a playground for develop and initial testing.
> Once I've updated my local workflow, I usually "export" it into another
> instance that is currently being used by QA. This QA instance is only
> modified after I validate that my local instance is stable enough. Given
> that there are a number of other developers using this QA instance as
> clients, the update process is not so easy. Before I introduce any change
> in QA, I need to coordinate with the QA department first. This is one of
> the biggest reasons I keep a local NiFi instance for development. I can do
> whatever I want whenever I want to in my local instance but I need to be
> very careful when I modify the QA instance.
>
> I have asked similar questions in this forum in the past and one of your
> recommendations was to use the same workflow for both QA and DEV. You
> suggested me to use an attribute in the FlowFiles as some kind of "flag" so
> it can be routed to the DEV branch of the workflow. I see 2 problems with
> this approach (maybe because of my own development workflow):
> 1.- Sometimes I don't have connectivity to the server where the QA NiFi
> instance is. I would like to be able to continue my development tasks even
> if I'm offline.
> 2.- My workflow is not stateless. I have a bunch of Controller Services
> that are used through the workflow that intentionally keep state. If I use
> the same NiFi instance for both QA and DEV, I will not only have to add a
> new "flag" attribute in my workflows, but I will also has to implement some
> kind of mechanism in my Controller Services as well to separate DEV and QA
> data. I don't think this is a really good idea.
>
> Regards,
>
>
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
>
> Esteban Aliverti
> - Blog @ http://ilesteban.wordpress.com
>
> On Tue, Apr 28, 2015 at 3:34 PM, Joe Witt <jo...@gmail.com> wrote:
>
> > Esteban,
> >
> > Thank you for providing so much detail to what you're trying to do. I
> > want to walk through this further with you to better understand the
> > perspective. But first I will say that the pain you refer to with
> > removing a flow is something we will sort out. You should be able to
> > simply right-click and delete so we will improve that until it is
> > true.
> >
> > Can you explain why you want to build on one instance, then make a
> > template, then go to another instance? I am wondering if this is a
> > sort of dev/integration/production workflow.
> >
> > I think our general philosophy has been to enable developers to easily
> > build new extensions which are effectively unit tested and then put
> > those out on the instance where the data is flowing. We then optimize
> > extensively around the operations/administrator experience. We've
> > seen a consistent theme of need though to clarify how the developer
> > experience can/should work. So happy to take more feedback on what
> > you're trying to do and also 'why' you're trying to do it so we can
> > help guide and so we can learn how to adjust as needed.
> >
> > Thanks
> > Joe
> >
> > On Tue, Apr 28, 2015 at 8:49 AM, Esteban Aliverti
> > <es...@gmail.com> wrote:
> > > Hi guys,
> > > I was wondering what is the best way to replace a complete workflow in
> > > NiFi.
> > > My scenario is that I have multiple instances of NiFi running the same
> > > workflow. At some point, when changes are introduced into one of the
> > > instances (let's say a new processor is introduced) I would like to
> > > replicate the changes to the other instances. For small changes, things
> > can
> > > be easily handled, but for big changes - introduction of a whole new
> > > branch, for example - things are not so easy.
> > > Given that the workflows I'm currently using are in dev stage, I don't
> > mind
> > > to lose any FlowFile that already exists in the NiFi instance I'm
> > updating.
> > >
> > > The way I'm currently handling this is:
> > > 1.- I create a new template from the NiFi instance where the changes
> were
> > > introduced.
> > > 2.- I export the template as an xml file.
> > > 3.- I import the template in the NiFi instance where I want the changes
> > to
> > > be applied.
> > > 4.- I manually remove any processor from the NiFi instance where I want
> > the
> > > changes to be applied. (for complex workflows, this step is a real
> > pita!).
> > > 5.- I use the template I've previously imported in the NiFi instance
> > where
> > > I want the changes to be applied.
> > >
> > > My question is: what is the best way to achieve this?
> > > Is there any way to completely delete the workflow that NiFi is
> currently
> > > executing?
> > >
> > > Regards,
> > >
> > > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> > >
> > > Esteban Aliverti
> > > - Blog @ http://ilesteban.wordpress.com
> >
>
Re: Best way to replace a workflow
Posted by Esteban Aliverti <es...@gmail.com>.
Joe,
thanks for your prompt response.
Let me give you more background about what I'm currently doing:
We are currently a group of 3 developers working on a NiFi workflow. My two
colleagues are in charge of building the individual processors required for
the workflow we are trying to create. My role, even if I'm also a Processor
developer myself, is kind of the "integrator" of these processors. I have a
local NiFi running and I'm in charge of the "workflow" design. I use this
local instance as a playground for develop and initial testing.
Once I've updated my local workflow, I usually "export" it into another
instance that is currently being used by QA. This QA instance is only
modified after I validate that my local instance is stable enough. Given
that there are a number of other developers using this QA instance as
clients, the update process is not so easy. Before I introduce any change
in QA, I need to coordinate with the QA department first. This is one of
the biggest reasons I keep a local NiFi instance for development. I can do
whatever I want whenever I want to in my local instance but I need to be
very careful when I modify the QA instance.
I have asked similar questions in this forum in the past and one of your
recommendations was to use the same workflow for both QA and DEV. You
suggested me to use an attribute in the FlowFiles as some kind of "flag" so
it can be routed to the DEV branch of the workflow. I see 2 problems with
this approach (maybe because of my own development workflow):
1.- Sometimes I don't have connectivity to the server where the QA NiFi
instance is. I would like to be able to continue my development tasks even
if I'm offline.
2.- My workflow is not stateless. I have a bunch of Controller Services
that are used through the workflow that intentionally keep state. If I use
the same NiFi instance for both QA and DEV, I will not only have to add a
new "flag" attribute in my workflows, but I will also has to implement some
kind of mechanism in my Controller Services as well to separate DEV and QA
data. I don't think this is a really good idea.
Regards,
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Esteban Aliverti
- Blog @ http://ilesteban.wordpress.com
On Tue, Apr 28, 2015 at 3:34 PM, Joe Witt <jo...@gmail.com> wrote:
> Esteban,
>
> Thank you for providing so much detail to what you're trying to do. I
> want to walk through this further with you to better understand the
> perspective. But first I will say that the pain you refer to with
> removing a flow is something we will sort out. You should be able to
> simply right-click and delete so we will improve that until it is
> true.
>
> Can you explain why you want to build on one instance, then make a
> template, then go to another instance? I am wondering if this is a
> sort of dev/integration/production workflow.
>
> I think our general philosophy has been to enable developers to easily
> build new extensions which are effectively unit tested and then put
> those out on the instance where the data is flowing. We then optimize
> extensively around the operations/administrator experience. We've
> seen a consistent theme of need though to clarify how the developer
> experience can/should work. So happy to take more feedback on what
> you're trying to do and also 'why' you're trying to do it so we can
> help guide and so we can learn how to adjust as needed.
>
> Thanks
> Joe
>
> On Tue, Apr 28, 2015 at 8:49 AM, Esteban Aliverti
> <es...@gmail.com> wrote:
> > Hi guys,
> > I was wondering what is the best way to replace a complete workflow in
> > NiFi.
> > My scenario is that I have multiple instances of NiFi running the same
> > workflow. At some point, when changes are introduced into one of the
> > instances (let's say a new processor is introduced) I would like to
> > replicate the changes to the other instances. For small changes, things
> can
> > be easily handled, but for big changes - introduction of a whole new
> > branch, for example - things are not so easy.
> > Given that the workflows I'm currently using are in dev stage, I don't
> mind
> > to lose any FlowFile that already exists in the NiFi instance I'm
> updating.
> >
> > The way I'm currently handling this is:
> > 1.- I create a new template from the NiFi instance where the changes were
> > introduced.
> > 2.- I export the template as an xml file.
> > 3.- I import the template in the NiFi instance where I want the changes
> to
> > be applied.
> > 4.- I manually remove any processor from the NiFi instance where I want
> the
> > changes to be applied. (for complex workflows, this step is a real
> pita!).
> > 5.- I use the template I've previously imported in the NiFi instance
> where
> > I want the changes to be applied.
> >
> > My question is: what is the best way to achieve this?
> > Is there any way to completely delete the workflow that NiFi is currently
> > executing?
> >
> > Regards,
> >
> > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> >
> > Esteban Aliverti
> > - Blog @ http://ilesteban.wordpress.com
>
Re: Best way to replace a workflow
Posted by Joe Witt <jo...@gmail.com>.
Esteban,
Thank you for providing so much detail to what you're trying to do. I
want to walk through this further with you to better understand the
perspective. But first I will say that the pain you refer to with
removing a flow is something we will sort out. You should be able to
simply right-click and delete so we will improve that until it is
true.
Can you explain why you want to build on one instance, then make a
template, then go to another instance? I am wondering if this is a
sort of dev/integration/production workflow.
I think our general philosophy has been to enable developers to easily
build new extensions which are effectively unit tested and then put
those out on the instance where the data is flowing. We then optimize
extensively around the operations/administrator experience. We've
seen a consistent theme of need though to clarify how the developer
experience can/should work. So happy to take more feedback on what
you're trying to do and also 'why' you're trying to do it so we can
help guide and so we can learn how to adjust as needed.
Thanks
Joe
On Tue, Apr 28, 2015 at 8:49 AM, Esteban Aliverti
<es...@gmail.com> wrote:
> Hi guys,
> I was wondering what is the best way to replace a complete workflow in
> NiFi.
> My scenario is that I have multiple instances of NiFi running the same
> workflow. At some point, when changes are introduced into one of the
> instances (let's say a new processor is introduced) I would like to
> replicate the changes to the other instances. For small changes, things can
> be easily handled, but for big changes - introduction of a whole new
> branch, for example - things are not so easy.
> Given that the workflows I'm currently using are in dev stage, I don't mind
> to lose any FlowFile that already exists in the NiFi instance I'm updating.
>
> The way I'm currently handling this is:
> 1.- I create a new template from the NiFi instance where the changes were
> introduced.
> 2.- I export the template as an xml file.
> 3.- I import the template in the NiFi instance where I want the changes to
> be applied.
> 4.- I manually remove any processor from the NiFi instance where I want the
> changes to be applied. (for complex workflows, this step is a real pita!).
> 5.- I use the template I've previously imported in the NiFi instance where
> I want the changes to be applied.
>
> My question is: what is the best way to achieve this?
> Is there any way to completely delete the workflow that NiFi is currently
> executing?
>
> Regards,
>
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
>
> Esteban Aliverti
> - Blog @ http://ilesteban.wordpress.com
Re: Best way to replace a workflow
Posted by Edgardo Vega <ed...@gmail.com>.
I think that just touches on the biggest issue I am struggling with, which
is how does nifi fit into my normal developer workflow. That is git,
Jenkins, etc. I have also done what you have described but that isn't very
developer friendly.
What is the prescribed development methodology is recommended for using
nifi in a group development environment?
On Tuesday, April 28, 2015, Esteban Aliverti <es...@gmail.com>
wrote:
> Hi guys,
> I was wondering what is the best way to replace a complete workflow in
> NiFi.
> My scenario is that I have multiple instances of NiFi running the same
> workflow. At some point, when changes are introduced into one of the
> instances (let's say a new processor is introduced) I would like to
> replicate the changes to the other instances. For small changes, things can
> be easily handled, but for big changes - introduction of a whole new
> branch, for example - things are not so easy.
> Given that the workflows I'm currently using are in dev stage, I don't mind
> to lose any FlowFile that already exists in the NiFi instance I'm updating.
>
> The way I'm currently handling this is:
> 1.- I create a new template from the NiFi instance where the changes were
> introduced.
> 2.- I export the template as an xml file.
> 3.- I import the template in the NiFi instance where I want the changes to
> be applied.
> 4.- I manually remove any processor from the NiFi instance where I want the
> changes to be applied. (for complex workflows, this step is a real pita!).
> 5.- I use the template I've previously imported in the NiFi instance where
> I want the changes to be applied.
>
> My question is: what is the best way to achieve this?
> Is there any way to completely delete the workflow that NiFi is currently
> executing?
>
> Regards,
>
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
>
> Esteban Aliverti
> - Blog @ http://ilesteban.wordpress.com
>
--
Cheers,
Edgardo
Sent from Gmail Mobile