You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Jorge Machado <jo...@me.com> on 2020/05/14 14:02:32 UTC
Re: Nifi - how to achieve a concurrent development and CI/CD
Hi,
Managing xml is always hard I think. Last time I need to do something similar we used https://nifi.apache.org/registry.html <https://nifi.apache.org/registry.html>
Works pretty well It was already 2 Years ago. Maybe now there is something better
> On 14. May 2020, at 15:57, Michal Slama (DHL IT Services), external <mi...@dhl.com> wrote:
>
> Hello,
>
> may I ask you for recommendations for development and CI/CD in NiFi?
> Pls let me describe our situation…I am a developer from DHL currently working on project including NiFi. It is part of our core and it is responsible for handling incoming data streams, data transormation and put it into various elastic search indexes (and queues).
>
> Up to now development was quite straightforward as only one developer did it. But we have extended our team recently and now we face problems how to correctly maintain development for more developers working in parallel and then CI/CD of it as we have classical dev/test/uat/project env. structure.
>
> Pls do you have any recommadation how to achive it? For now in general is enought. Its good to mention that currently we work with NiFi version 1.8…tried to upgrade to 1.9. but some of components failed so the upgrade was postponed. But with new features in 1.10. and 1.11. we head to uprade to these versions.
>
> Maybe if we can arrange a call it would be great!
>
> With regards,
> Michal Sláma
>
>
>
> This message is from DHL Information Services (Europe) s.r.o. and may contain confidential business information. It is intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient please contact the sender and delete this message and any attachment from your system. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its attachments is strictly prohibited.
Re: Nifi - how to achieve a concurrent development and CI/CD
Posted by Edward Armes <ed...@gmail.com>.
Hi Eric,
So I looked into this September last year. I found that a variant of master
driven development was the best course of action, for collaboration on a
single Nifi flow.
As far as I can tell the git integration deliberately designed to be as
simple a s possible, the down side of this is that git functionality like
branching just doesn't work with the git integration.
I briefly looked into seeing if I could sidestep a few of the issues by
changing how the flow version information was stored, but I ran into
several road blocks around adding support for history re-writes.......
Edward
On Fri, 15 May 2020, 01:17 Eric Secules, <es...@gmail.com> wrote:
> Hi Ami,
>
> I'm testing our NiFi application using a blackbox approach. We have a test
> harness that stages input files and waits for the corresponding outputs,
> then validates the content of the output. NiFi runs in a docker container
> (see https://hub.docker.com/r/apache/nifi/) and you can automate starting
> it up and orchestrating a test harness for your application in any
> competent CI/CD pipeline.
>
> Jorge, I don't understand your comment. NiFi registry can be synced to a
> git repo as a backup mechanism, but you can't simply make branches, work
> and merge them back, because even the smallest change in a flow will create
> a massive diff in the Registry's backup git repo. It would be great if NiFi
> had a way of highlighting the differences between flow versions on the
> canvas and allow you to choose which elements to keep and which to discard
> when resolving a merge conflict.
>
> -Eric
>
> On Thu, May 14, 2020 at 12:00 PM Jorge Machado <jo...@me.com> wrote:
>
>> I think we could improve Nifi by hooking up to GitHub somehow. So that we
>> don’t need the registry…
>>
>> On 14. May 2020, at 18:49, Ami Goldenberg <am...@gmail.com> wrote:
>>
>> Hi Eric,
>> Would love to know, what kind of tests do you write/run for NiFi? We were
>> just researching this topic.
>>
>> On Thu, May 14, 2020 at 6:38 PM Eric Secules <es...@gmail.com> wrote:
>>
>>> Hi Michal,
>>>
>>> I'm also using a single registry for development and production. It
>>> doesn't help with collaborating on the same process group as there is way
>>> for it to reconcile merge conflicts. Instead, the registry will earn you
>>> that you're about to overwrite someone else's changes. Another pain of
>>> concurrent development is there's no concept of a PR and no visual diff of
>>> your local changes making review difficult. I've backed our registry up to
>>> git I've set up a CI pipeline in Azure Devops which runs our tests every
>>> time a new version of a process group is checked in. It's better than
>>> nothing but I'd rather nifi implemented git flow. Developing as a team on
>>> nifi and nifi registry is like your whole team developing and pushing
>>> directly to master.
>>>
>>>
>>> -Eric
>>>
>>> On Thu., May 14, 2020, 7:02 a.m. Jorge Machado, <jo...@me.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Managing xml is always hard I think. Last time I need to do something
>>>> similar we used https://nifi.apache.org/registry.html
>>>> Works pretty well It was already 2 Years ago. Maybe now there is
>>>> something better
>>>>
>>>> On 14. May 2020, at 15:57, Michal Slama (DHL IT Services), external <
>>>> michal.slama@dhl.com> wrote:
>>>>
>>>> Hello,
>>>>
>>>> may I ask you for recommendations for development and CI/CD in NiFi?
>>>> Pls let me describe our situation…I am a developer from DHL currently
>>>> working on project including NiFi. It is part of our core and it is
>>>> responsible for handling incoming data streams, data transormation and
>>>> put it into various elastic search indexes (and queues).
>>>>
>>>> Up to now development was quite straightforward as only one developer
>>>> did it. But we have extended our team recently and now we face problems how
>>>> to correctly maintain development for more developers working in
>>>> parallel and then CI/CD of it as we have classical dev/test/uat/project
>>>> env. structure.
>>>>
>>>> Pls do you have any recommadation how to achive it? For now in general
>>>> is enought. Its good to mention that currently we work with NiFi version
>>>> 1.8…tried to upgrade to 1.9. but some of components failed so the
>>>> upgrade was postponed. But with new features in 1.10. and 1.11. we head
>>>> to uprade to these versions.
>>>>
>>>> Maybe if we can arrange a call it would be great!
>>>>
>>>> With regards,
>>>> Michal Sláma
>>>>
>>>>
>>>>
>>>> *This message is from DHL Information Services (Europe) s.r.o. and may
>>>> contain confidential business information. It is intended solely for the
>>>> use of the individual to whom it is addressed. If you are not the intended
>>>> recipient please contact the sender and delete this message and any
>>>> attachment from your system. Unauthorized publication, use, dissemination,
>>>> forwarding, printing or copying of this email and its attachments is
>>>> strictly prohibited.*
>>>>
>>>>
>>>>
>>
Re: Nifi - how to achieve a concurrent development and CI/CD
Posted by Eric Secules <es...@gmail.com>.
Hi Ami,
I'm testing our NiFi application using a blackbox approach. We have a test
harness that stages input files and waits for the corresponding outputs,
then validates the content of the output. NiFi runs in a docker container
(see https://hub.docker.com/r/apache/nifi/) and you can automate starting
it up and orchestrating a test harness for your application in any
competent CI/CD pipeline.
Jorge, I don't understand your comment. NiFi registry can be synced to a
git repo as a backup mechanism, but you can't simply make branches, work
and merge them back, because even the smallest change in a flow will create
a massive diff in the Registry's backup git repo. It would be great if NiFi
had a way of highlighting the differences between flow versions on the
canvas and allow you to choose which elements to keep and which to discard
when resolving a merge conflict.
-Eric
On Thu, May 14, 2020 at 12:00 PM Jorge Machado <jo...@me.com> wrote:
> I think we could improve Nifi by hooking up to GitHub somehow. So that we
> don’t need the registry…
>
> On 14. May 2020, at 18:49, Ami Goldenberg <am...@gmail.com> wrote:
>
> Hi Eric,
> Would love to know, what kind of tests do you write/run for NiFi? We were
> just researching this topic.
>
> On Thu, May 14, 2020 at 6:38 PM Eric Secules <es...@gmail.com> wrote:
>
>> Hi Michal,
>>
>> I'm also using a single registry for development and production. It
>> doesn't help with collaborating on the same process group as there is way
>> for it to reconcile merge conflicts. Instead, the registry will earn you
>> that you're about to overwrite someone else's changes. Another pain of
>> concurrent development is there's no concept of a PR and no visual diff of
>> your local changes making review difficult. I've backed our registry up to
>> git I've set up a CI pipeline in Azure Devops which runs our tests every
>> time a new version of a process group is checked in. It's better than
>> nothing but I'd rather nifi implemented git flow. Developing as a team on
>> nifi and nifi registry is like your whole team developing and pushing
>> directly to master.
>>
>>
>> -Eric
>>
>> On Thu., May 14, 2020, 7:02 a.m. Jorge Machado, <jo...@me.com> wrote:
>>
>>> Hi,
>>>
>>> Managing xml is always hard I think. Last time I need to do something
>>> similar we used https://nifi.apache.org/registry.html
>>> Works pretty well It was already 2 Years ago. Maybe now there is
>>> something better
>>>
>>> On 14. May 2020, at 15:57, Michal Slama (DHL IT Services), external <
>>> michal.slama@dhl.com> wrote:
>>>
>>> Hello,
>>>
>>> may I ask you for recommendations for development and CI/CD in NiFi?
>>> Pls let me describe our situation…I am a developer from DHL currently
>>> working on project including NiFi. It is part of our core and it is
>>> responsible for handling incoming data streams, data transormation and
>>> put it into various elastic search indexes (and queues).
>>>
>>> Up to now development was quite straightforward as only one developer
>>> did it. But we have extended our team recently and now we face problems how
>>> to correctly maintain development for more developers working in
>>> parallel and then CI/CD of it as we have classical dev/test/uat/project
>>> env. structure.
>>>
>>> Pls do you have any recommadation how to achive it? For now in general
>>> is enought. Its good to mention that currently we work with NiFi version
>>> 1.8…tried to upgrade to 1.9. but some of components failed so the
>>> upgrade was postponed. But with new features in 1.10. and 1.11. we head
>>> to uprade to these versions.
>>>
>>> Maybe if we can arrange a call it would be great!
>>>
>>> With regards,
>>> Michal Sláma
>>>
>>>
>>>
>>> *This message is from DHL Information Services (Europe) s.r.o. and may
>>> contain confidential business information. It is intended solely for the
>>> use of the individual to whom it is addressed. If you are not the intended
>>> recipient please contact the sender and delete this message and any
>>> attachment from your system. Unauthorized publication, use, dissemination,
>>> forwarding, printing or copying of this email and its attachments is
>>> strictly prohibited.*
>>>
>>>
>>>
>
Re: Nifi - how to achieve a concurrent development and CI/CD
Posted by Jorge Machado <jo...@me.com>.
I think we could improve Nifi by hooking up to GitHub somehow. So that we don’t need the registry…
> On 14. May 2020, at 18:49, Ami Goldenberg <am...@gmail.com> wrote:
>
> Hi Eric,
> Would love to know, what kind of tests do you write/run for NiFi? We were just researching this topic.
>
> On Thu, May 14, 2020 at 6:38 PM Eric Secules <esecules@gmail.com <ma...@gmail.com>> wrote:
> Hi Michal,
>
> I'm also using a single registry for development and production. It doesn't help with collaborating on the same process group as there is way for it to reconcile merge conflicts. Instead, the registry will earn you that you're about to overwrite someone else's changes. Another pain of concurrent development is there's no concept of a PR and no visual diff of your local changes making review difficult. I've backed our registry up to git I've set up a CI pipeline in Azure Devops which runs our tests every time a new version of a process group is checked in. It's better than nothing but I'd rather nifi implemented git flow. Developing as a team on nifi and nifi registry is like your whole team developing and pushing directly to master.
>
>
> -Eric
>
> On Thu., May 14, 2020, 7:02 a.m. Jorge Machado, <jomach@me.com <ma...@me.com>> wrote:
> Hi,
>
> Managing xml is always hard I think. Last time I need to do something similar we used https://nifi.apache.org/registry.html <https://nifi.apache.org/registry.html>
> Works pretty well It was already 2 Years ago. Maybe now there is something better
>
>> On 14. May 2020, at 15:57, Michal Slama (DHL IT Services), external <michal.slama@dhl.com <ma...@dhl.com>> wrote:
>>
>> Hello,
>>
>> may I ask you for recommendations for development and CI/CD in NiFi?
>> Pls let me describe our situation…I am a developer from DHL currently working on project including NiFi. It is part of our core and it is responsible for handling incoming data streams, data transormation and put it into various elastic search indexes (and queues).
>>
>> Up to now development was quite straightforward as only one developer did it. But we have extended our team recently and now we face problems how to correctly maintain development for more developers working in parallel and then CI/CD of it as we have classical dev/test/uat/project env. structure.
>>
>> Pls do you have any recommadation how to achive it? For now in general is enought. Its good to mention that currently we work with NiFi version 1.8…tried to upgrade to 1.9. but some of components failed so the upgrade was postponed. But with new features in 1.10. and 1.11. we head to uprade to these versions.
>>
>> Maybe if we can arrange a call it would be great!
>>
>> With regards,
>> Michal Sláma
>>
>>
>>
>> This message is from DHL Information Services (Europe) s.r.o. and may contain confidential business information. It is intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient please contact the sender and delete this message and any attachment from your system. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its attachments is strictly prohibited.
>
Re: Nifi - how to achieve a concurrent development and CI/CD
Posted by Ami Goldenberg <am...@gmail.com>.
Hi Eric,
Would love to know, what kind of tests do you write/run for NiFi? We were
just researching this topic.
On Thu, May 14, 2020 at 6:38 PM Eric Secules <es...@gmail.com> wrote:
> Hi Michal,
>
> I'm also using a single registry for development and production. It
> doesn't help with collaborating on the same process group as there is way
> for it to reconcile merge conflicts. Instead, the registry will earn you
> that you're about to overwrite someone else's changes. Another pain of
> concurrent development is there's no concept of a PR and no visual diff of
> your local changes making review difficult. I've backed our registry up to
> git I've set up a CI pipeline in Azure Devops which runs our tests every
> time a new version of a process group is checked in. It's better than
> nothing but I'd rather nifi implemented git flow. Developing as a team on
> nifi and nifi registry is like your whole team developing and pushing
> directly to master.
>
>
> -Eric
>
> On Thu., May 14, 2020, 7:02 a.m. Jorge Machado, <jo...@me.com> wrote:
>
>> Hi,
>>
>> Managing xml is always hard I think. Last time I need to do something
>> similar we used https://nifi.apache.org/registry.html
>> Works pretty well It was already 2 Years ago. Maybe now there is
>> something better
>>
>> On 14. May 2020, at 15:57, Michal Slama (DHL IT Services), external <
>> michal.slama@dhl.com> wrote:
>>
>> Hello,
>>
>> may I ask you for recommendations for development and CI/CD in NiFi?
>> Pls let me describe our situation…I am a developer from DHL currently
>> working on project including NiFi. It is part of our core and it is
>> responsible for handling incoming data streams, data transormation and
>> put it into various elastic search indexes (and queues).
>>
>> Up to now development was quite straightforward as only one developer did
>> it. But we have extended our team recently and now we face problems how to
>> correctly maintain development for more developers working in parallel
>> and then CI/CD of it as we have classical dev/test/uat/project env.
>> structure.
>>
>> Pls do you have any recommadation how to achive it? For now in general is
>> enought. Its good to mention that currently we work with NiFi version 1.8…tried
>> to upgrade to 1.9. but some of components failed so the upgrade was
>> postponed. But with new features in 1.10. and 1.11. we head to uprade to
>> these versions.
>>
>> Maybe if we can arrange a call it would be great!
>>
>> With regards,
>> Michal Sláma
>>
>>
>>
>> *This message is from DHL Information Services (Europe) s.r.o. and may
>> contain confidential business information. It is intended solely for the
>> use of the individual to whom it is addressed. If you are not the intended
>> recipient please contact the sender and delete this message and any
>> attachment from your system. Unauthorized publication, use, dissemination,
>> forwarding, printing or copying of this email and its attachments is
>> strictly prohibited.*
>>
>>
>>
Re: Nifi - how to achieve a concurrent development and CI/CD
Posted by Eric Secules <es...@gmail.com>.
Hi Michal,
I'm also using a single registry for development and production. It doesn't
help with collaborating on the same process group as there is way for it to
reconcile merge conflicts. Instead, the registry will earn you that you're
about to overwrite someone else's changes. Another pain of concurrent
development is there's no concept of a PR and no visual diff of your local
changes making review difficult. I've backed our registry up to git I've
set up a CI pipeline in Azure Devops which runs our tests every time a new
version of a process group is checked in. It's better than nothing but I'd
rather nifi implemented git flow. Developing as a team on nifi and nifi
registry is like your whole team developing and pushing directly to master.
-Eric
On Thu., May 14, 2020, 7:02 a.m. Jorge Machado, <jo...@me.com> wrote:
> Hi,
>
> Managing xml is always hard I think. Last time I need to do something
> similar we used https://nifi.apache.org/registry.html
> Works pretty well It was already 2 Years ago. Maybe now there is something
> better
>
> On 14. May 2020, at 15:57, Michal Slama (DHL IT Services), external <
> michal.slama@dhl.com> wrote:
>
> Hello,
>
> may I ask you for recommendations for development and CI/CD in NiFi?
> Pls let me describe our situation…I am a developer from DHL currently
> working on project including NiFi. It is part of our core and it is
> responsible for handling incoming data streams, data transormation and
> put it into various elastic search indexes (and queues).
>
> Up to now development was quite straightforward as only one developer did
> it. But we have extended our team recently and now we face problems how to
> correctly maintain development for more developers working in parallel
> and then CI/CD of it as we have classical dev/test/uat/project env.
> structure.
>
> Pls do you have any recommadation how to achive it? For now in general is
> enought. Its good to mention that currently we work with NiFi version 1.8…tried
> to upgrade to 1.9. but some of components failed so the upgrade was
> postponed. But with new features in 1.10. and 1.11. we head to uprade to
> these versions.
>
> Maybe if we can arrange a call it would be great!
>
> With regards,
> Michal Sláma
>
>
>
> *This message is from DHL Information Services (Europe) s.r.o. and may
> contain confidential business information. It is intended solely for the
> use of the individual to whom it is addressed. If you are not the intended
> recipient please contact the sender and delete this message and any
> attachment from your system. Unauthorized publication, use, dissemination,
> forwarding, printing or copying of this email and its attachments is
> strictly prohibited.*
>
>
>