You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Aldrin Piri <al...@gmail.com> on 2016/01/13 04:26:45 UTC
Direction for Integration Tests
All,
We have had a few conversations regarding integration tests in the past In
part, this issue has resurfaced thanks to the work going on with NIFI-1325
in extending AWS related processors and extensions. The issue at hand is
that, In this particular case, the contributor made test cases to
supplement their contribution but, because the integration level test
classes are ignored, had to create separate unit test classes to get these
tests to run. AbstractS3Test lays the ground work quite appropriately for
integration testing, but doesn't so much apply to unit tests where possible.
What I am interested in is establishing direction for how these integration
level tests should work within our codebase; superficially, what are the
conventions that make sense and how do we treat these within our codebase.
There is certainly grey area between unit and integration level tests at
spots within our codebase, but for the immediate future, I am looking to
establish a basis in terms of conventions so that we can avoid similar
issues moving forward and perhaps get some guidelines that the community
can use to provide these.
Following the pattern that is present in support of using the
maven-surefire-plugin, I could see a similar pattern of IT<Component> to
complement the Test<Component>. This is a default pattern that is
supported by the maven-failsafe-plugin, a likely candidate to drive this in
our build process under some associated profile [1].
In conjunction with this, I would like to see a profile that defaults to
being deactivated that runs integration level tests and migrate the
integration tests (AbstractS3Test and its subclasses) to an agreed upon
convention as previously mentioned.
We have an issue associated with this unit/integration level isolation,
NIFI-569 [2].
Any thoughts toward this would be most appreciated and I shall look to
incorporate the points made in the associated issues in JIRA.
Thanks!
[1]
http://maven.apache.org/surefire/maven-failsafe-plugin/examples/inclusion-exclusion.html
[2] https://issues.apache.org/jira/browse/NIFI-569
Re: Direction for Integration Tests
Posted by Joe Skora <js...@gmail.com>.
Aldrin,
+1 +++++
That sounds like a great approach, especially since it leverages existing
Surefire/Failsafe features.
Ideally, the TestXXX unit tests can filled out using Mocks and then the
ITXXX integration tests will follow up with the real service tests.
For the integration tests, since not every system will have all external
services available, have you given any thought to how to isolate tests to
services, so that if ITXXX needs S3 and ITYYY needs Hadoop I can run IT*
tests for available services but skip ones I don't have? I was thinking
Maven profiles might work, though I don't like that it requires
configuration to associate a profile with a set of tests I haven't yet
thought of a better solution.
Thanks,
Joe S
On Tue, Jan 12, 2016 at 10:26 PM, Aldrin Piri <al...@gmail.com> wrote:
> All,
>
> We have had a few conversations regarding integration tests in the past In
> part, this issue has resurfaced thanks to the work going on with NIFI-1325
> in extending AWS related processors and extensions. The issue at hand is
> that, In this particular case, the contributor made test cases to
> supplement their contribution but, because the integration level test
> classes are ignored, had to create separate unit test classes to get these
> tests to run. AbstractS3Test lays the ground work quite appropriately for
> integration testing, but doesn't so much apply to unit tests where
> possible.
>
> What I am interested in is establishing direction for how these integration
> level tests should work within our codebase; superficially, what are the
> conventions that make sense and how do we treat these within our codebase.
> There is certainly grey area between unit and integration level tests at
> spots within our codebase, but for the immediate future, I am looking to
> establish a basis in terms of conventions so that we can avoid similar
> issues moving forward and perhaps get some guidelines that the community
> can use to provide these.
>
> Following the pattern that is present in support of using the
> maven-surefire-plugin, I could see a similar pattern of IT<Component> to
> complement the Test<Component>. This is a default pattern that is
> supported by the maven-failsafe-plugin, a likely candidate to drive this in
> our build process under some associated profile [1].
>
> In conjunction with this, I would like to see a profile that defaults to
> being deactivated that runs integration level tests and migrate the
> integration tests (AbstractS3Test and its subclasses) to an agreed upon
> convention as previously mentioned.
>
> We have an issue associated with this unit/integration level isolation,
> NIFI-569 [2].
>
> Any thoughts toward this would be most appreciated and I shall look to
> incorporate the points made in the associated issues in JIRA.
>
> Thanks!
>
> [1]
>
> http://maven.apache.org/surefire/maven-failsafe-plugin/examples/inclusion-exclusion.html
> [2] https://issues.apache.org/jira/browse/NIFI-569
>
Re: Direction for Integration Tests
Posted by M Singh <ma...@yahoo.com.INVALID>.
Just another small note - I believe pure unit test are very imp because they are light weight and can give us 100% test coverage. The standalone/no ext dependencies gives us more confidence about interaction of the objects. They are not as light as unit tests and perhaps take a littler longer but can still be part of every build. Finally, the integration tests with ext dependencies/end to end test can run as required. For the last two integration tests it is difficult to get 100% coverage but they complement the unit test rather than replace them.
In my experience, this strategy has helped in many times in making sure that I catch bugs early/often in the dev cycle rather than in qa/prod.
On Wednesday, January 13, 2016 12:54 PM, Bryan Bende <bb...@gmail.com> wrote:
I also like Mans categories, as I know I have personally written some unit
tests that don't require any external dependencies, but might have veered
closer to an integration test than a true unit test.
Some examples I am thinking of are the tests for Put/GetSolr which use and
EmbeddedSolrServer, or TestListenSyslog which actually starts listening on
a port and receives messages over a socket.
I would still want those tests to run as part of the normal build, which I
think is what Mans is suggesting.
I'm not sure if the "no external dependencies" would always hold true as a
rule though. I could see there being some more complex tests between
internal components that might not have external
dependencies, but still makes sense to have in the IT part of the build. As
an example, if we ever create a mechanism to connect two processors
together and test the flow between them, this feels
more like a true integration test, yet it might not have any external
dependencies. So maybe it is a case by case basis.
On Wed, Jan 13, 2016 at 3:02 PM, Joe Skora <js...@gmail.com> wrote:
> I like that Mans' categories.
>
> Logically, I think the 2nd and 3rd categories fold together with "no
> dependency" integration tests being another case alongside "hadoop
> dependency", "jms dependency", "aws dependency", etc.
>
> Then I would expect that
>
> - every component should have unit tests,
> - most components (if not every) should have "no dependency" integration
> tests, and
> - externally connected components should have "external dependency"
> integration tests.
>
>
>
> On Wed, Jan 13, 2016 at 2:39 PM, M Singh <ma...@yahoo.com.invalid>
> wrote:
>
> > Hi:
> > My thought is that we can classify tests into 3 major categories -
> > - Pure unit test - class isolated using mocks etc
> > - Integration with no external dependencies - some interaction with
> > other classes/in memory db, or mocks, but no external resources.
> > - Integration with external resource upto and including end-to-end -
> > Which require real external resources (http endpoint, aws etc)
> >
> > In this way, we can always keep unit and integration w/o ext dependencies
> > tests while conditionally enable integration tests.
> > Mans
> >
> > On Wednesday, January 13, 2016 11:24 AM, Aldrin Piri <
> > aldrinpiri@gmail.com> wrote:
> >
> >
> > Definitely agree and can see value in those as well. The core issue I
> am
> > trying to address with this is that as highlighted by Joe.
> >
> > As an intermediate step, I have created a PR [1] that shows how I
> envision
> > a baseline of this. Currently this treats all ITs (in this external
> > resource context) the same without separate profiles. What I am trying
> to
> > do is not impede a place for unit tests (or, perhaps more generically,
> > those that can run as part of the build).
> >
> > [1] https://github.com/apache/nifi/pull/173
> >
> > On Wed, Jan 13, 2016 at 8:19 AM, Oleg Zhurakousky <
> > ozhurakousky@hortonworks.com> wrote:
> >
> > > Aldrin
> > >
> > > IMHO there are two types of integration testing; 1. Integration
> testing a
> > > Processor/ControllerService with the actual target system, 2.
> Integration
> > > testing of the flow or part of the flow.
> > > The second one essentially is the same as the first one but the target
> > > system is being NiFi itself and we are seriously lacking on that type
> of
> > > testing since NiFi is highly modularized and there is not a single
> module
> > > where such testing could be performed. As I’ve mentioned earlier in
> this
> > > list, I’ve already started such module on my fork
> > > https://github.com/olegz/nifi/tree/int-test which allowed me already
> to
> > > discover a few bugs and I think we need to start talking about pushing
> it
> > > to the trunk.
> > >
> > > So what I am essentially saying is that while profile and other maven
> > > tricks may set a convention to allow one to provide "conditional target
> > > system testing” (which is essentially what you mean by integration test
> > > with AWS etc), we still need to think about automating internal NiFi
> > > integration testing.
> > >
> > > Cheers
> > > Oleg
> > >
> > >
> > > On Jan 12, 2016, at 10:26 PM, Aldrin Piri <aldrinpiri@gmail.com
> <mailto:
> > > aldrinpiri@gmail.com>> wrote:
> > >
> > > All,
> > >
> > > We have had a few conversations regarding integration tests in the past
> > In
> > > part, this issue has resurfaced thanks to the work going on with
> > NIFI-1325
> > > in extending AWS related processors and extensions. The issue at hand
> is
> > > that, In this particular case, the contributor made test cases to
> > > supplement their contribution but, because the integration level test
> > > classes are ignored, had to create separate unit test classes to get
> > these
> > > tests to run. AbstractS3Test lays the ground work quite appropriately
> > for
> > > integration testing, but doesn't so much apply to unit tests where
> > > possible.
> > >
> > > What I am interested in is establishing direction for how these
> > integration
> > > level tests should work within our codebase; superficially, what are
> the
> > > conventions that make sense and how do we treat these within our
> > codebase.
> > > There is certainly grey area between unit and integration level tests
> at
> > > spots within our codebase, but for the immediate future, I am looking
> to
> > > establish a basis in terms of conventions so that we can avoid similar
> > > issues moving forward and perhaps get some guidelines that the
> community
> > > can use to provide these.
> > >
> > > Following the pattern that is present in support of using the
> > > maven-surefire-plugin, I could see a similar pattern of IT<Component>
> to
> > > complement the Test<Component>. This is a default pattern that is
> > > supported by the maven-failsafe-plugin, a likely candidate to drive
> this
> > in
> > > our build process under some associated profile [1].
> > >
> > > In conjunction with this, I would like to see a profile that defaults
> to
> > > being deactivated that runs integration level tests and migrate the
> > > integration tests (AbstractS3Test and its subclasses) to an agreed upon
> > > convention as previously mentioned.
> > >
> > > We have an issue associated with this unit/integration level isolation,
> > > NIFI-569 [2].
> > >
> > > Any thoughts toward this would be most appreciated and I shall look to
> > > incorporate the points made in the associated issues in JIRA.
> > >
> > > Thanks!
> > >
> > > [1]
> > >
> > >
> >
> http://maven.apache.org/surefire/maven-failsafe-plugin/examples/inclusion-exclusion.html
> > > [2] https://issues.apache.org/jira/browse/NIFI-569
> > >
> > >
> >
> >
> >
>
Re: Direction for Integration Tests
Posted by Bryan Bende <bb...@gmail.com>.
I also like Mans categories, as I know I have personally written some unit
tests that don't require any external dependencies, but might have veered
closer to an integration test than a true unit test.
Some examples I am thinking of are the tests for Put/GetSolr which use and
EmbeddedSolrServer, or TestListenSyslog which actually starts listening on
a port and receives messages over a socket.
I would still want those tests to run as part of the normal build, which I
think is what Mans is suggesting.
I'm not sure if the "no external dependencies" would always hold true as a
rule though. I could see there being some more complex tests between
internal components that might not have external
dependencies, but still makes sense to have in the IT part of the build. As
an example, if we ever create a mechanism to connect two processors
together and test the flow between them, this feels
more like a true integration test, yet it might not have any external
dependencies. So maybe it is a case by case basis.
On Wed, Jan 13, 2016 at 3:02 PM, Joe Skora <js...@gmail.com> wrote:
> I like that Mans' categories.
>
> Logically, I think the 2nd and 3rd categories fold together with "no
> dependency" integration tests being another case alongside "hadoop
> dependency", "jms dependency", "aws dependency", etc.
>
> Then I would expect that
>
> - every component should have unit tests,
> - most components (if not every) should have "no dependency" integration
> tests, and
> - externally connected components should have "external dependency"
> integration tests.
>
>
>
> On Wed, Jan 13, 2016 at 2:39 PM, M Singh <ma...@yahoo.com.invalid>
> wrote:
>
> > Hi:
> > My thought is that we can classify tests into 3 major categories -
> > - Pure unit test - class isolated using mocks etc
> > - Integration with no external dependencies - some interaction with
> > other classes/in memory db, or mocks, but no external resources.
> > - Integration with external resource upto and including end-to-end -
> > Which require real external resources (http endpoint, aws etc)
> >
> > In this way, we can always keep unit and integration w/o ext dependencies
> > tests while conditionally enable integration tests.
> > Mans
> >
> > On Wednesday, January 13, 2016 11:24 AM, Aldrin Piri <
> > aldrinpiri@gmail.com> wrote:
> >
> >
> > Definitely agree and can see value in those as well. The core issue I
> am
> > trying to address with this is that as highlighted by Joe.
> >
> > As an intermediate step, I have created a PR [1] that shows how I
> envision
> > a baseline of this. Currently this treats all ITs (in this external
> > resource context) the same without separate profiles. What I am trying
> to
> > do is not impede a place for unit tests (or, perhaps more generically,
> > those that can run as part of the build).
> >
> > [1] https://github.com/apache/nifi/pull/173
> >
> > On Wed, Jan 13, 2016 at 8:19 AM, Oleg Zhurakousky <
> > ozhurakousky@hortonworks.com> wrote:
> >
> > > Aldrin
> > >
> > > IMHO there are two types of integration testing; 1. Integration
> testing a
> > > Processor/ControllerService with the actual target system, 2.
> Integration
> > > testing of the flow or part of the flow.
> > > The second one essentially is the same as the first one but the target
> > > system is being NiFi itself and we are seriously lacking on that type
> of
> > > testing since NiFi is highly modularized and there is not a single
> module
> > > where such testing could be performed. As I’ve mentioned earlier in
> this
> > > list, I’ve already started such module on my fork
> > > https://github.com/olegz/nifi/tree/int-test which allowed me already
> to
> > > discover a few bugs and I think we need to start talking about pushing
> it
> > > to the trunk.
> > >
> > > So what I am essentially saying is that while profile and other maven
> > > tricks may set a convention to allow one to provide "conditional target
> > > system testing” (which is essentially what you mean by integration test
> > > with AWS etc), we still need to think about automating internal NiFi
> > > integration testing.
> > >
> > > Cheers
> > > Oleg
> > >
> > >
> > > On Jan 12, 2016, at 10:26 PM, Aldrin Piri <aldrinpiri@gmail.com
> <mailto:
> > > aldrinpiri@gmail.com>> wrote:
> > >
> > > All,
> > >
> > > We have had a few conversations regarding integration tests in the past
> > In
> > > part, this issue has resurfaced thanks to the work going on with
> > NIFI-1325
> > > in extending AWS related processors and extensions. The issue at hand
> is
> > > that, In this particular case, the contributor made test cases to
> > > supplement their contribution but, because the integration level test
> > > classes are ignored, had to create separate unit test classes to get
> > these
> > > tests to run. AbstractS3Test lays the ground work quite appropriately
> > for
> > > integration testing, but doesn't so much apply to unit tests where
> > > possible.
> > >
> > > What I am interested in is establishing direction for how these
> > integration
> > > level tests should work within our codebase; superficially, what are
> the
> > > conventions that make sense and how do we treat these within our
> > codebase.
> > > There is certainly grey area between unit and integration level tests
> at
> > > spots within our codebase, but for the immediate future, I am looking
> to
> > > establish a basis in terms of conventions so that we can avoid similar
> > > issues moving forward and perhaps get some guidelines that the
> community
> > > can use to provide these.
> > >
> > > Following the pattern that is present in support of using the
> > > maven-surefire-plugin, I could see a similar pattern of IT<Component>
> to
> > > complement the Test<Component>. This is a default pattern that is
> > > supported by the maven-failsafe-plugin, a likely candidate to drive
> this
> > in
> > > our build process under some associated profile [1].
> > >
> > > In conjunction with this, I would like to see a profile that defaults
> to
> > > being deactivated that runs integration level tests and migrate the
> > > integration tests (AbstractS3Test and its subclasses) to an agreed upon
> > > convention as previously mentioned.
> > >
> > > We have an issue associated with this unit/integration level isolation,
> > > NIFI-569 [2].
> > >
> > > Any thoughts toward this would be most appreciated and I shall look to
> > > incorporate the points made in the associated issues in JIRA.
> > >
> > > Thanks!
> > >
> > > [1]
> > >
> > >
> >
> http://maven.apache.org/surefire/maven-failsafe-plugin/examples/inclusion-exclusion.html
> > > [2] https://issues.apache.org/jira/browse/NIFI-569
> > >
> > >
> >
> >
> >
>
Re: Direction for Integration Tests
Posted by Joe Skora <js...@gmail.com>.
I like that Mans' categories.
Logically, I think the 2nd and 3rd categories fold together with "no
dependency" integration tests being another case alongside "hadoop
dependency", "jms dependency", "aws dependency", etc.
Then I would expect that
- every component should have unit tests,
- most components (if not every) should have "no dependency" integration
tests, and
- externally connected components should have "external dependency"
integration tests.
On Wed, Jan 13, 2016 at 2:39 PM, M Singh <ma...@yahoo.com.invalid>
wrote:
> Hi:
> My thought is that we can classify tests into 3 major categories -
> - Pure unit test - class isolated using mocks etc
> - Integration with no external dependencies - some interaction with
> other classes/in memory db, or mocks, but no external resources.
> - Integration with external resource upto and including end-to-end -
> Which require real external resources (http endpoint, aws etc)
>
> In this way, we can always keep unit and integration w/o ext dependencies
> tests while conditionally enable integration tests.
> Mans
>
> On Wednesday, January 13, 2016 11:24 AM, Aldrin Piri <
> aldrinpiri@gmail.com> wrote:
>
>
> Definitely agree and can see value in those as well. The core issue I am
> trying to address with this is that as highlighted by Joe.
>
> As an intermediate step, I have created a PR [1] that shows how I envision
> a baseline of this. Currently this treats all ITs (in this external
> resource context) the same without separate profiles. What I am trying to
> do is not impede a place for unit tests (or, perhaps more generically,
> those that can run as part of the build).
>
> [1] https://github.com/apache/nifi/pull/173
>
> On Wed, Jan 13, 2016 at 8:19 AM, Oleg Zhurakousky <
> ozhurakousky@hortonworks.com> wrote:
>
> > Aldrin
> >
> > IMHO there are two types of integration testing; 1. Integration testing a
> > Processor/ControllerService with the actual target system, 2. Integration
> > testing of the flow or part of the flow.
> > The second one essentially is the same as the first one but the target
> > system is being NiFi itself and we are seriously lacking on that type of
> > testing since NiFi is highly modularized and there is not a single module
> > where such testing could be performed. As I’ve mentioned earlier in this
> > list, I’ve already started such module on my fork
> > https://github.com/olegz/nifi/tree/int-test which allowed me already to
> > discover a few bugs and I think we need to start talking about pushing it
> > to the trunk.
> >
> > So what I am essentially saying is that while profile and other maven
> > tricks may set a convention to allow one to provide "conditional target
> > system testing” (which is essentially what you mean by integration test
> > with AWS etc), we still need to think about automating internal NiFi
> > integration testing.
> >
> > Cheers
> > Oleg
> >
> >
> > On Jan 12, 2016, at 10:26 PM, Aldrin Piri <aldrinpiri@gmail.com<mailto:
> > aldrinpiri@gmail.com>> wrote:
> >
> > All,
> >
> > We have had a few conversations regarding integration tests in the past
> In
> > part, this issue has resurfaced thanks to the work going on with
> NIFI-1325
> > in extending AWS related processors and extensions. The issue at hand is
> > that, In this particular case, the contributor made test cases to
> > supplement their contribution but, because the integration level test
> > classes are ignored, had to create separate unit test classes to get
> these
> > tests to run. AbstractS3Test lays the ground work quite appropriately
> for
> > integration testing, but doesn't so much apply to unit tests where
> > possible.
> >
> > What I am interested in is establishing direction for how these
> integration
> > level tests should work within our codebase; superficially, what are the
> > conventions that make sense and how do we treat these within our
> codebase.
> > There is certainly grey area between unit and integration level tests at
> > spots within our codebase, but for the immediate future, I am looking to
> > establish a basis in terms of conventions so that we can avoid similar
> > issues moving forward and perhaps get some guidelines that the community
> > can use to provide these.
> >
> > Following the pattern that is present in support of using the
> > maven-surefire-plugin, I could see a similar pattern of IT<Component> to
> > complement the Test<Component>. This is a default pattern that is
> > supported by the maven-failsafe-plugin, a likely candidate to drive this
> in
> > our build process under some associated profile [1].
> >
> > In conjunction with this, I would like to see a profile that defaults to
> > being deactivated that runs integration level tests and migrate the
> > integration tests (AbstractS3Test and its subclasses) to an agreed upon
> > convention as previously mentioned.
> >
> > We have an issue associated with this unit/integration level isolation,
> > NIFI-569 [2].
> >
> > Any thoughts toward this would be most appreciated and I shall look to
> > incorporate the points made in the associated issues in JIRA.
> >
> > Thanks!
> >
> > [1]
> >
> >
> http://maven.apache.org/surefire/maven-failsafe-plugin/examples/inclusion-exclusion.html
> > [2] https://issues.apache.org/jira/browse/NIFI-569
> >
> >
>
>
>
Re: Direction for Integration Tests
Posted by M Singh <ma...@yahoo.com.INVALID>.
Hi:
My thought is that we can classify tests into 3 major categories -
- Pure unit test - class isolated using mocks etc
- Integration with no external dependencies - some interaction with other classes/in memory db, or mocks, but no external resources.
- Integration with external resource upto and including end-to-end - Which require real external resources (http endpoint, aws etc)
In this way, we can always keep unit and integration w/o ext dependencies tests while conditionally enable integration tests.
Mans
On Wednesday, January 13, 2016 11:24 AM, Aldrin Piri <al...@gmail.com> wrote:
Definitely agree and can see value in those as well. The core issue I am
trying to address with this is that as highlighted by Joe.
As an intermediate step, I have created a PR [1] that shows how I envision
a baseline of this. Currently this treats all ITs (in this external
resource context) the same without separate profiles. What I am trying to
do is not impede a place for unit tests (or, perhaps more generically,
those that can run as part of the build).
[1] https://github.com/apache/nifi/pull/173
On Wed, Jan 13, 2016 at 8:19 AM, Oleg Zhurakousky <
ozhurakousky@hortonworks.com> wrote:
> Aldrin
>
> IMHO there are two types of integration testing; 1. Integration testing a
> Processor/ControllerService with the actual target system, 2. Integration
> testing of the flow or part of the flow.
> The second one essentially is the same as the first one but the target
> system is being NiFi itself and we are seriously lacking on that type of
> testing since NiFi is highly modularized and there is not a single module
> where such testing could be performed. As I’ve mentioned earlier in this
> list, I’ve already started such module on my fork
> https://github.com/olegz/nifi/tree/int-test which allowed me already to
> discover a few bugs and I think we need to start talking about pushing it
> to the trunk.
>
> So what I am essentially saying is that while profile and other maven
> tricks may set a convention to allow one to provide "conditional target
> system testing” (which is essentially what you mean by integration test
> with AWS etc), we still need to think about automating internal NiFi
> integration testing.
>
> Cheers
> Oleg
>
>
> On Jan 12, 2016, at 10:26 PM, Aldrin Piri <aldrinpiri@gmail.com<mailto:
> aldrinpiri@gmail.com>> wrote:
>
> All,
>
> We have had a few conversations regarding integration tests in the past In
> part, this issue has resurfaced thanks to the work going on with NIFI-1325
> in extending AWS related processors and extensions. The issue at hand is
> that, In this particular case, the contributor made test cases to
> supplement their contribution but, because the integration level test
> classes are ignored, had to create separate unit test classes to get these
> tests to run. AbstractS3Test lays the ground work quite appropriately for
> integration testing, but doesn't so much apply to unit tests where
> possible.
>
> What I am interested in is establishing direction for how these integration
> level tests should work within our codebase; superficially, what are the
> conventions that make sense and how do we treat these within our codebase.
> There is certainly grey area between unit and integration level tests at
> spots within our codebase, but for the immediate future, I am looking to
> establish a basis in terms of conventions so that we can avoid similar
> issues moving forward and perhaps get some guidelines that the community
> can use to provide these.
>
> Following the pattern that is present in support of using the
> maven-surefire-plugin, I could see a similar pattern of IT<Component> to
> complement the Test<Component>. This is a default pattern that is
> supported by the maven-failsafe-plugin, a likely candidate to drive this in
> our build process under some associated profile [1].
>
> In conjunction with this, I would like to see a profile that defaults to
> being deactivated that runs integration level tests and migrate the
> integration tests (AbstractS3Test and its subclasses) to an agreed upon
> convention as previously mentioned.
>
> We have an issue associated with this unit/integration level isolation,
> NIFI-569 [2].
>
> Any thoughts toward this would be most appreciated and I shall look to
> incorporate the points made in the associated issues in JIRA.
>
> Thanks!
>
> [1]
>
> http://maven.apache.org/surefire/maven-failsafe-plugin/examples/inclusion-exclusion.html
> [2] https://issues.apache.org/jira/browse/NIFI-569
>
>
Re: Direction for Integration Tests
Posted by Aldrin Piri <al...@gmail.com>.
Definitely agree and can see value in those as well. The core issue I am
trying to address with this is that as highlighted by Joe.
As an intermediate step, I have created a PR [1] that shows how I envision
a baseline of this. Currently this treats all ITs (in this external
resource context) the same without separate profiles. What I am trying to
do is not impede a place for unit tests (or, perhaps more generically,
those that can run as part of the build).
[1] https://github.com/apache/nifi/pull/173
On Wed, Jan 13, 2016 at 8:19 AM, Oleg Zhurakousky <
ozhurakousky@hortonworks.com> wrote:
> Aldrin
>
> IMHO there are two types of integration testing; 1. Integration testing a
> Processor/ControllerService with the actual target system, 2. Integration
> testing of the flow or part of the flow.
> The second one essentially is the same as the first one but the target
> system is being NiFi itself and we are seriously lacking on that type of
> testing since NiFi is highly modularized and there is not a single module
> where such testing could be performed. As I’ve mentioned earlier in this
> list, I’ve already started such module on my fork
> https://github.com/olegz/nifi/tree/int-test which allowed me already to
> discover a few bugs and I think we need to start talking about pushing it
> to the trunk.
>
> So what I am essentially saying is that while profile and other maven
> tricks may set a convention to allow one to provide "conditional target
> system testing” (which is essentially what you mean by integration test
> with AWS etc), we still need to think about automating internal NiFi
> integration testing.
>
> Cheers
> Oleg
>
>
> On Jan 12, 2016, at 10:26 PM, Aldrin Piri <aldrinpiri@gmail.com<mailto:
> aldrinpiri@gmail.com>> wrote:
>
> All,
>
> We have had a few conversations regarding integration tests in the past In
> part, this issue has resurfaced thanks to the work going on with NIFI-1325
> in extending AWS related processors and extensions. The issue at hand is
> that, In this particular case, the contributor made test cases to
> supplement their contribution but, because the integration level test
> classes are ignored, had to create separate unit test classes to get these
> tests to run. AbstractS3Test lays the ground work quite appropriately for
> integration testing, but doesn't so much apply to unit tests where
> possible.
>
> What I am interested in is establishing direction for how these integration
> level tests should work within our codebase; superficially, what are the
> conventions that make sense and how do we treat these within our codebase.
> There is certainly grey area between unit and integration level tests at
> spots within our codebase, but for the immediate future, I am looking to
> establish a basis in terms of conventions so that we can avoid similar
> issues moving forward and perhaps get some guidelines that the community
> can use to provide these.
>
> Following the pattern that is present in support of using the
> maven-surefire-plugin, I could see a similar pattern of IT<Component> to
> complement the Test<Component>. This is a default pattern that is
> supported by the maven-failsafe-plugin, a likely candidate to drive this in
> our build process under some associated profile [1].
>
> In conjunction with this, I would like to see a profile that defaults to
> being deactivated that runs integration level tests and migrate the
> integration tests (AbstractS3Test and its subclasses) to an agreed upon
> convention as previously mentioned.
>
> We have an issue associated with this unit/integration level isolation,
> NIFI-569 [2].
>
> Any thoughts toward this would be most appreciated and I shall look to
> incorporate the points made in the associated issues in JIRA.
>
> Thanks!
>
> [1]
>
> http://maven.apache.org/surefire/maven-failsafe-plugin/examples/inclusion-exclusion.html
> [2] https://issues.apache.org/jira/browse/NIFI-569
>
>
Re: Direction for Integration Tests
Posted by Oleg Zhurakousky <oz...@hortonworks.com>.
Aldrin
IMHO there are two types of integration testing; 1. Integration testing a Processor/ControllerService with the actual target system, 2. Integration testing of the flow or part of the flow.
The second one essentially is the same as the first one but the target system is being NiFi itself and we are seriously lacking on that type of testing since NiFi is highly modularized and there is not a single module where such testing could be performed. As I’ve mentioned earlier in this list, I’ve already started such module on my fork https://github.com/olegz/nifi/tree/int-test which allowed me already to discover a few bugs and I think we need to start talking about pushing it to the trunk.
So what I am essentially saying is that while profile and other maven tricks may set a convention to allow one to provide "conditional target system testing” (which is essentially what you mean by integration test with AWS etc), we still need to think about automating internal NiFi integration testing.
Cheers
Oleg
On Jan 12, 2016, at 10:26 PM, Aldrin Piri <al...@gmail.com>> wrote:
All,
We have had a few conversations regarding integration tests in the past In
part, this issue has resurfaced thanks to the work going on with NIFI-1325
in extending AWS related processors and extensions. The issue at hand is
that, In this particular case, the contributor made test cases to
supplement their contribution but, because the integration level test
classes are ignored, had to create separate unit test classes to get these
tests to run. AbstractS3Test lays the ground work quite appropriately for
integration testing, but doesn't so much apply to unit tests where possible.
What I am interested in is establishing direction for how these integration
level tests should work within our codebase; superficially, what are the
conventions that make sense and how do we treat these within our codebase.
There is certainly grey area between unit and integration level tests at
spots within our codebase, but for the immediate future, I am looking to
establish a basis in terms of conventions so that we can avoid similar
issues moving forward and perhaps get some guidelines that the community
can use to provide these.
Following the pattern that is present in support of using the
maven-surefire-plugin, I could see a similar pattern of IT<Component> to
complement the Test<Component>. This is a default pattern that is
supported by the maven-failsafe-plugin, a likely candidate to drive this in
our build process under some associated profile [1].
In conjunction with this, I would like to see a profile that defaults to
being deactivated that runs integration level tests and migrate the
integration tests (AbstractS3Test and its subclasses) to an agreed upon
convention as previously mentioned.
We have an issue associated with this unit/integration level isolation,
NIFI-569 [2].
Any thoughts toward this would be most appreciated and I shall look to
incorporate the points made in the associated issues in JIRA.
Thanks!
[1]
http://maven.apache.org/surefire/maven-failsafe-plugin/examples/inclusion-exclusion.html
[2] https://issues.apache.org/jira/browse/NIFI-569