You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pirk.apache.org by Darin Johnson <db...@gmail.com> on 2016/10/20 04:22:45 UTC
[Discuss] Submodules (Phase 2) observations
I threw up a WIP for phase 1 of the submodule refactor. This involved
pulling out the pieces for the storm, spark, and mapreduce responsers.
Mostly ran into some difficulties of hard coded conditionals for each
framework in DistributedTestSuite and/or BaseTests.
The next logical module to pull out is Responder. However, in order to do
so we need to pull DistributedTestSuite and BaseTests. While this seems OK
with the main code base, it's not so with the test code base as a few of
the tests explicitly rely on BaseTests and Inputs.
I'd like to get some comments on where people believe the BaseTests and
Inputs Classes belong within the code base, so that I can plan this out
accordingly.
Re: [Discuss] Submodules (Phase 2) observations
Posted by Darin Johnson <db...@gmail.com>.
I'd probably go with a little replicated code is tests in pirk-core and a
distributedtestsuite submodule, as it'd allow us to refactor the two pieces
independently. I'd suggest we take the first pr plus a minor refactor of
distributedtestsuite into a submodule. I could get that done next week.
On Oct 27, 2016 6:48 PM, "Ellison Anne Williams" <ea...@apache.org>
wrote:
> Hi Darin,
>
> A complete refactor of the tests would probably be ideal - however, it
> doesn't seem to make sense to tackle that as a part of the initial
> submodule refactor. Perhaps leaving the Responder refactor (and test
> refactor) to a subsequent PR makes more sense.
>
> Absent a complete test refactor, what are your thoughts on the best path
> forward?
>
> Thanks-
>
> Ellison Anne
>
> On Thu, Oct 27, 2016 at 5:33 PM, Darin Johnson <db...@gmail.com>
> wrote:
>
> > Ellison Anne,
> >
> > I'm not sure that's the ideal long term solution but could provide a way
> > forward. Though I'm not sure I'd want to use maven-jar plugin vs just
> > reuse the files. I view both as bad practice, but the later would allow
> us
> > to refactor the tests and the the DistributedTestSuite independently.
> This
> > should lead to them being better decoupled quicker. I started messing
> with
> > that and could push the changes to the WIP pretty quickly.
> >
> > Thoughts on the DistributedTestSuite - The driver cli needs reworked.
> > Ideally you'd specify a platform name ie "spark" this would allow
> > additional frameworks to be tested. Would also like to allow specifying
> > tests and adding new tests - ie. test the framework on a representation
> of
> > your data. Should be able to specify multiple jars on the class path
> for a
> > framework to distribute. (These should happen iteratively not at once).
> >
> > Thoughts?
> >
> > Darin
> >
> > On Wed, Oct 26, 2016 at 2:21 PM, Ellison Anne Williams <
> > eawilliams@apache.org> wrote:
> >
> > > Apologies for the delayed response.
> > >
> > > What would folks think about moving the org.apache.pirk.test package in
> > > src/main/java to src/test/java (rename/refactor the package
> > appropriately)
> > > and then use the maven-jar-plugin to create a test-jar from which we
> run
> > > the distributed tests?
> > >
> > > That would seem to resolve the test class issues (some of the
> > src/test/java
> > > tests relying on classes in src/main/java) and allow the responder to
> be
> > > refactored appropriately.
> > >
> > > On Thu, Oct 20, 2016 at 12:22 AM, Darin Johnson <
> dbjohnson1978@gmail.com
> > >
> > > wrote:
> > >
> > > > I threw up a WIP for phase 1 of the submodule refactor. This
> involved
> > > > pulling out the pieces for the storm, spark, and mapreduce
> responsers.
> > > > Mostly ran into some difficulties of hard coded conditionals for each
> > > > framework in DistributedTestSuite and/or BaseTests.
> > > >
> > > > The next logical module to pull out is Responder. However, in order
> to
> > > do
> > > > so we need to pull DistributedTestSuite and BaseTests. While this
> > seems
> > > OK
> > > > with the main code base, it's not so with the test code base as a few
> > of
> > > > the tests explicitly rely on BaseTests and Inputs.
> > > >
> > > > I'd like to get some comments on where people believe the BaseTests
> and
> > > > Inputs Classes belong within the code base, so that I can plan this
> out
> > > > accordingly.
> > > >
> > >
> >
>
Re: [Discuss] Submodules (Phase 2) observations
Posted by Ellison Anne Williams <ea...@apache.org>.
Hi Darin,
A complete refactor of the tests would probably be ideal - however, it
doesn't seem to make sense to tackle that as a part of the initial
submodule refactor. Perhaps leaving the Responder refactor (and test
refactor) to a subsequent PR makes more sense.
Absent a complete test refactor, what are your thoughts on the best path
forward?
Thanks-
Ellison Anne
On Thu, Oct 27, 2016 at 5:33 PM, Darin Johnson <db...@gmail.com>
wrote:
> Ellison Anne,
>
> I'm not sure that's the ideal long term solution but could provide a way
> forward. Though I'm not sure I'd want to use maven-jar plugin vs just
> reuse the files. I view both as bad practice, but the later would allow us
> to refactor the tests and the the DistributedTestSuite independently. This
> should lead to them being better decoupled quicker. I started messing with
> that and could push the changes to the WIP pretty quickly.
>
> Thoughts on the DistributedTestSuite - The driver cli needs reworked.
> Ideally you'd specify a platform name ie "spark" this would allow
> additional frameworks to be tested. Would also like to allow specifying
> tests and adding new tests - ie. test the framework on a representation of
> your data. Should be able to specify multiple jars on the class path for a
> framework to distribute. (These should happen iteratively not at once).
>
> Thoughts?
>
> Darin
>
> On Wed, Oct 26, 2016 at 2:21 PM, Ellison Anne Williams <
> eawilliams@apache.org> wrote:
>
> > Apologies for the delayed response.
> >
> > What would folks think about moving the org.apache.pirk.test package in
> > src/main/java to src/test/java (rename/refactor the package
> appropriately)
> > and then use the maven-jar-plugin to create a test-jar from which we run
> > the distributed tests?
> >
> > That would seem to resolve the test class issues (some of the
> src/test/java
> > tests relying on classes in src/main/java) and allow the responder to be
> > refactored appropriately.
> >
> > On Thu, Oct 20, 2016 at 12:22 AM, Darin Johnson <dbjohnson1978@gmail.com
> >
> > wrote:
> >
> > > I threw up a WIP for phase 1 of the submodule refactor. This involved
> > > pulling out the pieces for the storm, spark, and mapreduce responsers.
> > > Mostly ran into some difficulties of hard coded conditionals for each
> > > framework in DistributedTestSuite and/or BaseTests.
> > >
> > > The next logical module to pull out is Responder. However, in order to
> > do
> > > so we need to pull DistributedTestSuite and BaseTests. While this
> seems
> > OK
> > > with the main code base, it's not so with the test code base as a few
> of
> > > the tests explicitly rely on BaseTests and Inputs.
> > >
> > > I'd like to get some comments on where people believe the BaseTests and
> > > Inputs Classes belong within the code base, so that I can plan this out
> > > accordingly.
> > >
> >
>
Re: [Discuss] Submodules (Phase 2) observations
Posted by Darin Johnson <db...@gmail.com>.
Ellison Anne,
I'm not sure that's the ideal long term solution but could provide a way
forward. Though I'm not sure I'd want to use maven-jar plugin vs just
reuse the files. I view both as bad practice, but the later would allow us
to refactor the tests and the the DistributedTestSuite independently. This
should lead to them being better decoupled quicker. I started messing with
that and could push the changes to the WIP pretty quickly.
Thoughts on the DistributedTestSuite - The driver cli needs reworked.
Ideally you'd specify a platform name ie "spark" this would allow
additional frameworks to be tested. Would also like to allow specifying
tests and adding new tests - ie. test the framework on a representation of
your data. Should be able to specify multiple jars on the class path for a
framework to distribute. (These should happen iteratively not at once).
Thoughts?
Darin
On Wed, Oct 26, 2016 at 2:21 PM, Ellison Anne Williams <
eawilliams@apache.org> wrote:
> Apologies for the delayed response.
>
> What would folks think about moving the org.apache.pirk.test package in
> src/main/java to src/test/java (rename/refactor the package appropriately)
> and then use the maven-jar-plugin to create a test-jar from which we run
> the distributed tests?
>
> That would seem to resolve the test class issues (some of the src/test/java
> tests relying on classes in src/main/java) and allow the responder to be
> refactored appropriately.
>
> On Thu, Oct 20, 2016 at 12:22 AM, Darin Johnson <db...@gmail.com>
> wrote:
>
> > I threw up a WIP for phase 1 of the submodule refactor. This involved
> > pulling out the pieces for the storm, spark, and mapreduce responsers.
> > Mostly ran into some difficulties of hard coded conditionals for each
> > framework in DistributedTestSuite and/or BaseTests.
> >
> > The next logical module to pull out is Responder. However, in order to
> do
> > so we need to pull DistributedTestSuite and BaseTests. While this seems
> OK
> > with the main code base, it's not so with the test code base as a few of
> > the tests explicitly rely on BaseTests and Inputs.
> >
> > I'd like to get some comments on where people believe the BaseTests and
> > Inputs Classes belong within the code base, so that I can plan this out
> > accordingly.
> >
>
Re: [Discuss] Submodules (Phase 2) observations
Posted by Ellison Anne Williams <ea...@apache.org>.
Apologies for the delayed response.
What would folks think about moving the org.apache.pirk.test package in
src/main/java to src/test/java (rename/refactor the package appropriately)
and then use the maven-jar-plugin to create a test-jar from which we run
the distributed tests?
That would seem to resolve the test class issues (some of the src/test/java
tests relying on classes in src/main/java) and allow the responder to be
refactored appropriately.
On Thu, Oct 20, 2016 at 12:22 AM, Darin Johnson <db...@gmail.com>
wrote:
> I threw up a WIP for phase 1 of the submodule refactor. This involved
> pulling out the pieces for the storm, spark, and mapreduce responsers.
> Mostly ran into some difficulties of hard coded conditionals for each
> framework in DistributedTestSuite and/or BaseTests.
>
> The next logical module to pull out is Responder. However, in order to do
> so we need to pull DistributedTestSuite and BaseTests. While this seems OK
> with the main code base, it's not so with the test code base as a few of
> the tests explicitly rely on BaseTests and Inputs.
>
> I'd like to get some comments on where people believe the BaseTests and
> Inputs Classes belong within the code base, so that I can plan this out
> accordingly.
>