Heads up on MapR test framework and file ordering (specifically MD-185 tests)

[Jacques Nadeau <ja...@dremio.com>]

A large number of tests associated with MD-185 are unintentionally brittle.
Many of these tests reference more than one file. The tests are planning
tests and thus are prone to failure if the file ordering isn't the same as
what was used for test generation.

In the case of these types of tests, people should be cautious of adding
expected results that include multiple files in a certain order. I'm
working on a patch fix for these specific tests but wanted to let people
know to be cautious of these issues in the future.

thanks!

--
Jacques Nadeau
CTO and Co-Founder, Dremio

Re: Heads up on MapR test framework and file ordering (specifically MD-185 tests)

[Chun Chang <cc...@maprtech.com>]

Just verify numFiles may not be sufficient. I will see if I can enhance the
verification code to handle both. Thanks. -Chun

On Mon, Oct 26, 2015 at 10:01 AM, Jacques Nadeau <ja...@dremio.com> wrote:

> My current fix is to remove the specific lists. (A test which is flaky
> isn't very valuable.) You can see my current branch at [1]. You can look at
> my most recent commit to see the changes I made to the expected results
> files to get the tests to report the correct result.
>
> If you want to do file name verification (as opposed to simply numFiles
> verification), it seems like you should be doing a structured verification
> of the json plan rather than trying to do something with the text plan.
>
> [1] https://github.com/dremio/drill-test-framework
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Oct 26, 2015 at 9:36 AM, Chun Chang <cc...@maprtech.com> wrote:
>
> > Jacques,
> >
> > Thanks for the heads up. And what is your proposed fix? I think we can
> > easily enhance the test framework to handle file ordering cases. This
> way,
> > we don't need to modify any existing tests.
> >
> > Thanks,
> > Chun
> >
> > On Sat, Oct 24, 2015 at 9:11 PM, Jacques Nadeau <ja...@dremio.com>
> > wrote:
> >
> > > A large number of tests associated with MD-185 are unintentionally
> > brittle.
> > > Many of these tests reference more than one file. The tests are
> planning
> > > tests and thus are prone to failure if the file ordering isn't the same
> > as
> > > what was used for test generation.
> > >
> > > In the case of these types of tests, people should be cautious of
> adding
> > > expected results that include multiple files in a certain order. I'm
> > > working on a patch fix for these specific tests but wanted to let
> people
> > > know to be cautious of these issues in the future.
> > >
> > > thanks!
> > >
> > > --
> > > Jacques Nadeau
> > > CTO and Co-Founder, Dremio
> > >
> >
>

Re: Heads up on MapR test framework and file ordering (specifically MD-185 tests)

[Jacques Nadeau <ja...@dremio.com>]

My current fix is to remove the specific lists. (A test which is flaky
isn't very valuable.) You can see my current branch at [1]. You can look at
my most recent commit to see the changes I made to the expected results
files to get the tests to report the correct result.

If you want to do file name verification (as opposed to simply numFiles
verification), it seems like you should be doing a structured verification
of the json plan rather than trying to do something with the text plan.

[1] https://github.com/dremio/drill-test-framework

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Mon, Oct 26, 2015 at 9:36 AM, Chun Chang <cc...@maprtech.com> wrote:

> Jacques,
>
> Thanks for the heads up. And what is your proposed fix? I think we can
> easily enhance the test framework to handle file ordering cases. This way,
> we don't need to modify any existing tests.
>
> Thanks,
> Chun
>
> On Sat, Oct 24, 2015 at 9:11 PM, Jacques Nadeau <ja...@dremio.com>
> wrote:
>
> > A large number of tests associated with MD-185 are unintentionally
> brittle.
> > Many of these tests reference more than one file. The tests are planning
> > tests and thus are prone to failure if the file ordering isn't the same
> as
> > what was used for test generation.
> >
> > In the case of these types of tests, people should be cautious of adding
> > expected results that include multiple files in a certain order. I'm
> > working on a patch fix for these specific tests but wanted to let people
> > know to be cautious of these issues in the future.
> >
> > thanks!
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
>

Re: Heads up on MapR test framework and file ordering (specifically MD-185 tests)

[Chun Chang <cc...@maprtech.com>]

Jacques,

Thanks for the heads up. And what is your proposed fix? I think we can
easily enhance the test framework to handle file ordering cases. This way,
we don't need to modify any existing tests.

Thanks,
Chun

On Sat, Oct 24, 2015 at 9:11 PM, Jacques Nadeau <ja...@dremio.com> wrote:

> A large number of tests associated with MD-185 are unintentionally brittle.
> Many of these tests reference more than one file. The tests are planning
> tests and thus are prone to failure if the file ordering isn't the same as
> what was used for test generation.
>
> In the case of these types of tests, people should be cautious of adding
> expected results that include multiple files in a certain order. I'm
> working on a patch fix for these specific tests but wanted to let people
> know to be cautious of these issues in the future.
>
> thanks!
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>

Re: Heads up on MapR test framework and file ordering (specifically MD-185 tests)

[rahul challapalli <ch...@gmail.com>]

Jacques,

This is a known pain point. We cannot get away from adding planning tests
which scan more than one file. With many of the existing planning tests we
are verifying one of the following

1. Check the structure of the plan
2. Verify the scanned files to test pruning

Atleast for the second case a simple unordered list (of scanned files)
verification is sufficient. This might eliminate some of the hassle with
ordering files. Thoughts ?

Note : Because we use a regex based verification , its important to match
the "numFiles" option in the plan as well

- Rahul



On Sat, Oct 24, 2015 at 9:11 PM, Jacques Nadeau <ja...@dremio.com> wrote:

> A large number of tests associated with MD-185 are unintentionally brittle.
> Many of these tests reference more than one file. The tests are planning
> tests and thus are prone to failure if the file ordering isn't the same as
> what was used for test generation.
>
> In the case of these types of tests, people should be cautious of adding
> expected results that include multiple files in a certain order. I'm
> working on a patch fix for these specific tests but wanted to let people
> know to be cautious of these issues in the future.
>
> thanks!
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>