You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by "Jason Altekruse (JIRA)" <ji...@apache.org> on 2016/01/11 18:43:39 UTC
[jira] [Created] (DRILL-4259) Add new functional tests to ensure
that failures can be detected independent of the testing environment
Jason Altekruse created DRILL-4259:
--------------------------------------
Summary: Add new functional tests to ensure that failures can be detected independent of the testing environment
Key: DRILL-4259
URL: https://issues.apache.org/jira/browse/DRILL-4259
Project: Apache Drill
Issue Type: Test
Reporter: Jason Altekruse
In DRILL-4243 an out of memory issue was fixed after a change to the memory allocator made memory limits more strict. While the regression tests had been run by the team at Dremio prior to merging the patch, running the tests on a cluster with more cores changed the memory limits on the queries and caused several tests to fail.
While changes of this magnitude are not going to be common, we should have a test suite that reliably fails independent of the environment it is run (assuming that there are sufficient resources for the tests to run).
It would be good to at least try to reproduce this failure on a few different setups (cores, nodes in cluster) by adjusting available configuration options and adding tests with those different configurations so that the tests will fail in different environments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [jira] [Created] (DRILL-4259) Add new functional tests to ensure
that failures can be detected independent of the testing environment
Posted by Jason Altekruse <al...@gmail.com>.
Just realized replying to these messages does not put comments on the JIRA,
do you want to copy your comment there to consolidate the discussion on
this topic?
On Tue, Jan 12, 2016 at 8:01 AM, Jason Altekruse <al...@gmail.com>
wrote:
> That's what I was thinking, although I'm not very familiar with the setup
> of the regression tests. I know that some tests set options themselves by
> just including the alter sessions statements in the test files. I also know
> that we added an enhancement for the framework to set options globally, so
> that we could leave the decimal tests turned on [1] (even though we turned
> of the feature by default due to some lingering pref and correctness
> issues).
>
> I think what I would be looking to do would be configure a few different
> combinations of configurations that could be applied over an entire suite,
> so that we can run a number of tests with a few different configurations.
> Doing this by manually changing the test files to include near copes with
> the new options set doesn't seem like the right way to go. Let me know if
> this seems like a reasonable approach. I can work on the enhancement to the
> framework to enable it if necessary.
>
> [1] -
> https://github.com/mapr/drill-test-framework/blob/master/framework/before-run.sql
>
>
> On Mon, Jan 11, 2016 at 12:15 PM, Chun Chang <cc...@maprtech.com> wrote:
>
>> Can we come up with a list of different configuration option settings we
>> want to run? The automation framework supports running with different
>> system settings now.
>>
>> On Mon, Jan 11, 2016 at 9:43 AM, Jason Altekruse (JIRA) <ji...@apache.org>
>> wrote:
>>
>> > Jason Altekruse created DRILL-4259:
>> > --------------------------------------
>> >
>> > Summary: Add new functional tests to ensure that failures
>> can
>> > be detected independent of the testing environment
>> > Key: DRILL-4259
>> > URL: https://issues.apache.org/jira/browse/DRILL-4259
>> > Project: Apache Drill
>> > Issue Type: Test
>> > Reporter: Jason Altekruse
>> >
>> >
>> > In DRILL-4243 an out of memory issue was fixed after a change to the
>> > memory allocator made memory limits more strict. While the regression
>> tests
>> > had been run by the team at Dremio prior to merging the patch, running
>> the
>> > tests on a cluster with more cores changed the memory limits on the
>> queries
>> > and caused several tests to fail.
>> >
>> > While changes of this magnitude are not going to be common, we should
>> have
>> > a test suite that reliably fails independent of the environment it is
>> run
>> > (assuming that there are sufficient resources for the tests to run).
>> >
>> > It would be good to at least try to reproduce this failure on a few
>> > different setups (cores, nodes in cluster) by adjusting available
>> > configuration options and adding tests with those different
>> configurations
>> > so that the tests will fail in different environments.
>> >
>> >
>> >
>> > --
>> > This message was sent by Atlassian JIRA
>> > (v6.3.4#6332)
>> >
>>
>
>
Re: [jira] [Created] (DRILL-4259) Add new functional tests to ensure
that failures can be detected independent of the testing environment
Posted by Jason Altekruse <al...@gmail.com>.
That's what I was thinking, although I'm not very familiar with the setup
of the regression tests. I know that some tests set options themselves by
just including the alter sessions statements in the test files. I also know
that we added an enhancement for the framework to set options globally, so
that we could leave the decimal tests turned on [1] (even though we turned
of the feature by default due to some lingering pref and correctness
issues).
I think what I would be looking to do would be configure a few different
combinations of configurations that could be applied over an entire suite,
so that we can run a number of tests with a few different configurations.
Doing this by manually changing the test files to include near copes with
the new options set doesn't seem like the right way to go. Let me know if
this seems like a reasonable approach. I can work on the enhancement to the
framework to enable it if necessary.
[1] -
https://github.com/mapr/drill-test-framework/blob/master/framework/before-run.sql
On Mon, Jan 11, 2016 at 12:15 PM, Chun Chang <cc...@maprtech.com> wrote:
> Can we come up with a list of different configuration option settings we
> want to run? The automation framework supports running with different
> system settings now.
>
> On Mon, Jan 11, 2016 at 9:43 AM, Jason Altekruse (JIRA) <ji...@apache.org>
> wrote:
>
> > Jason Altekruse created DRILL-4259:
> > --------------------------------------
> >
> > Summary: Add new functional tests to ensure that failures
> can
> > be detected independent of the testing environment
> > Key: DRILL-4259
> > URL: https://issues.apache.org/jira/browse/DRILL-4259
> > Project: Apache Drill
> > Issue Type: Test
> > Reporter: Jason Altekruse
> >
> >
> > In DRILL-4243 an out of memory issue was fixed after a change to the
> > memory allocator made memory limits more strict. While the regression
> tests
> > had been run by the team at Dremio prior to merging the patch, running
> the
> > tests on a cluster with more cores changed the memory limits on the
> queries
> > and caused several tests to fail.
> >
> > While changes of this magnitude are not going to be common, we should
> have
> > a test suite that reliably fails independent of the environment it is run
> > (assuming that there are sufficient resources for the tests to run).
> >
> > It would be good to at least try to reproduce this failure on a few
> > different setups (cores, nodes in cluster) by adjusting available
> > configuration options and adding tests with those different
> configurations
> > so that the tests will fail in different environments.
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v6.3.4#6332)
> >
>
Re: [jira] [Created] (DRILL-4259) Add new functional tests to ensure
that failures can be detected independent of the testing environment
Posted by Chun Chang <cc...@maprtech.com>.
Can we come up with a list of different configuration option settings we
want to run? The automation framework supports running with different
system settings now.
On Mon, Jan 11, 2016 at 9:43 AM, Jason Altekruse (JIRA) <ji...@apache.org>
wrote:
> Jason Altekruse created DRILL-4259:
> --------------------------------------
>
> Summary: Add new functional tests to ensure that failures can
> be detected independent of the testing environment
> Key: DRILL-4259
> URL: https://issues.apache.org/jira/browse/DRILL-4259
> Project: Apache Drill
> Issue Type: Test
> Reporter: Jason Altekruse
>
>
> In DRILL-4243 an out of memory issue was fixed after a change to the
> memory allocator made memory limits more strict. While the regression tests
> had been run by the team at Dremio prior to merging the patch, running the
> tests on a cluster with more cores changed the memory limits on the queries
> and caused several tests to fail.
>
> While changes of this magnitude are not going to be common, we should have
> a test suite that reliably fails independent of the environment it is run
> (assuming that there are sufficient resources for the tests to run).
>
> It would be good to at least try to reproduce this failure on a few
> different setups (cores, nodes in cluster) by adjusting available
> configuration options and adding tests with those different configurations
> so that the tests will fail in different environments.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>