You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Gwen Shapira <gs...@cloudera.com> on 2015/06/05 01:29:44 UTC

Re: [DISCUSS] KIP-25 System test improvements

Hi,

Reviving the discussion a bit :)

I think it will be nice if each Kafka version that we release will also
have a separate "tests" artifact that users can download, untar and easily
run against a Kafka cluster of the same version.

The idea is that if someone downloads packages that claim to contain
something of a specific Kafka version (i.e. Kafka 0.8.2.0 + patches), users
can easily download the tests and verify that it indeed passes the tests
for this version and therefore behaves the way this version is expected to
behave.

Does it make sense?

Gwen

On Thu, May 21, 2015 at 3:26 PM, Geoffrey Anderson <ge...@confluent.io>
wrote:

> Hi Ashish,
>
> Looks like Ewen already hit the main points, but a few additions:
>
> 1. ducktape repo is here: https://github.com/confluentinc/ducktape
> ducktape itself will be pip installable in the near future, and Kafka
> system tests will be able to depend on a particular version of ducktape.
>
> 2.  The reporting is nothing fancy. We're definitely open to feedback, but
> it consists of:
> - top level summary of the test run (simple PASS/FAIL for each test)
> - top level info and debug logs
> - per-test info and debug logs
> - per-test "service" logs gathered from each service used in the test. For
> example, if your test pulls up a Kafka cluster with 5 brokers, the end
> result will have the Kafka logs from each of those 5 machines.
>
> Cheers,
> Geoff
>
> On Thu, May 21, 2015 at 3:15 PM, Ewen Cheslack-Postava <ew...@confluent.io>
> wrote:
>
> > Ashish,
> >
> > 1. That was the plan. We put some effort into cleanly separating the
> > framework so it would be reusable across many projects.
> > 2. I think you're seeing a test in progress where the final report hasn't
> > been created yet. If you visit one of the older ones you'll see it has a
> > landing page with links:
> > http://testing.confluent.io/confluent_platform/2015-05-20--001/
> Apparently
> > we need to adjust when we update the 'latest' symlink. The logs that are
> > collected for tests are configurable, and service implementations include
> > sane defaults (so, e.g., you will always get the normal log file for
> Kafka,
> > but only get the data files if the test asks for them).
> > 3. No code coverage support. Haven't looked into it, so I couldn't
> comment
> > on how hard it would be to add.
> >
> > -Ewen
> >
> > On Thu, May 21, 2015 at 2:38 PM, Ashish Singh <as...@cloudera.com>
> wrote:
> >
> > > Geoffrey,
> > >
> > > This looks great!
> > >
> > > A few questions.
> > > 1. Will ducktape be maintained separately as a github repo?
> > > 2. How easy is viewing the test results and logs. The link in KIP,
> > > http://testing.confluent.io/confluent_platform/latest/, lists a bunch
> of
> > > files and dirs. Could you add to KIP how the result and logs for the
> > tests
> > > will be organized.
> > > 3. Does it support code coverage? If not, how easy/ difficult would it
> > be?
> > >
> > > On Thu, May 21, 2015 at 2:03 PM, Geoffrey Anderson <geoff@confluent.io
> >
> > > wrote:
> > >
> > > > Great, I'll work on putting together a more detailed map of this
> > > > replacement process.
> > > >
> > > > On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira <
> gshapira@cloudera.com>
> > > > wrote:
> > > >
> > > > > Love this idea :)
> > > > >
> > > > > I took a look at Ducktape API and it looks like a good fit - clean
> > API,
> > > > > extensible, easy to use and powerful enough for our use-case.
> > > > >
> > > > > Something I'd like to see as part of the KIP is a map of what
> > > system-test
> > > > > currently tests, which ones we want to replace and a JIRA for
> > replacing
> > > > > (possibly one for each group of tests).
> > > > > Basically, I know we all want to use the new system for new test
> > cases
> > > > > (upgrades, etc), but I really want to make sure we don't get stuck
> > with
> > > > > both systems forever.
> > > > >
> > > > > Gwen
> > > > >
> > > > > On Thu, May 21, 2015 at 9:01 PM, Geoffrey Anderson <
> > geoff@confluent.io
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Just kicking off the discussion thread on KIP-25
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+improvements
> > > > > >
> > > > > > Thanks,
> > > > > > Geoff
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Regards,
> > > Ashish
> > >
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
>

Re: [DISCUSS] KIP-25 System test improvements

Posted by Geoffrey Anderson <ge...@confluent.io>.
Hi Aditya,

(1) There are a currently a few different ways to target a specific test or
subset of tests.

If for example tests were organized like the current system tests, where
suites are grouped by directory, you could run

cd <system_test_dir>
ducktape replication_testsuite/

You can also target tests in a particular file (ducktape <path_to_file>),
all tests in a test class (ducktape <path_to_file>::<test_class>), or a
particular test method in a class (ducktape
<path_to_file>::<test_class>.<test_method>)

(2) ducktape runs on a vagrant cluster by parsing the information returned
by the "vagrant ssh-config" command into JSON configuration used by the
JsonCluster class, so in that sense we are already using the JSON flavor.

I see a few potential challenges, but nothing too bad.
- There may be some work involved in getting ssh configs right
- A couple assumptions about where files are deployed on remote machines
are baked into some of the Service classes, so some minor refactoring may
be needed to make this a little more general. This would be a good thing.

In any case, we're happy to help out/take pull requests on ducktape etc.

Best,
Geoff








On Thu, Jun 11, 2015 at 10:13 AM, Aditya Auradkar <
aauradkar@linkedin.com.invalid> wrote:

> Hi Geoffrey,
>
> Thanks for the writeup. Couple of questions:
> - Is it possible to configure suites using ducktape? For example: assume
> all the tests in system_tests have been migrated to ducktape. Can I run a
> subset of all tests grouped by functional areas i.e. replication, broker
> failure etc.
>
> - Ducktape allows us to run tests on a vagrant cluster or on a static
> cluster configured via JSON. Once ported to ducktape, can we very easily
> run the existing system tests in both flavors?
>
> Thanks,
> Aditya
>
> ________________________________________
> From: Geoffrey Anderson [geoff@confluent.io]
> Sent: Monday, June 08, 2015 10:56 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-25 System test improvements
>
> Hi KIP-25 thread,
>
> I consolidated some of the questions from this thread and elsewhere.
>
> Q: Can we see a map of what system-test currently tests, which ones we want
> to replace and JIRAs for replacing?
> A: Initial draft here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Roadmap+-+port+existing+system+tests
>
> Q: Will ducktape be maintained separately as a github repo?
> A: Yes https://github.com/confluentinc/ducktape
>
> Q: How easy is viewing the test results and logs, how will test output be
> structured?
> A: Hierarchical structure as outlined here:
> https://github.com/confluentinc/ducktape/wiki/Design-overview#output
>
> Q: Does it support code coverage? If not, how easy/ difficult would it be
> to support?
> A: It does not, and we have no immediate plans to support this. Difficulty
> unclear.
>
> Q: It would be nice if each Kafka version that we release will also
> have a separate "tests" artifact that users can download, untar and easily
> run against a Kafka cluster of the same version.
> A: This seems reasonable and not too much extra work. Definitely open to
> discussion on this.
>
> Q: Why not share running services across multiple tests?
> A: Prefer to optimize for simplicity and correctness over what might be a
> questionable improvement in run-time.
>
> Q: Are regressions - in the road map?
> A: yes
>
> Q: Are Jepsen style tests involving network failures in the road map?
> A: yes
>
> Thanks much,
> Geoff
>
>
>
> On Mon, Jun 8, 2015 at 4:55 PM, Geoffrey Anderson <ge...@confluent.io>
> wrote:
>
> > Hi Gwen,
> >
> > I don't see any problem with this as long as we're convinced there's a
> > good use case, which seems to be true.
> >
> > Cheers,
> > Geoff
> >
> > On Thu, Jun 4, 2015 at 5:20 PM, Gwen Shapira <gs...@cloudera.com>
> > wrote:
> >
> >> Not completely random places :)
> >> People may use Cloudera / HWX distributions which include Kafka, but
> want
> >> to verify that these bits match a specific upstream release.
> >>
> >> I think having the tests separately will be useful for this. In this
> case,
> >> finding the tests are not a big issue - we'll add a download link :)
> >>
> >> On Thu, Jun 4, 2015 at 5:00 PM, Jiangjie Qin <jqin@linkedin.com.invalid
> >
> >> wrote:
> >>
> >> > Hey Gwen,
> >> >
> >> > Currently the test and code are downloaded at the same time.
> Supposedly
> >> > the tests in the same repository should cover match the code.
> >> > Are you saying people downloaded a release from some random place and
> >> want
> >> > to verify it? If that is the case, does that mean people still need to
> >> > find the correct place to download the right test artifact?
> >> >
> >> > Thanks,
> >> >
> >> > Jiangjie (Becket) Qin
> >> >
> >> >
> >> >
> >> > On 6/4/15, 4:29 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
> >> >
> >> > >Hi,
> >> > >
> >> > >Reviving the discussion a bit :)
> >> > >
> >> > >I think it will be nice if each Kafka version that we release will
> also
> >> > >have a separate "tests" artifact that users can download, untar and
> >> easily
> >> > >run against a Kafka cluster of the same version.
> >> > >
> >> > >The idea is that if someone downloads packages that claim to contain
> >> > >something of a specific Kafka version (i.e. Kafka 0.8.2.0 + patches),
> >> > >users
> >> > >can easily download the tests and verify that it indeed passes the
> >> tests
> >> > >for this version and therefore behaves the way this version is
> >> expected to
> >> > >behave.
> >> > >
> >> > >Does it make sense?
> >> > >
> >> > >Gwen
> >> > >
> >> > >On Thu, May 21, 2015 at 3:26 PM, Geoffrey Anderson <
> geoff@confluent.io
> >> >
> >> > >wrote:
> >> > >
> >> > >> Hi Ashish,
> >> > >>
> >> > >> Looks like Ewen already hit the main points, but a few additions:
> >> > >>
> >> > >> 1. ducktape repo is here: https://github.com/confluentinc/ducktape
> >> > >> ducktape itself will be pip installable in the near future, and
> Kafka
> >> > >> system tests will be able to depend on a particular version of
> >> ducktape.
> >> > >>
> >> > >> 2.  The reporting is nothing fancy. We're definitely open to
> >> feedback,
> >> > >>but
> >> > >> it consists of:
> >> > >> - top level summary of the test run (simple PASS/FAIL for each
> test)
> >> > >> - top level info and debug logs
> >> > >> - per-test info and debug logs
> >> > >> - per-test "service" logs gathered from each service used in the
> >> test.
> >> > >>For
> >> > >> example, if your test pulls up a Kafka cluster with 5 brokers, the
> >> end
> >> > >> result will have the Kafka logs from each of those 5 machines.
> >> > >>
> >> > >> Cheers,
> >> > >> Geoff
> >> > >>
> >> > >> On Thu, May 21, 2015 at 3:15 PM, Ewen Cheslack-Postava
> >> > >><ew...@confluent.io>
> >> > >> wrote:
> >> > >>
> >> > >> > Ashish,
> >> > >> >
> >> > >> > 1. That was the plan. We put some effort into cleanly separating
> >> the
> >> > >> > framework so it would be reusable across many projects.
> >> > >> > 2. I think you're seeing a test in progress where the final
> report
> >> > >>hasn't
> >> > >> > been created yet. If you visit one of the older ones you'll see
> it
> >> > >>has a
> >> > >> > landing page with links:
> >> > >> > http://testing.confluent.io/confluent_platform/2015-05-20--001/
> >> > >> Apparently
> >> > >> > we need to adjust when we update the 'latest' symlink. The logs
> >> that
> >> > >>are
> >> > >> > collected for tests are configurable, and service implementations
> >> > >>include
> >> > >> > sane defaults (so, e.g., you will always get the normal log file
> >> for
> >> > >> Kafka,
> >> > >> > but only get the data files if the test asks for them).
> >> > >> > 3. No code coverage support. Haven't looked into it, so I
> couldn't
> >> > >> comment
> >> > >> > on how hard it would be to add.
> >> > >> >
> >> > >> > -Ewen
> >> > >> >
> >> > >> > On Thu, May 21, 2015 at 2:38 PM, Ashish Singh <
> asingh@cloudera.com
> >> >
> >> > >> wrote:
> >> > >> >
> >> > >> > > Geoffrey,
> >> > >> > >
> >> > >> > > This looks great!
> >> > >> > >
> >> > >> > > A few questions.
> >> > >> > > 1. Will ducktape be maintained separately as a github repo?
> >> > >> > > 2. How easy is viewing the test results and logs. The link in
> >> KIP,
> >> > >> > > http://testing.confluent.io/confluent_platform/latest/, lists
> a
> >> > >>bunch
> >> > >> of
> >> > >> > > files and dirs. Could you add to KIP how the result and logs
> for
> >> the
> >> > >> > tests
> >> > >> > > will be organized.
> >> > >> > > 3. Does it support code coverage? If not, how easy/ difficult
> >> would
> >> > >>it
> >> > >> > be?
> >> > >> > >
> >> > >> > > On Thu, May 21, 2015 at 2:03 PM, Geoffrey Anderson
> >> > >><geoff@confluent.io
> >> > >> >
> >> > >> > > wrote:
> >> > >> > >
> >> > >> > > > Great, I'll work on putting together a more detailed map of
> >> this
> >> > >> > > > replacement process.
> >> > >> > > >
> >> > >> > > > On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira <
> >> > >> gshapira@cloudera.com>
> >> > >> > > > wrote:
> >> > >> > > >
> >> > >> > > > > Love this idea :)
> >> > >> > > > >
> >> > >> > > > > I took a look at Ducktape API and it looks like a good fit
> -
> >> > >>clean
> >> > >> > API,
> >> > >> > > > > extensible, easy to use and powerful enough for our
> use-case.
> >> > >> > > > >
> >> > >> > > > > Something I'd like to see as part of the KIP is a map of
> what
> >> > >> > > system-test
> >> > >> > > > > currently tests, which ones we want to replace and a JIRA
> for
> >> > >> > replacing
> >> > >> > > > > (possibly one for each group of tests).
> >> > >> > > > > Basically, I know we all want to use the new system for new
> >> test
> >> > >> > cases
> >> > >> > > > > (upgrades, etc), but I really want to make sure we don't
> get
> >> > >>stuck
> >> > >> > with
> >> > >> > > > > both systems forever.
> >> > >> > > > >
> >> > >> > > > > Gwen
> >> > >> > > > >
> >> > >> > > > > On Thu, May 21, 2015 at 9:01 PM, Geoffrey Anderson <
> >> > >> > geoff@confluent.io
> >> > >> > > >
> >> > >> > > > > wrote:
> >> > >> > > > >
> >> > >> > > > > > Hi,
> >> > >> > > > > >
> >> > >> > > > > > Just kicking off the discussion thread on KIP-25
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > >
> >> > >> > > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> > >>
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+im
> >> > >>provements
> >> > >> > > > > >
> >> > >> > > > > > Thanks,
> >> > >> > > > > > Geoff
> >> > >> > > > > >
> >> > >> > > > >
> >> > >> > > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > > --
> >> > >> > >
> >> > >> > > Regards,
> >> > >> > > Ashish
> >> > >> > >
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > --
> >> > >> > Thanks,
> >> > >> > Ewen
> >> > >> >
> >> > >>
> >> >
> >> >
> >>
> >
> >
>

RE: [DISCUSS] KIP-25 System test improvements

Posted by Aditya Auradkar <aa...@linkedin.com.INVALID>.
Hi Geoffrey,

Thanks for the writeup. Couple of questions:
- Is it possible to configure suites using ducktape? For example: assume all the tests in system_tests have been migrated to ducktape. Can I run a subset of all tests grouped by functional areas i.e. replication, broker failure etc.

- Ducktape allows us to run tests on a vagrant cluster or on a static cluster configured via JSON. Once ported to ducktape, can we very easily run the existing system tests in both flavors?

Thanks,
Aditya

________________________________________
From: Geoffrey Anderson [geoff@confluent.io]
Sent: Monday, June 08, 2015 10:56 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-25 System test improvements

Hi KIP-25 thread,

I consolidated some of the questions from this thread and elsewhere.

Q: Can we see a map of what system-test currently tests, which ones we want
to replace and JIRAs for replacing?
A: Initial draft here:
https://cwiki.apache.org/confluence/display/KAFKA/Roadmap+-+port+existing+system+tests

Q: Will ducktape be maintained separately as a github repo?
A: Yes https://github.com/confluentinc/ducktape

Q: How easy is viewing the test results and logs, how will test output be
structured?
A: Hierarchical structure as outlined here:
https://github.com/confluentinc/ducktape/wiki/Design-overview#output

Q: Does it support code coverage? If not, how easy/ difficult would it be
to support?
A: It does not, and we have no immediate plans to support this. Difficulty
unclear.

Q: It would be nice if each Kafka version that we release will also
have a separate "tests" artifact that users can download, untar and easily
run against a Kafka cluster of the same version.
A: This seems reasonable and not too much extra work. Definitely open to
discussion on this.

Q: Why not share running services across multiple tests?
A: Prefer to optimize for simplicity and correctness over what might be a
questionable improvement in run-time.

Q: Are regressions - in the road map?
A: yes

Q: Are Jepsen style tests involving network failures in the road map?
A: yes

Thanks much,
Geoff



On Mon, Jun 8, 2015 at 4:55 PM, Geoffrey Anderson <ge...@confluent.io>
wrote:

> Hi Gwen,
>
> I don't see any problem with this as long as we're convinced there's a
> good use case, which seems to be true.
>
> Cheers,
> Geoff
>
> On Thu, Jun 4, 2015 at 5:20 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
>
>> Not completely random places :)
>> People may use Cloudera / HWX distributions which include Kafka, but want
>> to verify that these bits match a specific upstream release.
>>
>> I think having the tests separately will be useful for this. In this case,
>> finding the tests are not a big issue - we'll add a download link :)
>>
>> On Thu, Jun 4, 2015 at 5:00 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
>> wrote:
>>
>> > Hey Gwen,
>> >
>> > Currently the test and code are downloaded at the same time. Supposedly
>> > the tests in the same repository should cover match the code.
>> > Are you saying people downloaded a release from some random place and
>> want
>> > to verify it? If that is the case, does that mean people still need to
>> > find the correct place to download the right test artifact?
>> >
>> > Thanks,
>> >
>> > Jiangjie (Becket) Qin
>> >
>> >
>> >
>> > On 6/4/15, 4:29 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
>> >
>> > >Hi,
>> > >
>> > >Reviving the discussion a bit :)
>> > >
>> > >I think it will be nice if each Kafka version that we release will also
>> > >have a separate "tests" artifact that users can download, untar and
>> easily
>> > >run against a Kafka cluster of the same version.
>> > >
>> > >The idea is that if someone downloads packages that claim to contain
>> > >something of a specific Kafka version (i.e. Kafka 0.8.2.0 + patches),
>> > >users
>> > >can easily download the tests and verify that it indeed passes the
>> tests
>> > >for this version and therefore behaves the way this version is
>> expected to
>> > >behave.
>> > >
>> > >Does it make sense?
>> > >
>> > >Gwen
>> > >
>> > >On Thu, May 21, 2015 at 3:26 PM, Geoffrey Anderson <geoff@confluent.io
>> >
>> > >wrote:
>> > >
>> > >> Hi Ashish,
>> > >>
>> > >> Looks like Ewen already hit the main points, but a few additions:
>> > >>
>> > >> 1. ducktape repo is here: https://github.com/confluentinc/ducktape
>> > >> ducktape itself will be pip installable in the near future, and Kafka
>> > >> system tests will be able to depend on a particular version of
>> ducktape.
>> > >>
>> > >> 2.  The reporting is nothing fancy. We're definitely open to
>> feedback,
>> > >>but
>> > >> it consists of:
>> > >> - top level summary of the test run (simple PASS/FAIL for each test)
>> > >> - top level info and debug logs
>> > >> - per-test info and debug logs
>> > >> - per-test "service" logs gathered from each service used in the
>> test.
>> > >>For
>> > >> example, if your test pulls up a Kafka cluster with 5 brokers, the
>> end
>> > >> result will have the Kafka logs from each of those 5 machines.
>> > >>
>> > >> Cheers,
>> > >> Geoff
>> > >>
>> > >> On Thu, May 21, 2015 at 3:15 PM, Ewen Cheslack-Postava
>> > >><ew...@confluent.io>
>> > >> wrote:
>> > >>
>> > >> > Ashish,
>> > >> >
>> > >> > 1. That was the plan. We put some effort into cleanly separating
>> the
>> > >> > framework so it would be reusable across many projects.
>> > >> > 2. I think you're seeing a test in progress where the final report
>> > >>hasn't
>> > >> > been created yet. If you visit one of the older ones you'll see it
>> > >>has a
>> > >> > landing page with links:
>> > >> > http://testing.confluent.io/confluent_platform/2015-05-20--001/
>> > >> Apparently
>> > >> > we need to adjust when we update the 'latest' symlink. The logs
>> that
>> > >>are
>> > >> > collected for tests are configurable, and service implementations
>> > >>include
>> > >> > sane defaults (so, e.g., you will always get the normal log file
>> for
>> > >> Kafka,
>> > >> > but only get the data files if the test asks for them).
>> > >> > 3. No code coverage support. Haven't looked into it, so I couldn't
>> > >> comment
>> > >> > on how hard it would be to add.
>> > >> >
>> > >> > -Ewen
>> > >> >
>> > >> > On Thu, May 21, 2015 at 2:38 PM, Ashish Singh <asingh@cloudera.com
>> >
>> > >> wrote:
>> > >> >
>> > >> > > Geoffrey,
>> > >> > >
>> > >> > > This looks great!
>> > >> > >
>> > >> > > A few questions.
>> > >> > > 1. Will ducktape be maintained separately as a github repo?
>> > >> > > 2. How easy is viewing the test results and logs. The link in
>> KIP,
>> > >> > > http://testing.confluent.io/confluent_platform/latest/, lists a
>> > >>bunch
>> > >> of
>> > >> > > files and dirs. Could you add to KIP how the result and logs for
>> the
>> > >> > tests
>> > >> > > will be organized.
>> > >> > > 3. Does it support code coverage? If not, how easy/ difficult
>> would
>> > >>it
>> > >> > be?
>> > >> > >
>> > >> > > On Thu, May 21, 2015 at 2:03 PM, Geoffrey Anderson
>> > >><geoff@confluent.io
>> > >> >
>> > >> > > wrote:
>> > >> > >
>> > >> > > > Great, I'll work on putting together a more detailed map of
>> this
>> > >> > > > replacement process.
>> > >> > > >
>> > >> > > > On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira <
>> > >> gshapira@cloudera.com>
>> > >> > > > wrote:
>> > >> > > >
>> > >> > > > > Love this idea :)
>> > >> > > > >
>> > >> > > > > I took a look at Ducktape API and it looks like a good fit -
>> > >>clean
>> > >> > API,
>> > >> > > > > extensible, easy to use and powerful enough for our use-case.
>> > >> > > > >
>> > >> > > > > Something I'd like to see as part of the KIP is a map of what
>> > >> > > system-test
>> > >> > > > > currently tests, which ones we want to replace and a JIRA for
>> > >> > replacing
>> > >> > > > > (possibly one for each group of tests).
>> > >> > > > > Basically, I know we all want to use the new system for new
>> test
>> > >> > cases
>> > >> > > > > (upgrades, etc), but I really want to make sure we don't get
>> > >>stuck
>> > >> > with
>> > >> > > > > both systems forever.
>> > >> > > > >
>> > >> > > > > Gwen
>> > >> > > > >
>> > >> > > > > On Thu, May 21, 2015 at 9:01 PM, Geoffrey Anderson <
>> > >> > geoff@confluent.io
>> > >> > > >
>> > >> > > > > wrote:
>> > >> > > > >
>> > >> > > > > > Hi,
>> > >> > > > > >
>> > >> > > > > > Just kicking off the discussion thread on KIP-25
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+im
>> > >>provements
>> > >> > > > > >
>> > >> > > > > > Thanks,
>> > >> > > > > > Geoff
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > > --
>> > >> > >
>> > >> > > Regards,
>> > >> > > Ashish
>> > >> > >
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > Thanks,
>> > >> > Ewen
>> > >> >
>> > >>
>> >
>> >
>>
>
>

Re: [DISCUSS] KIP-25 System test improvements

Posted by Geoffrey Anderson <ge...@confluent.io>.
Hi KIP-25 thread,

I consolidated some of the questions from this thread and elsewhere.

Q: Can we see a map of what system-test currently tests, which ones we want
to replace and JIRAs for replacing?
A: Initial draft here:
https://cwiki.apache.org/confluence/display/KAFKA/Roadmap+-+port+existing+system+tests

Q: Will ducktape be maintained separately as a github repo?
A: Yes https://github.com/confluentinc/ducktape

Q: How easy is viewing the test results and logs, how will test output be
structured?
A: Hierarchical structure as outlined here:
https://github.com/confluentinc/ducktape/wiki/Design-overview#output

Q: Does it support code coverage? If not, how easy/ difficult would it be
to support?
A: It does not, and we have no immediate plans to support this. Difficulty
unclear.

Q: It would be nice if each Kafka version that we release will also
have a separate "tests" artifact that users can download, untar and easily
run against a Kafka cluster of the same version.
A: This seems reasonable and not too much extra work. Definitely open to
discussion on this.

Q: Why not share running services across multiple tests?
A: Prefer to optimize for simplicity and correctness over what might be a
questionable improvement in run-time.

Q: Are regressions - in the road map?
A: yes

Q: Are Jepsen style tests involving network failures in the road map?
A: yes

Thanks much,
Geoff



On Mon, Jun 8, 2015 at 4:55 PM, Geoffrey Anderson <ge...@confluent.io>
wrote:

> Hi Gwen,
>
> I don't see any problem with this as long as we're convinced there's a
> good use case, which seems to be true.
>
> Cheers,
> Geoff
>
> On Thu, Jun 4, 2015 at 5:20 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
>
>> Not completely random places :)
>> People may use Cloudera / HWX distributions which include Kafka, but want
>> to verify that these bits match a specific upstream release.
>>
>> I think having the tests separately will be useful for this. In this case,
>> finding the tests are not a big issue - we'll add a download link :)
>>
>> On Thu, Jun 4, 2015 at 5:00 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
>> wrote:
>>
>> > Hey Gwen,
>> >
>> > Currently the test and code are downloaded at the same time. Supposedly
>> > the tests in the same repository should cover match the code.
>> > Are you saying people downloaded a release from some random place and
>> want
>> > to verify it? If that is the case, does that mean people still need to
>> > find the correct place to download the right test artifact?
>> >
>> > Thanks,
>> >
>> > Jiangjie (Becket) Qin
>> >
>> >
>> >
>> > On 6/4/15, 4:29 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
>> >
>> > >Hi,
>> > >
>> > >Reviving the discussion a bit :)
>> > >
>> > >I think it will be nice if each Kafka version that we release will also
>> > >have a separate "tests" artifact that users can download, untar and
>> easily
>> > >run against a Kafka cluster of the same version.
>> > >
>> > >The idea is that if someone downloads packages that claim to contain
>> > >something of a specific Kafka version (i.e. Kafka 0.8.2.0 + patches),
>> > >users
>> > >can easily download the tests and verify that it indeed passes the
>> tests
>> > >for this version and therefore behaves the way this version is
>> expected to
>> > >behave.
>> > >
>> > >Does it make sense?
>> > >
>> > >Gwen
>> > >
>> > >On Thu, May 21, 2015 at 3:26 PM, Geoffrey Anderson <geoff@confluent.io
>> >
>> > >wrote:
>> > >
>> > >> Hi Ashish,
>> > >>
>> > >> Looks like Ewen already hit the main points, but a few additions:
>> > >>
>> > >> 1. ducktape repo is here: https://github.com/confluentinc/ducktape
>> > >> ducktape itself will be pip installable in the near future, and Kafka
>> > >> system tests will be able to depend on a particular version of
>> ducktape.
>> > >>
>> > >> 2.  The reporting is nothing fancy. We're definitely open to
>> feedback,
>> > >>but
>> > >> it consists of:
>> > >> - top level summary of the test run (simple PASS/FAIL for each test)
>> > >> - top level info and debug logs
>> > >> - per-test info and debug logs
>> > >> - per-test "service" logs gathered from each service used in the
>> test.
>> > >>For
>> > >> example, if your test pulls up a Kafka cluster with 5 brokers, the
>> end
>> > >> result will have the Kafka logs from each of those 5 machines.
>> > >>
>> > >> Cheers,
>> > >> Geoff
>> > >>
>> > >> On Thu, May 21, 2015 at 3:15 PM, Ewen Cheslack-Postava
>> > >><ew...@confluent.io>
>> > >> wrote:
>> > >>
>> > >> > Ashish,
>> > >> >
>> > >> > 1. That was the plan. We put some effort into cleanly separating
>> the
>> > >> > framework so it would be reusable across many projects.
>> > >> > 2. I think you're seeing a test in progress where the final report
>> > >>hasn't
>> > >> > been created yet. If you visit one of the older ones you'll see it
>> > >>has a
>> > >> > landing page with links:
>> > >> > http://testing.confluent.io/confluent_platform/2015-05-20--001/
>> > >> Apparently
>> > >> > we need to adjust when we update the 'latest' symlink. The logs
>> that
>> > >>are
>> > >> > collected for tests are configurable, and service implementations
>> > >>include
>> > >> > sane defaults (so, e.g., you will always get the normal log file
>> for
>> > >> Kafka,
>> > >> > but only get the data files if the test asks for them).
>> > >> > 3. No code coverage support. Haven't looked into it, so I couldn't
>> > >> comment
>> > >> > on how hard it would be to add.
>> > >> >
>> > >> > -Ewen
>> > >> >
>> > >> > On Thu, May 21, 2015 at 2:38 PM, Ashish Singh <asingh@cloudera.com
>> >
>> > >> wrote:
>> > >> >
>> > >> > > Geoffrey,
>> > >> > >
>> > >> > > This looks great!
>> > >> > >
>> > >> > > A few questions.
>> > >> > > 1. Will ducktape be maintained separately as a github repo?
>> > >> > > 2. How easy is viewing the test results and logs. The link in
>> KIP,
>> > >> > > http://testing.confluent.io/confluent_platform/latest/, lists a
>> > >>bunch
>> > >> of
>> > >> > > files and dirs. Could you add to KIP how the result and logs for
>> the
>> > >> > tests
>> > >> > > will be organized.
>> > >> > > 3. Does it support code coverage? If not, how easy/ difficult
>> would
>> > >>it
>> > >> > be?
>> > >> > >
>> > >> > > On Thu, May 21, 2015 at 2:03 PM, Geoffrey Anderson
>> > >><geoff@confluent.io
>> > >> >
>> > >> > > wrote:
>> > >> > >
>> > >> > > > Great, I'll work on putting together a more detailed map of
>> this
>> > >> > > > replacement process.
>> > >> > > >
>> > >> > > > On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira <
>> > >> gshapira@cloudera.com>
>> > >> > > > wrote:
>> > >> > > >
>> > >> > > > > Love this idea :)
>> > >> > > > >
>> > >> > > > > I took a look at Ducktape API and it looks like a good fit -
>> > >>clean
>> > >> > API,
>> > >> > > > > extensible, easy to use and powerful enough for our use-case.
>> > >> > > > >
>> > >> > > > > Something I'd like to see as part of the KIP is a map of what
>> > >> > > system-test
>> > >> > > > > currently tests, which ones we want to replace and a JIRA for
>> > >> > replacing
>> > >> > > > > (possibly one for each group of tests).
>> > >> > > > > Basically, I know we all want to use the new system for new
>> test
>> > >> > cases
>> > >> > > > > (upgrades, etc), but I really want to make sure we don't get
>> > >>stuck
>> > >> > with
>> > >> > > > > both systems forever.
>> > >> > > > >
>> > >> > > > > Gwen
>> > >> > > > >
>> > >> > > > > On Thu, May 21, 2015 at 9:01 PM, Geoffrey Anderson <
>> > >> > geoff@confluent.io
>> > >> > > >
>> > >> > > > > wrote:
>> > >> > > > >
>> > >> > > > > > Hi,
>> > >> > > > > >
>> > >> > > > > > Just kicking off the discussion thread on KIP-25
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+im
>> > >>provements
>> > >> > > > > >
>> > >> > > > > > Thanks,
>> > >> > > > > > Geoff
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > > --
>> > >> > >
>> > >> > > Regards,
>> > >> > > Ashish
>> > >> > >
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > Thanks,
>> > >> > Ewen
>> > >> >
>> > >>
>> >
>> >
>>
>
>

Re: [DISCUSS] KIP-25 System test improvements

Posted by Geoffrey Anderson <ge...@confluent.io>.
Hi Gwen,

I don't see any problem with this as long as we're convinced there's a good
use case, which seems to be true.

Cheers,
Geoff

On Thu, Jun 4, 2015 at 5:20 PM, Gwen Shapira <gs...@cloudera.com> wrote:

> Not completely random places :)
> People may use Cloudera / HWX distributions which include Kafka, but want
> to verify that these bits match a specific upstream release.
>
> I think having the tests separately will be useful for this. In this case,
> finding the tests are not a big issue - we'll add a download link :)
>
> On Thu, Jun 4, 2015 at 5:00 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
> wrote:
>
> > Hey Gwen,
> >
> > Currently the test and code are downloaded at the same time. Supposedly
> > the tests in the same repository should cover match the code.
> > Are you saying people downloaded a release from some random place and
> want
> > to verify it? If that is the case, does that mean people still need to
> > find the correct place to download the right test artifact?
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> >
> > On 6/4/15, 4:29 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
> >
> > >Hi,
> > >
> > >Reviving the discussion a bit :)
> > >
> > >I think it will be nice if each Kafka version that we release will also
> > >have a separate "tests" artifact that users can download, untar and
> easily
> > >run against a Kafka cluster of the same version.
> > >
> > >The idea is that if someone downloads packages that claim to contain
> > >something of a specific Kafka version (i.e. Kafka 0.8.2.0 + patches),
> > >users
> > >can easily download the tests and verify that it indeed passes the tests
> > >for this version and therefore behaves the way this version is expected
> to
> > >behave.
> > >
> > >Does it make sense?
> > >
> > >Gwen
> > >
> > >On Thu, May 21, 2015 at 3:26 PM, Geoffrey Anderson <ge...@confluent.io>
> > >wrote:
> > >
> > >> Hi Ashish,
> > >>
> > >> Looks like Ewen already hit the main points, but a few additions:
> > >>
> > >> 1. ducktape repo is here: https://github.com/confluentinc/ducktape
> > >> ducktape itself will be pip installable in the near future, and Kafka
> > >> system tests will be able to depend on a particular version of
> ducktape.
> > >>
> > >> 2.  The reporting is nothing fancy. We're definitely open to feedback,
> > >>but
> > >> it consists of:
> > >> - top level summary of the test run (simple PASS/FAIL for each test)
> > >> - top level info and debug logs
> > >> - per-test info and debug logs
> > >> - per-test "service" logs gathered from each service used in the test.
> > >>For
> > >> example, if your test pulls up a Kafka cluster with 5 brokers, the end
> > >> result will have the Kafka logs from each of those 5 machines.
> > >>
> > >> Cheers,
> > >> Geoff
> > >>
> > >> On Thu, May 21, 2015 at 3:15 PM, Ewen Cheslack-Postava
> > >><ew...@confluent.io>
> > >> wrote:
> > >>
> > >> > Ashish,
> > >> >
> > >> > 1. That was the plan. We put some effort into cleanly separating the
> > >> > framework so it would be reusable across many projects.
> > >> > 2. I think you're seeing a test in progress where the final report
> > >>hasn't
> > >> > been created yet. If you visit one of the older ones you'll see it
> > >>has a
> > >> > landing page with links:
> > >> > http://testing.confluent.io/confluent_platform/2015-05-20--001/
> > >> Apparently
> > >> > we need to adjust when we update the 'latest' symlink. The logs that
> > >>are
> > >> > collected for tests are configurable, and service implementations
> > >>include
> > >> > sane defaults (so, e.g., you will always get the normal log file for
> > >> Kafka,
> > >> > but only get the data files if the test asks for them).
> > >> > 3. No code coverage support. Haven't looked into it, so I couldn't
> > >> comment
> > >> > on how hard it would be to add.
> > >> >
> > >> > -Ewen
> > >> >
> > >> > On Thu, May 21, 2015 at 2:38 PM, Ashish Singh <as...@cloudera.com>
> > >> wrote:
> > >> >
> > >> > > Geoffrey,
> > >> > >
> > >> > > This looks great!
> > >> > >
> > >> > > A few questions.
> > >> > > 1. Will ducktape be maintained separately as a github repo?
> > >> > > 2. How easy is viewing the test results and logs. The link in KIP,
> > >> > > http://testing.confluent.io/confluent_platform/latest/, lists a
> > >>bunch
> > >> of
> > >> > > files and dirs. Could you add to KIP how the result and logs for
> the
> > >> > tests
> > >> > > will be organized.
> > >> > > 3. Does it support code coverage? If not, how easy/ difficult
> would
> > >>it
> > >> > be?
> > >> > >
> > >> > > On Thu, May 21, 2015 at 2:03 PM, Geoffrey Anderson
> > >><geoff@confluent.io
> > >> >
> > >> > > wrote:
> > >> > >
> > >> > > > Great, I'll work on putting together a more detailed map of this
> > >> > > > replacement process.
> > >> > > >
> > >> > > > On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira <
> > >> gshapira@cloudera.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Love this idea :)
> > >> > > > >
> > >> > > > > I took a look at Ducktape API and it looks like a good fit -
> > >>clean
> > >> > API,
> > >> > > > > extensible, easy to use and powerful enough for our use-case.
> > >> > > > >
> > >> > > > > Something I'd like to see as part of the KIP is a map of what
> > >> > > system-test
> > >> > > > > currently tests, which ones we want to replace and a JIRA for
> > >> > replacing
> > >> > > > > (possibly one for each group of tests).
> > >> > > > > Basically, I know we all want to use the new system for new
> test
> > >> > cases
> > >> > > > > (upgrades, etc), but I really want to make sure we don't get
> > >>stuck
> > >> > with
> > >> > > > > both systems forever.
> > >> > > > >
> > >> > > > > Gwen
> > >> > > > >
> > >> > > > > On Thu, May 21, 2015 at 9:01 PM, Geoffrey Anderson <
> > >> > geoff@confluent.io
> > >> > > >
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Hi,
> > >> > > > > >
> > >> > > > > > Just kicking off the discussion thread on KIP-25
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+im
> > >>provements
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > > Geoff
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > >
> > >> > > Regards,
> > >> > > Ashish
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Thanks,
> > >> > Ewen
> > >> >
> > >>
> >
> >
>

Re: [DISCUSS] KIP-25 System test improvements

Posted by Gwen Shapira <gs...@cloudera.com>.
Not completely random places :)
People may use Cloudera / HWX distributions which include Kafka, but want
to verify that these bits match a specific upstream release.

I think having the tests separately will be useful for this. In this case,
finding the tests are not a big issue - we'll add a download link :)

On Thu, Jun 4, 2015 at 5:00 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
wrote:

> Hey Gwen,
>
> Currently the test and code are downloaded at the same time. Supposedly
> the tests in the same repository should cover match the code.
> Are you saying people downloaded a release from some random place and want
> to verify it? If that is the case, does that mean people still need to
> find the correct place to download the right test artifact?
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
>
> On 6/4/15, 4:29 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
>
> >Hi,
> >
> >Reviving the discussion a bit :)
> >
> >I think it will be nice if each Kafka version that we release will also
> >have a separate "tests" artifact that users can download, untar and easily
> >run against a Kafka cluster of the same version.
> >
> >The idea is that if someone downloads packages that claim to contain
> >something of a specific Kafka version (i.e. Kafka 0.8.2.0 + patches),
> >users
> >can easily download the tests and verify that it indeed passes the tests
> >for this version and therefore behaves the way this version is expected to
> >behave.
> >
> >Does it make sense?
> >
> >Gwen
> >
> >On Thu, May 21, 2015 at 3:26 PM, Geoffrey Anderson <ge...@confluent.io>
> >wrote:
> >
> >> Hi Ashish,
> >>
> >> Looks like Ewen already hit the main points, but a few additions:
> >>
> >> 1. ducktape repo is here: https://github.com/confluentinc/ducktape
> >> ducktape itself will be pip installable in the near future, and Kafka
> >> system tests will be able to depend on a particular version of ducktape.
> >>
> >> 2.  The reporting is nothing fancy. We're definitely open to feedback,
> >>but
> >> it consists of:
> >> - top level summary of the test run (simple PASS/FAIL for each test)
> >> - top level info and debug logs
> >> - per-test info and debug logs
> >> - per-test "service" logs gathered from each service used in the test.
> >>For
> >> example, if your test pulls up a Kafka cluster with 5 brokers, the end
> >> result will have the Kafka logs from each of those 5 machines.
> >>
> >> Cheers,
> >> Geoff
> >>
> >> On Thu, May 21, 2015 at 3:15 PM, Ewen Cheslack-Postava
> >><ew...@confluent.io>
> >> wrote:
> >>
> >> > Ashish,
> >> >
> >> > 1. That was the plan. We put some effort into cleanly separating the
> >> > framework so it would be reusable across many projects.
> >> > 2. I think you're seeing a test in progress where the final report
> >>hasn't
> >> > been created yet. If you visit one of the older ones you'll see it
> >>has a
> >> > landing page with links:
> >> > http://testing.confluent.io/confluent_platform/2015-05-20--001/
> >> Apparently
> >> > we need to adjust when we update the 'latest' symlink. The logs that
> >>are
> >> > collected for tests are configurable, and service implementations
> >>include
> >> > sane defaults (so, e.g., you will always get the normal log file for
> >> Kafka,
> >> > but only get the data files if the test asks for them).
> >> > 3. No code coverage support. Haven't looked into it, so I couldn't
> >> comment
> >> > on how hard it would be to add.
> >> >
> >> > -Ewen
> >> >
> >> > On Thu, May 21, 2015 at 2:38 PM, Ashish Singh <as...@cloudera.com>
> >> wrote:
> >> >
> >> > > Geoffrey,
> >> > >
> >> > > This looks great!
> >> > >
> >> > > A few questions.
> >> > > 1. Will ducktape be maintained separately as a github repo?
> >> > > 2. How easy is viewing the test results and logs. The link in KIP,
> >> > > http://testing.confluent.io/confluent_platform/latest/, lists a
> >>bunch
> >> of
> >> > > files and dirs. Could you add to KIP how the result and logs for the
> >> > tests
> >> > > will be organized.
> >> > > 3. Does it support code coverage? If not, how easy/ difficult would
> >>it
> >> > be?
> >> > >
> >> > > On Thu, May 21, 2015 at 2:03 PM, Geoffrey Anderson
> >><geoff@confluent.io
> >> >
> >> > > wrote:
> >> > >
> >> > > > Great, I'll work on putting together a more detailed map of this
> >> > > > replacement process.
> >> > > >
> >> > > > On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira <
> >> gshapira@cloudera.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Love this idea :)
> >> > > > >
> >> > > > > I took a look at Ducktape API and it looks like a good fit -
> >>clean
> >> > API,
> >> > > > > extensible, easy to use and powerful enough for our use-case.
> >> > > > >
> >> > > > > Something I'd like to see as part of the KIP is a map of what
> >> > > system-test
> >> > > > > currently tests, which ones we want to replace and a JIRA for
> >> > replacing
> >> > > > > (possibly one for each group of tests).
> >> > > > > Basically, I know we all want to use the new system for new test
> >> > cases
> >> > > > > (upgrades, etc), but I really want to make sure we don't get
> >>stuck
> >> > with
> >> > > > > both systems forever.
> >> > > > >
> >> > > > > Gwen
> >> > > > >
> >> > > > > On Thu, May 21, 2015 at 9:01 PM, Geoffrey Anderson <
> >> > geoff@confluent.io
> >> > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi,
> >> > > > > >
> >> > > > > > Just kicking off the discussion thread on KIP-25
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+im
> >>provements
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Geoff
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > >
> >> > > Regards,
> >> > > Ashish
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Thanks,
> >> > Ewen
> >> >
> >>
>
>

Re: [DISCUSS] KIP-25 System test improvements

Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
Hey Gwen,

Currently the test and code are downloaded at the same time. Supposedly
the tests in the same repository should cover match the code.
Are you saying people downloaded a release from some random place and want
to verify it? If that is the case, does that mean people still need to
find the correct place to download the right test artifact?

Thanks,

Jiangjie (Becket) Qin



On 6/4/15, 4:29 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:

>Hi,
>
>Reviving the discussion a bit :)
>
>I think it will be nice if each Kafka version that we release will also
>have a separate "tests" artifact that users can download, untar and easily
>run against a Kafka cluster of the same version.
>
>The idea is that if someone downloads packages that claim to contain
>something of a specific Kafka version (i.e. Kafka 0.8.2.0 + patches),
>users
>can easily download the tests and verify that it indeed passes the tests
>for this version and therefore behaves the way this version is expected to
>behave.
>
>Does it make sense?
>
>Gwen
>
>On Thu, May 21, 2015 at 3:26 PM, Geoffrey Anderson <ge...@confluent.io>
>wrote:
>
>> Hi Ashish,
>>
>> Looks like Ewen already hit the main points, but a few additions:
>>
>> 1. ducktape repo is here: https://github.com/confluentinc/ducktape
>> ducktape itself will be pip installable in the near future, and Kafka
>> system tests will be able to depend on a particular version of ducktape.
>>
>> 2.  The reporting is nothing fancy. We're definitely open to feedback,
>>but
>> it consists of:
>> - top level summary of the test run (simple PASS/FAIL for each test)
>> - top level info and debug logs
>> - per-test info and debug logs
>> - per-test "service" logs gathered from each service used in the test.
>>For
>> example, if your test pulls up a Kafka cluster with 5 brokers, the end
>> result will have the Kafka logs from each of those 5 machines.
>>
>> Cheers,
>> Geoff
>>
>> On Thu, May 21, 2015 at 3:15 PM, Ewen Cheslack-Postava
>><ew...@confluent.io>
>> wrote:
>>
>> > Ashish,
>> >
>> > 1. That was the plan. We put some effort into cleanly separating the
>> > framework so it would be reusable across many projects.
>> > 2. I think you're seeing a test in progress where the final report
>>hasn't
>> > been created yet. If you visit one of the older ones you'll see it
>>has a
>> > landing page with links:
>> > http://testing.confluent.io/confluent_platform/2015-05-20--001/
>> Apparently
>> > we need to adjust when we update the 'latest' symlink. The logs that
>>are
>> > collected for tests are configurable, and service implementations
>>include
>> > sane defaults (so, e.g., you will always get the normal log file for
>> Kafka,
>> > but only get the data files if the test asks for them).
>> > 3. No code coverage support. Haven't looked into it, so I couldn't
>> comment
>> > on how hard it would be to add.
>> >
>> > -Ewen
>> >
>> > On Thu, May 21, 2015 at 2:38 PM, Ashish Singh <as...@cloudera.com>
>> wrote:
>> >
>> > > Geoffrey,
>> > >
>> > > This looks great!
>> > >
>> > > A few questions.
>> > > 1. Will ducktape be maintained separately as a github repo?
>> > > 2. How easy is viewing the test results and logs. The link in KIP,
>> > > http://testing.confluent.io/confluent_platform/latest/, lists a
>>bunch
>> of
>> > > files and dirs. Could you add to KIP how the result and logs for the
>> > tests
>> > > will be organized.
>> > > 3. Does it support code coverage? If not, how easy/ difficult would
>>it
>> > be?
>> > >
>> > > On Thu, May 21, 2015 at 2:03 PM, Geoffrey Anderson
>><geoff@confluent.io
>> >
>> > > wrote:
>> > >
>> > > > Great, I'll work on putting together a more detailed map of this
>> > > > replacement process.
>> > > >
>> > > > On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira <
>> gshapira@cloudera.com>
>> > > > wrote:
>> > > >
>> > > > > Love this idea :)
>> > > > >
>> > > > > I took a look at Ducktape API and it looks like a good fit -
>>clean
>> > API,
>> > > > > extensible, easy to use and powerful enough for our use-case.
>> > > > >
>> > > > > Something I'd like to see as part of the KIP is a map of what
>> > > system-test
>> > > > > currently tests, which ones we want to replace and a JIRA for
>> > replacing
>> > > > > (possibly one for each group of tests).
>> > > > > Basically, I know we all want to use the new system for new test
>> > cases
>> > > > > (upgrades, etc), but I really want to make sure we don't get
>>stuck
>> > with
>> > > > > both systems forever.
>> > > > >
>> > > > > Gwen
>> > > > >
>> > > > > On Thu, May 21, 2015 at 9:01 PM, Geoffrey Anderson <
>> > geoff@confluent.io
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > Just kicking off the discussion thread on KIP-25
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> 
>>https://cwiki.apache.org/confluence/display/KAFKA/KIP+-+25+System+test+im
>>provements
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Geoff
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > >
>> > > Regards,
>> > > Ashish
>> > >
>> >
>> >
>> >
>> > --
>> > Thanks,
>> > Ewen
>> >
>>