You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sa Li <sa...@gmail.com> on 2015/01/22 21:00:09 UTC
kafka production server test
Hi, All
We are about to deliver kafka production server, I have been working on
different test, like performance test from linkedin. This is a 3-node
cluster, with 5 nodes zkEnsemble. I assume there are lots of tests I need
to do, like network, node failure, flush time, etc. Is there is completed
guide to instruct the tests for kafka production servers?
thanks
--
Alec Li
Re: kafka production server test
Posted by Guozhang Wang <wa...@gmail.com>.
Hmm, so you just want to start producer / consumer clients which points to
the existing brokers in production. I think what you described here should
work, but I may be wrong. Please let me know if that does not work.
Guozhang
On Mon, Jan 26, 2015 at 1:50 PM, Sa Li <sa...@gmail.com> wrote:
> Thank you for reply Guozhang, right now I can get it work out of box, run
> testcase_1 on VM and access production. However, from my point of view, we
> really like to test the existing configs on productions, which means for
> example, replica_basic_test.py won't start zookeeper&kafka since I want to
> test the started brokers. I am thinking if I can comment out this part:
>
> self.log_message("starting zookeepers")
>
> kafka_system_test_utils.start_zookeepers(self.systemTestEnv,
> self.testcaseEnv)
> self.anonLogger.info("sleeping for 2s")
> time.sleep(2)
> self.log_message("starting brokers")
> kafka_system_test_utils.start_brokers(self.systemTestEnv,
> self.testcaseEnv)
> self.anonLogger.info("sleeping for 5s")
> time.sleep(5)
>
> Now I plan to modify the properties files in
> /system_test/replication_testsuite/config/, and cluster_config.json and
> testcase_1_properties.json on
> /system_test/replication_testsuite/testcase_1/, to make these config files
> exactly same as what we have on production. Does it work or I need to
> change some other dependencies to get it work?
>
> thanks
>
> AL
>
> On Mon, Jan 26, 2015 at 12:16 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
>
> > Sa,
> >
> > I believe your questions have mostly been answered by Ewen, and sorry for
> > getting late on this.
> >
> > As you notice the current system test's out-of-the-box experience is not
> > very good, and we are proposing ways to improve that situation:
> >
> > KAFKA-1748 <https://issues.apache.org/jira/browse/KAFKA-1748>
> >
> > KAFKA-1589 <https://issues.apache.org/jira/browse/KAFKA-1589>
> >
> > And we are adding some more test cases at the same time:
> >
> > KAFKA-1888 <https://issues.apache.org/jira/browse/KAFKA-1888>
> >
> > So if you have new observations while using the package or if you are
> > willing to contribute to those tickets you are mostly welcomed.
> >
> > Guozhang
> >
> >
> > On Thu, Jan 22, 2015 at 3:02 PM, Sa Li <sa...@gmail.com> wrote:
> >
> > > Hi, Guozhang
> > >
> > > Can I run this package remotely test another server? which mean I run
> > this
> > > package on dev but testing kafka system on production?
> > >
> > > thanks
> > >
> > > AL
> > >
> > > On Thu, Jan 22, 2015 at 2:55 PM, Sa Li <sa...@gmail.com> wrote:
> > >
> > > > Hi, Guozhang,
> > > >
> > > > Good to know such package, will try it now. :-)
> > > >
> > > > thanks
> > > >
> > > > On Thu, Jan 22, 2015 at 2:40 PM, Guozhang Wang <wa...@gmail.com>
> > > wrote:
> > > >
> > > >> Hi Sa,
> > > >>
> > > >> Have you looked into the system test package? It contains a suite of
> > > tests
> > > >> on different failure modes of Kafka brokers.
> > > >>
> > > >> Guozhang
> > > >>
> > > >>
> > > >> On Thu, Jan 22, 2015 at 12:00 PM, Sa Li <sa...@gmail.com> wrote:
> > > >>
> > > >> > Hi, All
> > > >> >
> > > >> > We are about to deliver kafka production server, I have been
> working
> > > on
> > > >> > different test, like performance test from linkedin. This is a
> > 3-node
> > > >> > cluster, with 5 nodes zkEnsemble. I assume there are lots of
> tests I
> > > >> need
> > > >> > to do, like network, node failure, flush time, etc. Is there is
> > > >> completed
> > > >> > guide to instruct the tests for kafka production servers?
> > > >> >
> > > >> > thanks
> > > >> >
> > > >> > --
> > > >> >
> > > >> > Alec Li
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> -- Guozhang
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Alec Li
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Alec Li
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
>
> Alec Li
>
--
-- Guozhang
Re: kafka production server test
Posted by Sa Li <sa...@gmail.com>.
Thank you for reply Guozhang, right now I can get it work out of box, run
testcase_1 on VM and access production. However, from my point of view, we
really like to test the existing configs on productions, which means for
example, replica_basic_test.py won't start zookeeper&kafka since I want to
test the started brokers. I am thinking if I can comment out this part:
self.log_message("starting zookeepers")
kafka_system_test_utils.start_zookeepers(self.systemTestEnv,
self.testcaseEnv)
self.anonLogger.info("sleeping for 2s")
time.sleep(2)
self.log_message("starting brokers")
kafka_system_test_utils.start_brokers(self.systemTestEnv,
self.testcaseEnv)
self.anonLogger.info("sleeping for 5s")
time.sleep(5)
Now I plan to modify the properties files in
/system_test/replication_testsuite/config/, and cluster_config.json and
testcase_1_properties.json on
/system_test/replication_testsuite/testcase_1/, to make these config files
exactly same as what we have on production. Does it work or I need to
change some other dependencies to get it work?
thanks
AL
On Mon, Jan 26, 2015 at 12:16 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Sa,
>
> I believe your questions have mostly been answered by Ewen, and sorry for
> getting late on this.
>
> As you notice the current system test's out-of-the-box experience is not
> very good, and we are proposing ways to improve that situation:
>
> KAFKA-1748 <https://issues.apache.org/jira/browse/KAFKA-1748>
>
> KAFKA-1589 <https://issues.apache.org/jira/browse/KAFKA-1589>
>
> And we are adding some more test cases at the same time:
>
> KAFKA-1888 <https://issues.apache.org/jira/browse/KAFKA-1888>
>
> So if you have new observations while using the package or if you are
> willing to contribute to those tickets you are mostly welcomed.
>
> Guozhang
>
>
> On Thu, Jan 22, 2015 at 3:02 PM, Sa Li <sa...@gmail.com> wrote:
>
> > Hi, Guozhang
> >
> > Can I run this package remotely test another server? which mean I run
> this
> > package on dev but testing kafka system on production?
> >
> > thanks
> >
> > AL
> >
> > On Thu, Jan 22, 2015 at 2:55 PM, Sa Li <sa...@gmail.com> wrote:
> >
> > > Hi, Guozhang,
> > >
> > > Good to know such package, will try it now. :-)
> > >
> > > thanks
> > >
> > > On Thu, Jan 22, 2015 at 2:40 PM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> > >
> > >> Hi Sa,
> > >>
> > >> Have you looked into the system test package? It contains a suite of
> > tests
> > >> on different failure modes of Kafka brokers.
> > >>
> > >> Guozhang
> > >>
> > >>
> > >> On Thu, Jan 22, 2015 at 12:00 PM, Sa Li <sa...@gmail.com> wrote:
> > >>
> > >> > Hi, All
> > >> >
> > >> > We are about to deliver kafka production server, I have been working
> > on
> > >> > different test, like performance test from linkedin. This is a
> 3-node
> > >> > cluster, with 5 nodes zkEnsemble. I assume there are lots of tests I
> > >> need
> > >> > to do, like network, node failure, flush time, etc. Is there is
> > >> completed
> > >> > guide to instruct the tests for kafka production servers?
> > >> >
> > >> > thanks
> > >> >
> > >> > --
> > >> >
> > >> > Alec Li
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> -- Guozhang
> > >>
> > >
> > >
> > >
> > > --
> > >
> > > Alec Li
> > >
> >
> >
> >
> > --
> >
> > Alec Li
> >
>
>
>
> --
> -- Guozhang
>
--
Alec Li
Re: kafka production server test
Posted by Guozhang Wang <wa...@gmail.com>.
Sa,
I believe your questions have mostly been answered by Ewen, and sorry for
getting late on this.
As you notice the current system test's out-of-the-box experience is not
very good, and we are proposing ways to improve that situation:
KAFKA-1748 <https://issues.apache.org/jira/browse/KAFKA-1748>
KAFKA-1589 <https://issues.apache.org/jira/browse/KAFKA-1589>
And we are adding some more test cases at the same time:
KAFKA-1888 <https://issues.apache.org/jira/browse/KAFKA-1888>
So if you have new observations while using the package or if you are
willing to contribute to those tickets you are mostly welcomed.
Guozhang
On Thu, Jan 22, 2015 at 3:02 PM, Sa Li <sa...@gmail.com> wrote:
> Hi, Guozhang
>
> Can I run this package remotely test another server? which mean I run this
> package on dev but testing kafka system on production?
>
> thanks
>
> AL
>
> On Thu, Jan 22, 2015 at 2:55 PM, Sa Li <sa...@gmail.com> wrote:
>
> > Hi, Guozhang,
> >
> > Good to know such package, will try it now. :-)
> >
> > thanks
> >
> > On Thu, Jan 22, 2015 at 2:40 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> >> Hi Sa,
> >>
> >> Have you looked into the system test package? It contains a suite of
> tests
> >> on different failure modes of Kafka brokers.
> >>
> >> Guozhang
> >>
> >>
> >> On Thu, Jan 22, 2015 at 12:00 PM, Sa Li <sa...@gmail.com> wrote:
> >>
> >> > Hi, All
> >> >
> >> > We are about to deliver kafka production server, I have been working
> on
> >> > different test, like performance test from linkedin. This is a 3-node
> >> > cluster, with 5 nodes zkEnsemble. I assume there are lots of tests I
> >> need
> >> > to do, like network, node failure, flush time, etc. Is there is
> >> completed
> >> > guide to instruct the tests for kafka production servers?
> >> >
> >> > thanks
> >> >
> >> > --
> >> >
> >> > Alec Li
> >> >
> >>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
> >
> >
> > --
> >
> > Alec Li
> >
>
>
>
> --
>
> Alec Li
>
--
-- Guozhang
Re: kafka production server test
Posted by Sa Li <sa...@gmail.com>.
Hi, Guozhang
Can I run this package remotely test another server? which mean I run this
package on dev but testing kafka system on production?
thanks
AL
On Thu, Jan 22, 2015 at 2:55 PM, Sa Li <sa...@gmail.com> wrote:
> Hi, Guozhang,
>
> Good to know such package, will try it now. :-)
>
> thanks
>
> On Thu, Jan 22, 2015 at 2:40 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
>> Hi Sa,
>>
>> Have you looked into the system test package? It contains a suite of tests
>> on different failure modes of Kafka brokers.
>>
>> Guozhang
>>
>>
>> On Thu, Jan 22, 2015 at 12:00 PM, Sa Li <sa...@gmail.com> wrote:
>>
>> > Hi, All
>> >
>> > We are about to deliver kafka production server, I have been working on
>> > different test, like performance test from linkedin. This is a 3-node
>> > cluster, with 5 nodes zkEnsemble. I assume there are lots of tests I
>> need
>> > to do, like network, node failure, flush time, etc. Is there is
>> completed
>> > guide to instruct the tests for kafka production servers?
>> >
>> > thanks
>> >
>> > --
>> >
>> > Alec Li
>> >
>>
>>
>>
>> --
>> -- Guozhang
>>
>
>
>
> --
>
> Alec Li
>
--
Alec Li
Re: kafka production server test
Posted by Sa Li <sa...@gmail.com>.
Hi, Guozhang,
Good to know such package, will try it now. :-)
thanks
On Thu, Jan 22, 2015 at 2:40 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Sa,
>
> Have you looked into the system test package? It contains a suite of tests
> on different failure modes of Kafka brokers.
>
> Guozhang
>
>
> On Thu, Jan 22, 2015 at 12:00 PM, Sa Li <sa...@gmail.com> wrote:
>
> > Hi, All
> >
> > We are about to deliver kafka production server, I have been working on
> > different test, like performance test from linkedin. This is a 3-node
> > cluster, with 5 nodes zkEnsemble. I assume there are lots of tests I need
> > to do, like network, node failure, flush time, etc. Is there is completed
> > guide to instruct the tests for kafka production servers?
> >
> > thanks
> >
> > --
> >
> > Alec Li
> >
>
>
>
> --
> -- Guozhang
>
--
Alec Li
Re: kafka production server test
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Sa,
Have you looked into the system test package? It contains a suite of tests
on different failure modes of Kafka brokers.
Guozhang
On Thu, Jan 22, 2015 at 12:00 PM, Sa Li <sa...@gmail.com> wrote:
> Hi, All
>
> We are about to deliver kafka production server, I have been working on
> different test, like performance test from linkedin. This is a 3-node
> cluster, with 5 nodes zkEnsemble. I assume there are lots of tests I need
> to do, like network, node failure, flush time, etc. Is there is completed
> guide to instruct the tests for kafka production servers?
>
> thanks
>
> --
>
> Alec Li
>
--
-- Guozhang