You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Parviz deyhim <de...@gmail.com> on 2018/12/27 01:45:00 UTC
Kafka tests on a remote cluster
Hi,
I'm looking to see who has done this before and get some guidance. On
frequent basis I like to run basic tests on a remote Kafka cluster while
some random chaos/faults are being performed. In other words I like to run
chaos engineering tasks (network outage, disk outage, etc) and see how
Kafka behaves. For example:
1) bring some random Broker node down
2) send 2000 messages
3) consumes messages
4) confirm there's no data loss
My questions: I'm pretty sure most of the scenarios I'm looking to test
have been covered under Kafka's integration,unit and other existing tests.
What I cannot figure out is how to run those tests on a remote cluster vs.
a local one which the tests seems to run on. For example I like to run the
following command but the tests to be executed on a remote cluster of my
choice:
./gradlew cleanTest integrationTest
Any guidance/help would be appreciated.
Thanks
Re: Kafka tests on a remote cluster
Posted by Pere Urbón Bayes <pe...@gmail.com>.
Hi,
if I understand your question properly you are aiming to validate failure
scenarios. I usually see this done for learning purposes, basically to
answer how Kafka would react to such and such situation? or to validate the
current setup / configuration an Organisation have.
For the first one, or as well the second one but not directly on the
cluster, colleagues of mine actually wrote
https://github.com/Dabz/kafka-boom-boom, this uses docker to simulate the
scenarios. I've actually seen this approach used often.
This days I would only run a chaos testing if supported by a tooling like
the Netflix Chaos Monkey (https://github.com/Netflix/chaosmonkey) or the
other tooling available for the cloud providers, Kubernetes or Cloud
Foundry. A nice list is available from
https://github.com/dastergon/awesome-chaos-engineering#notable-tools
Cheers
-- Pere
Missatge de Parviz deyhim <de...@gmail.com> del dia dj., 27 de des. 2018 a
les 21:57:
> +dev@kafka.apache.org
>
> On Wed, Dec 26, 2018 at 8:53 PM Parviz deyhim <de...@gmail.com> wrote:
>
> > Thanks fair points. Probably best if I simplify the question: How does
> > Kafka community run tests besides using mocked local Kafka components?
> > Surely there are tests to confirm different failure scenarios such as
> > losing a broker in a real clustered environment (multi node cluster with
> > Ip, port, hostnsmes and etc). The answer would be a good starting point
> for
> > me.
> >
> > On Wed, Dec 26, 2018 at 6:11 PM Stephen Powis <sp...@salesforce.com>
> > wrote:
> >
> >> Without looking into how the integration tests work my best guess is
> >> within
> >> the context they were written to run in, it doesn't make sense to run
> them
> >> against a remote cluster. The "internal" cluster is running the same
> >> code,
> >> so why require having to coordinate with an external dependency?
> >>
> >> For the use case you gave, and I'm not sure if tests exist that cover
> this
> >> behavior or not -- Running the brokers locally in the context of the
> tests
> >> mean that those tests have control over the brokers (IE shut them off,
> >> restart them, etc.. programmatically) and validate behavior. To
> >> coordinate
> >> these operations on a remote broker would be significantly more
> difficult.
> >>
> >> Not sure this helps...but perhaps you're either asking the wrong
> questions
> >> or trying to go about solving your problem using the wrong set of tools?
> >> My gut feeling says if you want to do a full scale multi-server load /
> HA
> >> test, Kafka's test suite is not the best place to start.
> >>
> >> Stephen
> >>
> >>
> >>
> >> On Thu, Dec 27, 2018 at 10:53 AM Parviz deyhim <de...@gmail.com>
> wrote:
> >>
> >> > Hi,
> >> >
> >> > I'm looking to see who has done this before and get some guidance. On
> >> > frequent basis I like to run basic tests on a remote Kafka cluster
> while
> >> > some random chaos/faults are being performed. In other words I like to
> >> run
> >> > chaos engineering tasks (network outage, disk outage, etc) and see how
> >> > Kafka behaves. For example:
> >> >
> >> > 1) bring some random Broker node down
> >> > 2) send 2000 messages
> >> > 3) consumes messages
> >> > 4) confirm there's no data loss
> >> >
> >> > My questions: I'm pretty sure most of the scenarios I'm looking to
> test
> >> > have been covered under Kafka's integration,unit and other existing
> >> tests.
> >> > What I cannot figure out is how to run those tests on a remote cluster
> >> vs.
> >> > a local one which the tests seems to run on. For example I like to run
> >> the
> >> > following command but the tests to be executed on a remote cluster of
> my
> >> > choice:
> >> >
> >> > ./gradlew cleanTest integrationTest
> >> >
> >> > Any guidance/help would be appreciated.
> >> >
> >> > Thanks
> >> >
> >>
> >
>
--
Pere Urbon-Bayes
Software Architect
http://www.purbon.com
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/
Re: Kafka tests on a remote cluster
Posted by Pere Urbón Bayes <pe...@gmail.com>.
Hi,
if I understand your question properly you are aiming to validate failure
scenarios. I usually see this done for learning purposes, basically to
answer how Kafka would react to such and such situation? or to validate the
current setup / configuration an Organisation have.
For the first one, or as well the second one but not directly on the
cluster, colleagues of mine actually wrote
https://github.com/Dabz/kafka-boom-boom, this uses docker to simulate the
scenarios. I've actually seen this approach used often.
This days I would only run a chaos testing if supported by a tooling like
the Netflix Chaos Monkey (https://github.com/Netflix/chaosmonkey) or the
other tooling available for the cloud providers, Kubernetes or Cloud
Foundry. A nice list is available from
https://github.com/dastergon/awesome-chaos-engineering#notable-tools
Cheers
-- Pere
Missatge de Parviz deyhim <de...@gmail.com> del dia dj., 27 de des. 2018 a
les 21:57:
> +dev@kafka.apache.org
>
> On Wed, Dec 26, 2018 at 8:53 PM Parviz deyhim <de...@gmail.com> wrote:
>
> > Thanks fair points. Probably best if I simplify the question: How does
> > Kafka community run tests besides using mocked local Kafka components?
> > Surely there are tests to confirm different failure scenarios such as
> > losing a broker in a real clustered environment (multi node cluster with
> > Ip, port, hostnsmes and etc). The answer would be a good starting point
> for
> > me.
> >
> > On Wed, Dec 26, 2018 at 6:11 PM Stephen Powis <sp...@salesforce.com>
> > wrote:
> >
> >> Without looking into how the integration tests work my best guess is
> >> within
> >> the context they were written to run in, it doesn't make sense to run
> them
> >> against a remote cluster. The "internal" cluster is running the same
> >> code,
> >> so why require having to coordinate with an external dependency?
> >>
> >> For the use case you gave, and I'm not sure if tests exist that cover
> this
> >> behavior or not -- Running the brokers locally in the context of the
> tests
> >> mean that those tests have control over the brokers (IE shut them off,
> >> restart them, etc.. programmatically) and validate behavior. To
> >> coordinate
> >> these operations on a remote broker would be significantly more
> difficult.
> >>
> >> Not sure this helps...but perhaps you're either asking the wrong
> questions
> >> or trying to go about solving your problem using the wrong set of tools?
> >> My gut feeling says if you want to do a full scale multi-server load /
> HA
> >> test, Kafka's test suite is not the best place to start.
> >>
> >> Stephen
> >>
> >>
> >>
> >> On Thu, Dec 27, 2018 at 10:53 AM Parviz deyhim <de...@gmail.com>
> wrote:
> >>
> >> > Hi,
> >> >
> >> > I'm looking to see who has done this before and get some guidance. On
> >> > frequent basis I like to run basic tests on a remote Kafka cluster
> while
> >> > some random chaos/faults are being performed. In other words I like to
> >> run
> >> > chaos engineering tasks (network outage, disk outage, etc) and see how
> >> > Kafka behaves. For example:
> >> >
> >> > 1) bring some random Broker node down
> >> > 2) send 2000 messages
> >> > 3) consumes messages
> >> > 4) confirm there's no data loss
> >> >
> >> > My questions: I'm pretty sure most of the scenarios I'm looking to
> test
> >> > have been covered under Kafka's integration,unit and other existing
> >> tests.
> >> > What I cannot figure out is how to run those tests on a remote cluster
> >> vs.
> >> > a local one which the tests seems to run on. For example I like to run
> >> the
> >> > following command but the tests to be executed on a remote cluster of
> my
> >> > choice:
> >> >
> >> > ./gradlew cleanTest integrationTest
> >> >
> >> > Any guidance/help would be appreciated.
> >> >
> >> > Thanks
> >> >
> >>
> >
>
--
Pere Urbon-Bayes
Software Architect
http://www.purbon.com
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/
Re: Kafka tests on a remote cluster
Posted by Parviz deyhim <de...@gmail.com>.
+dev@kafka.apache.org
On Wed, Dec 26, 2018 at 8:53 PM Parviz deyhim <de...@gmail.com> wrote:
> Thanks fair points. Probably best if I simplify the question: How does
> Kafka community run tests besides using mocked local Kafka components?
> Surely there are tests to confirm different failure scenarios such as
> losing a broker in a real clustered environment (multi node cluster with
> Ip, port, hostnsmes and etc). The answer would be a good starting point for
> me.
>
> On Wed, Dec 26, 2018 at 6:11 PM Stephen Powis <sp...@salesforce.com>
> wrote:
>
>> Without looking into how the integration tests work my best guess is
>> within
>> the context they were written to run in, it doesn't make sense to run them
>> against a remote cluster. The "internal" cluster is running the same
>> code,
>> so why require having to coordinate with an external dependency?
>>
>> For the use case you gave, and I'm not sure if tests exist that cover this
>> behavior or not -- Running the brokers locally in the context of the tests
>> mean that those tests have control over the brokers (IE shut them off,
>> restart them, etc.. programmatically) and validate behavior. To
>> coordinate
>> these operations on a remote broker would be significantly more difficult.
>>
>> Not sure this helps...but perhaps you're either asking the wrong questions
>> or trying to go about solving your problem using the wrong set of tools?
>> My gut feeling says if you want to do a full scale multi-server load / HA
>> test, Kafka's test suite is not the best place to start.
>>
>> Stephen
>>
>>
>>
>> On Thu, Dec 27, 2018 at 10:53 AM Parviz deyhim <de...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I'm looking to see who has done this before and get some guidance. On
>> > frequent basis I like to run basic tests on a remote Kafka cluster while
>> > some random chaos/faults are being performed. In other words I like to
>> run
>> > chaos engineering tasks (network outage, disk outage, etc) and see how
>> > Kafka behaves. For example:
>> >
>> > 1) bring some random Broker node down
>> > 2) send 2000 messages
>> > 3) consumes messages
>> > 4) confirm there's no data loss
>> >
>> > My questions: I'm pretty sure most of the scenarios I'm looking to test
>> > have been covered under Kafka's integration,unit and other existing
>> tests.
>> > What I cannot figure out is how to run those tests on a remote cluster
>> vs.
>> > a local one which the tests seems to run on. For example I like to run
>> the
>> > following command but the tests to be executed on a remote cluster of my
>> > choice:
>> >
>> > ./gradlew cleanTest integrationTest
>> >
>> > Any guidance/help would be appreciated.
>> >
>> > Thanks
>> >
>>
>
Re: Kafka tests on a remote cluster
Posted by Parviz deyhim <de...@gmail.com>.
+dev@kafka.apache.org
On Wed, Dec 26, 2018 at 8:53 PM Parviz deyhim <de...@gmail.com> wrote:
> Thanks fair points. Probably best if I simplify the question: How does
> Kafka community run tests besides using mocked local Kafka components?
> Surely there are tests to confirm different failure scenarios such as
> losing a broker in a real clustered environment (multi node cluster with
> Ip, port, hostnsmes and etc). The answer would be a good starting point for
> me.
>
> On Wed, Dec 26, 2018 at 6:11 PM Stephen Powis <sp...@salesforce.com>
> wrote:
>
>> Without looking into how the integration tests work my best guess is
>> within
>> the context they were written to run in, it doesn't make sense to run them
>> against a remote cluster. The "internal" cluster is running the same
>> code,
>> so why require having to coordinate with an external dependency?
>>
>> For the use case you gave, and I'm not sure if tests exist that cover this
>> behavior or not -- Running the brokers locally in the context of the tests
>> mean that those tests have control over the brokers (IE shut them off,
>> restart them, etc.. programmatically) and validate behavior. To
>> coordinate
>> these operations on a remote broker would be significantly more difficult.
>>
>> Not sure this helps...but perhaps you're either asking the wrong questions
>> or trying to go about solving your problem using the wrong set of tools?
>> My gut feeling says if you want to do a full scale multi-server load / HA
>> test, Kafka's test suite is not the best place to start.
>>
>> Stephen
>>
>>
>>
>> On Thu, Dec 27, 2018 at 10:53 AM Parviz deyhim <de...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I'm looking to see who has done this before and get some guidance. On
>> > frequent basis I like to run basic tests on a remote Kafka cluster while
>> > some random chaos/faults are being performed. In other words I like to
>> run
>> > chaos engineering tasks (network outage, disk outage, etc) and see how
>> > Kafka behaves. For example:
>> >
>> > 1) bring some random Broker node down
>> > 2) send 2000 messages
>> > 3) consumes messages
>> > 4) confirm there's no data loss
>> >
>> > My questions: I'm pretty sure most of the scenarios I'm looking to test
>> > have been covered under Kafka's integration,unit and other existing
>> tests.
>> > What I cannot figure out is how to run those tests on a remote cluster
>> vs.
>> > a local one which the tests seems to run on. For example I like to run
>> the
>> > following command but the tests to be executed on a remote cluster of my
>> > choice:
>> >
>> > ./gradlew cleanTest integrationTest
>> >
>> > Any guidance/help would be appreciated.
>> >
>> > Thanks
>> >
>>
>
Re: Kafka tests on a remote cluster
Posted by Parviz deyhim <de...@gmail.com>.
Thanks fair points. Probably best if I simplify the question: How does
Kafka community run tests besides using mocked local Kafka components?
Surely there are tests to confirm different failure scenarios such as
losing a broker in a real clustered environment (multi node cluster with
Ip, port, hostnsmes and etc). The answer would be a good starting point for
me.
On Wed, Dec 26, 2018 at 6:11 PM Stephen Powis <sp...@salesforce.com> wrote:
> Without looking into how the integration tests work my best guess is within
> the context they were written to run in, it doesn't make sense to run them
> against a remote cluster. The "internal" cluster is running the same code,
> so why require having to coordinate with an external dependency?
>
> For the use case you gave, and I'm not sure if tests exist that cover this
> behavior or not -- Running the brokers locally in the context of the tests
> mean that those tests have control over the brokers (IE shut them off,
> restart them, etc.. programmatically) and validate behavior. To coordinate
> these operations on a remote broker would be significantly more difficult.
>
> Not sure this helps...but perhaps you're either asking the wrong questions
> or trying to go about solving your problem using the wrong set of tools?
> My gut feeling says if you want to do a full scale multi-server load / HA
> test, Kafka's test suite is not the best place to start.
>
> Stephen
>
>
>
> On Thu, Dec 27, 2018 at 10:53 AM Parviz deyhim <de...@gmail.com> wrote:
>
> > Hi,
> >
> > I'm looking to see who has done this before and get some guidance. On
> > frequent basis I like to run basic tests on a remote Kafka cluster while
> > some random chaos/faults are being performed. In other words I like to
> run
> > chaos engineering tasks (network outage, disk outage, etc) and see how
> > Kafka behaves. For example:
> >
> > 1) bring some random Broker node down
> > 2) send 2000 messages
> > 3) consumes messages
> > 4) confirm there's no data loss
> >
> > My questions: I'm pretty sure most of the scenarios I'm looking to test
> > have been covered under Kafka's integration,unit and other existing
> tests.
> > What I cannot figure out is how to run those tests on a remote cluster
> vs.
> > a local one which the tests seems to run on. For example I like to run
> the
> > following command but the tests to be executed on a remote cluster of my
> > choice:
> >
> > ./gradlew cleanTest integrationTest
> >
> > Any guidance/help would be appreciated.
> >
> > Thanks
> >
>
Re: Kafka tests on a remote cluster
Posted by Stephen Powis <sp...@salesforce.com>.
Without looking into how the integration tests work my best guess is within
the context they were written to run in, it doesn't make sense to run them
against a remote cluster. The "internal" cluster is running the same code,
so why require having to coordinate with an external dependency?
For the use case you gave, and I'm not sure if tests exist that cover this
behavior or not -- Running the brokers locally in the context of the tests
mean that those tests have control over the brokers (IE shut them off,
restart them, etc.. programmatically) and validate behavior. To coordinate
these operations on a remote broker would be significantly more difficult.
Not sure this helps...but perhaps you're either asking the wrong questions
or trying to go about solving your problem using the wrong set of tools?
My gut feeling says if you want to do a full scale multi-server load / HA
test, Kafka's test suite is not the best place to start.
Stephen
On Thu, Dec 27, 2018 at 10:53 AM Parviz deyhim <de...@gmail.com> wrote:
> Hi,
>
> I'm looking to see who has done this before and get some guidance. On
> frequent basis I like to run basic tests on a remote Kafka cluster while
> some random chaos/faults are being performed. In other words I like to run
> chaos engineering tasks (network outage, disk outage, etc) and see how
> Kafka behaves. For example:
>
> 1) bring some random Broker node down
> 2) send 2000 messages
> 3) consumes messages
> 4) confirm there's no data loss
>
> My questions: I'm pretty sure most of the scenarios I'm looking to test
> have been covered under Kafka's integration,unit and other existing tests.
> What I cannot figure out is how to run those tests on a remote cluster vs.
> a local one which the tests seems to run on. For example I like to run the
> following command but the tests to be executed on a remote cluster of my
> choice:
>
> ./gradlew cleanTest integrationTest
>
> Any guidance/help would be appreciated.
>
> Thanks
>