You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Nikolay Izhikov <ni...@apache.org> on 2020/09/03 08:06:39 UTC

Re: [DISCUSSION] Upgrade system tests to python 3

Hello! 

Just a friendly reminder.

Patch to resolve some kind of technical debt - python2 in system tests is ready!
Can someone, please, take a look?

https://github.com/apache/kafka/pull/9196

> 28 авг. 2020 г., в 11:19, Nikolay Izhikov <ni...@gmail.com> написал(а):
> 
> Hello!
> 
> Any feedback on this?
> What I should additionally do to prepare system tests migration?
> 
>> 24 авг. 2020 г., в 11:17, Nikolay Izhikov <ni...@gmail.com> написал(а):
>> 
>> Hello.
>> 
>> PR [1] is ready.
>> Please, review.
>> 
>> But, I need help with the two following questions:
>> 
>> 1. We need a new release of ducktape which includes fixes [2], [3] for python3.
>> I created the issue in ducktape repo [4].
>> Can someone help me with the release?
>> 
>> 2. I know that some companies run system tests for the trunk on a regular bases.
>> Can someone show me some results of these runs?
>> So, I can compare failures in my PR and in the trunk.
>> 
>> Results [5] of run all for my PR available in the ticket [6]
>> 
>> ```
>> SESSION REPORT (ALL TESTS)
>> ducktape version: 0.8.0
>> session_id:       2020-08-23--002
>> run time:         1010 minutes 46.483 seconds
>> tests run:        684
>> passed:           505
>> failed:           9
>> ignored:          170
>> ```
>> 
>> [1] https://github.com/apache/kafka/pull/9196
>> [2] https://github.com/confluentinc/ducktape/commit/23bd5ab53802e3a1e1da1ddf3630934f33b02305
>> [3] https://github.com/confluentinc/ducktape/commit/bfe53712f83b025832d29a43cde3de3d7803106f
>> [4] https://github.com/confluentinc/ducktape/issues/245
>> [5] https://issues.apache.org/jira/secure/attachment/13010366/report.txt
>> [6] https://issues.apache.org/jira/browse/KAFKA-10402
>> 
>>> 14 авг. 2020 г., в 21:26, Ismael Juma <is...@juma.me.uk> написал(а):
>>> 
>>> +1
>>> 
>>> On Fri, Aug 14, 2020 at 7:42 AM John Roesler <vv...@apache.org> wrote:
>>> 
>>>> Thanks Nikolay,
>>>> 
>>>> No objection. This would be very nice to have.
>>>> 
>>>> Thanks,
>>>> John
>>>> 
>>>> On Fri, Aug 14, 2020, at 09:18, Nikolay Izhikov wrote:
>>>>> Hello.
>>>>> 
>>>>>> If anyone's interested in porting it to Python 3 it would be a good
>>>> change.
>>>>> 
>>>>> I’ve created a ticket [1] to upgrade system tests to python3.
>>>>> Does someone have any additional inputs or objections for this change?
>>>>> 
>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-10402
>>>>> 
>>>>> 
>>>>>> 1 июля 2020 г., в 00:26, Gokul Ramanan Subramanian <
>>>> gokul2411s@gmail.com> написал(а):
>>>>>> 
>>>>>> Thanks Colin.
>>>>>> 
>>>>>> While at the subject of system tests, there are a few times I see tests
>>>>>> timed out (even on a large machine such as m5.4xlarge EC2 with Linux).
>>>> Are
>>>>>> there any knobs that system tests provide to control timeouts /
>>>> throughputs
>>>>>> across all tests?
>>>>>> Thanks.
>>>>>> 
>>>>>> On Tue, Jun 30, 2020 at 6:32 PM Colin McCabe <cm...@apache.org>
>>>> wrote:
>>>>>> 
>>>>>>> Ducktape runs on Python 2.  You can't use it with Python 3, as you are
>>>>>>> trying to do here.
>>>>>>> 
>>>>>>> If anyone's interested in porting it to Python 3 it would be a good
>>>> change.
>>>>>>> 
>>>>>>> Otherwise, using docker as suggested here seems to be the best way to
>>>> go.
>>>>>>> 
>>>>>>> best,
>>>>>>> Colin
>>>>>>> 
>>>>>>> On Mon, Jun 29, 2020, at 02:14, Gokul Ramanan Subramanian wrote:
>>>>>>>> Hi.
>>>>>>>> 
>>>>>>>> Has anyone had luck running Kafka system tests on a Mac. I have a
>>>> MacOS
>>>>>>>> Mojave 10.14.6. I got Python 3.6.9 using pyenv. However, the command
>>>>>>>> *ducktape tests/kafkatest/tests* yields the following error, making
>>>> it
>>>>>>> look
>>>>>>>> like some Python incompatibility issue.
>>>>>>>> 
>>>>>>>> $ ducktape tests/kafkatest/tests
>>>>>>>> Traceback (most recent call last):
>>>>>>>> File "/Users/gokusubr/.pyenv/versions/3.6.9/bin/ducktape", line 11,
>>>> in
>>>>>>>> <module>
>>>>>>>> load_entry_point('ducktape', 'console_scripts', 'ducktape')()
>>>>>>>> File
>>>>>>>> 
>>>>>>> 
>>>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>> line 487, in load_entry_point
>>>>>>>> return get_distribution(dist).load_entry_point(group, name)
>>>>>>>> File
>>>>>>>> 
>>>>>>> 
>>>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>> line 2728, in load_entry_point
>>>>>>>> return ep.load()
>>>>>>>> File
>>>>>>>> 
>>>>>>> 
>>>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>> line 2346, in load
>>>>>>>> return self.resolve()
>>>>>>>> File
>>>>>>>> 
>>>>>>> 
>>>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>> line 2352, in resolve
>>>>>>>> module = __import__(self.module_name, fromlist=['__name__'],
>>>>>>>> level=0)
>>>>>>>> File
>>>>>>>> 
>>>>>>> 
>>>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/ducktape-0.7.6-py3.6.egg/ducktape/command_line/main.py",
>>>>>>>> line 127
>>>>>>>> print "parameters are not valid json: " + str(e.message)
>>>>>>>>                                       ^
>>>>>>>> SyntaxError: invalid syntax
>>>>>>>> 
>>>>>>>> I followed the instructions in tests/README.md to setup a cluster of
>>>> 9
>>>>>>>> worker machines. That worked well. When I ran *python setup.py
>>>> develop*
>>>>>>> to
>>>>>>>> install the necessary dependencies (including ducktape), I got
>>>> similar
>>>>>>>> errors to above, but the overall command completed successfully.
>>>>>>>> 
>>>>>>>> Any help appreciated.
>>>>>>>> 
>>>>>>>> Thanks.
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>> 
>> 
> 


Re: [DISCUSSION] Upgrade system tests to python 3

Posted by Nikolay Izhikov <ni...@apache.org>.
Guozhang and all others who involved.
Thanks for your help!


> 7 окт. 2020 г., в 19:42, Guozhang Wang <wa...@gmail.com> написал(а):
> 
> Hello Nikolay,
> 
> I've merged the PR to trunk. Thanks for your huge effort and patience going
> through the review!
> 
> Guozhang
> 
> On Wed, Oct 7, 2020 at 6:52 AM Nikolay Izhikov <ni...@apache.org> wrote:
> 
>> Great news!
>> Thanks Magnus!
>> 
>> I’ve updated the PR.
>> 
>> Looks like we ready to merge it.
>> 
>>> 7 окт. 2020 г., в 15:29, Magnus Edenhill <ma...@edenhill.se>
>> написал(а):
>>> 
>>> Hi,
>>> 
>>> ducktape v0.8.0 is now released.
>>> 
>>> Regards,
>>> Magnus
>>> 
>>> 
>>> Den ons 7 okt. 2020 kl 10:50 skrev Nikolay Izhikov <nizhikov@apache.org
>>> :
>>> 
>>>> Hello.
>>>> 
>>>> Got 4 approvals for PR [1]
>>>> The only thing we need to be able to merge it is a ducktape 0.8 release.
>>>> If ducktape team need any help with the release, please, let me know.
>>>> 
>>>> [1] https://github.com/apache/kafka/pull/9196
>>>> 
>>>> 
>>>>> 21 сент. 2020 г., в 12:58, Nikolay Izhikov <ni...@gmail.com>
>>>> написал(а):
>>>>> 
>>>>> Hello.
>>>>> 
>>>>> I fixed two system tests that fails in trunk, also.
>>>>> 
>>>>> 
>> streams_upgrade_test.py::StreamsUpgradeTest.test_version_probing_upgrade
>>>>> streams_static_membership_test.py
>>>>> 
>>>>> Please, take a look at my PR [1]
>>>>> 
>>>>> [1] https://github.com/apache/kafka/pull/9312
>>>>> 
>>>>>> 20 сент. 2020 г., в 06:11, Guozhang Wang <wa...@gmail.com>
>>>> написал(а):
>>>>>> 
>>>>>> I've triggered a system test on top of your branch.
>>>>>> 
>>>>>> Maybe you could also re-run the jenkins unit tests since currently all
>>>> of
>>>>>> them fails but you've only touched on system tests, so I'd like to
>>>> confirm
>>>>>> at least one successful run.
>>>>>> 
>>>>>> On Wed, Sep 16, 2020 at 3:37 AM Nikolay Izhikov <ni...@apache.org>
>>>> wrote:
>>>>>> 
>>>>>>> Hello, Guozhang.
>>>>>>> 
>>>>>>>> I can help run the test suite once your PR is cleanly rebased to
>>>> verify
>>>>>>> the whole suite works
>>>>>>> 
>>>>>>> Thank you for joining to the review.
>>>>>>> 
>>>>>>> 1. PR rebased on the current trunk.
>>>>>>> 
>>>>>>> 2. I triggered all tests in my private environment to verify them
>> after
>>>>>>> rebase.
>>>>>>>     Will inform you once tests passed on my environment.
>>>>>>> 
>>>>>>> 3. We need a new ducktape release [1] to be able to merge PR [2].
>>>>>>>     For now, PR based on the ducktape trunk branch [3], not some
>>>>>>> specific release.
>>>>>>>     If ducktape team need any help with the release, please, let me
>>>>>>> know.
>>>>>>> 
>>>>>>> [1] https://github.com/confluentinc/ducktape/issues/245
>>>>>>> [2] https://github.com/apache/kafka/pull/9196
>>>>>>> [3]
>>>>>>> 
>>>> 
>> https://github.com/apache/kafka/pull/9196/files#diff-9235a7bdb1ca9268681c0e56f3f3609bR39
>>>>>>> 
>>>>>>>> 16 сент. 2020 г., в 07:32, Guozhang Wang <wa...@gmail.com>
>>>>>>> написал(а):
>>>>>>>> 
>>>>>>>> Hello Nikolay,
>>>>>>>> 
>>>>>>>> I can help run the test suite once your PR is cleanly rebased to
>>>> verify
>>>>>>> the
>>>>>>>> whole suite works and then I can merge (I'm trusting Ivan and Magnus
>>>> here
>>>>>>>> for their reviews :)
>>>>>>>> 
>>>>>>>> Guozhang
>>>>>>>> 
>>>>>>>> On Mon, Sep 14, 2020 at 3:56 AM Nikolay Izhikov <
>> nizhikov@apache.org>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hello!
>>>>>>>>> 
>>>>>>>>> I got 2 approvals from Ivan Daschinskiy and Magnus Edenhill.
>>>>>>>>> Committers, please, join the review.
>>>>>>>>> 
>>>>>>>>>> 3 сент. 2020 г., в 11:06, Nikolay Izhikov <nizhikov.dev@gmail.com
>>> 
>>>>>>>>> написал(а):
>>>>>>>>>> 
>>>>>>>>>> Hello!
>>>>>>>>>> 
>>>>>>>>>> Just a friendly reminder.
>>>>>>>>>> 
>>>>>>>>>> Patch to resolve some kind of technical debt - python2 in system
>>>> tests
>>>>>>>>> is ready!
>>>>>>>>>> Can someone, please, take a look?
>>>>>>>>>> 
>>>>>>>>>> https://github.com/apache/kafka/pull/9196
>>>>>>>>>> 
>>>>>>>>>>> 28 авг. 2020 г., в 11:19, Nikolay Izhikov <
>> nizhikov.dev@gmail.com>
>>>>>>>>> написал(а):
>>>>>>>>>>> 
>>>>>>>>>>> Hello!
>>>>>>>>>>> 
>>>>>>>>>>> Any feedback on this?
>>>>>>>>>>> What I should additionally do to prepare system tests migration?
>>>>>>>>>>> 
>>>>>>>>>>>> 24 авг. 2020 г., в 11:17, Nikolay Izhikov <
>> nizhikov.dev@gmail.com
>>>>> 
>>>>>>>>> написал(а):
>>>>>>>>>>>> 
>>>>>>>>>>>> Hello.
>>>>>>>>>>>> 
>>>>>>>>>>>> PR [1] is ready.
>>>>>>>>>>>> Please, review.
>>>>>>>>>>>> 
>>>>>>>>>>>> But, I need help with the two following questions:
>>>>>>>>>>>> 
>>>>>>>>>>>> 1. We need a new release of ducktape which includes fixes [2],
>> [3]
>>>>>>> for
>>>>>>>>> python3.
>>>>>>>>>>>> I created the issue in ducktape repo [4].
>>>>>>>>>>>> Can someone help me with the release?
>>>>>>>>>>>> 
>>>>>>>>>>>> 2. I know that some companies run system tests for the trunk on
>> a
>>>>>>>>> regular bases.
>>>>>>>>>>>> Can someone show me some results of these runs?
>>>>>>>>>>>> So, I can compare failures in my PR and in the trunk.
>>>>>>>>>>>> 
>>>>>>>>>>>> Results [5] of run all for my PR available in the ticket [6]
>>>>>>>>>>>> 
>>>>>>>>>>>> ```
>>>>>>>>>>>> SESSION REPORT (ALL TESTS)
>>>>>>>>>>>> ducktape version: 0.8.0
>>>>>>>>>>>> session_id:       2020-08-23--002
>>>>>>>>>>>> run time:         1010 minutes 46.483 seconds
>>>>>>>>>>>> tests run:        684
>>>>>>>>>>>> passed:           505
>>>>>>>>>>>> failed:           9
>>>>>>>>>>>> ignored:          170
>>>>>>>>>>>> ```
>>>>>>>>>>>> 
>>>>>>>>>>>> [1] https://github.com/apache/kafka/pull/9196
>>>>>>>>>>>> [2]
>>>>>>>>> 
>>>>>>> 
>>>> 
>> https://github.com/confluentinc/ducktape/commit/23bd5ab53802e3a1e1da1ddf3630934f33b02305
>>>>>>>>>>>> [3]
>>>>>>>>> 
>>>>>>> 
>>>> 
>> https://github.com/confluentinc/ducktape/commit/bfe53712f83b025832d29a43cde3de3d7803106f
>>>>>>>>>>>> [4] https://github.com/confluentinc/ducktape/issues/245
>>>>>>>>>>>> [5]
>>>>>>>>> 
>> https://issues.apache.org/jira/secure/attachment/13010366/report.txt
>>>>>>>>>>>> [6] https://issues.apache.org/jira/browse/KAFKA-10402
>>>>>>>>>>>> 
>>>>>>>>>>>>> 14 авг. 2020 г., в 21:26, Ismael Juma <is...@juma.me.uk>
>>>>>>> написал(а):
>>>>>>>>>>>>> 
>>>>>>>>>>>>> +1
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Aug 14, 2020 at 7:42 AM John Roesler <
>>>> vvcephei@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks Nikolay,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> No objection. This would be very nice to have.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> John
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Fri, Aug 14, 2020, at 09:18, Nikolay Izhikov wrote:
>>>>>>>>>>>>>>> Hello.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> If anyone's interested in porting it to Python 3 it would
>> be a
>>>>>>> good
>>>>>>>>>>>>>> change.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I’ve created a ticket [1] to upgrade system tests to python3.
>>>>>>>>>>>>>>> Does someone have any additional inputs or objections for
>> this
>>>>>>>>> change?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-10402
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 1 июля 2020 г., в 00:26, Gokul Ramanan Subramanian <
>>>>>>>>>>>>>> gokul2411s@gmail.com> написал(а):
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks Colin.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> While at the subject of system tests, there are a few times
>> I
>>>> see
>>>>>>>>> tests
>>>>>>>>>>>>>>>> timed out (even on a large machine such as m5.4xlarge EC2
>> with
>>>>>>>>> Linux).
>>>>>>>>>>>>>> Are
>>>>>>>>>>>>>>>> there any knobs that system tests provide to control
>> timeouts
>>>> /
>>>>>>>>>>>>>> throughputs
>>>>>>>>>>>>>>>> across all tests?
>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Tue, Jun 30, 2020 at 6:32 PM Colin McCabe <
>>>> cmccabe@apache.org
>>>>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Ducktape runs on Python 2.  You can't use it with Python 3,
>>>> as
>>>>>>>>> you are
>>>>>>>>>>>>>>>>> trying to do here.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> If anyone's interested in porting it to Python 3 it would
>> be
>>>> a
>>>>>>>>> good
>>>>>>>>>>>>>> change.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Otherwise, using docker as suggested here seems to be the
>>>> best
>>>>>>>>> way to
>>>>>>>>>>>>>> go.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> best,
>>>>>>>>>>>>>>>>> Colin
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Mon, Jun 29, 2020, at 02:14, Gokul Ramanan Subramanian
>>>> wrote:
>>>>>>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Has anyone had luck running Kafka system tests on a Mac. I
>>>>>>> have a
>>>>>>>>>>>>>> MacOS
>>>>>>>>>>>>>>>>>> Mojave 10.14.6. I got Python 3.6.9 using pyenv. However,
>> the
>>>>>>>>> command
>>>>>>>>>>>>>>>>>> *ducktape tests/kafkatest/tests* yields the following
>> error,
>>>>>>>>> making
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> look
>>>>>>>>>>>>>>>>>> like some Python incompatibility issue.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> $ ducktape tests/kafkatest/tests
>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>> File "/Users/gokusubr/.pyenv/versions/3.6.9/bin/ducktape",
>>>> line
>>>>>>>>> 11,
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> <module>
>>>>>>>>>>>>>>>>>> load_entry_point('ducktape', 'console_scripts',
>>>> 'ducktape')()
>>>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>>>>>>>>>>> line 487, in load_entry_point
>>>>>>>>>>>>>>>>>> return get_distribution(dist).load_entry_point(group,
>> name)
>>>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>>>>>>>>>>> line 2728, in load_entry_point
>>>>>>>>>>>>>>>>>> return ep.load()
>>>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>>>>>>>>>>> line 2346, in load
>>>>>>>>>>>>>>>>>> return self.resolve()
>>>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>>>>>>>>>>> line 2352, in resolve
>>>>>>>>>>>>>>>>>> module = __import__(self.module_name,
>> fromlist=['__name__'],
>>>>>>>>>>>>>>>>>> level=0)
>>>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/ducktape-0.7.6-py3.6.egg/ducktape/command_line/main.py",
>>>>>>>>>>>>>>>>>> line 127
>>>>>>>>>>>>>>>>>> print "parameters are not valid json: " + str(e.message)
>>>>>>>>>>>>>>>>>>                                 ^
>>>>>>>>>>>>>>>>>> SyntaxError: invalid syntax
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I followed the instructions in tests/README.md to setup a
>>>>>>>>> cluster of
>>>>>>>>>>>>>> 9
>>>>>>>>>>>>>>>>>> worker machines. That worked well. When I ran *python
>>>> setup.py
>>>>>>>>>>>>>> develop*
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> install the necessary dependencies (including ducktape), I
>>>> got
>>>>>>>>>>>>>> similar
>>>>>>>>>>>>>>>>>> errors to above, but the overall command completed
>>>>>>> successfully.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Any help appreciated.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> -- Guozhang
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> -- Guozhang
>>>>> 
>>>> 
>>>> 
>> 
>> 
> 
> -- 
> -- Guozhang


Re: [DISCUSSION] Upgrade system tests to python 3

Posted by Guozhang Wang <wa...@gmail.com>.
Hello Nikolay,

I've merged the PR to trunk. Thanks for your huge effort and patience going
through the review!

Guozhang

On Wed, Oct 7, 2020 at 6:52 AM Nikolay Izhikov <ni...@apache.org> wrote:

> Great news!
> Thanks Magnus!
>
> I’ve updated the PR.
>
> Looks like we ready to merge it.
>
> > 7 окт. 2020 г., в 15:29, Magnus Edenhill <ma...@edenhill.se>
> написал(а):
> >
> > Hi,
> >
> > ducktape v0.8.0 is now released.
> >
> > Regards,
> > Magnus
> >
> >
> > Den ons 7 okt. 2020 kl 10:50 skrev Nikolay Izhikov <nizhikov@apache.org
> >:
> >
> >> Hello.
> >>
> >> Got 4 approvals for PR [1]
> >> The only thing we need to be able to merge it is a ducktape 0.8 release.
> >> If ducktape team need any help with the release, please, let me know.
> >>
> >> [1] https://github.com/apache/kafka/pull/9196
> >>
> >>
> >>> 21 сент. 2020 г., в 12:58, Nikolay Izhikov <ni...@gmail.com>
> >> написал(а):
> >>>
> >>> Hello.
> >>>
> >>> I fixed two system tests that fails in trunk, also.
> >>>
> >>>
> streams_upgrade_test.py::StreamsUpgradeTest.test_version_probing_upgrade
> >>> streams_static_membership_test.py
> >>>
> >>> Please, take a look at my PR [1]
> >>>
> >>> [1] https://github.com/apache/kafka/pull/9312
> >>>
> >>>> 20 сент. 2020 г., в 06:11, Guozhang Wang <wa...@gmail.com>
> >> написал(а):
> >>>>
> >>>> I've triggered a system test on top of your branch.
> >>>>
> >>>> Maybe you could also re-run the jenkins unit tests since currently all
> >> of
> >>>> them fails but you've only touched on system tests, so I'd like to
> >> confirm
> >>>> at least one successful run.
> >>>>
> >>>> On Wed, Sep 16, 2020 at 3:37 AM Nikolay Izhikov <ni...@apache.org>
> >> wrote:
> >>>>
> >>>>> Hello, Guozhang.
> >>>>>
> >>>>>> I can help run the test suite once your PR is cleanly rebased to
> >> verify
> >>>>> the whole suite works
> >>>>>
> >>>>> Thank you for joining to the review.
> >>>>>
> >>>>> 1. PR rebased on the current trunk.
> >>>>>
> >>>>> 2. I triggered all tests in my private environment to verify them
> after
> >>>>> rebase.
> >>>>>      Will inform you once tests passed on my environment.
> >>>>>
> >>>>> 3. We need a new ducktape release [1] to be able to merge PR [2].
> >>>>>      For now, PR based on the ducktape trunk branch [3], not some
> >>>>> specific release.
> >>>>>      If ducktape team need any help with the release, please, let me
> >>>>> know.
> >>>>>
> >>>>> [1] https://github.com/confluentinc/ducktape/issues/245
> >>>>> [2] https://github.com/apache/kafka/pull/9196
> >>>>> [3]
> >>>>>
> >>
> https://github.com/apache/kafka/pull/9196/files#diff-9235a7bdb1ca9268681c0e56f3f3609bR39
> >>>>>
> >>>>>> 16 сент. 2020 г., в 07:32, Guozhang Wang <wa...@gmail.com>
> >>>>> написал(а):
> >>>>>>
> >>>>>> Hello Nikolay,
> >>>>>>
> >>>>>> I can help run the test suite once your PR is cleanly rebased to
> >> verify
> >>>>> the
> >>>>>> whole suite works and then I can merge (I'm trusting Ivan and Magnus
> >> here
> >>>>>> for their reviews :)
> >>>>>>
> >>>>>> Guozhang
> >>>>>>
> >>>>>> On Mon, Sep 14, 2020 at 3:56 AM Nikolay Izhikov <
> nizhikov@apache.org>
> >>>>> wrote:
> >>>>>>
> >>>>>>> Hello!
> >>>>>>>
> >>>>>>> I got 2 approvals from Ivan Daschinskiy and Magnus Edenhill.
> >>>>>>> Committers, please, join the review.
> >>>>>>>
> >>>>>>>> 3 сент. 2020 г., в 11:06, Nikolay Izhikov <nizhikov.dev@gmail.com
> >
> >>>>>>> написал(а):
> >>>>>>>>
> >>>>>>>> Hello!
> >>>>>>>>
> >>>>>>>> Just a friendly reminder.
> >>>>>>>>
> >>>>>>>> Patch to resolve some kind of technical debt - python2 in system
> >> tests
> >>>>>>> is ready!
> >>>>>>>> Can someone, please, take a look?
> >>>>>>>>
> >>>>>>>> https://github.com/apache/kafka/pull/9196
> >>>>>>>>
> >>>>>>>>> 28 авг. 2020 г., в 11:19, Nikolay Izhikov <
> nizhikov.dev@gmail.com>
> >>>>>>> написал(а):
> >>>>>>>>>
> >>>>>>>>> Hello!
> >>>>>>>>>
> >>>>>>>>> Any feedback on this?
> >>>>>>>>> What I should additionally do to prepare system tests migration?
> >>>>>>>>>
> >>>>>>>>>> 24 авг. 2020 г., в 11:17, Nikolay Izhikov <
> nizhikov.dev@gmail.com
> >>>
> >>>>>>> написал(а):
> >>>>>>>>>>
> >>>>>>>>>> Hello.
> >>>>>>>>>>
> >>>>>>>>>> PR [1] is ready.
> >>>>>>>>>> Please, review.
> >>>>>>>>>>
> >>>>>>>>>> But, I need help with the two following questions:
> >>>>>>>>>>
> >>>>>>>>>> 1. We need a new release of ducktape which includes fixes [2],
> [3]
> >>>>> for
> >>>>>>> python3.
> >>>>>>>>>> I created the issue in ducktape repo [4].
> >>>>>>>>>> Can someone help me with the release?
> >>>>>>>>>>
> >>>>>>>>>> 2. I know that some companies run system tests for the trunk on
> a
> >>>>>>> regular bases.
> >>>>>>>>>> Can someone show me some results of these runs?
> >>>>>>>>>> So, I can compare failures in my PR and in the trunk.
> >>>>>>>>>>
> >>>>>>>>>> Results [5] of run all for my PR available in the ticket [6]
> >>>>>>>>>>
> >>>>>>>>>> ```
> >>>>>>>>>> SESSION REPORT (ALL TESTS)
> >>>>>>>>>> ducktape version: 0.8.0
> >>>>>>>>>> session_id:       2020-08-23--002
> >>>>>>>>>> run time:         1010 minutes 46.483 seconds
> >>>>>>>>>> tests run:        684
> >>>>>>>>>> passed:           505
> >>>>>>>>>> failed:           9
> >>>>>>>>>> ignored:          170
> >>>>>>>>>> ```
> >>>>>>>>>>
> >>>>>>>>>> [1] https://github.com/apache/kafka/pull/9196
> >>>>>>>>>> [2]
> >>>>>>>
> >>>>>
> >>
> https://github.com/confluentinc/ducktape/commit/23bd5ab53802e3a1e1da1ddf3630934f33b02305
> >>>>>>>>>> [3]
> >>>>>>>
> >>>>>
> >>
> https://github.com/confluentinc/ducktape/commit/bfe53712f83b025832d29a43cde3de3d7803106f
> >>>>>>>>>> [4] https://github.com/confluentinc/ducktape/issues/245
> >>>>>>>>>> [5]
> >>>>>>>
> https://issues.apache.org/jira/secure/attachment/13010366/report.txt
> >>>>>>>>>> [6] https://issues.apache.org/jira/browse/KAFKA-10402
> >>>>>>>>>>
> >>>>>>>>>>> 14 авг. 2020 г., в 21:26, Ismael Juma <is...@juma.me.uk>
> >>>>> написал(а):
> >>>>>>>>>>>
> >>>>>>>>>>> +1
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Aug 14, 2020 at 7:42 AM John Roesler <
> >> vvcephei@apache.org>
> >>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Thanks Nikolay,
> >>>>>>>>>>>>
> >>>>>>>>>>>> No objection. This would be very nice to have.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> John
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Aug 14, 2020, at 09:18, Nikolay Izhikov wrote:
> >>>>>>>>>>>>> Hello.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> If anyone's interested in porting it to Python 3 it would
> be a
> >>>>> good
> >>>>>>>>>>>> change.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I’ve created a ticket [1] to upgrade system tests to python3.
> >>>>>>>>>>>>> Does someone have any additional inputs or objections for
> this
> >>>>>>> change?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-10402
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> 1 июля 2020 г., в 00:26, Gokul Ramanan Subramanian <
> >>>>>>>>>>>> gokul2411s@gmail.com> написал(а):
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks Colin.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> While at the subject of system tests, there are a few times
> I
> >> see
> >>>>>>> tests
> >>>>>>>>>>>>>> timed out (even on a large machine such as m5.4xlarge EC2
> with
> >>>>>>> Linux).
> >>>>>>>>>>>> Are
> >>>>>>>>>>>>>> there any knobs that system tests provide to control
> timeouts
> >> /
> >>>>>>>>>>>> throughputs
> >>>>>>>>>>>>>> across all tests?
> >>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Tue, Jun 30, 2020 at 6:32 PM Colin McCabe <
> >> cmccabe@apache.org
> >>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Ducktape runs on Python 2.  You can't use it with Python 3,
> >> as
> >>>>>>> you are
> >>>>>>>>>>>>>>> trying to do here.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> If anyone's interested in porting it to Python 3 it would
> be
> >> a
> >>>>>>> good
> >>>>>>>>>>>> change.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Otherwise, using docker as suggested here seems to be the
> >> best
> >>>>>>> way to
> >>>>>>>>>>>> go.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> best,
> >>>>>>>>>>>>>>> Colin
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Mon, Jun 29, 2020, at 02:14, Gokul Ramanan Subramanian
> >> wrote:
> >>>>>>>>>>>>>>>> Hi.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Has anyone had luck running Kafka system tests on a Mac. I
> >>>>> have a
> >>>>>>>>>>>> MacOS
> >>>>>>>>>>>>>>>> Mojave 10.14.6. I got Python 3.6.9 using pyenv. However,
> the
> >>>>>>> command
> >>>>>>>>>>>>>>>> *ducktape tests/kafkatest/tests* yields the following
> error,
> >>>>>>> making
> >>>>>>>>>>>> it
> >>>>>>>>>>>>>>> look
> >>>>>>>>>>>>>>>> like some Python incompatibility issue.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> $ ducktape tests/kafkatest/tests
> >>>>>>>>>>>>>>>> Traceback (most recent call last):
> >>>>>>>>>>>>>>>> File "/Users/gokusubr/.pyenv/versions/3.6.9/bin/ducktape",
> >> line
> >>>>>>> 11,
> >>>>>>>>>>>> in
> >>>>>>>>>>>>>>>> <module>
> >>>>>>>>>>>>>>>> load_entry_point('ducktape', 'console_scripts',
> >> 'ducktape')()
> >>>>>>>>>>>>>>>> File
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>
> >>>>>
> >>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> >>>>>>>>>>>>>>>> line 487, in load_entry_point
> >>>>>>>>>>>>>>>> return get_distribution(dist).load_entry_point(group,
> name)
> >>>>>>>>>>>>>>>> File
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>
> >>>>>
> >>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> >>>>>>>>>>>>>>>> line 2728, in load_entry_point
> >>>>>>>>>>>>>>>> return ep.load()
> >>>>>>>>>>>>>>>> File
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>
> >>>>>
> >>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> >>>>>>>>>>>>>>>> line 2346, in load
> >>>>>>>>>>>>>>>> return self.resolve()
> >>>>>>>>>>>>>>>> File
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>
> >>>>>
> >>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> >>>>>>>>>>>>>>>> line 2352, in resolve
> >>>>>>>>>>>>>>>> module = __import__(self.module_name,
> fromlist=['__name__'],
> >>>>>>>>>>>>>>>> level=0)
> >>>>>>>>>>>>>>>> File
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>
> >>>>>
> >>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/ducktape-0.7.6-py3.6.egg/ducktape/command_line/main.py",
> >>>>>>>>>>>>>>>> line 127
> >>>>>>>>>>>>>>>> print "parameters are not valid json: " + str(e.message)
> >>>>>>>>>>>>>>>>                                  ^
> >>>>>>>>>>>>>>>> SyntaxError: invalid syntax
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I followed the instructions in tests/README.md to setup a
> >>>>>>> cluster of
> >>>>>>>>>>>> 9
> >>>>>>>>>>>>>>>> worker machines. That worked well. When I ran *python
> >> setup.py
> >>>>>>>>>>>> develop*
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> install the necessary dependencies (including ducktape), I
> >> got
> >>>>>>>>>>>> similar
> >>>>>>>>>>>>>>>> errors to above, but the overall command completed
> >>>>> successfully.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Any help appreciated.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> -- Guozhang
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> -- Guozhang
> >>>
> >>
> >>
>
>

-- 
-- Guozhang

Re: [DISCUSSION] Upgrade system tests to python 3

Posted by Nikolay Izhikov <ni...@apache.org>.
Great news! 
Thanks Magnus!

I’ve updated the PR.

Looks like we ready to merge it.

> 7 окт. 2020 г., в 15:29, Magnus Edenhill <ma...@edenhill.se> написал(а):
> 
> Hi,
> 
> ducktape v0.8.0 is now released.
> 
> Regards,
> Magnus
> 
> 
> Den ons 7 okt. 2020 kl 10:50 skrev Nikolay Izhikov <ni...@apache.org>:
> 
>> Hello.
>> 
>> Got 4 approvals for PR [1]
>> The only thing we need to be able to merge it is a ducktape 0.8 release.
>> If ducktape team need any help with the release, please, let me know.
>> 
>> [1] https://github.com/apache/kafka/pull/9196
>> 
>> 
>>> 21 сент. 2020 г., в 12:58, Nikolay Izhikov <ni...@gmail.com>
>> написал(а):
>>> 
>>> Hello.
>>> 
>>> I fixed two system tests that fails in trunk, also.
>>> 
>>> streams_upgrade_test.py::StreamsUpgradeTest.test_version_probing_upgrade
>>> streams_static_membership_test.py
>>> 
>>> Please, take a look at my PR [1]
>>> 
>>> [1] https://github.com/apache/kafka/pull/9312
>>> 
>>>> 20 сент. 2020 г., в 06:11, Guozhang Wang <wa...@gmail.com>
>> написал(а):
>>>> 
>>>> I've triggered a system test on top of your branch.
>>>> 
>>>> Maybe you could also re-run the jenkins unit tests since currently all
>> of
>>>> them fails but you've only touched on system tests, so I'd like to
>> confirm
>>>> at least one successful run.
>>>> 
>>>> On Wed, Sep 16, 2020 at 3:37 AM Nikolay Izhikov <ni...@apache.org>
>> wrote:
>>>> 
>>>>> Hello, Guozhang.
>>>>> 
>>>>>> I can help run the test suite once your PR is cleanly rebased to
>> verify
>>>>> the whole suite works
>>>>> 
>>>>> Thank you for joining to the review.
>>>>> 
>>>>> 1. PR rebased on the current trunk.
>>>>> 
>>>>> 2. I triggered all tests in my private environment to verify them after
>>>>> rebase.
>>>>>      Will inform you once tests passed on my environment.
>>>>> 
>>>>> 3. We need a new ducktape release [1] to be able to merge PR [2].
>>>>>      For now, PR based on the ducktape trunk branch [3], not some
>>>>> specific release.
>>>>>      If ducktape team need any help with the release, please, let me
>>>>> know.
>>>>> 
>>>>> [1] https://github.com/confluentinc/ducktape/issues/245
>>>>> [2] https://github.com/apache/kafka/pull/9196
>>>>> [3]
>>>>> 
>> https://github.com/apache/kafka/pull/9196/files#diff-9235a7bdb1ca9268681c0e56f3f3609bR39
>>>>> 
>>>>>> 16 сент. 2020 г., в 07:32, Guozhang Wang <wa...@gmail.com>
>>>>> написал(а):
>>>>>> 
>>>>>> Hello Nikolay,
>>>>>> 
>>>>>> I can help run the test suite once your PR is cleanly rebased to
>> verify
>>>>> the
>>>>>> whole suite works and then I can merge (I'm trusting Ivan and Magnus
>> here
>>>>>> for their reviews :)
>>>>>> 
>>>>>> Guozhang
>>>>>> 
>>>>>> On Mon, Sep 14, 2020 at 3:56 AM Nikolay Izhikov <ni...@apache.org>
>>>>> wrote:
>>>>>> 
>>>>>>> Hello!
>>>>>>> 
>>>>>>> I got 2 approvals from Ivan Daschinskiy and Magnus Edenhill.
>>>>>>> Committers, please, join the review.
>>>>>>> 
>>>>>>>> 3 сент. 2020 г., в 11:06, Nikolay Izhikov <ni...@gmail.com>
>>>>>>> написал(а):
>>>>>>>> 
>>>>>>>> Hello!
>>>>>>>> 
>>>>>>>> Just a friendly reminder.
>>>>>>>> 
>>>>>>>> Patch to resolve some kind of technical debt - python2 in system
>> tests
>>>>>>> is ready!
>>>>>>>> Can someone, please, take a look?
>>>>>>>> 
>>>>>>>> https://github.com/apache/kafka/pull/9196
>>>>>>>> 
>>>>>>>>> 28 авг. 2020 г., в 11:19, Nikolay Izhikov <ni...@gmail.com>
>>>>>>> написал(а):
>>>>>>>>> 
>>>>>>>>> Hello!
>>>>>>>>> 
>>>>>>>>> Any feedback on this?
>>>>>>>>> What I should additionally do to prepare system tests migration?
>>>>>>>>> 
>>>>>>>>>> 24 авг. 2020 г., в 11:17, Nikolay Izhikov <nizhikov.dev@gmail.com
>>> 
>>>>>>> написал(а):
>>>>>>>>>> 
>>>>>>>>>> Hello.
>>>>>>>>>> 
>>>>>>>>>> PR [1] is ready.
>>>>>>>>>> Please, review.
>>>>>>>>>> 
>>>>>>>>>> But, I need help with the two following questions:
>>>>>>>>>> 
>>>>>>>>>> 1. We need a new release of ducktape which includes fixes [2], [3]
>>>>> for
>>>>>>> python3.
>>>>>>>>>> I created the issue in ducktape repo [4].
>>>>>>>>>> Can someone help me with the release?
>>>>>>>>>> 
>>>>>>>>>> 2. I know that some companies run system tests for the trunk on a
>>>>>>> regular bases.
>>>>>>>>>> Can someone show me some results of these runs?
>>>>>>>>>> So, I can compare failures in my PR and in the trunk.
>>>>>>>>>> 
>>>>>>>>>> Results [5] of run all for my PR available in the ticket [6]
>>>>>>>>>> 
>>>>>>>>>> ```
>>>>>>>>>> SESSION REPORT (ALL TESTS)
>>>>>>>>>> ducktape version: 0.8.0
>>>>>>>>>> session_id:       2020-08-23--002
>>>>>>>>>> run time:         1010 minutes 46.483 seconds
>>>>>>>>>> tests run:        684
>>>>>>>>>> passed:           505
>>>>>>>>>> failed:           9
>>>>>>>>>> ignored:          170
>>>>>>>>>> ```
>>>>>>>>>> 
>>>>>>>>>> [1] https://github.com/apache/kafka/pull/9196
>>>>>>>>>> [2]
>>>>>>> 
>>>>> 
>> https://github.com/confluentinc/ducktape/commit/23bd5ab53802e3a1e1da1ddf3630934f33b02305
>>>>>>>>>> [3]
>>>>>>> 
>>>>> 
>> https://github.com/confluentinc/ducktape/commit/bfe53712f83b025832d29a43cde3de3d7803106f
>>>>>>>>>> [4] https://github.com/confluentinc/ducktape/issues/245
>>>>>>>>>> [5]
>>>>>>> https://issues.apache.org/jira/secure/attachment/13010366/report.txt
>>>>>>>>>> [6] https://issues.apache.org/jira/browse/KAFKA-10402
>>>>>>>>>> 
>>>>>>>>>>> 14 авг. 2020 г., в 21:26, Ismael Juma <is...@juma.me.uk>
>>>>> написал(а):
>>>>>>>>>>> 
>>>>>>>>>>> +1
>>>>>>>>>>> 
>>>>>>>>>>> On Fri, Aug 14, 2020 at 7:42 AM John Roesler <
>> vvcephei@apache.org>
>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Thanks Nikolay,
>>>>>>>>>>>> 
>>>>>>>>>>>> No objection. This would be very nice to have.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> John
>>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, Aug 14, 2020, at 09:18, Nikolay Izhikov wrote:
>>>>>>>>>>>>> Hello.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> If anyone's interested in porting it to Python 3 it would be a
>>>>> good
>>>>>>>>>>>> change.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I’ve created a ticket [1] to upgrade system tests to python3.
>>>>>>>>>>>>> Does someone have any additional inputs or objections for this
>>>>>>> change?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-10402
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 1 июля 2020 г., в 00:26, Gokul Ramanan Subramanian <
>>>>>>>>>>>> gokul2411s@gmail.com> написал(а):
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks Colin.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> While at the subject of system tests, there are a few times I
>> see
>>>>>>> tests
>>>>>>>>>>>>>> timed out (even on a large machine such as m5.4xlarge EC2 with
>>>>>>> Linux).
>>>>>>>>>>>> Are
>>>>>>>>>>>>>> there any knobs that system tests provide to control timeouts
>> /
>>>>>>>>>>>> throughputs
>>>>>>>>>>>>>> across all tests?
>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Tue, Jun 30, 2020 at 6:32 PM Colin McCabe <
>> cmccabe@apache.org
>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Ducktape runs on Python 2.  You can't use it with Python 3,
>> as
>>>>>>> you are
>>>>>>>>>>>>>>> trying to do here.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> If anyone's interested in porting it to Python 3 it would be
>> a
>>>>>>> good
>>>>>>>>>>>> change.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Otherwise, using docker as suggested here seems to be the
>> best
>>>>>>> way to
>>>>>>>>>>>> go.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> best,
>>>>>>>>>>>>>>> Colin
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Mon, Jun 29, 2020, at 02:14, Gokul Ramanan Subramanian
>> wrote:
>>>>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Has anyone had luck running Kafka system tests on a Mac. I
>>>>> have a
>>>>>>>>>>>> MacOS
>>>>>>>>>>>>>>>> Mojave 10.14.6. I got Python 3.6.9 using pyenv. However, the
>>>>>>> command
>>>>>>>>>>>>>>>> *ducktape tests/kafkatest/tests* yields the following error,
>>>>>>> making
>>>>>>>>>>>> it
>>>>>>>>>>>>>>> look
>>>>>>>>>>>>>>>> like some Python incompatibility issue.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> $ ducktape tests/kafkatest/tests
>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>> File "/Users/gokusubr/.pyenv/versions/3.6.9/bin/ducktape",
>> line
>>>>>>> 11,
>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> <module>
>>>>>>>>>>>>>>>> load_entry_point('ducktape', 'console_scripts',
>> 'ducktape')()
>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>> 
>>>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>>>>>>>>> line 487, in load_entry_point
>>>>>>>>>>>>>>>> return get_distribution(dist).load_entry_point(group, name)
>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>> 
>>>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>>>>>>>>> line 2728, in load_entry_point
>>>>>>>>>>>>>>>> return ep.load()
>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>> 
>>>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>>>>>>>>> line 2346, in load
>>>>>>>>>>>>>>>> return self.resolve()
>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>> 
>>>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>>>>>>>>> line 2352, in resolve
>>>>>>>>>>>>>>>> module = __import__(self.module_name, fromlist=['__name__'],
>>>>>>>>>>>>>>>> level=0)
>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>> 
>>>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/ducktape-0.7.6-py3.6.egg/ducktape/command_line/main.py",
>>>>>>>>>>>>>>>> line 127
>>>>>>>>>>>>>>>> print "parameters are not valid json: " + str(e.message)
>>>>>>>>>>>>>>>>                                  ^
>>>>>>>>>>>>>>>> SyntaxError: invalid syntax
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I followed the instructions in tests/README.md to setup a
>>>>>>> cluster of
>>>>>>>>>>>> 9
>>>>>>>>>>>>>>>> worker machines. That worked well. When I ran *python
>> setup.py
>>>>>>>>>>>> develop*
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> install the necessary dependencies (including ducktape), I
>> got
>>>>>>>>>>>> similar
>>>>>>>>>>>>>>>> errors to above, but the overall command completed
>>>>> successfully.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Any help appreciated.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> -- Guozhang
>>>>> 
>>>>> 
>>>> 
>>>> --
>>>> -- Guozhang
>>> 
>> 
>> 


Re: [DISCUSSION] Upgrade system tests to python 3

Posted by Magnus Edenhill <ma...@edenhill.se>.
Hi,

ducktape v0.8.0 is now released.

Regards,
Magnus


Den ons 7 okt. 2020 kl 10:50 skrev Nikolay Izhikov <ni...@apache.org>:

> Hello.
>
> Got 4 approvals for PR [1]
> The only thing we need to be able to merge it is a ducktape 0.8 release.
>  If ducktape team need any help with the release, please, let me know.
>
> [1] https://github.com/apache/kafka/pull/9196
>
>
> > 21 сент. 2020 г., в 12:58, Nikolay Izhikov <ni...@gmail.com>
> написал(а):
> >
> > Hello.
> >
> > I fixed two system tests that fails in trunk, also.
> >
> > streams_upgrade_test.py::StreamsUpgradeTest.test_version_probing_upgrade
> > streams_static_membership_test.py
> >
> > Please, take a look at my PR [1]
> >
> > [1] https://github.com/apache/kafka/pull/9312
> >
> >> 20 сент. 2020 г., в 06:11, Guozhang Wang <wa...@gmail.com>
> написал(а):
> >>
> >> I've triggered a system test on top of your branch.
> >>
> >> Maybe you could also re-run the jenkins unit tests since currently all
> of
> >> them fails but you've only touched on system tests, so I'd like to
> confirm
> >> at least one successful run.
> >>
> >> On Wed, Sep 16, 2020 at 3:37 AM Nikolay Izhikov <ni...@apache.org>
> wrote:
> >>
> >>> Hello, Guozhang.
> >>>
> >>>> I can help run the test suite once your PR is cleanly rebased to
> verify
> >>> the whole suite works
> >>>
> >>> Thank you for joining to the review.
> >>>
> >>> 1. PR rebased on the current trunk.
> >>>
> >>> 2. I triggered all tests in my private environment to verify them after
> >>> rebase.
> >>>       Will inform you once tests passed on my environment.
> >>>
> >>> 3. We need a new ducktape release [1] to be able to merge PR [2].
> >>>       For now, PR based on the ducktape trunk branch [3], not some
> >>> specific release.
> >>>       If ducktape team need any help with the release, please, let me
> >>> know.
> >>>
> >>> [1] https://github.com/confluentinc/ducktape/issues/245
> >>> [2] https://github.com/apache/kafka/pull/9196
> >>> [3]
> >>>
> https://github.com/apache/kafka/pull/9196/files#diff-9235a7bdb1ca9268681c0e56f3f3609bR39
> >>>
> >>>> 16 сент. 2020 г., в 07:32, Guozhang Wang <wa...@gmail.com>
> >>> написал(а):
> >>>>
> >>>> Hello Nikolay,
> >>>>
> >>>> I can help run the test suite once your PR is cleanly rebased to
> verify
> >>> the
> >>>> whole suite works and then I can merge (I'm trusting Ivan and Magnus
> here
> >>>> for their reviews :)
> >>>>
> >>>> Guozhang
> >>>>
> >>>> On Mon, Sep 14, 2020 at 3:56 AM Nikolay Izhikov <ni...@apache.org>
> >>> wrote:
> >>>>
> >>>>> Hello!
> >>>>>
> >>>>> I got 2 approvals from Ivan Daschinskiy and Magnus Edenhill.
> >>>>> Committers, please, join the review.
> >>>>>
> >>>>>> 3 сент. 2020 г., в 11:06, Nikolay Izhikov <ni...@gmail.com>
> >>>>> написал(а):
> >>>>>>
> >>>>>> Hello!
> >>>>>>
> >>>>>> Just a friendly reminder.
> >>>>>>
> >>>>>> Patch to resolve some kind of technical debt - python2 in system
> tests
> >>>>> is ready!
> >>>>>> Can someone, please, take a look?
> >>>>>>
> >>>>>> https://github.com/apache/kafka/pull/9196
> >>>>>>
> >>>>>>> 28 авг. 2020 г., в 11:19, Nikolay Izhikov <ni...@gmail.com>
> >>>>> написал(а):
> >>>>>>>
> >>>>>>> Hello!
> >>>>>>>
> >>>>>>> Any feedback on this?
> >>>>>>> What I should additionally do to prepare system tests migration?
> >>>>>>>
> >>>>>>>> 24 авг. 2020 г., в 11:17, Nikolay Izhikov <nizhikov.dev@gmail.com
> >
> >>>>> написал(а):
> >>>>>>>>
> >>>>>>>> Hello.
> >>>>>>>>
> >>>>>>>> PR [1] is ready.
> >>>>>>>> Please, review.
> >>>>>>>>
> >>>>>>>> But, I need help with the two following questions:
> >>>>>>>>
> >>>>>>>> 1. We need a new release of ducktape which includes fixes [2], [3]
> >>> for
> >>>>> python3.
> >>>>>>>> I created the issue in ducktape repo [4].
> >>>>>>>> Can someone help me with the release?
> >>>>>>>>
> >>>>>>>> 2. I know that some companies run system tests for the trunk on a
> >>>>> regular bases.
> >>>>>>>> Can someone show me some results of these runs?
> >>>>>>>> So, I can compare failures in my PR and in the trunk.
> >>>>>>>>
> >>>>>>>> Results [5] of run all for my PR available in the ticket [6]
> >>>>>>>>
> >>>>>>>> ```
> >>>>>>>> SESSION REPORT (ALL TESTS)
> >>>>>>>> ducktape version: 0.8.0
> >>>>>>>> session_id:       2020-08-23--002
> >>>>>>>> run time:         1010 minutes 46.483 seconds
> >>>>>>>> tests run:        684
> >>>>>>>> passed:           505
> >>>>>>>> failed:           9
> >>>>>>>> ignored:          170
> >>>>>>>> ```
> >>>>>>>>
> >>>>>>>> [1] https://github.com/apache/kafka/pull/9196
> >>>>>>>> [2]
> >>>>>
> >>>
> https://github.com/confluentinc/ducktape/commit/23bd5ab53802e3a1e1da1ddf3630934f33b02305
> >>>>>>>> [3]
> >>>>>
> >>>
> https://github.com/confluentinc/ducktape/commit/bfe53712f83b025832d29a43cde3de3d7803106f
> >>>>>>>> [4] https://github.com/confluentinc/ducktape/issues/245
> >>>>>>>> [5]
> >>>>> https://issues.apache.org/jira/secure/attachment/13010366/report.txt
> >>>>>>>> [6] https://issues.apache.org/jira/browse/KAFKA-10402
> >>>>>>>>
> >>>>>>>>> 14 авг. 2020 г., в 21:26, Ismael Juma <is...@juma.me.uk>
> >>> написал(а):
> >>>>>>>>>
> >>>>>>>>> +1
> >>>>>>>>>
> >>>>>>>>> On Fri, Aug 14, 2020 at 7:42 AM John Roesler <
> vvcephei@apache.org>
> >>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Thanks Nikolay,
> >>>>>>>>>>
> >>>>>>>>>> No objection. This would be very nice to have.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> John
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Aug 14, 2020, at 09:18, Nikolay Izhikov wrote:
> >>>>>>>>>>> Hello.
> >>>>>>>>>>>
> >>>>>>>>>>>> If anyone's interested in porting it to Python 3 it would be a
> >>> good
> >>>>>>>>>> change.
> >>>>>>>>>>>
> >>>>>>>>>>> I’ve created a ticket [1] to upgrade system tests to python3.
> >>>>>>>>>>> Does someone have any additional inputs or objections for this
> >>>>> change?
> >>>>>>>>>>>
> >>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-10402
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> 1 июля 2020 г., в 00:26, Gokul Ramanan Subramanian <
> >>>>>>>>>> gokul2411s@gmail.com> написал(а):
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks Colin.
> >>>>>>>>>>>>
> >>>>>>>>>>>> While at the subject of system tests, there are a few times I
> see
> >>>>> tests
> >>>>>>>>>>>> timed out (even on a large machine such as m5.4xlarge EC2 with
> >>>>> Linux).
> >>>>>>>>>> Are
> >>>>>>>>>>>> there any knobs that system tests provide to control timeouts
> /
> >>>>>>>>>> throughputs
> >>>>>>>>>>>> across all tests?
> >>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Jun 30, 2020 at 6:32 PM Colin McCabe <
> cmccabe@apache.org
> >>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Ducktape runs on Python 2.  You can't use it with Python 3,
> as
> >>>>> you are
> >>>>>>>>>>>>> trying to do here.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> If anyone's interested in porting it to Python 3 it would be
> a
> >>>>> good
> >>>>>>>>>> change.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Otherwise, using docker as suggested here seems to be the
> best
> >>>>> way to
> >>>>>>>>>> go.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> best,
> >>>>>>>>>>>>> Colin
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Mon, Jun 29, 2020, at 02:14, Gokul Ramanan Subramanian
> wrote:
> >>>>>>>>>>>>>> Hi.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Has anyone had luck running Kafka system tests on a Mac. I
> >>> have a
> >>>>>>>>>> MacOS
> >>>>>>>>>>>>>> Mojave 10.14.6. I got Python 3.6.9 using pyenv. However, the
> >>>>> command
> >>>>>>>>>>>>>> *ducktape tests/kafkatest/tests* yields the following error,
> >>>>> making
> >>>>>>>>>> it
> >>>>>>>>>>>>> look
> >>>>>>>>>>>>>> like some Python incompatibility issue.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> $ ducktape tests/kafkatest/tests
> >>>>>>>>>>>>>> Traceback (most recent call last):
> >>>>>>>>>>>>>> File "/Users/gokusubr/.pyenv/versions/3.6.9/bin/ducktape",
> line
> >>>>> 11,
> >>>>>>>>>> in
> >>>>>>>>>>>>>> <module>
> >>>>>>>>>>>>>> load_entry_point('ducktape', 'console_scripts',
> 'ducktape')()
> >>>>>>>>>>>>>> File
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>
> >>>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> >>>>>>>>>>>>>> line 487, in load_entry_point
> >>>>>>>>>>>>>> return get_distribution(dist).load_entry_point(group, name)
> >>>>>>>>>>>>>> File
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>
> >>>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> >>>>>>>>>>>>>> line 2728, in load_entry_point
> >>>>>>>>>>>>>> return ep.load()
> >>>>>>>>>>>>>> File
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>
> >>>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> >>>>>>>>>>>>>> line 2346, in load
> >>>>>>>>>>>>>> return self.resolve()
> >>>>>>>>>>>>>> File
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>
> >>>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> >>>>>>>>>>>>>> line 2352, in resolve
> >>>>>>>>>>>>>> module = __import__(self.module_name, fromlist=['__name__'],
> >>>>>>>>>>>>>> level=0)
> >>>>>>>>>>>>>> File
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>
> >>>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/ducktape-0.7.6-py3.6.egg/ducktape/command_line/main.py",
> >>>>>>>>>>>>>> line 127
> >>>>>>>>>>>>>> print "parameters are not valid json: " + str(e.message)
> >>>>>>>>>>>>>>                                   ^
> >>>>>>>>>>>>>> SyntaxError: invalid syntax
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I followed the instructions in tests/README.md to setup a
> >>>>> cluster of
> >>>>>>>>>> 9
> >>>>>>>>>>>>>> worker machines. That worked well. When I ran *python
> setup.py
> >>>>>>>>>> develop*
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>> install the necessary dependencies (including ducktape), I
> got
> >>>>>>>>>> similar
> >>>>>>>>>>>>>> errors to above, but the overall command completed
> >>> successfully.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Any help appreciated.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> -- Guozhang
> >>>
> >>>
> >>
> >> --
> >> -- Guozhang
> >
>
>

Re: [DISCUSSION] Upgrade system tests to python 3

Posted by Nikolay Izhikov <ni...@apache.org>.
Hello.

Got 4 approvals for PR [1]
The only thing we need to be able to merge it is a ducktape 0.8 release.
 If ducktape team need any help with the release, please, let me know.

[1] https://github.com/apache/kafka/pull/9196


> 21 сент. 2020 г., в 12:58, Nikolay Izhikov <ni...@gmail.com> написал(а):
> 
> Hello.
> 
> I fixed two system tests that fails in trunk, also.
> 
> streams_upgrade_test.py::StreamsUpgradeTest.test_version_probing_upgrade
> streams_static_membership_test.py
> 
> Please, take a look at my PR [1]
> 
> [1] https://github.com/apache/kafka/pull/9312
> 
>> 20 сент. 2020 г., в 06:11, Guozhang Wang <wa...@gmail.com> написал(а):
>> 
>> I've triggered a system test on top of your branch.
>> 
>> Maybe you could also re-run the jenkins unit tests since currently all of
>> them fails but you've only touched on system tests, so I'd like to confirm
>> at least one successful run.
>> 
>> On Wed, Sep 16, 2020 at 3:37 AM Nikolay Izhikov <ni...@apache.org> wrote:
>> 
>>> Hello, Guozhang.
>>> 
>>>> I can help run the test suite once your PR is cleanly rebased to verify
>>> the whole suite works
>>> 
>>> Thank you for joining to the review.
>>> 
>>> 1. PR rebased on the current trunk.
>>> 
>>> 2. I triggered all tests in my private environment to verify them after
>>> rebase.
>>>       Will inform you once tests passed on my environment.
>>> 
>>> 3. We need a new ducktape release [1] to be able to merge PR [2].
>>>       For now, PR based on the ducktape trunk branch [3], not some
>>> specific release.
>>>       If ducktape team need any help with the release, please, let me
>>> know.
>>> 
>>> [1] https://github.com/confluentinc/ducktape/issues/245
>>> [2] https://github.com/apache/kafka/pull/9196
>>> [3]
>>> https://github.com/apache/kafka/pull/9196/files#diff-9235a7bdb1ca9268681c0e56f3f3609bR39
>>> 
>>>> 16 сент. 2020 г., в 07:32, Guozhang Wang <wa...@gmail.com>
>>> написал(а):
>>>> 
>>>> Hello Nikolay,
>>>> 
>>>> I can help run the test suite once your PR is cleanly rebased to verify
>>> the
>>>> whole suite works and then I can merge (I'm trusting Ivan and Magnus here
>>>> for their reviews :)
>>>> 
>>>> Guozhang
>>>> 
>>>> On Mon, Sep 14, 2020 at 3:56 AM Nikolay Izhikov <ni...@apache.org>
>>> wrote:
>>>> 
>>>>> Hello!
>>>>> 
>>>>> I got 2 approvals from Ivan Daschinskiy and Magnus Edenhill.
>>>>> Committers, please, join the review.
>>>>> 
>>>>>> 3 сент. 2020 г., в 11:06, Nikolay Izhikov <ni...@gmail.com>
>>>>> написал(а):
>>>>>> 
>>>>>> Hello!
>>>>>> 
>>>>>> Just a friendly reminder.
>>>>>> 
>>>>>> Patch to resolve some kind of technical debt - python2 in system tests
>>>>> is ready!
>>>>>> Can someone, please, take a look?
>>>>>> 
>>>>>> https://github.com/apache/kafka/pull/9196
>>>>>> 
>>>>>>> 28 авг. 2020 г., в 11:19, Nikolay Izhikov <ni...@gmail.com>
>>>>> написал(а):
>>>>>>> 
>>>>>>> Hello!
>>>>>>> 
>>>>>>> Any feedback on this?
>>>>>>> What I should additionally do to prepare system tests migration?
>>>>>>> 
>>>>>>>> 24 авг. 2020 г., в 11:17, Nikolay Izhikov <ni...@gmail.com>
>>>>> написал(а):
>>>>>>>> 
>>>>>>>> Hello.
>>>>>>>> 
>>>>>>>> PR [1] is ready.
>>>>>>>> Please, review.
>>>>>>>> 
>>>>>>>> But, I need help with the two following questions:
>>>>>>>> 
>>>>>>>> 1. We need a new release of ducktape which includes fixes [2], [3]
>>> for
>>>>> python3.
>>>>>>>> I created the issue in ducktape repo [4].
>>>>>>>> Can someone help me with the release?
>>>>>>>> 
>>>>>>>> 2. I know that some companies run system tests for the trunk on a
>>>>> regular bases.
>>>>>>>> Can someone show me some results of these runs?
>>>>>>>> So, I can compare failures in my PR and in the trunk.
>>>>>>>> 
>>>>>>>> Results [5] of run all for my PR available in the ticket [6]
>>>>>>>> 
>>>>>>>> ```
>>>>>>>> SESSION REPORT (ALL TESTS)
>>>>>>>> ducktape version: 0.8.0
>>>>>>>> session_id:       2020-08-23--002
>>>>>>>> run time:         1010 minutes 46.483 seconds
>>>>>>>> tests run:        684
>>>>>>>> passed:           505
>>>>>>>> failed:           9
>>>>>>>> ignored:          170
>>>>>>>> ```
>>>>>>>> 
>>>>>>>> [1] https://github.com/apache/kafka/pull/9196
>>>>>>>> [2]
>>>>> 
>>> https://github.com/confluentinc/ducktape/commit/23bd5ab53802e3a1e1da1ddf3630934f33b02305
>>>>>>>> [3]
>>>>> 
>>> https://github.com/confluentinc/ducktape/commit/bfe53712f83b025832d29a43cde3de3d7803106f
>>>>>>>> [4] https://github.com/confluentinc/ducktape/issues/245
>>>>>>>> [5]
>>>>> https://issues.apache.org/jira/secure/attachment/13010366/report.txt
>>>>>>>> [6] https://issues.apache.org/jira/browse/KAFKA-10402
>>>>>>>> 
>>>>>>>>> 14 авг. 2020 г., в 21:26, Ismael Juma <is...@juma.me.uk>
>>> написал(а):
>>>>>>>>> 
>>>>>>>>> +1
>>>>>>>>> 
>>>>>>>>> On Fri, Aug 14, 2020 at 7:42 AM John Roesler <vv...@apache.org>
>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Thanks Nikolay,
>>>>>>>>>> 
>>>>>>>>>> No objection. This would be very nice to have.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> John
>>>>>>>>>> 
>>>>>>>>>> On Fri, Aug 14, 2020, at 09:18, Nikolay Izhikov wrote:
>>>>>>>>>>> Hello.
>>>>>>>>>>> 
>>>>>>>>>>>> If anyone's interested in porting it to Python 3 it would be a
>>> good
>>>>>>>>>> change.
>>>>>>>>>>> 
>>>>>>>>>>> I’ve created a ticket [1] to upgrade system tests to python3.
>>>>>>>>>>> Does someone have any additional inputs or objections for this
>>>>> change?
>>>>>>>>>>> 
>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-10402
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> 1 июля 2020 г., в 00:26, Gokul Ramanan Subramanian <
>>>>>>>>>> gokul2411s@gmail.com> написал(а):
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks Colin.
>>>>>>>>>>>> 
>>>>>>>>>>>> While at the subject of system tests, there are a few times I see
>>>>> tests
>>>>>>>>>>>> timed out (even on a large machine such as m5.4xlarge EC2 with
>>>>> Linux).
>>>>>>>>>> Are
>>>>>>>>>>>> there any knobs that system tests provide to control timeouts /
>>>>>>>>>> throughputs
>>>>>>>>>>>> across all tests?
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Jun 30, 2020 at 6:32 PM Colin McCabe <cmccabe@apache.org
>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Ducktape runs on Python 2.  You can't use it with Python 3, as
>>>>> you are
>>>>>>>>>>>>> trying to do here.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> If anyone's interested in porting it to Python 3 it would be a
>>>>> good
>>>>>>>>>> change.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Otherwise, using docker as suggested here seems to be the best
>>>>> way to
>>>>>>>>>> go.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> best,
>>>>>>>>>>>>> Colin
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Mon, Jun 29, 2020, at 02:14, Gokul Ramanan Subramanian wrote:
>>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Has anyone had luck running Kafka system tests on a Mac. I
>>> have a
>>>>>>>>>> MacOS
>>>>>>>>>>>>>> Mojave 10.14.6. I got Python 3.6.9 using pyenv. However, the
>>>>> command
>>>>>>>>>>>>>> *ducktape tests/kafkatest/tests* yields the following error,
>>>>> making
>>>>>>>>>> it
>>>>>>>>>>>>> look
>>>>>>>>>>>>>> like some Python incompatibility issue.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> $ ducktape tests/kafkatest/tests
>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>> File "/Users/gokusubr/.pyenv/versions/3.6.9/bin/ducktape", line
>>>>> 11,
>>>>>>>>>> in
>>>>>>>>>>>>>> <module>
>>>>>>>>>>>>>> load_entry_point('ducktape', 'console_scripts', 'ducktape')()
>>>>>>>>>>>>>> File
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>> 
>>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>>>>>>> line 487, in load_entry_point
>>>>>>>>>>>>>> return get_distribution(dist).load_entry_point(group, name)
>>>>>>>>>>>>>> File
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>> 
>>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>>>>>>> line 2728, in load_entry_point
>>>>>>>>>>>>>> return ep.load()
>>>>>>>>>>>>>> File
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>> 
>>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>>>>>>> line 2346, in load
>>>>>>>>>>>>>> return self.resolve()
>>>>>>>>>>>>>> File
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>> 
>>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>>>>>>> line 2352, in resolve
>>>>>>>>>>>>>> module = __import__(self.module_name, fromlist=['__name__'],
>>>>>>>>>>>>>> level=0)
>>>>>>>>>>>>>> File
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>> 
>>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/ducktape-0.7.6-py3.6.egg/ducktape/command_line/main.py",
>>>>>>>>>>>>>> line 127
>>>>>>>>>>>>>> print "parameters are not valid json: " + str(e.message)
>>>>>>>>>>>>>>                                   ^
>>>>>>>>>>>>>> SyntaxError: invalid syntax
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I followed the instructions in tests/README.md to setup a
>>>>> cluster of
>>>>>>>>>> 9
>>>>>>>>>>>>>> worker machines. That worked well. When I ran *python setup.py
>>>>>>>>>> develop*
>>>>>>>>>>>>> to
>>>>>>>>>>>>>> install the necessary dependencies (including ducktape), I got
>>>>>>>>>> similar
>>>>>>>>>>>>>> errors to above, but the overall command completed
>>> successfully.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Any help appreciated.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> --
>>>> -- Guozhang
>>> 
>>> 
>> 
>> -- 
>> -- Guozhang
> 


Re: [DISCUSSION] Upgrade system tests to python 3

Posted by Nikolay Izhikov <ni...@apache.org>.
Hello.

I fixed two system tests that fails in trunk, also.

streams_upgrade_test.py::StreamsUpgradeTest.test_version_probing_upgrade
streams_static_membership_test.py

Please, take a look at my PR [1]

[1] https://github.com/apache/kafka/pull/9312

> 20 сент. 2020 г., в 06:11, Guozhang Wang <wa...@gmail.com> написал(а):
> 
> I've triggered a system test on top of your branch.
> 
> Maybe you could also re-run the jenkins unit tests since currently all of
> them fails but you've only touched on system tests, so I'd like to confirm
> at least one successful run.
> 
> On Wed, Sep 16, 2020 at 3:37 AM Nikolay Izhikov <ni...@apache.org> wrote:
> 
>> Hello, Guozhang.
>> 
>>> I can help run the test suite once your PR is cleanly rebased to verify
>> the whole suite works
>> 
>> Thank you for joining to the review.
>> 
>> 1. PR rebased on the current trunk.
>> 
>> 2. I triggered all tests in my private environment to verify them after
>> rebase.
>>        Will inform you once tests passed on my environment.
>> 
>> 3. We need a new ducktape release [1] to be able to merge PR [2].
>>        For now, PR based on the ducktape trunk branch [3], not some
>> specific release.
>>        If ducktape team need any help with the release, please, let me
>> know.
>> 
>> [1] https://github.com/confluentinc/ducktape/issues/245
>> [2] https://github.com/apache/kafka/pull/9196
>> [3]
>> https://github.com/apache/kafka/pull/9196/files#diff-9235a7bdb1ca9268681c0e56f3f3609bR39
>> 
>>> 16 сент. 2020 г., в 07:32, Guozhang Wang <wa...@gmail.com>
>> написал(а):
>>> 
>>> Hello Nikolay,
>>> 
>>> I can help run the test suite once your PR is cleanly rebased to verify
>> the
>>> whole suite works and then I can merge (I'm trusting Ivan and Magnus here
>>> for their reviews :)
>>> 
>>> Guozhang
>>> 
>>> On Mon, Sep 14, 2020 at 3:56 AM Nikolay Izhikov <ni...@apache.org>
>> wrote:
>>> 
>>>> Hello!
>>>> 
>>>> I got 2 approvals from Ivan Daschinskiy and Magnus Edenhill.
>>>> Committers, please, join the review.
>>>> 
>>>>> 3 сент. 2020 г., в 11:06, Nikolay Izhikov <ni...@gmail.com>
>>>> написал(а):
>>>>> 
>>>>> Hello!
>>>>> 
>>>>> Just a friendly reminder.
>>>>> 
>>>>> Patch to resolve some kind of technical debt - python2 in system tests
>>>> is ready!
>>>>> Can someone, please, take a look?
>>>>> 
>>>>> https://github.com/apache/kafka/pull/9196
>>>>> 
>>>>>> 28 авг. 2020 г., в 11:19, Nikolay Izhikov <ni...@gmail.com>
>>>> написал(а):
>>>>>> 
>>>>>> Hello!
>>>>>> 
>>>>>> Any feedback on this?
>>>>>> What I should additionally do to prepare system tests migration?
>>>>>> 
>>>>>>> 24 авг. 2020 г., в 11:17, Nikolay Izhikov <ni...@gmail.com>
>>>> написал(а):
>>>>>>> 
>>>>>>> Hello.
>>>>>>> 
>>>>>>> PR [1] is ready.
>>>>>>> Please, review.
>>>>>>> 
>>>>>>> But, I need help with the two following questions:
>>>>>>> 
>>>>>>> 1. We need a new release of ducktape which includes fixes [2], [3]
>> for
>>>> python3.
>>>>>>> I created the issue in ducktape repo [4].
>>>>>>> Can someone help me with the release?
>>>>>>> 
>>>>>>> 2. I know that some companies run system tests for the trunk on a
>>>> regular bases.
>>>>>>> Can someone show me some results of these runs?
>>>>>>> So, I can compare failures in my PR and in the trunk.
>>>>>>> 
>>>>>>> Results [5] of run all for my PR available in the ticket [6]
>>>>>>> 
>>>>>>> ```
>>>>>>> SESSION REPORT (ALL TESTS)
>>>>>>> ducktape version: 0.8.0
>>>>>>> session_id:       2020-08-23--002
>>>>>>> run time:         1010 minutes 46.483 seconds
>>>>>>> tests run:        684
>>>>>>> passed:           505
>>>>>>> failed:           9
>>>>>>> ignored:          170
>>>>>>> ```
>>>>>>> 
>>>>>>> [1] https://github.com/apache/kafka/pull/9196
>>>>>>> [2]
>>>> 
>> https://github.com/confluentinc/ducktape/commit/23bd5ab53802e3a1e1da1ddf3630934f33b02305
>>>>>>> [3]
>>>> 
>> https://github.com/confluentinc/ducktape/commit/bfe53712f83b025832d29a43cde3de3d7803106f
>>>>>>> [4] https://github.com/confluentinc/ducktape/issues/245
>>>>>>> [5]
>>>> https://issues.apache.org/jira/secure/attachment/13010366/report.txt
>>>>>>> [6] https://issues.apache.org/jira/browse/KAFKA-10402
>>>>>>> 
>>>>>>>> 14 авг. 2020 г., в 21:26, Ismael Juma <is...@juma.me.uk>
>> написал(а):
>>>>>>>> 
>>>>>>>> +1
>>>>>>>> 
>>>>>>>> On Fri, Aug 14, 2020 at 7:42 AM John Roesler <vv...@apache.org>
>>>> wrote:
>>>>>>>> 
>>>>>>>>> Thanks Nikolay,
>>>>>>>>> 
>>>>>>>>> No objection. This would be very nice to have.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> John
>>>>>>>>> 
>>>>>>>>> On Fri, Aug 14, 2020, at 09:18, Nikolay Izhikov wrote:
>>>>>>>>>> Hello.
>>>>>>>>>> 
>>>>>>>>>>> If anyone's interested in porting it to Python 3 it would be a
>> good
>>>>>>>>> change.
>>>>>>>>>> 
>>>>>>>>>> I’ve created a ticket [1] to upgrade system tests to python3.
>>>>>>>>>> Does someone have any additional inputs or objections for this
>>>> change?
>>>>>>>>>> 
>>>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-10402
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> 1 июля 2020 г., в 00:26, Gokul Ramanan Subramanian <
>>>>>>>>> gokul2411s@gmail.com> написал(а):
>>>>>>>>>>> 
>>>>>>>>>>> Thanks Colin.
>>>>>>>>>>> 
>>>>>>>>>>> While at the subject of system tests, there are a few times I see
>>>> tests
>>>>>>>>>>> timed out (even on a large machine such as m5.4xlarge EC2 with
>>>> Linux).
>>>>>>>>> Are
>>>>>>>>>>> there any knobs that system tests provide to control timeouts /
>>>>>>>>> throughputs
>>>>>>>>>>> across all tests?
>>>>>>>>>>> Thanks.
>>>>>>>>>>> 
>>>>>>>>>>> On Tue, Jun 30, 2020 at 6:32 PM Colin McCabe <cmccabe@apache.org
>>> 
>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Ducktape runs on Python 2.  You can't use it with Python 3, as
>>>> you are
>>>>>>>>>>>> trying to do here.
>>>>>>>>>>>> 
>>>>>>>>>>>> If anyone's interested in porting it to Python 3 it would be a
>>>> good
>>>>>>>>> change.
>>>>>>>>>>>> 
>>>>>>>>>>>> Otherwise, using docker as suggested here seems to be the best
>>>> way to
>>>>>>>>> go.
>>>>>>>>>>>> 
>>>>>>>>>>>> best,
>>>>>>>>>>>> Colin
>>>>>>>>>>>> 
>>>>>>>>>>>> On Mon, Jun 29, 2020, at 02:14, Gokul Ramanan Subramanian wrote:
>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Has anyone had luck running Kafka system tests on a Mac. I
>> have a
>>>>>>>>> MacOS
>>>>>>>>>>>>> Mojave 10.14.6. I got Python 3.6.9 using pyenv. However, the
>>>> command
>>>>>>>>>>>>> *ducktape tests/kafkatest/tests* yields the following error,
>>>> making
>>>>>>>>> it
>>>>>>>>>>>> look
>>>>>>>>>>>>> like some Python incompatibility issue.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> $ ducktape tests/kafkatest/tests
>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>> File "/Users/gokusubr/.pyenv/versions/3.6.9/bin/ducktape", line
>>>> 11,
>>>>>>>>> in
>>>>>>>>>>>>> <module>
>>>>>>>>>>>>> load_entry_point('ducktape', 'console_scripts', 'ducktape')()
>>>>>>>>>>>>> File
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>> 
>>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>>>>>> line 487, in load_entry_point
>>>>>>>>>>>>> return get_distribution(dist).load_entry_point(group, name)
>>>>>>>>>>>>> File
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>> 
>>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>>>>>> line 2728, in load_entry_point
>>>>>>>>>>>>> return ep.load()
>>>>>>>>>>>>> File
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>> 
>>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>>>>>> line 2346, in load
>>>>>>>>>>>>> return self.resolve()
>>>>>>>>>>>>> File
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>> 
>>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>>>>>> line 2352, in resolve
>>>>>>>>>>>>> module = __import__(self.module_name, fromlist=['__name__'],
>>>>>>>>>>>>> level=0)
>>>>>>>>>>>>> File
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>> 
>>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/ducktape-0.7.6-py3.6.egg/ducktape/command_line/main.py",
>>>>>>>>>>>>> line 127
>>>>>>>>>>>>> print "parameters are not valid json: " + str(e.message)
>>>>>>>>>>>>>                                    ^
>>>>>>>>>>>>> SyntaxError: invalid syntax
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I followed the instructions in tests/README.md to setup a
>>>> cluster of
>>>>>>>>> 9
>>>>>>>>>>>>> worker machines. That worked well. When I ran *python setup.py
>>>>>>>>> develop*
>>>>>>>>>>>> to
>>>>>>>>>>>>> install the necessary dependencies (including ducktape), I got
>>>>>>>>> similar
>>>>>>>>>>>>> errors to above, but the overall command completed
>> successfully.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Any help appreciated.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> --
>>> -- Guozhang
>> 
>> 
> 
> -- 
> -- Guozhang


Re: [DISCUSSION] Upgrade system tests to python 3

Posted by Guozhang Wang <wa...@gmail.com>.
I've triggered a system test on top of your branch.

Maybe you could also re-run the jenkins unit tests since currently all of
them fails but you've only touched on system tests, so I'd like to confirm
at least one successful run.

On Wed, Sep 16, 2020 at 3:37 AM Nikolay Izhikov <ni...@apache.org> wrote:

> Hello, Guozhang.
>
> > I can help run the test suite once your PR is cleanly rebased to verify
> the whole suite works
>
> Thank you for joining to the review.
>
> 1. PR rebased on the current trunk.
>
> 2. I triggered all tests in my private environment to verify them after
> rebase.
>         Will inform you once tests passed on my environment.
>
> 3. We need a new ducktape release [1] to be able to merge PR [2].
>         For now, PR based on the ducktape trunk branch [3], not some
> specific release.
>         If ducktape team need any help with the release, please, let me
> know.
>
> [1] https://github.com/confluentinc/ducktape/issues/245
> [2] https://github.com/apache/kafka/pull/9196
> [3]
> https://github.com/apache/kafka/pull/9196/files#diff-9235a7bdb1ca9268681c0e56f3f3609bR39
>
> > 16 сент. 2020 г., в 07:32, Guozhang Wang <wa...@gmail.com>
> написал(а):
> >
> > Hello Nikolay,
> >
> > I can help run the test suite once your PR is cleanly rebased to verify
> the
> > whole suite works and then I can merge (I'm trusting Ivan and Magnus here
> > for their reviews :)
> >
> > Guozhang
> >
> > On Mon, Sep 14, 2020 at 3:56 AM Nikolay Izhikov <ni...@apache.org>
> wrote:
> >
> >> Hello!
> >>
> >> I got 2 approvals from Ivan Daschinskiy and Magnus Edenhill.
> >> Committers, please, join the review.
> >>
> >>> 3 сент. 2020 г., в 11:06, Nikolay Izhikov <ni...@gmail.com>
> >> написал(а):
> >>>
> >>> Hello!
> >>>
> >>> Just a friendly reminder.
> >>>
> >>> Patch to resolve some kind of technical debt - python2 in system tests
> >> is ready!
> >>> Can someone, please, take a look?
> >>>
> >>> https://github.com/apache/kafka/pull/9196
> >>>
> >>>> 28 авг. 2020 г., в 11:19, Nikolay Izhikov <ni...@gmail.com>
> >> написал(а):
> >>>>
> >>>> Hello!
> >>>>
> >>>> Any feedback on this?
> >>>> What I should additionally do to prepare system tests migration?
> >>>>
> >>>>> 24 авг. 2020 г., в 11:17, Nikolay Izhikov <ni...@gmail.com>
> >> написал(а):
> >>>>>
> >>>>> Hello.
> >>>>>
> >>>>> PR [1] is ready.
> >>>>> Please, review.
> >>>>>
> >>>>> But, I need help with the two following questions:
> >>>>>
> >>>>> 1. We need a new release of ducktape which includes fixes [2], [3]
> for
> >> python3.
> >>>>> I created the issue in ducktape repo [4].
> >>>>> Can someone help me with the release?
> >>>>>
> >>>>> 2. I know that some companies run system tests for the trunk on a
> >> regular bases.
> >>>>> Can someone show me some results of these runs?
> >>>>> So, I can compare failures in my PR and in the trunk.
> >>>>>
> >>>>> Results [5] of run all for my PR available in the ticket [6]
> >>>>>
> >>>>> ```
> >>>>> SESSION REPORT (ALL TESTS)
> >>>>> ducktape version: 0.8.0
> >>>>> session_id:       2020-08-23--002
> >>>>> run time:         1010 minutes 46.483 seconds
> >>>>> tests run:        684
> >>>>> passed:           505
> >>>>> failed:           9
> >>>>> ignored:          170
> >>>>> ```
> >>>>>
> >>>>> [1] https://github.com/apache/kafka/pull/9196
> >>>>> [2]
> >>
> https://github.com/confluentinc/ducktape/commit/23bd5ab53802e3a1e1da1ddf3630934f33b02305
> >>>>> [3]
> >>
> https://github.com/confluentinc/ducktape/commit/bfe53712f83b025832d29a43cde3de3d7803106f
> >>>>> [4] https://github.com/confluentinc/ducktape/issues/245
> >>>>> [5]
> >> https://issues.apache.org/jira/secure/attachment/13010366/report.txt
> >>>>> [6] https://issues.apache.org/jira/browse/KAFKA-10402
> >>>>>
> >>>>>> 14 авг. 2020 г., в 21:26, Ismael Juma <is...@juma.me.uk>
> написал(а):
> >>>>>>
> >>>>>> +1
> >>>>>>
> >>>>>> On Fri, Aug 14, 2020 at 7:42 AM John Roesler <vv...@apache.org>
> >> wrote:
> >>>>>>
> >>>>>>> Thanks Nikolay,
> >>>>>>>
> >>>>>>> No objection. This would be very nice to have.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> John
> >>>>>>>
> >>>>>>> On Fri, Aug 14, 2020, at 09:18, Nikolay Izhikov wrote:
> >>>>>>>> Hello.
> >>>>>>>>
> >>>>>>>>> If anyone's interested in porting it to Python 3 it would be a
> good
> >>>>>>> change.
> >>>>>>>>
> >>>>>>>> I’ve created a ticket [1] to upgrade system tests to python3.
> >>>>>>>> Does someone have any additional inputs or objections for this
> >> change?
> >>>>>>>>
> >>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-10402
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> 1 июля 2020 г., в 00:26, Gokul Ramanan Subramanian <
> >>>>>>> gokul2411s@gmail.com> написал(а):
> >>>>>>>>>
> >>>>>>>>> Thanks Colin.
> >>>>>>>>>
> >>>>>>>>> While at the subject of system tests, there are a few times I see
> >> tests
> >>>>>>>>> timed out (even on a large machine such as m5.4xlarge EC2 with
> >> Linux).
> >>>>>>> Are
> >>>>>>>>> there any knobs that system tests provide to control timeouts /
> >>>>>>> throughputs
> >>>>>>>>> across all tests?
> >>>>>>>>> Thanks.
> >>>>>>>>>
> >>>>>>>>> On Tue, Jun 30, 2020 at 6:32 PM Colin McCabe <cmccabe@apache.org
> >
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Ducktape runs on Python 2.  You can't use it with Python 3, as
> >> you are
> >>>>>>>>>> trying to do here.
> >>>>>>>>>>
> >>>>>>>>>> If anyone's interested in porting it to Python 3 it would be a
> >> good
> >>>>>>> change.
> >>>>>>>>>>
> >>>>>>>>>> Otherwise, using docker as suggested here seems to be the best
> >> way to
> >>>>>>> go.
> >>>>>>>>>>
> >>>>>>>>>> best,
> >>>>>>>>>> Colin
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Jun 29, 2020, at 02:14, Gokul Ramanan Subramanian wrote:
> >>>>>>>>>>> Hi.
> >>>>>>>>>>>
> >>>>>>>>>>> Has anyone had luck running Kafka system tests on a Mac. I
> have a
> >>>>>>> MacOS
> >>>>>>>>>>> Mojave 10.14.6. I got Python 3.6.9 using pyenv. However, the
> >> command
> >>>>>>>>>>> *ducktape tests/kafkatest/tests* yields the following error,
> >> making
> >>>>>>> it
> >>>>>>>>>> look
> >>>>>>>>>>> like some Python incompatibility issue.
> >>>>>>>>>>>
> >>>>>>>>>>> $ ducktape tests/kafkatest/tests
> >>>>>>>>>>> Traceback (most recent call last):
> >>>>>>>>>>> File "/Users/gokusubr/.pyenv/versions/3.6.9/bin/ducktape", line
> >> 11,
> >>>>>>> in
> >>>>>>>>>>> <module>
> >>>>>>>>>>> load_entry_point('ducktape', 'console_scripts', 'ducktape')()
> >>>>>>>>>>> File
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> >>>>>>>>>>> line 487, in load_entry_point
> >>>>>>>>>>> return get_distribution(dist).load_entry_point(group, name)
> >>>>>>>>>>> File
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> >>>>>>>>>>> line 2728, in load_entry_point
> >>>>>>>>>>> return ep.load()
> >>>>>>>>>>> File
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> >>>>>>>>>>> line 2346, in load
> >>>>>>>>>>> return self.resolve()
> >>>>>>>>>>> File
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> >>>>>>>>>>> line 2352, in resolve
> >>>>>>>>>>> module = __import__(self.module_name, fromlist=['__name__'],
> >>>>>>>>>>> level=0)
> >>>>>>>>>>> File
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/ducktape-0.7.6-py3.6.egg/ducktape/command_line/main.py",
> >>>>>>>>>>> line 127
> >>>>>>>>>>> print "parameters are not valid json: " + str(e.message)
> >>>>>>>>>>>                                     ^
> >>>>>>>>>>> SyntaxError: invalid syntax
> >>>>>>>>>>>
> >>>>>>>>>>> I followed the instructions in tests/README.md to setup a
> >> cluster of
> >>>>>>> 9
> >>>>>>>>>>> worker machines. That worked well. When I ran *python setup.py
> >>>>>>> develop*
> >>>>>>>>>> to
> >>>>>>>>>>> install the necessary dependencies (including ducktape), I got
> >>>>>>> similar
> >>>>>>>>>>> errors to above, but the overall command completed
> successfully.
> >>>>>>>>>>>
> >>>>>>>>>>> Any help appreciated.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >
> > --
> > -- Guozhang
>
>

-- 
-- Guozhang

Re: [DISCUSSION] Upgrade system tests to python 3

Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Guozhang.

> I can help run the test suite once your PR is cleanly rebased to verify the whole suite works

Thank you for joining to the review.

1. PR rebased on the current trunk.

2. I triggered all tests in my private environment to verify them after rebase.
	Will inform you once tests passed on my environment.

3. We need a new ducktape release [1] to be able to merge PR [2]. 
	For now, PR based on the ducktape trunk branch [3], not some specific release.
	If ducktape team need any help with the release, please, let me know.

[1] https://github.com/confluentinc/ducktape/issues/245
[2] https://github.com/apache/kafka/pull/9196
[3] https://github.com/apache/kafka/pull/9196/files#diff-9235a7bdb1ca9268681c0e56f3f3609bR39

> 16 сент. 2020 г., в 07:32, Guozhang Wang <wa...@gmail.com> написал(а):
> 
> Hello Nikolay,
> 
> I can help run the test suite once your PR is cleanly rebased to verify the
> whole suite works and then I can merge (I'm trusting Ivan and Magnus here
> for their reviews :)
> 
> Guozhang
> 
> On Mon, Sep 14, 2020 at 3:56 AM Nikolay Izhikov <ni...@apache.org> wrote:
> 
>> Hello!
>> 
>> I got 2 approvals from Ivan Daschinskiy and Magnus Edenhill.
>> Committers, please, join the review.
>> 
>>> 3 сент. 2020 г., в 11:06, Nikolay Izhikov <ni...@gmail.com>
>> написал(а):
>>> 
>>> Hello!
>>> 
>>> Just a friendly reminder.
>>> 
>>> Patch to resolve some kind of technical debt - python2 in system tests
>> is ready!
>>> Can someone, please, take a look?
>>> 
>>> https://github.com/apache/kafka/pull/9196
>>> 
>>>> 28 авг. 2020 г., в 11:19, Nikolay Izhikov <ni...@gmail.com>
>> написал(а):
>>>> 
>>>> Hello!
>>>> 
>>>> Any feedback on this?
>>>> What I should additionally do to prepare system tests migration?
>>>> 
>>>>> 24 авг. 2020 г., в 11:17, Nikolay Izhikov <ni...@gmail.com>
>> написал(а):
>>>>> 
>>>>> Hello.
>>>>> 
>>>>> PR [1] is ready.
>>>>> Please, review.
>>>>> 
>>>>> But, I need help with the two following questions:
>>>>> 
>>>>> 1. We need a new release of ducktape which includes fixes [2], [3] for
>> python3.
>>>>> I created the issue in ducktape repo [4].
>>>>> Can someone help me with the release?
>>>>> 
>>>>> 2. I know that some companies run system tests for the trunk on a
>> regular bases.
>>>>> Can someone show me some results of these runs?
>>>>> So, I can compare failures in my PR and in the trunk.
>>>>> 
>>>>> Results [5] of run all for my PR available in the ticket [6]
>>>>> 
>>>>> ```
>>>>> SESSION REPORT (ALL TESTS)
>>>>> ducktape version: 0.8.0
>>>>> session_id:       2020-08-23--002
>>>>> run time:         1010 minutes 46.483 seconds
>>>>> tests run:        684
>>>>> passed:           505
>>>>> failed:           9
>>>>> ignored:          170
>>>>> ```
>>>>> 
>>>>> [1] https://github.com/apache/kafka/pull/9196
>>>>> [2]
>> https://github.com/confluentinc/ducktape/commit/23bd5ab53802e3a1e1da1ddf3630934f33b02305
>>>>> [3]
>> https://github.com/confluentinc/ducktape/commit/bfe53712f83b025832d29a43cde3de3d7803106f
>>>>> [4] https://github.com/confluentinc/ducktape/issues/245
>>>>> [5]
>> https://issues.apache.org/jira/secure/attachment/13010366/report.txt
>>>>> [6] https://issues.apache.org/jira/browse/KAFKA-10402
>>>>> 
>>>>>> 14 авг. 2020 г., в 21:26, Ismael Juma <is...@juma.me.uk> написал(а):
>>>>>> 
>>>>>> +1
>>>>>> 
>>>>>> On Fri, Aug 14, 2020 at 7:42 AM John Roesler <vv...@apache.org>
>> wrote:
>>>>>> 
>>>>>>> Thanks Nikolay,
>>>>>>> 
>>>>>>> No objection. This would be very nice to have.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> John
>>>>>>> 
>>>>>>> On Fri, Aug 14, 2020, at 09:18, Nikolay Izhikov wrote:
>>>>>>>> Hello.
>>>>>>>> 
>>>>>>>>> If anyone's interested in porting it to Python 3 it would be a good
>>>>>>> change.
>>>>>>>> 
>>>>>>>> I’ve created a ticket [1] to upgrade system tests to python3.
>>>>>>>> Does someone have any additional inputs or objections for this
>> change?
>>>>>>>> 
>>>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-10402
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> 1 июля 2020 г., в 00:26, Gokul Ramanan Subramanian <
>>>>>>> gokul2411s@gmail.com> написал(а):
>>>>>>>>> 
>>>>>>>>> Thanks Colin.
>>>>>>>>> 
>>>>>>>>> While at the subject of system tests, there are a few times I see
>> tests
>>>>>>>>> timed out (even on a large machine such as m5.4xlarge EC2 with
>> Linux).
>>>>>>> Are
>>>>>>>>> there any knobs that system tests provide to control timeouts /
>>>>>>> throughputs
>>>>>>>>> across all tests?
>>>>>>>>> Thanks.
>>>>>>>>> 
>>>>>>>>> On Tue, Jun 30, 2020 at 6:32 PM Colin McCabe <cm...@apache.org>
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Ducktape runs on Python 2.  You can't use it with Python 3, as
>> you are
>>>>>>>>>> trying to do here.
>>>>>>>>>> 
>>>>>>>>>> If anyone's interested in porting it to Python 3 it would be a
>> good
>>>>>>> change.
>>>>>>>>>> 
>>>>>>>>>> Otherwise, using docker as suggested here seems to be the best
>> way to
>>>>>>> go.
>>>>>>>>>> 
>>>>>>>>>> best,
>>>>>>>>>> Colin
>>>>>>>>>> 
>>>>>>>>>> On Mon, Jun 29, 2020, at 02:14, Gokul Ramanan Subramanian wrote:
>>>>>>>>>>> Hi.
>>>>>>>>>>> 
>>>>>>>>>>> Has anyone had luck running Kafka system tests on a Mac. I have a
>>>>>>> MacOS
>>>>>>>>>>> Mojave 10.14.6. I got Python 3.6.9 using pyenv. However, the
>> command
>>>>>>>>>>> *ducktape tests/kafkatest/tests* yields the following error,
>> making
>>>>>>> it
>>>>>>>>>> look
>>>>>>>>>>> like some Python incompatibility issue.
>>>>>>>>>>> 
>>>>>>>>>>> $ ducktape tests/kafkatest/tests
>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>> File "/Users/gokusubr/.pyenv/versions/3.6.9/bin/ducktape", line
>> 11,
>>>>>>> in
>>>>>>>>>>> <module>
>>>>>>>>>>> load_entry_point('ducktape', 'console_scripts', 'ducktape')()
>>>>>>>>>>> File
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>>>> line 487, in load_entry_point
>>>>>>>>>>> return get_distribution(dist).load_entry_point(group, name)
>>>>>>>>>>> File
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>>>> line 2728, in load_entry_point
>>>>>>>>>>> return ep.load()
>>>>>>>>>>> File
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>>>> line 2346, in load
>>>>>>>>>>> return self.resolve()
>>>>>>>>>>> File
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>>>> line 2352, in resolve
>>>>>>>>>>> module = __import__(self.module_name, fromlist=['__name__'],
>>>>>>>>>>> level=0)
>>>>>>>>>>> File
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>> 
>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/ducktape-0.7.6-py3.6.egg/ducktape/command_line/main.py",
>>>>>>>>>>> line 127
>>>>>>>>>>> print "parameters are not valid json: " + str(e.message)
>>>>>>>>>>>                                     ^
>>>>>>>>>>> SyntaxError: invalid syntax
>>>>>>>>>>> 
>>>>>>>>>>> I followed the instructions in tests/README.md to setup a
>> cluster of
>>>>>>> 9
>>>>>>>>>>> worker machines. That worked well. When I ran *python setup.py
>>>>>>> develop*
>>>>>>>>>> to
>>>>>>>>>>> install the necessary dependencies (including ducktape), I got
>>>>>>> similar
>>>>>>>>>>> errors to above, but the overall command completed successfully.
>>>>>>>>>>> 
>>>>>>>>>>> Any help appreciated.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks.
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>> 
> 
> -- 
> -- Guozhang


Re: [DISCUSSION] Upgrade system tests to python 3

Posted by Guozhang Wang <wa...@gmail.com>.
Hello Nikolay,

I can help run the test suite once your PR is cleanly rebased to verify the
whole suite works and then I can merge (I'm trusting Ivan and Magnus here
for their reviews :)

Guozhang

On Mon, Sep 14, 2020 at 3:56 AM Nikolay Izhikov <ni...@apache.org> wrote:

> Hello!
>
> I got 2 approvals from Ivan Daschinskiy and Magnus Edenhill.
> Committers, please, join the review.
>
> > 3 сент. 2020 г., в 11:06, Nikolay Izhikov <ni...@gmail.com>
> написал(а):
> >
> > Hello!
> >
> > Just a friendly reminder.
> >
> > Patch to resolve some kind of technical debt - python2 in system tests
> is ready!
> > Can someone, please, take a look?
> >
> > https://github.com/apache/kafka/pull/9196
> >
> >> 28 авг. 2020 г., в 11:19, Nikolay Izhikov <ni...@gmail.com>
> написал(а):
> >>
> >> Hello!
> >>
> >> Any feedback on this?
> >> What I should additionally do to prepare system tests migration?
> >>
> >>> 24 авг. 2020 г., в 11:17, Nikolay Izhikov <ni...@gmail.com>
> написал(а):
> >>>
> >>> Hello.
> >>>
> >>> PR [1] is ready.
> >>> Please, review.
> >>>
> >>> But, I need help with the two following questions:
> >>>
> >>> 1. We need a new release of ducktape which includes fixes [2], [3] for
> python3.
> >>> I created the issue in ducktape repo [4].
> >>> Can someone help me with the release?
> >>>
> >>> 2. I know that some companies run system tests for the trunk on a
> regular bases.
> >>> Can someone show me some results of these runs?
> >>> So, I can compare failures in my PR and in the trunk.
> >>>
> >>> Results [5] of run all for my PR available in the ticket [6]
> >>>
> >>> ```
> >>> SESSION REPORT (ALL TESTS)
> >>> ducktape version: 0.8.0
> >>> session_id:       2020-08-23--002
> >>> run time:         1010 minutes 46.483 seconds
> >>> tests run:        684
> >>> passed:           505
> >>> failed:           9
> >>> ignored:          170
> >>> ```
> >>>
> >>> [1] https://github.com/apache/kafka/pull/9196
> >>> [2]
> https://github.com/confluentinc/ducktape/commit/23bd5ab53802e3a1e1da1ddf3630934f33b02305
> >>> [3]
> https://github.com/confluentinc/ducktape/commit/bfe53712f83b025832d29a43cde3de3d7803106f
> >>> [4] https://github.com/confluentinc/ducktape/issues/245
> >>> [5]
> https://issues.apache.org/jira/secure/attachment/13010366/report.txt
> >>> [6] https://issues.apache.org/jira/browse/KAFKA-10402
> >>>
> >>>> 14 авг. 2020 г., в 21:26, Ismael Juma <is...@juma.me.uk> написал(а):
> >>>>
> >>>> +1
> >>>>
> >>>> On Fri, Aug 14, 2020 at 7:42 AM John Roesler <vv...@apache.org>
> wrote:
> >>>>
> >>>>> Thanks Nikolay,
> >>>>>
> >>>>> No objection. This would be very nice to have.
> >>>>>
> >>>>> Thanks,
> >>>>> John
> >>>>>
> >>>>> On Fri, Aug 14, 2020, at 09:18, Nikolay Izhikov wrote:
> >>>>>> Hello.
> >>>>>>
> >>>>>>> If anyone's interested in porting it to Python 3 it would be a good
> >>>>> change.
> >>>>>>
> >>>>>> I’ve created a ticket [1] to upgrade system tests to python3.
> >>>>>> Does someone have any additional inputs or objections for this
> change?
> >>>>>>
> >>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-10402
> >>>>>>
> >>>>>>
> >>>>>>> 1 июля 2020 г., в 00:26, Gokul Ramanan Subramanian <
> >>>>> gokul2411s@gmail.com> написал(а):
> >>>>>>>
> >>>>>>> Thanks Colin.
> >>>>>>>
> >>>>>>> While at the subject of system tests, there are a few times I see
> tests
> >>>>>>> timed out (even on a large machine such as m5.4xlarge EC2 with
> Linux).
> >>>>> Are
> >>>>>>> there any knobs that system tests provide to control timeouts /
> >>>>> throughputs
> >>>>>>> across all tests?
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>> On Tue, Jun 30, 2020 at 6:32 PM Colin McCabe <cm...@apache.org>
> >>>>> wrote:
> >>>>>>>
> >>>>>>>> Ducktape runs on Python 2.  You can't use it with Python 3, as
> you are
> >>>>>>>> trying to do here.
> >>>>>>>>
> >>>>>>>> If anyone's interested in porting it to Python 3 it would be a
> good
> >>>>> change.
> >>>>>>>>
> >>>>>>>> Otherwise, using docker as suggested here seems to be the best
> way to
> >>>>> go.
> >>>>>>>>
> >>>>>>>> best,
> >>>>>>>> Colin
> >>>>>>>>
> >>>>>>>> On Mon, Jun 29, 2020, at 02:14, Gokul Ramanan Subramanian wrote:
> >>>>>>>>> Hi.
> >>>>>>>>>
> >>>>>>>>> Has anyone had luck running Kafka system tests on a Mac. I have a
> >>>>> MacOS
> >>>>>>>>> Mojave 10.14.6. I got Python 3.6.9 using pyenv. However, the
> command
> >>>>>>>>> *ducktape tests/kafkatest/tests* yields the following error,
> making
> >>>>> it
> >>>>>>>> look
> >>>>>>>>> like some Python incompatibility issue.
> >>>>>>>>>
> >>>>>>>>> $ ducktape tests/kafkatest/tests
> >>>>>>>>> Traceback (most recent call last):
> >>>>>>>>> File "/Users/gokusubr/.pyenv/versions/3.6.9/bin/ducktape", line
> 11,
> >>>>> in
> >>>>>>>>> <module>
> >>>>>>>>> load_entry_point('ducktape', 'console_scripts', 'ducktape')()
> >>>>>>>>> File
> >>>>>>>>>
> >>>>>>>>
> >>>>>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> >>>>>>>>> line 487, in load_entry_point
> >>>>>>>>> return get_distribution(dist).load_entry_point(group, name)
> >>>>>>>>> File
> >>>>>>>>>
> >>>>>>>>
> >>>>>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> >>>>>>>>> line 2728, in load_entry_point
> >>>>>>>>> return ep.load()
> >>>>>>>>> File
> >>>>>>>>>
> >>>>>>>>
> >>>>>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> >>>>>>>>> line 2346, in load
> >>>>>>>>> return self.resolve()
> >>>>>>>>> File
> >>>>>>>>>
> >>>>>>>>
> >>>>>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
> >>>>>>>>> line 2352, in resolve
> >>>>>>>>> module = __import__(self.module_name, fromlist=['__name__'],
> >>>>>>>>> level=0)
> >>>>>>>>> File
> >>>>>>>>>
> >>>>>>>>
> >>>>>
> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/ducktape-0.7.6-py3.6.egg/ducktape/command_line/main.py",
> >>>>>>>>> line 127
> >>>>>>>>> print "parameters are not valid json: " + str(e.message)
> >>>>>>>>>                                      ^
> >>>>>>>>> SyntaxError: invalid syntax
> >>>>>>>>>
> >>>>>>>>> I followed the instructions in tests/README.md to setup a
> cluster of
> >>>>> 9
> >>>>>>>>> worker machines. That worked well. When I ran *python setup.py
> >>>>> develop*
> >>>>>>>> to
> >>>>>>>>> install the necessary dependencies (including ducktape), I got
> >>>>> similar
> >>>>>>>>> errors to above, but the overall command completed successfully.
> >>>>>>>>>
> >>>>>>>>> Any help appreciated.
> >>>>>>>>>
> >>>>>>>>> Thanks.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> >
>
>

-- 
-- Guozhang

Re: [DISCUSSION] Upgrade system tests to python 3

Posted by Nikolay Izhikov <ni...@apache.org>.
Hello!

I got 2 approvals from Ivan Daschinskiy and Magnus Edenhill.
Committers, please, join the review.

> 3 сент. 2020 г., в 11:06, Nikolay Izhikov <ni...@gmail.com> написал(а):
> 
> Hello! 
> 
> Just a friendly reminder.
> 
> Patch to resolve some kind of technical debt - python2 in system tests is ready!
> Can someone, please, take a look?
> 
> https://github.com/apache/kafka/pull/9196
> 
>> 28 авг. 2020 г., в 11:19, Nikolay Izhikov <ni...@gmail.com> написал(а):
>> 
>> Hello!
>> 
>> Any feedback on this?
>> What I should additionally do to prepare system tests migration?
>> 
>>> 24 авг. 2020 г., в 11:17, Nikolay Izhikov <ni...@gmail.com> написал(а):
>>> 
>>> Hello.
>>> 
>>> PR [1] is ready.
>>> Please, review.
>>> 
>>> But, I need help with the two following questions:
>>> 
>>> 1. We need a new release of ducktape which includes fixes [2], [3] for python3.
>>> I created the issue in ducktape repo [4].
>>> Can someone help me with the release?
>>> 
>>> 2. I know that some companies run system tests for the trunk on a regular bases.
>>> Can someone show me some results of these runs?
>>> So, I can compare failures in my PR and in the trunk.
>>> 
>>> Results [5] of run all for my PR available in the ticket [6]
>>> 
>>> ```
>>> SESSION REPORT (ALL TESTS)
>>> ducktape version: 0.8.0
>>> session_id:       2020-08-23--002
>>> run time:         1010 minutes 46.483 seconds
>>> tests run:        684
>>> passed:           505
>>> failed:           9
>>> ignored:          170
>>> ```
>>> 
>>> [1] https://github.com/apache/kafka/pull/9196
>>> [2] https://github.com/confluentinc/ducktape/commit/23bd5ab53802e3a1e1da1ddf3630934f33b02305
>>> [3] https://github.com/confluentinc/ducktape/commit/bfe53712f83b025832d29a43cde3de3d7803106f
>>> [4] https://github.com/confluentinc/ducktape/issues/245
>>> [5] https://issues.apache.org/jira/secure/attachment/13010366/report.txt
>>> [6] https://issues.apache.org/jira/browse/KAFKA-10402
>>> 
>>>> 14 авг. 2020 г., в 21:26, Ismael Juma <is...@juma.me.uk> написал(а):
>>>> 
>>>> +1
>>>> 
>>>> On Fri, Aug 14, 2020 at 7:42 AM John Roesler <vv...@apache.org> wrote:
>>>> 
>>>>> Thanks Nikolay,
>>>>> 
>>>>> No objection. This would be very nice to have.
>>>>> 
>>>>> Thanks,
>>>>> John
>>>>> 
>>>>> On Fri, Aug 14, 2020, at 09:18, Nikolay Izhikov wrote:
>>>>>> Hello.
>>>>>> 
>>>>>>> If anyone's interested in porting it to Python 3 it would be a good
>>>>> change.
>>>>>> 
>>>>>> I’ve created a ticket [1] to upgrade system tests to python3.
>>>>>> Does someone have any additional inputs or objections for this change?
>>>>>> 
>>>>>> [1] https://issues.apache.org/jira/browse/KAFKA-10402
>>>>>> 
>>>>>> 
>>>>>>> 1 июля 2020 г., в 00:26, Gokul Ramanan Subramanian <
>>>>> gokul2411s@gmail.com> написал(а):
>>>>>>> 
>>>>>>> Thanks Colin.
>>>>>>> 
>>>>>>> While at the subject of system tests, there are a few times I see tests
>>>>>>> timed out (even on a large machine such as m5.4xlarge EC2 with Linux).
>>>>> Are
>>>>>>> there any knobs that system tests provide to control timeouts /
>>>>> throughputs
>>>>>>> across all tests?
>>>>>>> Thanks.
>>>>>>> 
>>>>>>> On Tue, Jun 30, 2020 at 6:32 PM Colin McCabe <cm...@apache.org>
>>>>> wrote:
>>>>>>> 
>>>>>>>> Ducktape runs on Python 2.  You can't use it with Python 3, as you are
>>>>>>>> trying to do here.
>>>>>>>> 
>>>>>>>> If anyone's interested in porting it to Python 3 it would be a good
>>>>> change.
>>>>>>>> 
>>>>>>>> Otherwise, using docker as suggested here seems to be the best way to
>>>>> go.
>>>>>>>> 
>>>>>>>> best,
>>>>>>>> Colin
>>>>>>>> 
>>>>>>>> On Mon, Jun 29, 2020, at 02:14, Gokul Ramanan Subramanian wrote:
>>>>>>>>> Hi.
>>>>>>>>> 
>>>>>>>>> Has anyone had luck running Kafka system tests on a Mac. I have a
>>>>> MacOS
>>>>>>>>> Mojave 10.14.6. I got Python 3.6.9 using pyenv. However, the command
>>>>>>>>> *ducktape tests/kafkatest/tests* yields the following error, making
>>>>> it
>>>>>>>> look
>>>>>>>>> like some Python incompatibility issue.
>>>>>>>>> 
>>>>>>>>> $ ducktape tests/kafkatest/tests
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>> File "/Users/gokusubr/.pyenv/versions/3.6.9/bin/ducktape", line 11,
>>>>> in
>>>>>>>>> <module>
>>>>>>>>> load_entry_point('ducktape', 'console_scripts', 'ducktape')()
>>>>>>>>> File
>>>>>>>>> 
>>>>>>>> 
>>>>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>> line 487, in load_entry_point
>>>>>>>>> return get_distribution(dist).load_entry_point(group, name)
>>>>>>>>> File
>>>>>>>>> 
>>>>>>>> 
>>>>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>> line 2728, in load_entry_point
>>>>>>>>> return ep.load()
>>>>>>>>> File
>>>>>>>>> 
>>>>>>>> 
>>>>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>> line 2346, in load
>>>>>>>>> return self.resolve()
>>>>>>>>> File
>>>>>>>>> 
>>>>>>>> 
>>>>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/pkg_resources/__init__.py",
>>>>>>>>> line 2352, in resolve
>>>>>>>>> module = __import__(self.module_name, fromlist=['__name__'],
>>>>>>>>> level=0)
>>>>>>>>> File
>>>>>>>>> 
>>>>>>>> 
>>>>> "/Users/gokusubr/.pyenv/versions/3.6.9/lib/python3.6/site-packages/ducktape-0.7.6-py3.6.egg/ducktape/command_line/main.py",
>>>>>>>>> line 127
>>>>>>>>> print "parameters are not valid json: " + str(e.message)
>>>>>>>>>                                      ^
>>>>>>>>> SyntaxError: invalid syntax
>>>>>>>>> 
>>>>>>>>> I followed the instructions in tests/README.md to setup a cluster of
>>>>> 9
>>>>>>>>> worker machines. That worked well. When I ran *python setup.py
>>>>> develop*
>>>>>>>> to
>>>>>>>>> install the necessary dependencies (including ducktape), I got
>>>>> similar
>>>>>>>>> errors to above, but the overall command completed successfully.
>>>>>>>>> 
>>>>>>>>> Any help appreciated.
>>>>>>>>> 
>>>>>>>>> Thanks.
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> 
>