[jira] [Updated] (KAFKA-16578) Revert changes to connect_distributed_test.py for the new async Consumer

["Kirk True (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-16578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kirk True updated KAFKA-16578:
------------------------------
    Description: 
To test the new, asynchronous Kafka {{Consumer}} implementation, we migrated a slew of system tests to run both the "old" and "new" implementations. KAFKA-16272 updated the system tests in {{connect_distributed_test.py}} so it could test the new consumer with Connect. However, we are not supporting Connect with the new consumer in 3.8.0. Unsurprisingly, when we run the Connect system tests with the new {{AsyncKafkaConsumer}}, we get errors like the following:

{code}
test_id:    kafkatest.tests.connect.connect_distributed_test.ConnectDistributedTest.test_exactly_once_source.clean=False.connect_protocol=eager.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=True.group_protocol=consumer
status:     FAIL
run time:   6 minutes 3.899 seconds


    InsufficientResourcesError('Not enough nodes available to allocate. linux nodes requested: 1. linux nodes available: 0')
Traceback (most recent call last):
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/connect/connect_distributed_test.py", line 919, in test_exactly_once_source
    consumer_validator = ConsoleConsumer(self.test_context, 1, self.kafka, self.source.topic, consumer_timeout_ms=1000, print_key=True)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/services/console_consumer.py", line 97, in __init__
    BackgroundThreadService.__init__(self, context, num_nodes)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/background_thread.py", line 26, in __init__
    super(BackgroundThreadService, self).__init__(context, num_nodes, cluster_spec, *args, **kwargs)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/service.py", line 107, in __init__
    self.allocate_nodes()
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/service.py", line 217, in allocate_nodes
    self.nodes = self.cluster.alloc(self.cluster_spec)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/cluster.py", line 54, in alloc
    allocated = self.do_alloc(cluster_spec)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/finite_subcluster.py", line 31, in do_alloc
    allocated = self._available_nodes.remove_spec(cluster_spec)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/node_container.py", line 117, in remove_spec
    raise InsufficientResourcesError("Not enough nodes available to allocate. " + msg)
ducktape.cluster.node_container.InsufficientResourcesError: Not enough nodes available to allocate. linux nodes requested: 1. linux nodes available: 0
{code}

The task here is to revert the changes made in KAFKA-16272 [PR 15576|https://github.com/apache/kafka/pull/15576].

  was:
To test the new, asynchronous Kafka {{Consumer}} implementation, we migrated a slew of system tests to run both the "old" and "new" implementations. KAFKA-16272 updated the system tests in {{connect_distributed_test.py}} so it could test the new consumer with Connect. However, we are not supporting Connect with the new consumer in 3.8.0. Unsurprisingly, when we run the Connect system tests with the new {{{}AsyncKafkaConsumer{}}}, we get errors like the following:
{code:java}
test_id:    kafkatest.tests.connect.connect_distributed_test.ConnectDistributedTest.test_exactly_once_source.clean=False.connect_protocol=eager.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=True.group_protocol=consumer
status:     FAIL
run time:   6 minutes 3.899 seconds


    InsufficientResourcesError('Not enough nodes available to allocate. linux nodes requested: 1. linux nodes available: 0')
Traceback (most recent call last):
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
    return self.test_context.function(self.test)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/connect/connect_distributed_test.py", line 919, in test_exactly_once_source
    consumer_validator = ConsoleConsumer(self.test_context, 1, self.kafka, self.source.topic, consumer_timeout_ms=1000, print_key=True)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/services/console_consumer.py", line 97, in __init__
    BackgroundThreadService.__init__(self, context, num_nodes)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/background_thread.py", line 26, in __init__
    super(BackgroundThreadService, self).__init__(context, num_nodes, cluster_spec, *args, **kwargs)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/service.py", line 107, in __init__
    self.allocate_nodes()
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/service.py", line 217, in allocate_nodes
    self.nodes = self.cluster.alloc(self.cluster_spec)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/cluster.py", line 54, in alloc
    allocated = self.do_alloc(cluster_spec)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/finite_subcluster.py", line 31, in do_alloc
    allocated = self._available_nodes.remove_spec(cluster_spec)
  File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/node_container.py", line 117, in remove_spec
    raise InsufficientResourcesError("Not enough nodes available to allocate. " + msg)
ducktape.cluster.node_container.InsufficientResourcesError: Not enough nodes available to allocate. linux nodes requested: 1. linux nodes available: 0
{code}
The task here is to revert the changes made in KAFKA-16272 [PR 15576|https://github.com/apache/kafka/pull/15576].


> Revert changes to connect_distributed_test.py for the new async Consumer
> ------------------------------------------------------------------------
>
>                 Key: KAFKA-16578
>                 URL: https://issues.apache.org/jira/browse/KAFKA-16578
>             Project: Kafka
>          Issue Type: Task
>          Components: clients, consumer, system tests
>    Affects Versions: 3.8.0
>            Reporter: Kirk True
>            Assignee: Kirk True
>            Priority: Critical
>              Labels: kip-848-client-support, system-tests
>             Fix For: 3.8.0
>
>
> To test the new, asynchronous Kafka {{Consumer}} implementation, we migrated a slew of system tests to run both the "old" and "new" implementations. KAFKA-16272 updated the system tests in {{connect_distributed_test.py}} so it could test the new consumer with Connect. However, we are not supporting Connect with the new consumer in 3.8.0. Unsurprisingly, when we run the Connect system tests with the new {{AsyncKafkaConsumer}}, we get errors like the following:
> {code}
> test_id:    kafkatest.tests.connect.connect_distributed_test.ConnectDistributedTest.test_exactly_once_source.clean=False.connect_protocol=eager.metadata_quorum=ISOLATED_KRAFT.use_new_coordinator=True.group_protocol=consumer
> status:     FAIL
> run time:   6 minutes 3.899 seconds
>     InsufficientResourcesError('Not enough nodes available to allocate. linux nodes requested: 1. linux nodes available: 0')
> Traceback (most recent call last):
>   File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
>     data = self.run_test()
>   File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/tests/runner_client.py", line 262, in run_test
>     return self.test_context.function(self.test)
>   File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/mark/_mark.py", line 433, in wrapper
>     return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/tests/connect/connect_distributed_test.py", line 919, in test_exactly_once_source
>     consumer_validator = ConsoleConsumer(self.test_context, 1, self.kafka, self.source.topic, consumer_timeout_ms=1000, print_key=True)
>   File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/tests/kafkatest/services/console_consumer.py", line 97, in __init__
>     BackgroundThreadService.__init__(self, context, num_nodes)
>   File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/background_thread.py", line 26, in __init__
>     super(BackgroundThreadService, self).__init__(context, num_nodes, cluster_spec, *args, **kwargs)
>   File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/service.py", line 107, in __init__
>     self.allocate_nodes()
>   File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/services/service.py", line 217, in allocate_nodes
>     self.nodes = self.cluster.alloc(self.cluster_spec)
>   File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/cluster.py", line 54, in alloc
>     allocated = self.do_alloc(cluster_spec)
>   File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/finite_subcluster.py", line 31, in do_alloc
>     allocated = self._available_nodes.remove_spec(cluster_spec)
>   File "/home/jenkins/workspace/system-test-kafka-branch-builder/kafka/venv/lib/python3.7/site-packages/ducktape/cluster/node_container.py", line 117, in remove_spec
>     raise InsufficientResourcesError("Not enough nodes available to allocate. " + msg)
> ducktape.cluster.node_container.InsufficientResourcesError: Not enough nodes available to allocate. linux nodes requested: 1. linux nodes available: 0
> {code}
> The task here is to revert the changes made in KAFKA-16272 [PR 15576|https://github.com/apache/kafka/pull/15576].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)