You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "david-streamlio (via GitHub)" <gi...@apache.org> on 2026/07/15 13:25:16 UTC

[I] [improve][test] Add integration tests for the Kafka Connect Adaptor connector [pulsar-connectors]

david-streamlio opened a new issue, #111:
URL: https://github.com/apache/pulsar-connectors/issues/111

   ## Current state
   
   `kafka-connect-adaptor` has substantial unit-test coverage (`KafkaConnectSinkTest`, `KafkaConnectSourceTest`, offset/schema converters), but these drive the adaptor with file-stream/mock Kafka Connect tasks. There is no integration test against a real Kafka broker.
   
   ## Proposal
   
   Add a Testcontainers-based integration test using a Kafka broker container that exercises the adaptor with a real Kafka Connect connector (e.g. the file or a simple JDBC connector) end-to-end: `KafkaConnectSource` reading from Kafka into Pulsar, and `KafkaConnectSink` writing from Pulsar into Kafka, asserting records and offsets round-trip correctly.
   
   Related: #29 (converter configs ignored), #15 (back-pressure/OOM) — an integration harness would give both a place to reproduce.
   
   _Part of an integration-test coverage sweep of all connector modules._
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [improve][test] Add integration tests for the Kafka Connect Adaptor connector [pulsar-connectors]

Posted by "david-streamlio (via GitHub)" <gi...@apache.org>.
david-streamlio closed issue #111: [improve][test] Add integration tests for the Kafka Connect Adaptor connector
URL: https://github.com/apache/pulsar-connectors/issues/111


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [improve][test] Add integration tests for the Kafka Connect Adaptor connector [pulsar-connectors]

Posted by "david-streamlio (via GitHub)" <gi...@apache.org>.
david-streamlio commented on issue #111:
URL: https://github.com/apache/pulsar-connectors/issues/111#issuecomment-4981328116

   Closing: this connector already has integration coverage, and the premise of this issue was mistaken.
   
   Both `KafkaConnectSourceTest` and `KafkaConnectSinkTest` extend `ProducerConsumerBase`, which stands up a real in-process Pulsar broker (real `PulsarClient`, producers/consumers), and run real Kafka Connect tasks (`FileStreamSourceConnector`/`FileStreamSourceTask`) end-to-end through the adaptor.
   
   Also, the adaptor is designed to run Kafka Connect connectors *inside* Pulsar without a Kafka broker, so 'add a Testcontainers Kafka broker' was the wrong integration surface — a live Pulsar broker running real Connect tasks (which already exists) is the correct one.
   
   The linked concerns #29 (converter configs) and #15 (back-pressure) remain valid on their own.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [improve][test] Add integration tests for the Kafka Connect Adaptor connector [pulsar-connectors]

Posted by "david-streamlio (via GitHub)" <gi...@apache.org>.
david-streamlio commented on issue #111:
URL: https://github.com/apache/pulsar-connectors/issues/111#issuecomment-4981543282

   Correction to my earlier closing comment: I said this adaptor 'already has integration coverage,' but that overstated the *running* state.
   
   The comprehensive `ProducerConsumerBase`-based sink/source/offset tests exist in source, but 5 of the 7 test files are **excluded from compilation** in `build.gradle.kts` (`brokerDependentTestSources`, added in #31) because they depend on Pulsar broker internals not present in the pinned `pulsar = 4.2.0`. So they don't run in CI today.
   
   A JaCoCo run of the tests that *do* execute shows **8.7% line coverage**, with `KafkaConnectSink` and `KafkaConnectSource` at **0%**.
   
   This doesn't reopen the original 'add a Kafka broker' premise (still not the right integration surface — the adaptor runs Connect tasks against Pulsar, not Kafka). The real, actionable item is **re-enabling** the already-written tests, now tracked in #118. Leaving this closed; #118 supersedes it for the coverage concern.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org