[I] [improve][kafka-connect-adaptor] An unproductive source is indistinguishable from an idle one [pulsar-connectors]

["david-streamlio (via GitHub)" <gi...@apache.org>]

david-streamlio opened a new issue, #125:
URL: https://github.com/apache/pulsar-connectors/issues/125

   
   ## What happens
   
   A source built on `kafka-connect-adaptor` whose wrapped Kafka Connect task can never
   produce a record — wrong host, unreachable database, bad credentials — reports itself as
   perfectly healthy for as long as it runs. `pulsar-admin sources status` shows:
   
   ```json
   {"running": true, "error": "", "numRestarts": 0,
    "numReceivedFromSource": 0, "lastReceivedTime": 0}
   ```
   
   There is no restart, no exception, and no error string. Operationally this is
   indistinguishable from a correctly configured source that simply has no traffic yet.
   
   ## Why
   
   `AbstractKafkaConnectSource.read()` discards the fact that a poll produced nothing:
   
   ```java
   public synchronized Record<T> read() throws Exception {
       while (true) {
           if (currentBatch == null) {
               List<SourceRecord> recordList = sourceTask.poll();
               if (recordList == null || recordList.isEmpty()) {
                   continue;                       // <-- nothing counted, timed or logged
               }
   ```
   
   Nothing counts consecutive empty polls or timestamps the last non-empty one, so there is
   no state for the runtime to surface. `lastReceivedTime` stays `0`, which is the same value
   a genuinely idle source reports, so it cannot be used to tell the two apart.
   
   ## Evidence
   
   Reproduced deterministically on Pulsar 3.3.7 against a Debezium-based source pointed at an
   unreachable database, sampling `sources status` every 10 s for 60 s. Two failure shapes were
   tested and behave identically:
   
   - **blackholed** — `192.0.2.10` (RFC 5737 TEST-NET-1), packets discarded, connects hang
   - **refused** — `127.0.0.1:7100`, immediate `ECONNREFUSED`
   
   Both held `running: true` / `error: ""` / `numRestarts: 0` / `numReceivedFromSource: 0` for
   the entire window. The instance log over the same period contained 37 INFO, 10 WARN and
   **zero ERROR** lines. Startup ran cleanly all the way through `KafkaConnectSource`,
   `PulsarKafkaWorkerConfig` and `PulsarOffsetBackingStore` before stalling, so every signal
   the adaptor emits says the source is fine.
   
   ## Scope
   
   This is not specific to one connector. Any source using the adaptor inherits it, including
   the Debezium MySQL/Postgres connectors in this repo — a Debezium source with a wrong
   `database.hostname` should reproduce it.
   
   To be explicit about what is *not* being reported here: the underlying client retrying
   forever without escalating is the connector's own behaviour, not the adaptor's, and stock
   Kafka Connect would also report such a task as RUNNING. The gap being raised is that the
   adaptor holds the information needed to distinguish "has never produced anything" from
   "idle" and currently throws it away.
   
   ## Suggested direction
   
   Something cheap and non-behavioural would be enough:
   
   - track consecutive empty polls and the time of the last non-empty poll in
     `AbstractKafkaConnectSource`
   - expose them via the existing source metrics, so `sources status` can distinguish
     "never produced a record since start" from "idle"
   - optionally log at a decaying interval once the empty-poll streak crosses a threshold
   
   Happy to put up a PR if the direction seems right.
   
   ## Environment
   
   - Pulsar 3.3.7 (`kafka-connect-adaptor`); the `read()` loop above is unchanged on master
   - JDK 17, source deployed as a NAR to a Pulsar worker
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [improve][kafka-connect-adaptor] An unproductive source is indistinguishable from an idle one [pulsar-connectors]

["david-streamlio (via GitHub)" <gi...@apache.org>]

david-streamlio commented on issue #125:
URL: https://github.com/apache/pulsar-connectors/issues/125#issuecomment-5169281990

   For cross-reference: the connector-side half of the case that prompted this is filed at https://github.com/yugabyte/debezium-connector-yugabytedb/issues/416 — a client that retries an unreachable endpoint indefinitely at WARN without ever failing the task.
   
   The two are independent. That report is about one connector's retry policy; this one is about the adaptor discarding the empty-poll information regardless of why the polls are empty. A connector that fails loudly would still be reported as `running: true` with `numReceivedFromSource: 0` here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org