You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@pekko.apache.org by "pjfanning (via GitHub)" <gi...@apache.org> on 2026/09/06 23:05:51 UTC

[PR] create the Kafka metadata client when the source is materialized [pekko-projection]

pjfanning opened a new pull request, #621:
URL: https://github.com/apache/pekko-projection/pull/621

   ### Motivation
   `KafkaSourceProviderImpl.source` creates the `MetadataClientAdapter` - and with it a `KafkaConsumerActor` and its Kafka consumer - before returning the source:
   
   ```scala
   val metadataClient = metadataClientFactory()
   val numPartitionsF = metadataClient.numPartitions(topics)
   numPartitionsF.failed.foreach(_ => metadataClient.stop())
   numPartitionsF.map { numPartitions =>
     _source(readOffsets, numPartitions, metadataClient)
       ...
       .mapMaterializedValue { terminated => terminated.onComplete(_ => metadataClient.stop()); NotUsed }
   }
   ```
   
   It is stopped only when `numPartitions` fails, or when the materialized stream terminates. If the returned source is never materialized - for example when the projection is stopped between `source()` completing and the stream being run, which the restart backoff makes possible on every restart - the client and its consumer actor are never stopped.
   
   ### Modification
   Create the metadata client inside `Source.lazyFutureSource`, so it is only created when the source is materialized and demand arrives, and is then always stopped by the existing `watchTermination` callback. The `numPartitions` lookup (used for the `flatMapMerge` breadth) moves with it, so a failure now fails the stream instead of the returned future; both end up failing the projection stream, which the restart backoff handles.
   
   ### Result
   No orphaned metadata client, and no orphaned Kafka consumer actor, for a source that is never materialized.
   
   ### Tests
   - New `KafkaSourceProviderImplSpec` case: asserts no metadata client is created for a source that is not materialized, and that one is created and then stopped when the stream is run. It fails before the fix (the client is created eagerly).
   - `sbt "kafka/test"` - 2 passed
   - `sbt "kafka/mimaReportBinaryIssues"` - success
   - `sbt checkCodeStyle` - success
   - The `kafka-test` integration specs were not run locally (they need Kafka testcontainers); they exercise this path via `Source.futureSource(provider.source(...))` and are covered by the CI Kafka integration jobs.
   
   ### References
   None - found during a resource-leak audit of the main sources


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] create the Kafka metadata client when the source is materialized [pekko-projection]

Posted by "Philippus (via GitHub)" <gi...@apache.org>.
Philippus commented on code in PR #621:
URL: https://github.com/apache/pekko-projection/pull/621#discussion_r3946569249


##########
kafka/src/main/scala/org/apache/pekko/projection/kafka/internal/KafkaSourceProviderImpl.scala:
##########
@@ -94,23 +94,30 @@ import org.apache.kafka.common.record.TimestampType
         })
 
   override def source(readOffsets: ReadOffsets): Future[Source[ConsumerRecord[K, V], NotUsed]] = {
-    // get the total number of partitions to configure the `breadth` parameter, or we could just use a really large
-    // number.  i don't think using a large number would present a problem.
-    val metadataClient = metadataClientFactory()
-    val numPartitionsF = metadataClient.numPartitions(topics)
-    numPartitionsF.failed.foreach(_ => metadataClient.stop())
-    numPartitionsF.map { numPartitions =>
-      _source(readOffsets, numPartitions, metadataClient)
-        .mapMaterializedValue { m =>
-          control = Some(m)
-          m
+    // The metadata client is created when the source is materialized, and stopped when the stream terminates.
+    // Creating it eagerly here would leak the client, and its Kafka consumer actor, when the returned source is
+    // never materialized, such as when the projection is stopped while it is starting up.
+    val source =
+      Source.lazyFutureSource[ConsumerRecord[K, V], NotUsed] { () =>
+        // get the total number of partitions to configure the `breadth` parameter, or we could just use a really large
+        // number.  i don't think using a large number would present a problem.

Review Comment:
   ```suggestion
           // number. I don't think using a large number would present a problem.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] create the Kafka metadata client when the source is materialized [pekko-projection]

Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #621:
URL: https://github.com/apache/pekko-projection/pull/621#issuecomment-6437193270

   @Philippus would you have time to re-review this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] create the Kafka metadata client when the source is materialized [pekko-projection]

Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #621:
URL: https://github.com/apache/pekko-projection/pull/621#issuecomment-5566748284

   Thanks @Philippus, I merged your suggestion.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [PR] create the Kafka metadata client when the source is materialized [pekko-projection]

Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning merged PR #621:
URL: https://github.com/apache/pekko-projection/pull/621


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org