[jira] [Commented] (BEAM-11102) Unable to use KafkaIO with SSL via Python API

["Michael Wizner (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-11102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17293802#comment-17293802 ] 

Michael Wizner commented on BEAM-11102:
---------------------------------------

Hey, thanks everyone for chipping in.
{quote}one option might be to predefine a Java function for this purpose in cross-language builder and configure it appropriately through parameters received from Python side (for example, the location of the local credentials file). Will this work ?
{quote}
Yeah, that sounds reasonable to me (y)
[~chamikara], are there any examples in the codebase of something like that done for other cross-language transforms? (Is that a well established pattern?) I'm not much familiar with the internals, but perhaps if someone could point me in the right direction I might have a go at this one day when I feel brave enough :)
{quote}Alternatively props python factory function executed on a worker might do the job nicely.
{quote}
Hey Maxim, from my understanding that wouldn't be enough, because the Python API is just a wrapper around the external Java transform, so without the ability to pass the config to it you'd not be able to configure the Java consumer/producer correctly.

> Unable to use KafkaIO with SSL via Python API
> ---------------------------------------------
>
>                 Key: BEAM-11102
>                 URL: https://issues.apache.org/jira/browse/BEAM-11102
>             Project: Beam
>          Issue Type: Improvement
>          Components: cross-language, io-py-kafka
>            Reporter: Michael Wizner
>            Priority: P2
>
> I've come across the same issue described in BEAM-9627 while trying to use KafkaIO with SSL via the Python API.
> While the solution in the Java API is to use withConsumerFactoryFn or withProducerFactoryFn and provide a custom function to download or generate key/trust stores before the consumer or producer gets created, there is no such functionality exposed via in the Python API since it's just a wrapper around the external transform.
> Is there currently any other way to work around this issue in Python or would it be possible to expose a similar functionality in the API? (I'm using the Dataflow runner, but hopefully it shouldn't matter.)
> As far as my understanding goes, alternatively one could write an external transform that's a thin wrapper around the Java's KafkaIO API that handles key/trust stores under the hood and make it available via an external expansion service, but that's definitely way less ideal than being able to achieve the same thing via the built-in API.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)