You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Paul Nimusiima <pa...@gmail.com> on 2021/02/02 11:26:04 UTC
Using the Beam Python SDK and PortableRunner with Flink to connect to
Kafka with SSL
Hello Beam community,
I am wondering whether it is possible to connect to a secure Kafka broker
with the python beam SDK. So for example, in the code below, how would I
make the "ssl.truststore.location" and "ssl.keystore.location" accessible
inside the Java SDK harness which runs the code.
ReadFromKafka(
consumer_config={
"bootstrap.servers": "bootstrap-server:17032",
"security.protocol": "SSL",
"ssl.truststore.location":
"/opt/keys/client.truststore.jks", # how do I make this available to
the Java SDK harness
"ssl.truststore.password": "password",
"ssl.keystore.type": "PKCS12",
"ssl.keystore.location": "/opt/keys/client.keystore.p12",
# how do I make this available to the Java SDK harness
"ssl.keystore.password": "password",
"group.id": "group",
"basic.auth.credentials.source": "USER_INFO",
"schema.registry.basic.auth.user.info": "user:password"
},
topics=["topic"],
max_num_records=2,
# expansion_service="localhost:56938"
)
I tried building a custom image that has pulls credentials into the
container based on "apache/beam_java11_sdk:2.27.0" and using the
"--sdk_harness_container_image_overrides=.*java.*,{image}:{tag}"
argument, but it does not appear to be used when the pipeline runs on Flink.
Thank you for the great tool and any help or advice is greatly appreciated.
--
paul
Re: Using the Beam Python SDK and PortableRunner with Flink to
connect to Kafka with SSL
Posted by Paul Nimusiima <pa...@gmail.com>.
Thanks for the information Chamikara
On Wed, Feb 3, 2021 at 12:04 AM Chamikara Jayalath <ch...@google.com>
wrote:
> Currently 'sdk_harness_container_image_overrides' is Dataflow only.
>
> Also I think options might be getting dropped due to
> https://issues.apache.org/jira/browse/BEAM-9449.
>
> Unfortunately, till one of the above is fixed I think you might have to
> update the code at the following location to override the container and
> rebuild the SDK.
>
> https://github.com/apache/beam/blob/0078bb35ba4aef9410d681d8b4e2c16d9f56433d/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java#L397
>
> Thanks,
> Cham
>
> On Tue, Feb 2, 2021 at 2:54 PM Paul Nimusiima <pa...@gmail.com>
> wrote:
>
>> Those two arguments appear to be discarded
>> WARNING:apache_beam.options.pipeline_options:Discarding unparseable args:
>> ['--default_environment_type=DOCKER',
>> '--default_environment_config=beam_java_sdk:0.0.1']
>> and the default apache/beam_java11_sdk:2.27.0 is used.
>>
>> INFO
>> org.apache.beam.runners.fnexecution.environment.DockerEnvironmentFactory -
>> Still waiting for startup of environment apache/beam_java11_sdk:2.27.0 for
>> worker id 1-2
>>
>> and when the when the harness is being removed
>>
>> INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory
>> - Closing environment urn: "beam:env:docker:v1"
>> payload: "\n\035apache/beam_java11_sdk:2.27.0"
>> capabilities: "beam:coder:bytes:v1"
>> ...
>> capabilities: "beam:transform:sdf_truncate_sized_restrictions:v1"
>> dependencies {
>> type_urn: "beam:artifact:type:file:v1"
>> type_payload:
>> "\n\354\001/var/folders/yj/njlsyhss5sb7t36fklhxjbvc0000gp/T/beam-artifact-staging/11d9c289a5c41d77ecf6d4eff0cc95928c241ec416c8e88ea67fbc931463a0c5/1-external_1beam:env:docker-beam-sdks-java-io-expansion-service-2.27.0-OrWYGnEG6c2yQRgdLP3AI3hLQ_1lT"
>> role_urn: "beam:artifact:role:staging_to:v1"
>> role_payload:
>> "\nZbeam-sdks-java-io-expansion-service-2.27.0-OrWYGnEG6c2yQRgdLP3AI3hLQ_1lTDuSw_K_Lc45B5c.jar"
>> }
>>
>> On Tue, Feb 2, 2021 at 10:14 PM Kyle Weaver <kc...@google.com> wrote:
>>
>>> > Regarding Kyle Weaver's suggestion, I tried a combination of
>>> "--environment_type=DOCKER" and
>>> '--environment_config=beam_python_sdk:0.0.1' where beam_python_sdk:0.0.1 is
>>> an image I built. That made it possible to override the python sdk harness,
>>> but I was wondering if there is something similar for overriding the Java
>>> sdk harness?
>>>
>>> The Java options are named differently, despite appearing to have
>>> identical semantics. Can you try default_environment_type and
>>> default_environment_config?
>>>
>>> On Tue, Feb 2, 2021 at 2:04 PM Paul Nimusiima <pa...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>> Thanks for the responses.
>>>> Regarding Kyle Weaver's suggestion, I tried a combination of
>>>> "--environment_type=DOCKER" and
>>>> '--environment_config=beam_python_sdk:0.0.1' where
>>>> beam_python_sdk:0.0.1 is an image I built. That made it possible to
>>>> override the python sdk harness, but I was wondering if there is something
>>>> similar for overriding the Java sdk harness?
>>>>
>>>> Thanks again.
>>>>
>>>> On Tue, Feb 2, 2021 at 9:08 PM Kyle Weaver <kc...@google.com> wrote:
>>>>
>>>>> AFAIK sdk_harness_container_image_overrides only works for the
>>>>> Dataflow runner. For other runners I think you will have to change the
>>>>> default environment config:
>>>>> https://github.com/apache/beam/blob/0078bb35ba4aef9410d681d8b4e2c16d9f56433d/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L53-L71
>>>>>
>>>>> On Tue, Feb 2, 2021 at 11:56 AM Brian Hulette <bh...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Hm I would expect that to work. Can you tell what container Flink is
>>>>>> using if it's not using the one you specified? +Chamikara Jayalath
>>>>>> <ch...@google.com> may have some insight here
>>>>>>
>>>>>> Brian
>>>>>>
>>>>>> On Tue, Feb 2, 2021 at 3:27 AM Paul Nimusiima <pa...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello Beam community,
>>>>>>>
>>>>>>> I am wondering whether it is possible to connect to a secure Kafka
>>>>>>> broker with the python beam SDK. So for example, in the code below, how
>>>>>>> would I make the "ssl.truststore.location" and "ssl.keystore.location"
>>>>>>> accessible inside the Java SDK harness which runs the code.
>>>>>>>
>>>>>>> ReadFromKafka(
>>>>>>> consumer_config={
>>>>>>> "bootstrap.servers": "bootstrap-server:17032",
>>>>>>> "security.protocol": "SSL",
>>>>>>> "ssl.truststore.location": "/opt/keys/client.truststore.jks", # how do I make this available to the Java SDK harness
>>>>>>> "ssl.truststore.password": "password",
>>>>>>> "ssl.keystore.type": "PKCS12",
>>>>>>> "ssl.keystore.location": "/opt/keys/client.keystore.p12", # how do I make this available to the Java SDK harness
>>>>>>> "ssl.keystore.password": "password",
>>>>>>> "group.id": "group",
>>>>>>> "basic.auth.credentials.source": "USER_INFO",
>>>>>>> "schema.registry.basic.auth.user.info": "user:password"
>>>>>>> },
>>>>>>> topics=["topic"],
>>>>>>> max_num_records=2,
>>>>>>> # expansion_service="localhost:56938"
>>>>>>> )
>>>>>>>
>>>>>>> I tried building a custom image that has pulls credentials into the
>>>>>>> container based on "apache/beam_java11_sdk:2.27.0" and using the "--sdk_harness_container_image_overrides=.*java.*,{image}:{tag}"
>>>>>>> argument, but it does not appear to be used when the pipeline runs
>>>>>>> on Flink.
>>>>>>>
>>>>>>> Thank you for the great tool and any help or advice is greatly
>>>>>>> appreciated.
>>>>>>>
>>>>>>> --
>>>>>>> paul
>>>>>>>
>>>>>>
>>>>
>>>> --
>>>> paul
>>>>
>>>
>>
>> --
>> paul
>>
>
--
paul
Re: Using the Beam Python SDK and PortableRunner with Flink to
connect to Kafka with SSL
Posted by Chamikara Jayalath <ch...@google.com>.
Currently 'sdk_harness_container_image_overrides' is Dataflow only.
Also I think options might be getting dropped due to
https://issues.apache.org/jira/browse/BEAM-9449.
Unfortunately, till one of the above is fixed I think you might have to
update the code at the following location to override the container and
rebuild the SDK.
https://github.com/apache/beam/blob/0078bb35ba4aef9410d681d8b4e2c16d9f56433d/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/Environments.java#L397
Thanks,
Cham
On Tue, Feb 2, 2021 at 2:54 PM Paul Nimusiima <pa...@gmail.com> wrote:
> Those two arguments appear to be discarded
> WARNING:apache_beam.options.pipeline_options:Discarding unparseable args:
> ['--default_environment_type=DOCKER',
> '--default_environment_config=beam_java_sdk:0.0.1']
> and the default apache/beam_java11_sdk:2.27.0 is used.
>
> INFO
> org.apache.beam.runners.fnexecution.environment.DockerEnvironmentFactory -
> Still waiting for startup of environment apache/beam_java11_sdk:2.27.0 for
> worker id 1-2
>
> and when the when the harness is being removed
>
> INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory
> - Closing environment urn: "beam:env:docker:v1"
> payload: "\n\035apache/beam_java11_sdk:2.27.0"
> capabilities: "beam:coder:bytes:v1"
> ...
> capabilities: "beam:transform:sdf_truncate_sized_restrictions:v1"
> dependencies {
> type_urn: "beam:artifact:type:file:v1"
> type_payload:
> "\n\354\001/var/folders/yj/njlsyhss5sb7t36fklhxjbvc0000gp/T/beam-artifact-staging/11d9c289a5c41d77ecf6d4eff0cc95928c241ec416c8e88ea67fbc931463a0c5/1-external_1beam:env:docker-beam-sdks-java-io-expansion-service-2.27.0-OrWYGnEG6c2yQRgdLP3AI3hLQ_1lT"
> role_urn: "beam:artifact:role:staging_to:v1"
> role_payload:
> "\nZbeam-sdks-java-io-expansion-service-2.27.0-OrWYGnEG6c2yQRgdLP3AI3hLQ_1lTDuSw_K_Lc45B5c.jar"
> }
>
> On Tue, Feb 2, 2021 at 10:14 PM Kyle Weaver <kc...@google.com> wrote:
>
>> > Regarding Kyle Weaver's suggestion, I tried a combination of
>> "--environment_type=DOCKER" and
>> '--environment_config=beam_python_sdk:0.0.1' where beam_python_sdk:0.0.1 is
>> an image I built. That made it possible to override the python sdk harness,
>> but I was wondering if there is something similar for overriding the Java
>> sdk harness?
>>
>> The Java options are named differently, despite appearing to have
>> identical semantics. Can you try default_environment_type and
>> default_environment_config?
>>
>> On Tue, Feb 2, 2021 at 2:04 PM Paul Nimusiima <pa...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>> Thanks for the responses.
>>> Regarding Kyle Weaver's suggestion, I tried a combination of
>>> "--environment_type=DOCKER" and
>>> '--environment_config=beam_python_sdk:0.0.1' where beam_python_sdk:0.0.1
>>> is an image I built. That made it possible to override the python sdk
>>> harness, but I was wondering if there is something similar for overriding
>>> the Java sdk harness?
>>>
>>> Thanks again.
>>>
>>> On Tue, Feb 2, 2021 at 9:08 PM Kyle Weaver <kc...@google.com> wrote:
>>>
>>>> AFAIK sdk_harness_container_image_overrides only works for the Dataflow
>>>> runner. For other runners I think you will have to change the default
>>>> environment config:
>>>> https://github.com/apache/beam/blob/0078bb35ba4aef9410d681d8b4e2c16d9f56433d/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L53-L71
>>>>
>>>> On Tue, Feb 2, 2021 at 11:56 AM Brian Hulette <bh...@google.com>
>>>> wrote:
>>>>
>>>>> Hm I would expect that to work. Can you tell what container Flink is
>>>>> using if it's not using the one you specified? +Chamikara Jayalath
>>>>> <ch...@google.com> may have some insight here
>>>>>
>>>>> Brian
>>>>>
>>>>> On Tue, Feb 2, 2021 at 3:27 AM Paul Nimusiima <pa...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello Beam community,
>>>>>>
>>>>>> I am wondering whether it is possible to connect to a secure Kafka
>>>>>> broker with the python beam SDK. So for example, in the code below, how
>>>>>> would I make the "ssl.truststore.location" and "ssl.keystore.location"
>>>>>> accessible inside the Java SDK harness which runs the code.
>>>>>>
>>>>>> ReadFromKafka(
>>>>>> consumer_config={
>>>>>> "bootstrap.servers": "bootstrap-server:17032",
>>>>>> "security.protocol": "SSL",
>>>>>> "ssl.truststore.location": "/opt/keys/client.truststore.jks", # how do I make this available to the Java SDK harness
>>>>>> "ssl.truststore.password": "password",
>>>>>> "ssl.keystore.type": "PKCS12",
>>>>>> "ssl.keystore.location": "/opt/keys/client.keystore.p12", # how do I make this available to the Java SDK harness
>>>>>> "ssl.keystore.password": "password",
>>>>>> "group.id": "group",
>>>>>> "basic.auth.credentials.source": "USER_INFO",
>>>>>> "schema.registry.basic.auth.user.info": "user:password"
>>>>>> },
>>>>>> topics=["topic"],
>>>>>> max_num_records=2,
>>>>>> # expansion_service="localhost:56938"
>>>>>> )
>>>>>>
>>>>>> I tried building a custom image that has pulls credentials into the
>>>>>> container based on "apache/beam_java11_sdk:2.27.0" and using the "--sdk_harness_container_image_overrides=.*java.*,{image}:{tag}"
>>>>>> argument, but it does not appear to be used when the pipeline runs on
>>>>>> Flink.
>>>>>>
>>>>>> Thank you for the great tool and any help or advice is greatly
>>>>>> appreciated.
>>>>>>
>>>>>> --
>>>>>> paul
>>>>>>
>>>>>
>>>
>>> --
>>> paul
>>>
>>
>
> --
> paul
>
Re: Using the Beam Python SDK and PortableRunner with Flink to
connect to Kafka with SSL
Posted by Paul Nimusiima <pa...@gmail.com>.
Those two arguments appear to be discarded
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args:
['--default_environment_type=DOCKER',
'--default_environment_config=beam_java_sdk:0.0.1']
and the default apache/beam_java11_sdk:2.27.0 is used.
INFO
org.apache.beam.runners.fnexecution.environment.DockerEnvironmentFactory -
Still waiting for startup of environment apache/beam_java11_sdk:2.27.0 for
worker id 1-2
and when the when the harness is being removed
INFO org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory
- Closing environment urn: "beam:env:docker:v1"
payload: "\n\035apache/beam_java11_sdk:2.27.0"
capabilities: "beam:coder:bytes:v1"
...
capabilities: "beam:transform:sdf_truncate_sized_restrictions:v1"
dependencies {
type_urn: "beam:artifact:type:file:v1"
type_payload:
"\n\354\001/var/folders/yj/njlsyhss5sb7t36fklhxjbvc0000gp/T/beam-artifact-staging/11d9c289a5c41d77ecf6d4eff0cc95928c241ec416c8e88ea67fbc931463a0c5/1-external_1beam:env:docker-beam-sdks-java-io-expansion-service-2.27.0-OrWYGnEG6c2yQRgdLP3AI3hLQ_1lT"
role_urn: "beam:artifact:role:staging_to:v1"
role_payload:
"\nZbeam-sdks-java-io-expansion-service-2.27.0-OrWYGnEG6c2yQRgdLP3AI3hLQ_1lTDuSw_K_Lc45B5c.jar"
}
On Tue, Feb 2, 2021 at 10:14 PM Kyle Weaver <kc...@google.com> wrote:
> > Regarding Kyle Weaver's suggestion, I tried a combination of
> "--environment_type=DOCKER" and
> '--environment_config=beam_python_sdk:0.0.1' where beam_python_sdk:0.0.1 is
> an image I built. That made it possible to override the python sdk harness,
> but I was wondering if there is something similar for overriding the Java
> sdk harness?
>
> The Java options are named differently, despite appearing to have
> identical semantics. Can you try default_environment_type and
> default_environment_config?
>
> On Tue, Feb 2, 2021 at 2:04 PM Paul Nimusiima <pa...@gmail.com>
> wrote:
>
>> Hi all,
>> Thanks for the responses.
>> Regarding Kyle Weaver's suggestion, I tried a combination of
>> "--environment_type=DOCKER" and
>> '--environment_config=beam_python_sdk:0.0.1' where beam_python_sdk:0.0.1
>> is an image I built. That made it possible to override the python sdk
>> harness, but I was wondering if there is something similar for overriding
>> the Java sdk harness?
>>
>> Thanks again.
>>
>> On Tue, Feb 2, 2021 at 9:08 PM Kyle Weaver <kc...@google.com> wrote:
>>
>>> AFAIK sdk_harness_container_image_overrides only works for the Dataflow
>>> runner. For other runners I think you will have to change the default
>>> environment config:
>>> https://github.com/apache/beam/blob/0078bb35ba4aef9410d681d8b4e2c16d9f56433d/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L53-L71
>>>
>>> On Tue, Feb 2, 2021 at 11:56 AM Brian Hulette <bh...@google.com>
>>> wrote:
>>>
>>>> Hm I would expect that to work. Can you tell what container Flink is
>>>> using if it's not using the one you specified? +Chamikara Jayalath
>>>> <ch...@google.com> may have some insight here
>>>>
>>>> Brian
>>>>
>>>> On Tue, Feb 2, 2021 at 3:27 AM Paul Nimusiima <pa...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello Beam community,
>>>>>
>>>>> I am wondering whether it is possible to connect to a secure Kafka
>>>>> broker with the python beam SDK. So for example, in the code below, how
>>>>> would I make the "ssl.truststore.location" and "ssl.keystore.location"
>>>>> accessible inside the Java SDK harness which runs the code.
>>>>>
>>>>> ReadFromKafka(
>>>>> consumer_config={
>>>>> "bootstrap.servers": "bootstrap-server:17032",
>>>>> "security.protocol": "SSL",
>>>>> "ssl.truststore.location": "/opt/keys/client.truststore.jks", # how do I make this available to the Java SDK harness
>>>>> "ssl.truststore.password": "password",
>>>>> "ssl.keystore.type": "PKCS12",
>>>>> "ssl.keystore.location": "/opt/keys/client.keystore.p12", # how do I make this available to the Java SDK harness
>>>>> "ssl.keystore.password": "password",
>>>>> "group.id": "group",
>>>>> "basic.auth.credentials.source": "USER_INFO",
>>>>> "schema.registry.basic.auth.user.info": "user:password"
>>>>> },
>>>>> topics=["topic"],
>>>>> max_num_records=2,
>>>>> # expansion_service="localhost:56938"
>>>>> )
>>>>>
>>>>> I tried building a custom image that has pulls credentials into the
>>>>> container based on "apache/beam_java11_sdk:2.27.0" and using the "--sdk_harness_container_image_overrides=.*java.*,{image}:{tag}"
>>>>> argument, but it does not appear to be used when the pipeline runs on
>>>>> Flink.
>>>>>
>>>>> Thank you for the great tool and any help or advice is greatly
>>>>> appreciated.
>>>>>
>>>>> --
>>>>> paul
>>>>>
>>>>
>>
>> --
>> paul
>>
>
--
paul
Re: Using the Beam Python SDK and PortableRunner with Flink to
connect to Kafka with SSL
Posted by Kyle Weaver <kc...@google.com>.
> Regarding Kyle Weaver's suggestion, I tried a combination of
"--environment_type=DOCKER" and '--environment_config=beam_python_sdk:0.0.1'
where beam_python_sdk:0.0.1 is an image I built. That made it possible to
override the python sdk harness, but I was wondering if there is something
similar for overriding the Java sdk harness?
The Java options are named differently, despite appearing to have identical
semantics. Can you try default_environment_type and
default_environment_config?
On Tue, Feb 2, 2021 at 2:04 PM Paul Nimusiima <pa...@gmail.com> wrote:
> Hi all,
> Thanks for the responses.
> Regarding Kyle Weaver's suggestion, I tried a combination of
> "--environment_type=DOCKER" and
> '--environment_config=beam_python_sdk:0.0.1' where beam_python_sdk:0.0.1
> is an image I built. That made it possible to override the python sdk
> harness, but I was wondering if there is something similar for overriding
> the Java sdk harness?
>
> Thanks again.
>
> On Tue, Feb 2, 2021 at 9:08 PM Kyle Weaver <kc...@google.com> wrote:
>
>> AFAIK sdk_harness_container_image_overrides only works for the Dataflow
>> runner. For other runners I think you will have to change the default
>> environment config:
>> https://github.com/apache/beam/blob/0078bb35ba4aef9410d681d8b4e2c16d9f56433d/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L53-L71
>>
>> On Tue, Feb 2, 2021 at 11:56 AM Brian Hulette <bh...@google.com>
>> wrote:
>>
>>> Hm I would expect that to work. Can you tell what container Flink is
>>> using if it's not using the one you specified? +Chamikara Jayalath
>>> <ch...@google.com> may have some insight here
>>>
>>> Brian
>>>
>>> On Tue, Feb 2, 2021 at 3:27 AM Paul Nimusiima <pa...@gmail.com>
>>> wrote:
>>>
>>>> Hello Beam community,
>>>>
>>>> I am wondering whether it is possible to connect to a secure Kafka
>>>> broker with the python beam SDK. So for example, in the code below, how
>>>> would I make the "ssl.truststore.location" and "ssl.keystore.location"
>>>> accessible inside the Java SDK harness which runs the code.
>>>>
>>>> ReadFromKafka(
>>>> consumer_config={
>>>> "bootstrap.servers": "bootstrap-server:17032",
>>>> "security.protocol": "SSL",
>>>> "ssl.truststore.location": "/opt/keys/client.truststore.jks", # how do I make this available to the Java SDK harness
>>>> "ssl.truststore.password": "password",
>>>> "ssl.keystore.type": "PKCS12",
>>>> "ssl.keystore.location": "/opt/keys/client.keystore.p12", # how do I make this available to the Java SDK harness
>>>> "ssl.keystore.password": "password",
>>>> "group.id": "group",
>>>> "basic.auth.credentials.source": "USER_INFO",
>>>> "schema.registry.basic.auth.user.info": "user:password"
>>>> },
>>>> topics=["topic"],
>>>> max_num_records=2,
>>>> # expansion_service="localhost:56938"
>>>> )
>>>>
>>>> I tried building a custom image that has pulls credentials into the
>>>> container based on "apache/beam_java11_sdk:2.27.0" and using the "--sdk_harness_container_image_overrides=.*java.*,{image}:{tag}"
>>>> argument, but it does not appear to be used when the pipeline runs on
>>>> Flink.
>>>>
>>>> Thank you for the great tool and any help or advice is greatly
>>>> appreciated.
>>>>
>>>> --
>>>> paul
>>>>
>>>
>
> --
> paul
>
Re: Using the Beam Python SDK and PortableRunner with Flink to
connect to Kafka with SSL
Posted by Paul Nimusiima <pa...@gmail.com>.
Hi all,
Thanks for the responses.
Regarding Kyle Weaver's suggestion, I tried a combination of
"--environment_type=DOCKER" and '--environment_config=beam_python_sdk:0.0.1'
where beam_python_sdk:0.0.1 is an image I built. That made it possible to
override the python sdk harness, but I was wondering if there is something
similar for overriding the Java sdk harness?
Thanks again.
On Tue, Feb 2, 2021 at 9:08 PM Kyle Weaver <kc...@google.com> wrote:
> AFAIK sdk_harness_container_image_overrides only works for the Dataflow
> runner. For other runners I think you will have to change the default
> environment config:
> https://github.com/apache/beam/blob/0078bb35ba4aef9410d681d8b4e2c16d9f56433d/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L53-L71
>
> On Tue, Feb 2, 2021 at 11:56 AM Brian Hulette <bh...@google.com> wrote:
>
>> Hm I would expect that to work. Can you tell what container Flink is
>> using if it's not using the one you specified? +Chamikara Jayalath
>> <ch...@google.com> may have some insight here
>>
>> Brian
>>
>> On Tue, Feb 2, 2021 at 3:27 AM Paul Nimusiima <pa...@gmail.com>
>> wrote:
>>
>>> Hello Beam community,
>>>
>>> I am wondering whether it is possible to connect to a secure Kafka
>>> broker with the python beam SDK. So for example, in the code below, how
>>> would I make the "ssl.truststore.location" and "ssl.keystore.location"
>>> accessible inside the Java SDK harness which runs the code.
>>>
>>> ReadFromKafka(
>>> consumer_config={
>>> "bootstrap.servers": "bootstrap-server:17032",
>>> "security.protocol": "SSL",
>>> "ssl.truststore.location": "/opt/keys/client.truststore.jks", # how do I make this available to the Java SDK harness
>>> "ssl.truststore.password": "password",
>>> "ssl.keystore.type": "PKCS12",
>>> "ssl.keystore.location": "/opt/keys/client.keystore.p12", # how do I make this available to the Java SDK harness
>>> "ssl.keystore.password": "password",
>>> "group.id": "group",
>>> "basic.auth.credentials.source": "USER_INFO",
>>> "schema.registry.basic.auth.user.info": "user:password"
>>> },
>>> topics=["topic"],
>>> max_num_records=2,
>>> # expansion_service="localhost:56938"
>>> )
>>>
>>> I tried building a custom image that has pulls credentials into the
>>> container based on "apache/beam_java11_sdk:2.27.0" and using the "--sdk_harness_container_image_overrides=.*java.*,{image}:{tag}"
>>> argument, but it does not appear to be used when the pipeline runs on
>>> Flink.
>>>
>>> Thank you for the great tool and any help or advice is greatly
>>> appreciated.
>>>
>>> --
>>> paul
>>>
>>
--
paul
Re: Using the Beam Python SDK and PortableRunner with Flink to
connect to Kafka with SSL
Posted by Kyle Weaver <kc...@google.com>.
AFAIK sdk_harness_container_image_overrides only works for the Dataflow
runner. For other runners I think you will have to change the default
environment config:
https://github.com/apache/beam/blob/0078bb35ba4aef9410d681d8b4e2c16d9f56433d/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L53-L71
On Tue, Feb 2, 2021 at 11:56 AM Brian Hulette <bh...@google.com> wrote:
> Hm I would expect that to work. Can you tell what container Flink is using
> if it's not using the one you specified? +Chamikara Jayalath
> <ch...@google.com> may have some insight here
>
> Brian
>
> On Tue, Feb 2, 2021 at 3:27 AM Paul Nimusiima <pa...@gmail.com>
> wrote:
>
>> Hello Beam community,
>>
>> I am wondering whether it is possible to connect to a secure Kafka
>> broker with the python beam SDK. So for example, in the code below, how
>> would I make the "ssl.truststore.location" and "ssl.keystore.location"
>> accessible inside the Java SDK harness which runs the code.
>>
>> ReadFromKafka(
>> consumer_config={
>> "bootstrap.servers": "bootstrap-server:17032",
>> "security.protocol": "SSL",
>> "ssl.truststore.location": "/opt/keys/client.truststore.jks", # how do I make this available to the Java SDK harness
>> "ssl.truststore.password": "password",
>> "ssl.keystore.type": "PKCS12",
>> "ssl.keystore.location": "/opt/keys/client.keystore.p12", # how do I make this available to the Java SDK harness
>> "ssl.keystore.password": "password",
>> "group.id": "group",
>> "basic.auth.credentials.source": "USER_INFO",
>> "schema.registry.basic.auth.user.info": "user:password"
>> },
>> topics=["topic"],
>> max_num_records=2,
>> # expansion_service="localhost:56938"
>> )
>>
>> I tried building a custom image that has pulls credentials into the
>> container based on "apache/beam_java11_sdk:2.27.0" and using the "--sdk_harness_container_image_overrides=.*java.*,{image}:{tag}"
>> argument, but it does not appear to be used when the pipeline runs on
>> Flink.
>>
>> Thank you for the great tool and any help or advice is greatly
>> appreciated.
>>
>> --
>> paul
>>
>
Re: Using the Beam Python SDK and PortableRunner with Flink to
connect to Kafka with SSL
Posted by Brian Hulette <bh...@google.com>.
Hm I would expect that to work. Can you tell what container Flink is using
if it's not using the one you specified? +Chamikara Jayalath
<ch...@google.com> may have some insight here
Brian
On Tue, Feb 2, 2021 at 3:27 AM Paul Nimusiima <pa...@gmail.com> wrote:
> Hello Beam community,
>
> I am wondering whether it is possible to connect to a secure Kafka broker
> with the python beam SDK. So for example, in the code below, how would I
> make the "ssl.truststore.location" and "ssl.keystore.location" accessible
> inside the Java SDK harness which runs the code.
>
> ReadFromKafka(
> consumer_config={
> "bootstrap.servers": "bootstrap-server:17032",
> "security.protocol": "SSL",
> "ssl.truststore.location": "/opt/keys/client.truststore.jks", # how do I make this available to the Java SDK harness
> "ssl.truststore.password": "password",
> "ssl.keystore.type": "PKCS12",
> "ssl.keystore.location": "/opt/keys/client.keystore.p12", # how do I make this available to the Java SDK harness
> "ssl.keystore.password": "password",
> "group.id": "group",
> "basic.auth.credentials.source": "USER_INFO",
> "schema.registry.basic.auth.user.info": "user:password"
> },
> topics=["topic"],
> max_num_records=2,
> # expansion_service="localhost:56938"
> )
>
> I tried building a custom image that has pulls credentials into the
> container based on "apache/beam_java11_sdk:2.27.0" and using the "--sdk_harness_container_image_overrides=.*java.*,{image}:{tag}"
> argument, but it does not appear to be used when the pipeline runs on
> Flink.
>
> Thank you for the great tool and any help or advice is greatly appreciated.
>
> --
> paul
>