You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Kaur Jaanson <kj...@gmail.com> on 2025/03/26 15:01:49 UTC

Specifying java dependencies for multi-language Beam pipeline (using java transforms from python)

Hello!

Hoping someone can point me in the right direction. I am trying to use PEM
certificates for authenticating to Kafka topic using Python and Java
Expansion Service. However I am running into an error:

 Caused by: java.security.NoSuchAlgorithmException: PEM KeyStore not
> available


I am using FlinkRunner and starting expansion service myself with the
expansion service jar taken from Maven. I suspect that the issue is older
version of Java-s kafka-client that has been packaged into expansion
service jar (and job server?). From Apache Beam docs page on Java SDK
dependencies, it seems that default libraries that get used during building
jars are set in
https://github.com/apache/beam/blob/867d70c1689bc4b16141043616f912ce0b10b74c/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L626
and kafka-client version there is quite old (2.4.1). PEM support is
supposed to be there after 2.7.

With that I wonder what would be a good way to specify Java
dependencies for multi language pipeline, eg using KafkaIO with newer
kafka-client from Python?

PS: I tried updating the kafka-client version in BeamModulePlugin.groovy
and creating expansion-service jar and packaging into my python beam image,
however it seems that the error still remained. It might be skill issue, I
am not very familiar with java build tooling (maven, gradle) and might have
missed something. Beam version I am using is 2.61.0.

Thank you,
Kaur Jaanson

Re: Specifying java dependencies for multi-language Beam pipeline (using java transforms from python)

Posted by Robert Bradshaw via user <us...@beam.apache.org>.
Creating your own jar with updated dependencies is the right solution,
to ensure it gets used you may need to set Python's --beam_services
pipeline option, e.g.

--beam_services='{"sdks:java:io:expansion-service:shadowJar":
"/path/to/your.jar"}'

See https://github.com/apache/beam/blob/release-2.46.0/sdks/python/apache_beam/options/pipeline_options.py#L512
Equivalently, you can pass a manually created expansion service into
your ReadFromKafka constructor.

Long term the nicest solution is to update the version used by Beam.
Pull requests welcome!

- Robert


On Wed, Mar 26, 2025 at 8:02 AM Kaur Jaanson <kj...@gmail.com> wrote:
>
> Hello!
>
> Hoping someone can point me in the right direction. I am trying to use PEM certificates for authenticating to Kafka topic using Python and Java Expansion Service. However I am running into an error:
>
>>  Caused by: java.security.NoSuchAlgorithmException: PEM KeyStore not available
>
>
> I am using FlinkRunner and starting expansion service myself with the expansion service jar taken from Maven. I suspect that the issue is older version of Java-s kafka-client that has been packaged into expansion service jar (and job server?). From Apache Beam docs page on Java SDK dependencies, it seems that default libraries that get used during building jars are set in https://github.com/apache/beam/blob/867d70c1689bc4b16141043616f912ce0b10b74c/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L626 and kafka-client version there is quite old (2.4.1). PEM support is supposed to be there after 2.7.
>
> With that I wonder what would be a good way to specify Java dependencies for multi language pipeline, eg using KafkaIO with newer kafka-client from Python?
>
> PS: I tried updating the kafka-client version in BeamModulePlugin.groovy and creating expansion-service jar and packaging into my python beam image, however it seems that the error still remained. It might be skill issue, I am not very familiar with java build tooling (maven, gradle) and might have missed something. Beam version I am using is 2.61.0.
>
> Thank you,
> Kaur Jaanson