Build python Beam from source

[Поротиков Станислав Вячеславович via dev <de...@beam.apache.org>]

Hello!
How to properly install/build apache-beam python package from source?
I've tried running:
                pip install .
from skds/python<https://github.com/apache/beam/tree/master/sdks/python> directory
It's installed successfully, but when I try to run python beam pipeline, it complains:
                RuntimeError: <MY_VIRTUAL_ENVIRONMENT>/lib/sdks/java/io/expansion-service/build/libs/beam-sdks-java-io-expansion-service-2.52.0-SNAPSHOT.jar not found. Please build the server with
                 cd <MY_VIRTUAL_ENVIRONMENT>/lib; ./gradlew sdks:java:io:expansion-service:shadowJar

Glad to any help!

Best regards,
Stanislav Porotikov

Re: Build python Beam from source

[Robert Bradshaw via dev <de...@beam.apache.org>]

To use cross language capabilities from a non-release branch you'll
have to build the cross-language bits yourself as well. This can be
done by

(1) Making sure Java (for java dependencies) is installed.
(2) In the top level of the repository, running .//gradlew
sdks:java:io:expansion-service:shadowJar

For released versions of Beam, it will automatically fetch the
pre-built, released artifacts for you from maven. You can manually
request those of a previous release by passing something like

--beam_services='{"sdks:java:extensions:sql:expansion-service:shadowJar":
"https://repository.apache.org/content/repositories/orgapachebeam-1361/org/apache/beam/beam-sdks-java-extensions-sql-expansion-service/2.52.0/beam-sdks-java-extensions-sql-expansion-service-2.52.0.jar"}'

which basically says "when looking for this target, use that jar"
though as this is using an out-of-date copy of the libraries this may
not always work.


On Tue, Dec 5, 2023 at 6:14 AM Поротиков Станислав Вячеславович via
dev <de...@beam.apache.org> wrote:
>
> Hello!
> How to properly install/build apache-beam python package from source?
>
> I've tried running:
>
>                 pip install .
>
> from skds/python directory
>
> It's installed successfully, but when I try to run python beam pipeline, it complains:
>                 RuntimeError: <MY_VIRTUAL_ENVIRONMENT>/lib/sdks/java/io/expansion-service/build/libs/beam-sdks-java-io-expansion-service-2.52.0-SNAPSHOT.jar not found. Please build the server with
>
>                  cd <MY_VIRTUAL_ENVIRONMENT>/lib; ./gradlew sdks:java:io:expansion-service:shadowJar
>
>
>
> Glad to any help!
>
>
>
> Best regards,
>
> Stanislav Porotikov
>
>