You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Robert Bradshaw (Jira)" <ji...@apache.org> on 2020/10/01 00:25:00 UTC
[jira] [Commented] (BEAM-10986) :build task doesn't build shaded
jar with shadow >5.0.0
[ https://issues.apache.org/jira/browse/BEAM-10986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17205163#comment-17205163 ]
Robert Bradshaw commented on BEAM-10986:
----------------------------------------
Correct, the full contract for naming and starting up expansion
services (or job services) is not well specified, despite the protocol
(once you have a port) being well established.
For beam java-provided, python-used expansion services, we have been naming
them based on gradle targets, e.g.
https://github.com/apache/beam/blob/v2.24.0/sdks/python/apache_beam/io/kafka.py#L107
If 'dev' is in the version name, we compute the relative path of the
gradle-build jar (and if not present, give instructions how to use it).
Otherwise we try to fetch the relevant artifacts from the apache beam maven
repository itself. We use this tactic for job services as well, e.g. when
running with the FlinkRunner it'll automatically download the Flink job
server jar and invoke it with java. (We also provide the ability to map
gradle tasks to manually provided jar paths as well, which can be useful
for testing or non-traditional setups.)
This is a pragmatic solution that provides a decent development and
end-user experience, but of course isn't fully general. We could also
offer these as docker images, though that has other issues (e.g. docker is
harder to install and generally less widely available than Java (though
that might be changing) and one doesn't inherit the host's environment
(including the filesystem and credentials) and may be more heavy-weight,
but has the advantage that it is portable to arbitrary languages. We could
also explore using the image that provides the expansion be the same image
that is used as the worker (provided it can reliably name itself).
Certainly room for improvement here.
On Wed, Sep 30, 2020 at 4:06 PM Kenneth Knowles (Jira) <ji...@apache.org>
> :build task doesn't build shaded jar with shadow >5.0.0
> -------------------------------------------------------
>
> Key: BEAM-10986
> URL: https://issues.apache.org/jira/browse/BEAM-10986
> Project: Beam
> Issue Type: Bug
> Components: build-system
> Reporter: Brian Hulette
> Assignee: Brian Hulette
> Priority: P0
> Fix For: 2.25.0
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> Not 100% sure but I believe this will prevent us from releasing shaded jars.
> See [https://lists.apache.org/thread.html/r12c785c2b01017329204ad816740e80b70c19c5f2bb1ea01535a987d%40%3Cdev.beam.apache.org%3E] for details.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)