You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Kenneth Knowles (Jira)" <ji...@apache.org> on 2022/01/14 05:18:00 UTC
[jira] [Updated] (BEAM-12836) Flink JobServer should allow setting the default environment for the expansion service
[ https://issues.apache.org/jira/browse/BEAM-12836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kenneth Knowles updated BEAM-12836:
-----------------------------------
Status: Open (was: Triage Needed)
> Flink JobServer should allow setting the default environment for the expansion service
> --------------------------------------------------------------------------------------
>
> Key: BEAM-12836
> URL: https://issues.apache.org/jira/browse/BEAM-12836
> Project: Beam
> Issue Type: Improvement
> Components: jobserver
> Reporter: Jeremy Lewi
> Priority: P3
>
> I think the [FlinkJobServer|https://github.com/apache/beam/blob/3a7b8e757d0c220b5838ca6a0805ee463b24f85d/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java#L72] needs to expose command line arguments to allow configuring the default environment for the expansion service.
> IIUC the expansion service in the job server is needed when submitting jobs using the portable runner that have cross language transforms (e.g. a Python pipeline that uses KafkaIO which is a Java language transform). In this case, we may need to configure the expansion service to change the default environment (e.g. the docker image used for Java transforms).
> It looks like [https://github.com/apache/beam/pull/15082/files] added appropriate command line flags to the main entry point for a standalone expansion service. However, I don't believe this code path is used when running the expansion service as part of the job server.
> I believe the [FlinkJobServer|https://github.com/apache/beam/blob/3a7b8e757d0c220b5838ca6a0805ee463b24f85d/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java#L72] would need to take similar command line arguments and use them to configure the expansion service.
> More discussion in this [thread|https://lists.apache.org/thread.html/r3cf56464c157da840f46e777c1b3f307f65ec2cf6b2278728f2cd1e1%40%3Cuser.beam.apache.org%3E]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)