You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 20:56:09 UTC

[GitHub] [beam] damccorm opened a new issue, #21021: Flink JobServer should allow setting the default environment for the expansion service

damccorm opened a new issue, #21021:
URL: https://github.com/apache/beam/issues/21021

   I think the [FlinkJobServer](https://github.com/apache/beam/blob/3a7b8e757d0c220b5838ca6a0805ee463b24f85d/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java#L72) needs to expose command line arguments to allow configuring the default environment for the expansion service.
   
   IIUC the expansion service in the job server is needed when submitting jobs using the portable runner that have cross language transforms (e.g. a Python pipeline that uses KafkaIO which is a Java language transform). In this case, we may need to configure the expansion service to change the default environment (e.g. the docker image used for Java transforms).
   
   It looks like [https://github.com/apache/beam/pull/15082/files](https://github.com/apache/beam/pull/15082/files) added appropriate command line flags to the main entry point for a standalone expansion service. However, I don't believe this code path is used when running the expansion service as part of the job server. 
   
   I believe the [FlinkJobServer](https://github.com/apache/beam/blob/3a7b8e757d0c220b5838ca6a0805ee463b24f85d/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java#L72) would need to take similar command line arguments and use them to configure the expansion service.
   
   More discussion in this [thread](https://lists.apache.org/thread.html/r3cf56464c157da840f46e777c1b3f307f65ec2cf6b2278728f2cd1e1%40%3Cuser.beam.apache.org%3E)
   
   Imported from Jira [BEAM-12836](https://issues.apache.org/jira/browse/BEAM-12836). Original Jira may contain additional context.
   Reported by: jeremy@lewi.us.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org