You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Eddie Wang (Jira)" <ji...@apache.org> on 2021/09/26 06:42:00 UTC
[jira] [Commented] (BEAM-12814) Support External Environment For
Java SDK Harness
[ https://issues.apache.org/jira/browse/BEAM-12814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17420217#comment-17420217 ]
Eddie Wang commented on BEAM-12814:
-----------------------------------
This is critical for allowing k8s to run Flink workloads.
Having this broken makes the promise of Beam fall short for all Python users. All important IOs depend on the Java expansion service, and the inability to deploy this workload to K8s (without enabling dind) makes Beam pretty locked down to GCP Dataflow :(
> Support External Environment For Java SDK Harness
> -------------------------------------------------
>
> Key: BEAM-12814
> URL: https://issues.apache.org/jira/browse/BEAM-12814
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-harness
> Reporter: Jeremy Lewi
> Priority: P2
>
> As mentioned in this [thread|https://lists.apache.org/x/thread.html/r3cf56464c157da840f46e777c1b3f307f65ec2cf6b2278728f2cd1e1@%3Cuser.beam.apache.org%3E] the Java SDK harness only supports environments docker and embedded for the Java SDK harness.
> It would be nice if the Java SDK also supported external environments similar to what the Python SDK harness does.
> The use case would be running in Kubernetes. On Kubernetes I think its better to run the SDK harness containers as side cars of the taskmanager pods rather than relying on docker in docker. Docker in docker isn't great from a security standpoint because it requires privileged containers. It also makes it harder to debug and operate on Kubernetes.
>
> This was also mentioned in this thread on the developer mailing list; [thread|https://lists.apache.org/thread.html/r43fbff58c0d11b05a74617b175ed2d0565b9e65717e92f1dd908f4fa%40%3Cdev.beam.apache.org%3E]
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)