[jira] [Created] (BEAM-8591) Exception is thrown when running Beam Pipeline on Kubernetes Flink Cluster.

["Mingliang Gong (Jira)" <ji...@apache.org>]

Mingliang Gong created BEAM-8591:
------------------------------------

             Summary: Exception is thrown when running Beam Pipeline on Kubernetes Flink Cluster.
                 Key: BEAM-8591
                 URL: https://issues.apache.org/jira/browse/BEAM-8591
             Project: Beam
          Issue Type: Bug
          Components: runner-flink
            Reporter: Mingliang Gong


h2. Setup Clusters
 # Setup Local Flink Cluster: [https://ci.apache.org/projects/flink/flink-docs-release-1.8/tutorials/local_setup.html]
 # Setup Kubernetes Flink Cluster with Minikube: [https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/deployment/kubernetes.html]

h2. [|https://github.com/sql-machine-learning/elasticdl/wiki/Test-Walkthrough-Apache-Beam-and-Flink#verify-clusters]Verify Clusters

Execute command “./bin/flink run examples/streaming/WordCount.jar”. Both Local and K8S Flink Cluster work fine.
h2. [|https://github.com/sql-machine-learning/elasticdl/wiki/Test-Walkthrough-Apache-Beam-and-Flink#apache-beam-flink-runner]Apache Beam Flink Runner

Instruction: [https://beam.apache.org/documentation/runners/flink/]

Sample Pipeline Code:
import apache_beam as beam from apache_beam.options.pipeline_options import PipelineOptions options = PipelineOptions([ "--runner=PortableRunner", "--job_endpoint=localhost:8099", "--environment_type=LOOPBACK" ]) with beam.Pipeline(options=options) as pipeline: data = ["Sample data", "Sample data - 0", "Sample data - 1"] raw_data = (pipeline | 'CreateHardCodeData' >> beam.Create(data) | 'Map' >> beam.Map(lambda line : line + '.') | 'Print' >> beam.Map(print))
Verfiy different environment_type in Python SDK Harness Configuration
*environment_type=LOOKBACK*
 # Run pipeline on local cluster: Works Fine
 # Run pipeline on K8S cluster, Exceptions are thrown:
java.lang.Exception: The user defined 'open()' method caused an exception: org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception Caused by: org.apache.beam.vendor.grpc.v1p21p0.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:51017

*environment_type=DOCKER*
 # Run pipeline on local cluster: Work fine
 # Run pipeline on K8S cluster, Exception are thrown:
Caused by: java.io.IOException: Cannot run program "docker": error=2, No such file or directory.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)