You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Mingliang Gong (Jira)" <ji...@apache.org> on 2019/11/08 02:46:00 UTC
[jira] [Updated] (BEAM-8591) Exception is thrown when running Beam
Pipeline on Kubernetes Flink Cluster.
[ https://issues.apache.org/jira/browse/BEAM-8591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mingliang Gong updated BEAM-8591:
---------------------------------
Description:
# Apache Beam and Flink Walkthrough Test
## Setup Clusters
1. Setup Local Flink Cluster: https://ci.apache.org/projects/flink/flink-docs-release-1.8/tutorials/local_setup.html
2. Setup Kubernetes Flink Cluster: https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/deployment/kubernetes.html
## Verify Clusters
Execute command “./bin/flink run examples/streaming/WordCount.jar”. Both Local and K8S Flink Cluster work fine.
## Apache Beam Flink Runner
Instrution: https://beam.apache.org/documentation/runners/flink/
Sample Pipeline Code:
```python
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
options = PipelineOptions([
"--runner=PortableRunner",
"--job_endpoint=localhost:8099",
"--environment_type=LOOPBACK"
])
with beam.Pipeline(options=options) as pipeline:
data = ["Sample data",
"Sample data - 0",
"Sample data - 1"]
raw_data = (pipeline
| 'CreateHardCodeData' >> beam.Create(data)
| 'Map' >> beam.Map(lambda line : line + '.')
| 'Print' >> beam.Map(print))
```
Verfiy different environment_type in Python SDK Harness Configuration
**environment_type=LOOKBACK**
1. Run pipeline on local cluster: Works Fine
2. Run pipeline on K8S cluster, Exceptions are thrown:
java.lang.Exception: The user defined 'open()' method caused an exception: org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
Caused by: org.apache.beam.vendor.grpc.v1p21p0.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:51017
**environment_type=DOCKER**
1. Run pipeline on local cluster: Work fine
2. Run pipeline on K8S cluster, Exception are thrown:
Caused by: java.io.IOException: Cannot run program "docker": error=2, No such file or directory.
was:
h2. Setup Clusters
# Setup Local Flink Cluster: [https://ci.apache.org/projects/flink/flink-docs-release-1.8/tutorials/local_setup.html]
# Setup Kubernetes Flink Cluster with Minikube: [https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/deployment/kubernetes.html]
h2. [|https://github.com/sql-machine-learning/elasticdl/wiki/Test-Walkthrough-Apache-Beam-and-Flink#verify-clusters]Verify Clusters
Execute command “./bin/flink run examples/streaming/WordCount.jar”. Both Local and K8S Flink Cluster work fine.
h2. [|https://github.com/sql-machine-learning/elasticdl/wiki/Test-Walkthrough-Apache-Beam-and-Flink#apache-beam-flink-runner]Apache Beam Flink Runner
Instruction: [https://beam.apache.org/documentation/runners/flink/]
Sample Pipeline Code:
import apache_beam as beam from apache_beam.options.pipeline_options import PipelineOptions options = PipelineOptions([ "--runner=PortableRunner", "--job_endpoint=localhost:8099", "--environment_type=LOOPBACK" ]) with beam.Pipeline(options=options) as pipeline: data = ["Sample data", "Sample data - 0", "Sample data - 1"] raw_data = (pipeline | 'CreateHardCodeData' >> beam.Create(data) | 'Map' >> beam.Map(lambda line : line + '.') | 'Print' >> beam.Map(print))
Verfiy different environment_type in Python SDK Harness Configuration
*environment_type=LOOKBACK*
# Run pipeline on local cluster: Works Fine
# Run pipeline on K8S cluster, Exceptions are thrown:
java.lang.Exception: The user defined 'open()' method caused an exception: org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception Caused by: org.apache.beam.vendor.grpc.v1p21p0.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:51017
*environment_type=DOCKER*
# Run pipeline on local cluster: Work fine
# Run pipeline on K8S cluster, Exception are thrown:
Caused by: java.io.IOException: Cannot run program "docker": error=2, No such file or directory.
> Exception is thrown when running Beam Pipeline on Kubernetes Flink Cluster.
> ---------------------------------------------------------------------------
>
> Key: BEAM-8591
> URL: https://issues.apache.org/jira/browse/BEAM-8591
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Reporter: Mingliang Gong
> Priority: Major
>
> # Apache Beam and Flink Walkthrough Test
> ## Setup Clusters
> 1. Setup Local Flink Cluster: https://ci.apache.org/projects/flink/flink-docs-release-1.8/tutorials/local_setup.html
> 2. Setup Kubernetes Flink Cluster: https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/deployment/kubernetes.html
> ## Verify Clusters
> Execute command “./bin/flink run examples/streaming/WordCount.jar”. Both Local and K8S Flink Cluster work fine.
> ## Apache Beam Flink Runner
> Instrution: https://beam.apache.org/documentation/runners/flink/
> Sample Pipeline Code:
> ```python
> import apache_beam as beam
> from apache_beam.options.pipeline_options import PipelineOptions
> options = PipelineOptions([
> "--runner=PortableRunner",
> "--job_endpoint=localhost:8099",
> "--environment_type=LOOPBACK"
> ])
> with beam.Pipeline(options=options) as pipeline:
> data = ["Sample data",
> "Sample data - 0",
> "Sample data - 1"]
> raw_data = (pipeline
> | 'CreateHardCodeData' >> beam.Create(data)
> | 'Map' >> beam.Map(lambda line : line + '.')
> | 'Print' >> beam.Map(print))
> ```
> Verfiy different environment_type in Python SDK Harness Configuration
> **environment_type=LOOKBACK**
> 1. Run pipeline on local cluster: Works Fine
> 2. Run pipeline on K8S cluster, Exceptions are thrown:
> java.lang.Exception: The user defined 'open()' method caused an exception: org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> Caused by: org.apache.beam.vendor.grpc.v1p21p0.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:51017
> **environment_type=DOCKER**
> 1. Run pipeline on local cluster: Work fine
> 2. Run pipeline on K8S cluster, Exception are thrown:
> Caused by: java.io.IOException: Cannot run program "docker": error=2, No such file or directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)