You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Matthew Patterson <mp...@aligntech.com> on 2019/09/17 16:40:41 UTC
Word-count example
Folks,
Am sure I am doing it wrong, but have been beating head against wall long enough I need to ask for help.
My goal is to use the python SDK to run jobs on AWS flink cluster, but very little is working, so I am trying to back down to the very simplest thing that works, and build back up: not going so well.
Now I am running, or trying to, on a minikube link cluster. Now running from just java/mvn, so I go through the following:
1. bring-up-cluster (see below)
2. grab-the-code (see below)
3. maven-submit (see below)
Eventual failure is
“[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project word-count-beam: An exception occured while executing the Java class. Pipeline execution failed: Could not retrieve the execution result. (JobID: 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to deserialize JobGraph.]”
and log is attached (I hope)
Other details:
on osx
➜ minikube version
minikube version: v1.3.1
commit: ca60a424ce69a4d79f502650199ca2b52f29e631
➜ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
➜ flink --version
Version: 1.8.1, Commit ID: 7297bac
➜ python -c"import apache_beam; print(apache_beam.__version__)"
2.15.0
➜ curl localhost:8081
<!--
~ Licensed to the Apache Software Foundation (ASF) under one
~ or more contributor license agreements. See the NOTICE file
~ distributed with this work for additional information
~ regarding copyright ownership. The ASF licenses this file
~ to you under the Apache License, Version 2.0 (the
~ "License"); you may not use this file except in compliance
~ with the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Apache Flink Web Dashboard</title>
<link rel="apple-touch-icon" sizes="180x180" href="assets/favicon/apple-touch-icon.png">
<link rel="icon" type="image/png" href="assets/favicon/favicon-32x32.png" sizes="32x32">
<link rel="icon" type="image/png" href="assets/favicon/favicon-16x16.png" sizes="16x16">
<link rel="manifest" href="assets/favicon/manifest.json">
<link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg" color="#aa1919">
<link rel="shortcut icon" href="assets/favicon/favicon.ico">
<meta name="msapplication-config" content="assets/favicon/browserconfig.xml">
<meta name="theme-color" content="#ffffff">
<base href="./"><link rel="stylesheet" href="styles.30d0912c1ece284d8d9a.css"></head>
<body>
<flink-root></flink-root>
<script type="text/javascript" src="runtime.ebbfed06bb8e06aaa889.js"></script><script type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js" nomodule></script><script type="text/javascript" src="polyfills.b37850e8279bc3caafc9.js"></script><script type="text/javascript" src="main.dc6e0997d092df051db0.js"></script></body>
</html>
Thanks for any ideas,
Matt
>>> bring-up-cluster
# [yamls from here](https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/kubernetes.html)
minikube start
minikube ssh 'sudo ip link set docker0 promisc on'
kubectl create -f flink-configuration-configmap.yaml
kubectl create -f jobmanager-service.yaml
kubectl create -f jobmanager-deployment.yaml
kubectl create -f taskmanager-deployment.yaml
kubectl apply -f jobmanager-rest-service.yaml
>>> grab-the-code
mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
-DarchetypeVersion=2.15.0 \
-DgroupId=org.example \
-DartifactId=word-count-beam \
-Dversion="0.1" \
-Dpackage=org.apache.beam.examples \
-DinteractiveMode=false
>>> maven-submit
# kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in separate shell, as blocking
mvn package exec:java -X -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
-Dexec.args="--runner=FlinkRunner\
--flinkMaster="localhost:8081"\
--filesToStage=target/word-count-beam-bundled-0.1.jar\
--output=<directory-that-exists>" -Pflink-runner
Re: Word-count example
Posted by Kyle Weaver <kc...@google.com>.
> The only requirement is that TaskManager nodes must have access to Docker.
That's not strictly true anymore; there is also the PROCESS option, but
I'll warn you it's still under-documented. See
https://beam.apache.org/roadmap/portability/#sdk-harness-config
> Sorry if/that this is stupid question, but is there a preferred method to
fortify Flink taskmanger with docker?
Definitely not a stupid question! It's actually a difficult one that's been
discussed a lot, such as here:
https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.hr29wtiqrupv
Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com
On Tue, Sep 24, 2019 at 10:21 AM Matthew Patterson <mp...@aligntech.com>
wrote:
> OK: after restarting & following directions better, paying attention to
> versions/branches/runner-option-flags, I am able to get jobs submitted
> (submitted: not running to fruition) to Flink cluster (AWS EMR). Further, I
> can get the basic example (a slight riff on word count example) to work
> using loopback environment. Now, to scope Beam-on-Flink for production, I
> want to get jobs on cluster to finish successfully.
>
>
>
> Currently, I am working through this:
> “Flink Deployment
>
> The Flink cluster itself is deployed as normal. For example, it might be
> deployed on a standalone Flink cluster, on Kubernetes, or on YARN. The only
> requirement is that TaskManager nodes must have access to Docker.
>
> ”
>
>
>
> from [Flink Portable Runner Overview](
> https://docs.google.com/document/d/17hL9h_q9aX6dA4lzX5IcV-AgdduxDcj3xTzjbLJyJoQ/edit#
> )
>
>
>
> Sorry if/that this is stupid question, but is there a preferred method to
> fortify Flink taskmanger with docker? Docker in docker (dind)? Just install
> in image? Docker-out-of-docker? Any working example folks could point me to
> would be great.
>
>
>
> Thanks All,
>
> Matt
>
>
>
> [dind?](
> https://medium.com/hootsuite-engineering/building-docker-images-inside-kubernetes-42c6af855f25
> )
>
>
>
>
>
> *From: *Kyle Weaver <kc...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Monday, September 23, 2019 at 4:19 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> If there are no jobs or logs appearing in the Flink UI, it is likely that
> the pipeline is not being submitted to the Flink cluster successfully.
> (Right now, there is no timeout on pipeline submission, but I would like to
> see that changed; see https://issues.apache.org/jira/browse/BEAM-7933
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-7933&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759357273&sdata=pYz%2FPe1nxq%2BXG5YvNOa3JmlIgZYB3h4OtBYd4tXHddQ%3D&reserved=0>
> .)
>
>
>
> I would start by making sure there are no unneeded Flink clusters, job
> servers, etc. left running on your machine, as port conflicts can cause
> silent failures.
>
>
>
> Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759367268&sdata=ttv1XygS09045JCmq0%2BNfw9PGW8504YQhzVpFVxT9%2Bc%3D&reserved=0>
> | kcweaver@google.com
>
>
>
>
>
> On Mon, Sep 23, 2019 at 12:58 PM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Kyle, right: saw that. Grabbing at anything.
>
>
>
> The process on the client seems to submit, or at least not bomb, but the cluster does not acknowledge a running job. It is possible I have not waited long enough yet, and images are still being downloaded & launched.
>
>
>
> 2.16 section on FlinkRunner does not have (that I found) help on gathering
> logs: might you point me at that as well?
>
>
>
> Thanks for the help, Kyle.
>
> Matt
>
>
>
> *From: *Kyle Weaver <kc...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Monday, September 23, 2019 at 3:49 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> "WARNING:root:No unique name set for transform ..." should not affect the pipeline's ability to complete successfully. Is the pipeline failing? If so, could you share more logs?
>
>
>
> Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759367268&sdata=ttv1XygS09045JCmq0%2BNfw9PGW8504YQhzVpFVxT9%2Bc%3D&reserved=0>
> | kcweaver@google.com
>
>
>
>
>
> On Mon, Sep 23, 2019 at 11:43 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> After setting runner, adding *'--environment_type=LOOPBACK'*, I now get
>
>
>
> “
>
> …
>
> WARNING:root:No unique name set for transform fn/read/ref_PCollection_PCollection_17:0
>
> WARNING:root:No unique name set for transform fn/read/ref_PCollection_PCollection_17:0
>
> WARNING:root:No unique name set for transform fn/write/ref_PCollection_PCollection_24:0
>
> …
>
> ”
>
> when running wordcount example:
>
>
>
>
>
> *# We use the save_main_session option because one or more DoFn's in this
> # workflow rely on global context (e.g., a module imported at module
> level). *pipeline_options = runner_options
> pipeline_options.view_as(SetupOptions).save_main_session =
>
> *True *p = beam.Pipeline(options=pipeline_options)
>
>
>
> *# Read the text file[pattern] into a PCollection. *lines = p | *'read' *>>
> ReadFromText(_input)
>
>
> *# Count the occurrences of each word. **def *count_ones(word_ones):
> (word, ones) = word_ones
> *return *(word, sum(ones))
>
> counts = (lines
> | *'split' *>>
> (beam.ParDo(WordExtractingDoFn()).with_output_types(unicode))
> | *'pair_with_one' *>> beam.Map(*lambda *x: (x, 1))
> | *'group' *>> beam.GroupByKey()
> | *'count' *>> beam.Map(count_ones))
>
>
> *# Format the counts into a PCollection of strings. **def *
> format_result(word_count):
> (word, count) = word_count
> *return **'%s: %d' *% (word, count)
>
> output = counts | *'format' *>> beam.Map(format_result)
>
>
>
> *# Write the output using a "Write" transform that has side effects. #
> pylint: disable=expression-not-assigned *output | *'write' *>>
> WriteToText(_output)
>
> result = p.run()
> result.wait_until_finish()
>
>
>
>
>
>
>
> *From: *Matthew Patterson <mp...@aligntech.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Monday, September 23, 2019 at 2:22 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> Sorry: needed to add LOOPBACK
>
>
>
> *From: *Matthew Patterson <mp...@aligntech.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Monday, September 23, 2019 at 2:18 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> OK, now using release-2.15.0, calling word-count example with runner,
> after building & calling
>
>
>
> ./gradlew :runners:flink:1.5:job-server:runShadow (tried 1.8 as well)
>
>
>
> , as
>
>
>
> “*f'--runner=**PortableRunner**'*, *f'--job_endpoint=localhost:8099'*
>
> ”
>
> …and getting
>
>
>
> File
> "/Users/mpatterson/miniconda/envs/aws/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py",
> line 446, in wait_until_finish
>
> self._job_id, self._state, self._last_error_message()))
>
> RuntimeError: Pipeline
> BeamApp-mpatterson-0923181154-d9c8a662_737ea859-53e2-4d7e-ba94-2955a1642f94
> failed in state FAILED:
> org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException:
> CANCELLED: cancelled before receiving half close
>
>
>
> Ideas? Thanks!
>
>
>
> *From: *Matthew Patterson <mp...@aligntech.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Friday, September 20, 2019 at 9:27 AM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> 411:
>
>
>
> Although it does not seem correct, adding a symbolic link to `virtualenv`
> in ` <path-to-beam>/sdks/python` solves the issue of not finding
> `virtualenv`.
>
>
>
> *From: *Matthew Patterson <mp...@aligntech.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 2:56 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> Re Kyle:
>
>
>
> Just so you may believe me, see below(virtualenv installed, error occurs
> nonetheless): currently I am trying to figure out if it is a difference
> between .bash_profile and .bashrc.
>
>
>
> Thanks,
>
> Matt
>
>
>
> “””
>
> Caused by: net.rubygrapefruit.platform.NativeException: Could not start
> 'virtualenv'
>
> at
> net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
>
> at
> net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
>
> at
> org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:97)
>
> at
> org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:70)
>
> ... 4 more
>
> Caused by: java.io.IOException: Cannot run program "virtualenv" (in
> directory "/Users/mpatterson/dev/beam/sdks/python"): error=2, No such file
> or directory
>
> at
> net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
>
> ... 7 more
>
> Caused by: java.io.IOException: error=2, No such file or directory
>
> ... 8 more
>
>
>
>
>
> * Get more help at *https://help.gradle.org
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.gradle.org&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759377266&sdata=ZL7jkJnzVSvtRAKMtbs65fGiWW39fedlgmdOel0OGF0%3D&reserved=0>*
>
>
>
> Deprecated Gradle features were used in this build, making it incompatible
> with Gradle 6.0.
>
> Use '--warning-mode all' to show the individual deprecation warnings.
>
> See
> https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.gradle.org%2F5.2.1%2Fuserguide%2Fcommand_line_interface.html%23sec%3Acommand_line_warnings&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759387263&sdata=lE%2BAqbgitqeDTEu2gN5IqMDcfOJofz1K0ojyZPLineE%3D&reserved=0>
>
>
>
> *BUILD FAILED* in 0s
>
> 5 actionable tasks: 3 executed, 2 up-to-date
>
> bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
>
> MANIFEST.in build.gradle
> gen_protos.py setup.py tox.ini
>
> OWNERS container
> scripts test-suites
>
> apache_beam findSupportedPython.groovy
> setup.cfg test_config.py
>
> bash-3.2$ pip install virtualenv
>
> Requirement already satisfied: virtualenv in
> /Users/mpatterson/miniconda/lib/python3.7/site-packages (16.7.5)
>
> bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
>
> MANIFEST.in build.gradle
> gen_protos.py setup.py tox.ini
>
> OWNERS container scripts
> test-suites
>
> apache_beam findSupportedPython.groovy
> setup.cfg test_config.py
>
> bash-3.2$
>
>
>
> ”””
>
>
>
> *From: *Matthew Patterson <mp...@aligntech.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 2:31 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> Have done: hard to get gradle to respect env (conda? Bash? Fish? Zsh?):
>
>
>
> *From: *Kyle Weaver <kc...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 2:29 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> I'm guessing you need to install virtualenv: `pip install virtualenv`
>
>
> Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759387263&sdata=mB1tJqnT6v%2BT4QRSNn3w7pC%2BqR5d9DCFx49tybSyZk0%3D&reserved=0>
> | kcweaver@google.com
>
>
>
>
>
> On Thu, Sep 19, 2019 at 11:27 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Kyle,
>
>
>
> Excellent, will do: unfortunately switch to 2.16 was only thing that fixed
>
>
>
> FAILURE: Build failed with an exception.
>
>
>
> * What went wrong:
>
> Execution failed for task ':sdks:python:setupVirtualenv'.
>
> > A problem occurred starting process 'command 'virtualenv''
>
>
>
>
>
> …if you have suggestions there, I would be most appreciative.
>
> Matt
>
>
>
>
>
> *From: *Kyle Weaver <kc...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 2:25 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> You should probably use 2.15, since 2.16 release artifacts have not been
> published yet. Just follow the instructions that
> say --runner=PortableRunner, not --runner=FlinkRunner, otherwise you'll hit
> that other deserialization bug that was mentioned..
>
>
> Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759397256&sdata=ea4%2F99%2BY8KZKdt3LJfyJXBScDe4oWeR60ZpeKc3zBM0%3D&reserved=0>
> | kcweaver@google.com
>
>
>
>
>
> On Thu, Sep 19, 2019 at 11:19 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Kyle,
>
>
>
> Happily: should I be working from (source) release-2.15.0 or
> release-2.16.0: presumably still specifying flink-1.8.
>
> Matt
>
>
>
> *From: *Kyle Weaver <kc...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 2:16 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> Re Benjamin & Ankur: I don't think `--experiments=beam_fn_api` applies
> here. If I understand correctly, Matthew is just trying to run an
> old-fashioned Beam Java jar, nothing to do with portability/Python, and
> judging by the stack trace provided,
> https://issues.apache.org/jira/browse/BEAM-8037
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-8037&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759407250&sdata=C6R%2Fir5a9%2BFFL69RisTiQq0TgRpoUg8M6oDH6WZ%2B6VM%3D&reserved=0> is
> not the issue here.
>
>
>
> Re Matthew: I suspect a Flink version mismatch between the Flink cluster
> and the Beam runner. However, I am not familiar with the Maven commands
> you're using. Since you're planning on running Python eventually anyway,
> could you try following the "portable"/Python instructions on
> https://beam.apache.org/documentation/runners/flink/
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Fdocumentation%2Frunners%2Fflink%2F&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759407250&sdata=jme4hrasy7l293aC6tyrg1w0kHj6Ax8t1zIU9fP%2BfeI%3D&reserved=0> and
> let us know how that goes?
>
>
> Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759417243&sdata=4%2FFNrCIWrYZv7UdgtRrqoL0zHWDce3%2BbZ%2B35nYuGRM0%3D&reserved=0>
> | kcweaver@google.com
>
>
>
>
>
> On Thu, Sep 19, 2019 at 10:52 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Thanks Ankur,
>
>
>
> As one who speaks almost no gradle, is there a preferred way to get gradle
> to respect the conda configured python on `gradlew build` ?
>
>
>
> Matt
>
>
>
> *From: *Ankur Goenka <go...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 1:50 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> We have this bug in 2.15 which is discussed here
> https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759417243&sdata=zQ9x6Y53O4zfBWAKoh%2BB9UAkdR2N9zPvuak3Tl7bpT0%3D&reserved=0>
>
>
>
> For now, please add "--experiments=beam_fn_api" to your pipeline to make
> it work.
>
>
>
> On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Hi Ankur,
>
>
>
> Yes, I was using 2.15, but was getting failure to deserialize.
>
>
>
> Thanks,
>
> Matt
>
>
>
> *From: *Ankur Goenka <go...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Wednesday, September 18, 2019 at 9:34 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> Hi Matthew,
>
>
>
> Beam 2.16.0 is not yet released hence you are getting the error. Can you
> try using 2.15.0 version?
>
>
>
> Thanks,
>
> Ankur
>
>
>
> On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Tried
>
> "
> mvn archetype:generate \
> -DarchetypeGroupId=org.apache.beam \
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> -DarchetypeVersion=2.16.0 \
> -DgroupId=org.example \
> -DartifactId=word-count-beam \
> -Dversion="0.1" \
> -Dpackage=org.apache.beam.examples \
> -DinteractiveMode=false
> "
> got
>
> -> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate
> (default-cli) on project standalone-pom: The desired archetype does not
> exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) ->
> [Help 1]
>
>
> On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com> wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
>
> Ah. So maybe try 2.16.0?
>
> On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com> wrote:
> > --experiments=beam_fn_api doesn't apply here, as this is a Java
> pipeline
> > using the non-portable version of the Flink runner.
> >
> > Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759427242&sdata=i2gvTK2I0Nysq%2FBD2%2B8IJNcTw9yojDU4wPMp8%2Fgd%2BUg%3D&reserved=0>
> | kcweaver@google.com
> >
> >
> > On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <
> benjamintanweihao@gmail.com>
> > wrote:
> >
> > > Could you try adding "--experiments=beam_fn_api" to the runner
> options?
> > > Kyle did mention that it should be fixed in 2.16.0.
> > >
> > > On 2019/09/17 16:40:41, Matthew Patterson <
> mpatterson@aligntech.com>
> > > wrote:
> > > > Folks,
> > > >
> > > > Am sure I am doing it wrong, but have been beating head against
> wall
> > > long enough I need to ask for help.
> > > >
> > > > My goal is to use the python SDK to run jobs on AWS flink
> cluster, but
> > > very little is working, so I am trying to back down to the very
> simplest
> > > thing that works, and build back up: not going so well.
> > > >
> > > > Now I am running, or trying to, on a minikube link cluster. Now
> running
> > > from just java/mvn, so I go through the following:
> > > >
> > > > 1. bring-up-cluster (see below)
> > > > 2. grab-the-code (see below)
> > > > 3. maven-submit (see below)
> > > >
> > > > Eventual failure is
> > > >
> > > > “[ERROR] Failed to execute goal
> > > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on
> project
> > > word-count-beam: An exception occured while executing the Java
> class.
> > > Pipeline execution failed: Could not retrieve the execution
> result. (JobID:
> > > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph.
> [Failed to
> > > deserialize JobGraph.]”
> > > >
> > > > and log is attached (I hope)
> > > >
> > > > Other details:
> > > >
> > > > on osx
> > > >
> > > >
> > > >
> > > > ➜ minikube version
> > > >
> > > > minikube version: v1.3.1
> > > >
> > > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > > >
> > > >
> > > > ➜ kubectl version
> > > >
> > > > Client Version: version.Info{Major:"1", Minor:"14",
> > > GitVersion:"v1.14.6",
> GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > > >
> > > > Server Version: version.Info{Major:"1", Minor:"15",
> > > GitVersion:"v1.15.2",
> GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > > >
> > > >
> > > > ➜ flink --version
> > > >
> > > > Version: 1.8.1, Commit ID: 7297bac
> > > >
> > > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > > 2.15.0
> > > >
> > > > ➜ curl localhost:8081
> > > > <!--
> > > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > > ~ or more contributor license agreements. See the NOTICE file
> > > > ~ distributed with this work for additional information
> > > > ~ regarding copyright ownership. The ASF licenses this file
> > > > ~ to you under the Apache License, Version 2.0 (the
> > > > ~ "License"); you may not use this file except in compliance
> > > > ~ with the License. You may obtain a copy of the License at
> > > > ~
> > > > ~
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759437240&sdata=x1UpsfgZh699aMI89ZcwBPky2mC2mp2jBkKHkOTIm7Y%3D&reserved=0>
> > > > ~
> > > > ~ Unless required by applicable law or agreed to in writing,
> software
> > > > ~ distributed under the License is distributed on an "AS IS"
> BASIS,
> > > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > > implied.
> > > > ~ See the License for the specific language governing
> permissions and
> > > > ~ limitations under the License.
> > > > -->
> > > >
> > > > <!doctype html>
> > > > <html lang="en">
> > > > <head>
> > > > <meta charset="utf-8">
> > > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > > <meta name="viewport" content="width=device-width,
> initial-scale=1">
> > > > <title>Apache Flink Web Dashboard</title>
> > > > <link rel="apple-touch-icon" sizes="180x180"
> > > href="assets/favicon/apple-touch-icon.png">
> > > > <link rel="icon" type="image/png"
> > > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > > <link rel="icon" type="image/png"
> > > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > > <link rel="mask-icon"
> href="assets/favicon/safari-pinned-tab.svg"
> > > color="#aa1919">
> > > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > > <meta name="msapplication-config"
> > > content="assets/favicon/browserconfig.xml">
> > > > <meta name="theme-color" content="#ffffff">
> > > > <base href="./"><link rel="stylesheet"
> > > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > > <body>
> > > > <flink-root></flink-root>
> > > > <script type="text/javascript"
> > > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > > type="text/javascript"
> src="es2015-polyfills.5e343224e81eefb7658e.js"
> > > nomodule></script><script type="text/javascript"
> > > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > > type="text/javascript"
> src="main.dc6e0997d092df051db0.js"></script></body>
> > > > </html>
> > > >
> > > > Thanks for any ideas,
> > > > Matt
> > > >
> > > > >>> bring-up-cluster
> > > >
> > > > # [yamls from here](
> > >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759437240&sdata=QaFx9%2BoIs8EoaxOzyzNlgKQwmajQAuVZG%2BWR5uaSuSM%3D&reserved=0>
> > > )
> > > > minikube start
> > > > minikube ssh 'sudo ip link set docker0 promisc on'
> > > >
> > > > kubectl create -f flink-configuration-configmap.yaml
> > > > kubectl create -f jobmanager-service.yaml
> > > > kubectl create -f jobmanager-deployment.yaml
> > > > kubectl create -f taskmanager-deployment.yaml
> > > > kubectl apply -f jobmanager-rest-service.yaml
> > > >
> > > > >>> grab-the-code
> > > > mvn archetype:generate \
> > > > -DarchetypeGroupId=org.apache.beam \
> > > >
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > > -DarchetypeVersion=2.15.0 \
> > > > -DgroupId=org.example \
> > > > -DartifactId=word-count-beam \
> > > > -Dversion="0.1" \
> > > > -Dpackage=org.apache.beam.examples \
> > > > -DinteractiveMode=false
> > > >
> > > > >>> maven-submit
> > > >
> > > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in
> separate
> > > shell, as blocking
> > > > mvn package exec:java -X
> > > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > > -Dexec.args="--runner=FlinkRunner\
> > > > --flinkMaster="localhost:8081"\
> > > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > > --output=<directory-that-exists>" -Pflink-runner
> > > >
> > > >
> > > >
> > >
> >
>
>
Re: Word-count example
Posted by Matthew Patterson <mp...@aligntech.com>.
OK: after restarting & following directions better, paying attention to versions/branches/runner-option-flags, I am able to get jobs submitted (submitted: not running to fruition) to Flink cluster (AWS EMR). Further, I can get the basic example (a slight riff on word count example) to work using loopback environment. Now, to scope Beam-on-Flink for production, I want to get jobs on cluster to finish successfully.
Currently, I am working through this:
“Flink Deployment
The Flink cluster itself is deployed as normal. For example, it might be deployed on a standalone Flink cluster, on Kubernetes, or on YARN. The only requirement is that TaskManager nodes must have access to Docker.
”
from [Flink Portable Runner Overview](https://docs.google.com/document/d/17hL9h_q9aX6dA4lzX5IcV-AgdduxDcj3xTzjbLJyJoQ/edit#)
Sorry if/that this is stupid question, but is there a preferred method to fortify Flink taskmanger with docker? Docker in docker (dind)? Just install in image? Docker-out-of-docker? Any working example folks could point me to would be great.
Thanks All,
Matt
[dind?]( https://medium.com/hootsuite-engineering/building-docker-images-inside-kubernetes-42c6af855f25)
From: Kyle Weaver <kc...@google.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Monday, September 23, 2019 at 4:19 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
If there are no jobs or logs appearing in the Flink UI, it is likely that the pipeline is not being submitted to the Flink cluster successfully. (Right now, there is no timeout on pipeline submission, but I would like to see that changed; see https://issues.apache.org/jira/browse/BEAM-7933<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-7933&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759357273&sdata=pYz%2FPe1nxq%2BXG5YvNOa3JmlIgZYB3h4OtBYd4tXHddQ%3D&reserved=0>.)
I would start by making sure there are no unneeded Flink clusters, job servers, etc. left running on your machine, as port conflicts can cause silent failures.
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759367268&sdata=ttv1XygS09045JCmq0%2BNfw9PGW8504YQhzVpFVxT9%2Bc%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Mon, Sep 23, 2019 at 12:58 PM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle, right: saw that. Grabbing at anything.
The process on the client seems to submit, or at least not bomb, but the cluster does not acknowledge a running job. It is possible I have not waited long enough yet, and images are still being downloaded & launched.
2.16 section on FlinkRunner does not have (that I found) help on gathering logs: might you point me at that as well?
Thanks for the help, Kyle.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Monday, September 23, 2019 at 3:49 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
"WARNING:root:No unique name set for transform ..." should not affect the pipeline's ability to complete successfully. Is the pipeline failing? If so, could you share more logs?
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759367268&sdata=ttv1XygS09045JCmq0%2BNfw9PGW8504YQhzVpFVxT9%2Bc%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Mon, Sep 23, 2019 at 11:43 AM Matthew Patterson <mp...@aligntech.com>> wrote:
After setting runner, adding '--environment_type=LOOPBACK', I now get
“
…
WARNING:root:No unique name set for transform fn/read/ref_PCollection_PCollection_17:0
WARNING:root:No unique name set for transform fn/read/ref_PCollection_PCollection_17:0
WARNING:root:No unique name set for transform fn/write/ref_PCollection_PCollection_24:0
…
”
when running wordcount example:
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = runner_options
pipeline_options.view_as(SetupOptions).save_main_session = True
p = beam.Pipeline(options=pipeline_options)
# Read the text file[pattern] into a PCollection.
lines = p | 'read' >> ReadFromText(_input)
# Count the occurrences of each word.
def count_ones(word_ones):
(word, ones) = word_ones
return (word, sum(ones))
counts = (lines
| 'split' >> (beam.ParDo(WordExtractingDoFn()).with_output_types(unicode))
| 'pair_with_one' >> beam.Map(lambda x: (x, 1))
| 'group' >> beam.GroupByKey()
| 'count' >> beam.Map(count_ones))
# Format the counts into a PCollection of strings.
def format_result(word_count):
(word, count) = word_count
return '%s: %d' % (word, count)
output = counts | 'format' >> beam.Map(format_result)
# Write the output using a "Write" transform that has side effects.
# pylint: disable=expression-not-assigned
output | 'write' >> WriteToText(_output)
result = p.run()
result.wait_until_finish()
From: Matthew Patterson <mp...@aligntech.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Monday, September 23, 2019 at 2:22 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
Sorry: needed to add LOOPBACK
From: Matthew Patterson <mp...@aligntech.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Monday, September 23, 2019 at 2:18 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
OK, now using release-2.15.0, calling word-count example with runner, after building & calling
./gradlew :runners:flink:1.5:job-server:runShadow (tried 1.8 as well)
, as
“f'--runner=PortableRunner', f'--job_endpoint=localhost:8099'
”
…and getting
File "/Users/mpatterson/miniconda/envs/aws/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 446, in wait_until_finish
self._job_id, self._state, self._last_error_message()))
RuntimeError: Pipeline BeamApp-mpatterson-0923181154-d9c8a662_737ea859-53e2-4d7e-ba94-2955a1642f94 failed in state FAILED: org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
Ideas? Thanks!
From: Matthew Patterson <mp...@aligntech.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Friday, September 20, 2019 at 9:27 AM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
411:
Although it does not seem correct, adding a symbolic link to `virtualenv` in ` <path-to-beam>/sdks/python` solves the issue of not finding `virtualenv`.
From: Matthew Patterson <mp...@aligntech.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:56 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
Re Kyle:
Just so you may believe me, see below(virtualenv installed, error occurs nonetheless): currently I am trying to figure out if it is a difference between .bash_profile and .bashrc.
Thanks,
Matt
“””
Caused by: net.rubygrapefruit.platform.NativeException: Could not start 'virtualenv'
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
at net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
at org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:97)
at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:70)
... 4 more
Caused by: java.io.IOException: Cannot run program "virtualenv" (in directory "/Users/mpatterson/dev/beam/sdks/python"): error=2, No such file or directory
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
... 8 more
* Get more help at https://help.gradle.org<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.gradle.org&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759377266&sdata=ZL7jkJnzVSvtRAKMtbs65fGiWW39fedlgmdOel0OGF0%3D&reserved=0>
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.gradle.org%2F5.2.1%2Fuserguide%2Fcommand_line_interface.html%23sec%3Acommand_line_warnings&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759387263&sdata=lE%2BAqbgitqeDTEu2gN5IqMDcfOJofz1K0ojyZPLineE%3D&reserved=0>
BUILD FAILED in 0s
5 actionable tasks: 3 executed, 2 up-to-date
bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
MANIFEST.in build.gradle gen_protos.py setup.py tox.ini
OWNERS container scripts test-suites
apache_beam findSupportedPython.groovy setup.cfg test_config.py
bash-3.2$ pip install virtualenv
Requirement already satisfied: virtualenv in /Users/mpatterson/miniconda/lib/python3.7/site-packages (16.7.5)
bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
MANIFEST.in build.gradle gen_protos.py setup.py tox.ini
OWNERS container scripts test-suites
apache_beam findSupportedPython.groovy setup.cfg test_config.py
bash-3.2$
”””
From: Matthew Patterson <mp...@aligntech.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:31 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
Have done: hard to get gradle to respect env (conda? Bash? Fish? Zsh?):
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:29 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I'm guessing you need to install virtualenv: `pip install virtualenv`
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759387263&sdata=mB1tJqnT6v%2BT4QRSNn3w7pC%2BqR5d9DCFx49tybSyZk0%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 11:27 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle,
Excellent, will do: unfortunately switch to 2.16 was only thing that fixed
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:setupVirtualenv'.
> A problem occurred starting process 'command 'virtualenv''
…if you have suggestions there, I would be most appreciative.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:25 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
You should probably use 2.15, since 2.16 release artifacts have not been published yet. Just follow the instructions that say --runner=PortableRunner, not --runner=FlinkRunner, otherwise you'll hit that other deserialization bug that was mentioned..
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759397256&sdata=ea4%2F99%2BY8KZKdt3LJfyJXBScDe4oWeR60ZpeKc3zBM0%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 11:19 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle,
Happily: should I be working from (source) release-2.15.0 or release-2.16.0: presumably still specifying flink-1.8.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:16 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re Benjamin & Ankur: I don't think `--experiments=beam_fn_api` applies here. If I understand correctly, Matthew is just trying to run an old-fashioned Beam Java jar, nothing to do with portability/Python, and judging by the stack trace provided, https://issues.apache.org/jira/browse/BEAM-8037<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-8037&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759407250&sdata=C6R%2Fir5a9%2BFFL69RisTiQq0TgRpoUg8M6oDH6WZ%2B6VM%3D&reserved=0> is not the issue here.
Re Matthew: I suspect a Flink version mismatch between the Flink cluster and the Beam runner. However, I am not familiar with the Maven commands you're using. Since you're planning on running Python eventually anyway, could you try following the "portable"/Python instructions on https://beam.apache.org/documentation/runners/flink/<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Fdocumentation%2Frunners%2Fflink%2F&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759407250&sdata=jme4hrasy7l293aC6tyrg1w0kHj6Ax8t1zIU9fP%2BfeI%3D&reserved=0> and let us know how that goes?
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759417243&sdata=4%2FFNrCIWrYZv7UdgtRrqoL0zHWDce3%2BbZ%2B35nYuGRM0%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 10:52 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Thanks Ankur,
As one who speaks almost no gradle, is there a preferred way to get gradle to respect the conda configured python on `gradlew build` ?
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 1:50 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
We have this bug in 2.15 which is discussed here https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759417243&sdata=zQ9x6Y53O4zfBWAKoh%2BB9UAkdR2N9zPvuak3Tl7bpT0%3D&reserved=0>
For now, please add "--experiments=beam_fn_api" to your pipeline to make it work.
On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Hi Ankur,
Yes, I was using 2.15, but was getting failure to deserialize.
Thanks,
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Wednesday, September 18, 2019 at 9:34 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Matthew,
Beam 2.16.0 is not yet released hence you are getting the error. Can you try using 2.15.0 version?
Thanks,
Ankur
On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Tried
"
mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
-DarchetypeVersion=2.16.0 \
-DgroupId=org.example \
-DartifactId=word-count-beam \
-Dversion="0.1" \
-Dpackage=org.apache.beam.examples \
-DinteractiveMode=false
"
got
-> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate (default-cli) on project standalone-pom: The desired archetype does not exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) -> [Help 1]
On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com>> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Ah. So maybe try 2.16.0?
On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com>> wrote:
> --experiments=beam_fn_api doesn't apply here, as this is a Java pipeline
> using the non-portable version of the Flink runner.
>
> Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759427242&sdata=i2gvTK2I0Nysq%2FBD2%2B8IJNcTw9yojDU4wPMp8%2Fgd%2BUg%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
>
>
> On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <be...@gmail.com>>
> wrote:
>
> > Could you try adding "--experiments=beam_fn_api" to the runner options?
> > Kyle did mention that it should be fixed in 2.16.0.
> >
> > On 2019/09/17 16:40:41, Matthew Patterson <mp...@aligntech.com>>
> > wrote:
> > > Folks,
> > >
> > > Am sure I am doing it wrong, but have been beating head against wall
> > long enough I need to ask for help.
> > >
> > > My goal is to use the python SDK to run jobs on AWS flink cluster, but
> > very little is working, so I am trying to back down to the very simplest
> > thing that works, and build back up: not going so well.
> > >
> > > Now I am running, or trying to, on a minikube link cluster. Now running
> > from just java/mvn, so I go through the following:
> > >
> > > 1. bring-up-cluster (see below)
> > > 2. grab-the-code (see below)
> > > 3. maven-submit (see below)
> > >
> > > Eventual failure is
> > >
> > > “[ERROR] Failed to execute goal
> > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project
> > word-count-beam: An exception occured while executing the Java class.
> > Pipeline execution failed: Could not retrieve the execution result. (JobID:
> > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to
> > deserialize JobGraph.]”
> > >
> > > and log is attached (I hope)
> > >
> > > Other details:
> > >
> > > on osx
> > >
> > >
> > >
> > > ➜ minikube version
> > >
> > > minikube version: v1.3.1
> > >
> > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > >
> > >
> > > ➜ kubectl version
> > >
> > > Client Version: version.Info{Major:"1", Minor:"14",
> > GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > >
> > > Server Version: version.Info{Major:"1", Minor:"15",
> > GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > >
> > >
> > > ➜ flink --version
> > >
> > > Version: 1.8.1, Commit ID: 7297bac
> > >
> > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > 2.15.0
> > >
> > > ➜ curl localhost:8081
> > > <!--
> > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > ~ or more contributor license agreements. See the NOTICE file
> > > ~ distributed with this work for additional information
> > > ~ regarding copyright ownership. The ASF licenses this file
> > > ~ to you under the Apache License, Version 2.0 (the
> > > ~ "License"); you may not use this file except in compliance
> > > ~ with the License. You may obtain a copy of the License at
> > > ~
> > > ~ https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759437240&sdata=x1UpsfgZh699aMI89ZcwBPky2mC2mp2jBkKHkOTIm7Y%3D&reserved=0>
> > > ~
> > > ~ Unless required by applicable law or agreed to in writing, software
> > > ~ distributed under the License is distributed on an "AS IS" BASIS,
> > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > implied.
> > > ~ See the License for the specific language governing permissions and
> > > ~ limitations under the License.
> > > -->
> > >
> > > <!doctype html>
> > > <html lang="en">
> > > <head>
> > > <meta charset="utf-8">
> > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > <meta name="viewport" content="width=device-width, initial-scale=1">
> > > <title>Apache Flink Web Dashboard</title>
> > > <link rel="apple-touch-icon" sizes="180x180"
> > href="assets/favicon/apple-touch-icon.png">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > <link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg"
> > color="#aa1919">
> > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > <meta name="msapplication-config"
> > content="assets/favicon/browserconfig.xml">
> > > <meta name="theme-color" content="#ffffff">
> > > <base href="./"><link rel="stylesheet"
> > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > <body>
> > > <flink-root></flink-root>
> > > <script type="text/javascript"
> > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js"
> > nomodule></script><script type="text/javascript"
> > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > type="text/javascript" src="main.dc6e0997d092df051db0.js"></script></body>
> > > </html>
> > >
> > > Thanks for any ideas,
> > > Matt
> > >
> > > >>> bring-up-cluster
> > >
> > > # [yamls from here](
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759437240&sdata=QaFx9%2BoIs8EoaxOzyzNlgKQwmajQAuVZG%2BWR5uaSuSM%3D&reserved=0>
> > )
> > > minikube start
> > > minikube ssh 'sudo ip link set docker0 promisc on'
> > >
> > > kubectl create -f flink-configuration-configmap.yaml
> > > kubectl create -f jobmanager-service.yaml
> > > kubectl create -f jobmanager-deployment.yaml
> > > kubectl create -f taskmanager-deployment.yaml
> > > kubectl apply -f jobmanager-rest-service.yaml
> > >
> > > >>> grab-the-code
> > > mvn archetype:generate \
> > > -DarchetypeGroupId=org.apache.beam \
> > > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > -DarchetypeVersion=2.15.0 \
> > > -DgroupId=org.example \
> > > -DartifactId=word-count-beam \
> > > -Dversion="0.1" \
> > > -Dpackage=org.apache.beam.examples \
> > > -DinteractiveMode=false
> > >
> > > >>> maven-submit
> > >
> > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in separate
> > shell, as blocking
> > > mvn package exec:java -X
> > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > -Dexec.args="--runner=FlinkRunner\
> > > --flinkMaster="localhost:8081"\
> > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > --output=<directory-that-exists>" -Pflink-runner
> > >
> > >
> > >
> >
>
Re: Word-count example
Posted by Matthew Patterson <mp...@aligntech.com>.
Nice diagnosis, Kyle.
…was not going to think of that, thank you.
sudo lsof -i -n -P | grep 8081
firefox 2018 mpatterson 115u IPv4 0xc79caf16386**** 0t0 TCP 192.168.2.107:55582->18.209.126.59:8081 (ESTABLISHED)
firefox 2018 mpatterson 234u IPv4 0xc79caf164d6**** 0t0 TCP 10.30.32.214:56132->54.144.67.115:8081 (ESTABLISHED)
On to the next thing!
From: Kyle Weaver <kc...@google.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Monday, September 23, 2019 at 4:19 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
If there are no jobs or logs appearing in the Flink UI, it is likely that the pipeline is not being submitted to the Flink cluster successfully. (Right now, there is no timeout on pipeline submission, but I would like to see that changed; see https://issues.apache.org/jira/browse/BEAM-7933<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-7933&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759357273&sdata=pYz%2FPe1nxq%2BXG5YvNOa3JmlIgZYB3h4OtBYd4tXHddQ%3D&reserved=0>.)
I would start by making sure there are no unneeded Flink clusters, job servers, etc. left running on your machine, as port conflicts can cause silent failures.
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759367268&sdata=ttv1XygS09045JCmq0%2BNfw9PGW8504YQhzVpFVxT9%2Bc%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Mon, Sep 23, 2019 at 12:58 PM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle, right: saw that. Grabbing at anything.
The process on the client seems to submit, or at least not bomb, but the cluster does not acknowledge a running job. It is possible I have not waited long enough yet, and images are still being downloaded & launched.
2.16 section on FlinkRunner does not have (that I found) help on gathering logs: might you point me at that as well?
Thanks for the help, Kyle.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Monday, September 23, 2019 at 3:49 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
"WARNING:root:No unique name set for transform ..." should not affect the pipeline's ability to complete successfully. Is the pipeline failing? If so, could you share more logs?
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759367268&sdata=ttv1XygS09045JCmq0%2BNfw9PGW8504YQhzVpFVxT9%2Bc%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Mon, Sep 23, 2019 at 11:43 AM Matthew Patterson <mp...@aligntech.com>> wrote:
After setting runner, adding '--environment_type=LOOPBACK', I now get
“
…
WARNING:root:No unique name set for transform fn/read/ref_PCollection_PCollection_17:0
WARNING:root:No unique name set for transform fn/read/ref_PCollection_PCollection_17:0
WARNING:root:No unique name set for transform fn/write/ref_PCollection_PCollection_24:0
…
”
when running wordcount example:
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = runner_options
pipeline_options.view_as(SetupOptions).save_main_session = True
p = beam.Pipeline(options=pipeline_options)
# Read the text file[pattern] into a PCollection.
lines = p | 'read' >> ReadFromText(_input)
# Count the occurrences of each word.
def count_ones(word_ones):
(word, ones) = word_ones
return (word, sum(ones))
counts = (lines
| 'split' >> (beam.ParDo(WordExtractingDoFn()).with_output_types(unicode))
| 'pair_with_one' >> beam.Map(lambda x: (x, 1))
| 'group' >> beam.GroupByKey()
| 'count' >> beam.Map(count_ones))
# Format the counts into a PCollection of strings.
def format_result(word_count):
(word, count) = word_count
return '%s: %d' % (word, count)
output = counts | 'format' >> beam.Map(format_result)
# Write the output using a "Write" transform that has side effects.
# pylint: disable=expression-not-assigned
output | 'write' >> WriteToText(_output)
result = p.run()
result.wait_until_finish()
From: Matthew Patterson <mp...@aligntech.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Monday, September 23, 2019 at 2:22 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
Sorry: needed to add LOOPBACK
From: Matthew Patterson <mp...@aligntech.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Monday, September 23, 2019 at 2:18 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
OK, now using release-2.15.0, calling word-count example with runner, after building & calling
./gradlew :runners:flink:1.5:job-server:runShadow (tried 1.8 as well)
, as
“f'--runner=PortableRunner', f'--job_endpoint=localhost:8099'
”
…and getting
File "/Users/mpatterson/miniconda/envs/aws/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 446, in wait_until_finish
self._job_id, self._state, self._last_error_message()))
RuntimeError: Pipeline BeamApp-mpatterson-0923181154-d9c8a662_737ea859-53e2-4d7e-ba94-2955a1642f94 failed in state FAILED: org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
Ideas? Thanks!
From: Matthew Patterson <mp...@aligntech.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Friday, September 20, 2019 at 9:27 AM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
411:
Although it does not seem correct, adding a symbolic link to `virtualenv` in ` <path-to-beam>/sdks/python` solves the issue of not finding `virtualenv`.
From: Matthew Patterson <mp...@aligntech.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:56 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
Re Kyle:
Just so you may believe me, see below(virtualenv installed, error occurs nonetheless): currently I am trying to figure out if it is a difference between .bash_profile and .bashrc.
Thanks,
Matt
“””
Caused by: net.rubygrapefruit.platform.NativeException: Could not start 'virtualenv'
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
at net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
at org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:97)
at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:70)
... 4 more
Caused by: java.io.IOException: Cannot run program "virtualenv" (in directory "/Users/mpatterson/dev/beam/sdks/python"): error=2, No such file or directory
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
... 8 more
* Get more help at https://help.gradle.org<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.gradle.org&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759377266&sdata=ZL7jkJnzVSvtRAKMtbs65fGiWW39fedlgmdOel0OGF0%3D&reserved=0>
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.gradle.org%2F5.2.1%2Fuserguide%2Fcommand_line_interface.html%23sec%3Acommand_line_warnings&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759387263&sdata=lE%2BAqbgitqeDTEu2gN5IqMDcfOJofz1K0ojyZPLineE%3D&reserved=0>
BUILD FAILED in 0s
5 actionable tasks: 3 executed, 2 up-to-date
bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
MANIFEST.in build.gradle gen_protos.py setup.py tox.ini
OWNERS container scripts test-suites
apache_beam findSupportedPython.groovy setup.cfg test_config.py
bash-3.2$ pip install virtualenv
Requirement already satisfied: virtualenv in /Users/mpatterson/miniconda/lib/python3.7/site-packages (16.7.5)
bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
MANIFEST.in build.gradle gen_protos.py setup.py tox.ini
OWNERS container scripts test-suites
apache_beam findSupportedPython.groovy setup.cfg test_config.py
bash-3.2$
”””
From: Matthew Patterson <mp...@aligntech.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:31 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
Have done: hard to get gradle to respect env (conda? Bash? Fish? Zsh?):
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:29 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I'm guessing you need to install virtualenv: `pip install virtualenv`
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759387263&sdata=mB1tJqnT6v%2BT4QRSNn3w7pC%2BqR5d9DCFx49tybSyZk0%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 11:27 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle,
Excellent, will do: unfortunately switch to 2.16 was only thing that fixed
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:setupVirtualenv'.
> A problem occurred starting process 'command 'virtualenv''
…if you have suggestions there, I would be most appreciative.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:25 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
You should probably use 2.15, since 2.16 release artifacts have not been published yet. Just follow the instructions that say --runner=PortableRunner, not --runner=FlinkRunner, otherwise you'll hit that other deserialization bug that was mentioned..
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759397256&sdata=ea4%2F99%2BY8KZKdt3LJfyJXBScDe4oWeR60ZpeKc3zBM0%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 11:19 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle,
Happily: should I be working from (source) release-2.15.0 or release-2.16.0: presumably still specifying flink-1.8.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:16 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re Benjamin & Ankur: I don't think `--experiments=beam_fn_api` applies here. If I understand correctly, Matthew is just trying to run an old-fashioned Beam Java jar, nothing to do with portability/Python, and judging by the stack trace provided, https://issues.apache.org/jira/browse/BEAM-8037<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-8037&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759407250&sdata=C6R%2Fir5a9%2BFFL69RisTiQq0TgRpoUg8M6oDH6WZ%2B6VM%3D&reserved=0> is not the issue here.
Re Matthew: I suspect a Flink version mismatch between the Flink cluster and the Beam runner. However, I am not familiar with the Maven commands you're using. Since you're planning on running Python eventually anyway, could you try following the "portable"/Python instructions on https://beam.apache.org/documentation/runners/flink/<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Fdocumentation%2Frunners%2Fflink%2F&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759407250&sdata=jme4hrasy7l293aC6tyrg1w0kHj6Ax8t1zIU9fP%2BfeI%3D&reserved=0> and let us know how that goes?
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759417243&sdata=4%2FFNrCIWrYZv7UdgtRrqoL0zHWDce3%2BbZ%2B35nYuGRM0%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 10:52 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Thanks Ankur,
As one who speaks almost no gradle, is there a preferred way to get gradle to respect the conda configured python on `gradlew build` ?
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 1:50 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
We have this bug in 2.15 which is discussed here https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759417243&sdata=zQ9x6Y53O4zfBWAKoh%2BB9UAkdR2N9zPvuak3Tl7bpT0%3D&reserved=0>
For now, please add "--experiments=beam_fn_api" to your pipeline to make it work.
On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Hi Ankur,
Yes, I was using 2.15, but was getting failure to deserialize.
Thanks,
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Wednesday, September 18, 2019 at 9:34 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Matthew,
Beam 2.16.0 is not yet released hence you are getting the error. Can you try using 2.15.0 version?
Thanks,
Ankur
On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Tried
"
mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
-DarchetypeVersion=2.16.0 \
-DgroupId=org.example \
-DartifactId=word-count-beam \
-Dversion="0.1" \
-Dpackage=org.apache.beam.examples \
-DinteractiveMode=false
"
got
-> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate (default-cli) on project standalone-pom: The desired archetype does not exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) -> [Help 1]
On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com>> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Ah. So maybe try 2.16.0?
On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com>> wrote:
> --experiments=beam_fn_api doesn't apply here, as this is a Java pipeline
> using the non-portable version of the Flink runner.
>
> Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759427242&sdata=i2gvTK2I0Nysq%2FBD2%2B8IJNcTw9yojDU4wPMp8%2Fgd%2BUg%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
>
>
> On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <be...@gmail.com>>
> wrote:
>
> > Could you try adding "--experiments=beam_fn_api" to the runner options?
> > Kyle did mention that it should be fixed in 2.16.0.
> >
> > On 2019/09/17 16:40:41, Matthew Patterson <mp...@aligntech.com>>
> > wrote:
> > > Folks,
> > >
> > > Am sure I am doing it wrong, but have been beating head against wall
> > long enough I need to ask for help.
> > >
> > > My goal is to use the python SDK to run jobs on AWS flink cluster, but
> > very little is working, so I am trying to back down to the very simplest
> > thing that works, and build back up: not going so well.
> > >
> > > Now I am running, or trying to, on a minikube link cluster. Now running
> > from just java/mvn, so I go through the following:
> > >
> > > 1. bring-up-cluster (see below)
> > > 2. grab-the-code (see below)
> > > 3. maven-submit (see below)
> > >
> > > Eventual failure is
> > >
> > > “[ERROR] Failed to execute goal
> > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project
> > word-count-beam: An exception occured while executing the Java class.
> > Pipeline execution failed: Could not retrieve the execution result. (JobID:
> > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to
> > deserialize JobGraph.]”
> > >
> > > and log is attached (I hope)
> > >
> > > Other details:
> > >
> > > on osx
> > >
> > >
> > >
> > > ➜ minikube version
> > >
> > > minikube version: v1.3.1
> > >
> > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > >
> > >
> > > ➜ kubectl version
> > >
> > > Client Version: version.Info{Major:"1", Minor:"14",
> > GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > >
> > > Server Version: version.Info{Major:"1", Minor:"15",
> > GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > >
> > >
> > > ➜ flink --version
> > >
> > > Version: 1.8.1, Commit ID: 7297bac
> > >
> > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > 2.15.0
> > >
> > > ➜ curl localhost:8081
> > > <!--
> > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > ~ or more contributor license agreements. See the NOTICE file
> > > ~ distributed with this work for additional information
> > > ~ regarding copyright ownership. The ASF licenses this file
> > > ~ to you under the Apache License, Version 2.0 (the
> > > ~ "License"); you may not use this file except in compliance
> > > ~ with the License. You may obtain a copy of the License at
> > > ~
> > > ~ https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759437240&sdata=x1UpsfgZh699aMI89ZcwBPky2mC2mp2jBkKHkOTIm7Y%3D&reserved=0>
> > > ~
> > > ~ Unless required by applicable law or agreed to in writing, software
> > > ~ distributed under the License is distributed on an "AS IS" BASIS,
> > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > implied.
> > > ~ See the License for the specific language governing permissions and
> > > ~ limitations under the License.
> > > -->
> > >
> > > <!doctype html>
> > > <html lang="en">
> > > <head>
> > > <meta charset="utf-8">
> > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > <meta name="viewport" content="width=device-width, initial-scale=1">
> > > <title>Apache Flink Web Dashboard</title>
> > > <link rel="apple-touch-icon" sizes="180x180"
> > href="assets/favicon/apple-touch-icon.png">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > <link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg"
> > color="#aa1919">
> > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > <meta name="msapplication-config"
> > content="assets/favicon/browserconfig.xml">
> > > <meta name="theme-color" content="#ffffff">
> > > <base href="./"><link rel="stylesheet"
> > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > <body>
> > > <flink-root></flink-root>
> > > <script type="text/javascript"
> > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js"
> > nomodule></script><script type="text/javascript"
> > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > type="text/javascript" src="main.dc6e0997d092df051db0.js"></script></body>
> > > </html>
> > >
> > > Thanks for any ideas,
> > > Matt
> > >
> > > >>> bring-up-cluster
> > >
> > > # [yamls from here](
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7Cabb0305d58fe46d786a208d7406358c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048667759437240&sdata=QaFx9%2BoIs8EoaxOzyzNlgKQwmajQAuVZG%2BWR5uaSuSM%3D&reserved=0>
> > )
> > > minikube start
> > > minikube ssh 'sudo ip link set docker0 promisc on'
> > >
> > > kubectl create -f flink-configuration-configmap.yaml
> > > kubectl create -f jobmanager-service.yaml
> > > kubectl create -f jobmanager-deployment.yaml
> > > kubectl create -f taskmanager-deployment.yaml
> > > kubectl apply -f jobmanager-rest-service.yaml
> > >
> > > >>> grab-the-code
> > > mvn archetype:generate \
> > > -DarchetypeGroupId=org.apache.beam \
> > > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > -DarchetypeVersion=2.15.0 \
> > > -DgroupId=org.example \
> > > -DartifactId=word-count-beam \
> > > -Dversion="0.1" \
> > > -Dpackage=org.apache.beam.examples \
> > > -DinteractiveMode=false
> > >
> > > >>> maven-submit
> > >
> > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in separate
> > shell, as blocking
> > > mvn package exec:java -X
> > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > -Dexec.args="--runner=FlinkRunner\
> > > --flinkMaster="localhost:8081"\
> > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > --output=<directory-that-exists>" -Pflink-runner
> > >
> > >
> > >
> >
>
Re: Word-count example
Posted by Kyle Weaver <kc...@google.com>.
If there are no jobs or logs appearing in the Flink UI, it is likely that
the pipeline is not being submitted to the Flink cluster successfully.
(Right now, there is no timeout on pipeline submission, but I would like to
see that changed; see https://issues.apache.org/jira/browse/BEAM-7933.)
I would start by making sure there are no unneeded Flink clusters, job
servers, etc. left running on your machine, as port conflicts can cause
silent failures.
Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com
On Mon, Sep 23, 2019 at 12:58 PM Matthew Patterson <mp...@aligntech.com>
wrote:
> Kyle, right: saw that. Grabbing at anything.
>
>
>
> The process on the client seems to submit, or at least not bomb, but the cluster does not acknowledge a running job. It is possible I have not waited long enough yet, and images are still being downloaded & launched.
>
>
>
> 2.16 section on FlinkRunner does not have (that I found) help on gathering
> logs: might you point me at that as well?
>
>
>
> Thanks for the help, Kyle.
>
> Matt
>
>
>
> *From: *Kyle Weaver <kc...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Monday, September 23, 2019 at 3:49 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> "WARNING:root:No unique name set for transform ..." should not affect the pipeline's ability to complete successfully. Is the pipeline failing? If so, could you share more logs?
>
>
>
> Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624449524&sdata=F5tue28BCAg712rWTOO%2F2Kvbd91lqOYsGR0%2BQ25vcrY%3D&reserved=0>
> | kcweaver@google.com
>
>
>
>
>
> On Mon, Sep 23, 2019 at 11:43 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> After setting runner, adding *'--environment_type=LOOPBACK'*, I now get
>
>
>
> “
>
> …
>
> WARNING:root:No unique name set for transform fn/read/ref_PCollection_PCollection_17:0
>
> WARNING:root:No unique name set for transform fn/read/ref_PCollection_PCollection_17:0
>
> WARNING:root:No unique name set for transform fn/write/ref_PCollection_PCollection_24:0
>
> …
>
> ”
>
> when running wordcount example:
>
>
>
>
>
> *# We use the save_main_session option because one or more DoFn's in this
> # workflow rely on global context (e.g., a module imported at module
> level). *pipeline_options = runner_options
> pipeline_options.view_as(SetupOptions).save_main_session =
>
> *True *p = beam.Pipeline(options=pipeline_options)
>
>
>
> *# Read the text file[pattern] into a PCollection. *lines = p | *'read' *>>
> ReadFromText(_input)
>
>
> *# Count the occurrences of each word. **def *count_ones(word_ones):
> (word, ones) = word_ones
> *return *(word, sum(ones))
>
> counts = (lines
> | *'split' *>>
> (beam.ParDo(WordExtractingDoFn()).with_output_types(unicode))
> | *'pair_with_one' *>> beam.Map(*lambda *x: (x, 1))
> | *'group' *>> beam.GroupByKey()
> | *'count' *>> beam.Map(count_ones))
>
>
> *# Format the counts into a PCollection of strings. **def *
> format_result(word_count):
> (word, count) = word_count
> *return **'%s: %d' *% (word, count)
>
> output = counts | *'format' *>> beam.Map(format_result)
>
>
>
> *# Write the output using a "Write" transform that has side effects. #
> pylint: disable=expression-not-assigned *output | *'write' *>>
> WriteToText(_output)
>
> result = p.run()
> result.wait_until_finish()
>
>
>
>
>
>
>
> *From: *Matthew Patterson <mp...@aligntech.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Monday, September 23, 2019 at 2:22 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> Sorry: needed to add LOOPBACK
>
>
>
> *From: *Matthew Patterson <mp...@aligntech.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Monday, September 23, 2019 at 2:18 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> OK, now using release-2.15.0, calling word-count example with runner,
> after building & calling
>
>
>
> ./gradlew :runners:flink:1.5:job-server:runShadow (tried 1.8 as well)
>
>
>
> , as
>
>
>
> “*f'--runner=**PortableRunner**'*, *f'--job_endpoint=localhost:8099'*
>
> ”
>
> …and getting
>
>
>
> File
> "/Users/mpatterson/miniconda/envs/aws/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py",
> line 446, in wait_until_finish
>
> self._job_id, self._state, self._last_error_message()))
>
> RuntimeError: Pipeline
> BeamApp-mpatterson-0923181154-d9c8a662_737ea859-53e2-4d7e-ba94-2955a1642f94
> failed in state FAILED:
> org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException:
> CANCELLED: cancelled before receiving half close
>
>
>
> Ideas? Thanks!
>
>
>
> *From: *Matthew Patterson <mp...@aligntech.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Friday, September 20, 2019 at 9:27 AM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> 411:
>
>
>
> Although it does not seem correct, adding a symbolic link to `virtualenv`
> in ` <path-to-beam>/sdks/python` solves the issue of not finding
> `virtualenv`.
>
>
>
> *From: *Matthew Patterson <mp...@aligntech.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 2:56 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> Re Kyle:
>
>
>
> Just so you may believe me, see below(virtualenv installed, error occurs
> nonetheless): currently I am trying to figure out if it is a difference
> between .bash_profile and .bashrc.
>
>
>
> Thanks,
>
> Matt
>
>
>
> “””
>
> Caused by: net.rubygrapefruit.platform.NativeException: Could not start
> 'virtualenv'
>
> at
> net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
>
> at
> net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
>
> at
> org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:97)
>
> at
> org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:70)
>
> ... 4 more
>
> Caused by: java.io.IOException: Cannot run program "virtualenv" (in
> directory "/Users/mpatterson/dev/beam/sdks/python"): error=2, No such file
> or directory
>
> at
> net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
>
> ... 7 more
>
> Caused by: java.io.IOException: error=2, No such file or directory
>
> ... 8 more
>
>
>
>
>
> * Get more help at *https://help.gradle.org
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.gradle.org&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624459522&sdata=uYZSmBb0wHW6gPiKSWd%2BWEJFBiOPanl5Np%2B9fm7SbAs%3D&reserved=0>*
>
>
>
> Deprecated Gradle features were used in this build, making it incompatible
> with Gradle 6.0.
>
> Use '--warning-mode all' to show the individual deprecation warnings.
>
> See
> https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.gradle.org%2F5.2.1%2Fuserguide%2Fcommand_line_interface.html%23sec%3Acommand_line_warnings&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624459522&sdata=fUTdCyr5oIURuUQOn3K27%2BSnPDotCBqSzfhfmvppE5E%3D&reserved=0>
>
>
>
> *BUILD FAILED* in 0s
>
> 5 actionable tasks: 3 executed, 2 up-to-date
>
> bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
>
> MANIFEST.in build.gradle gen_protos.py
> setup.py tox.ini
>
> OWNERS container scripts
> test-suites
>
> apache_beam findSupportedPython.groovy setup.cfg
> test_config.py
>
> bash-3.2$ pip install virtualenv
>
> Requirement already satisfied: virtualenv in
> /Users/mpatterson/miniconda/lib/python3.7/site-packages (16.7.5)
>
> bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
>
> MANIFEST.in build.gradle gen_protos.py
> setup.py tox.ini
>
> OWNERS container scripts
> test-suites
>
> apache_beam findSupportedPython.groovy setup.cfg
> test_config.py
>
> bash-3.2$
>
>
>
> ”””
>
>
>
> *From: *Matthew Patterson <mp...@aligntech.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 2:31 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> Have done: hard to get gradle to respect env (conda? Bash? Fish? Zsh?):
>
>
>
> *From: *Kyle Weaver <kc...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 2:29 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> I'm guessing you need to install virtualenv: `pip install virtualenv`
>
>
> Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624469516&sdata=L5LJ3FG9He8%2BZAUGrjq6QbGE49FKUCuJawvFbHFfdys%3D&reserved=0>
> | kcweaver@google.com
>
>
>
>
>
> On Thu, Sep 19, 2019 at 11:27 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Kyle,
>
>
>
> Excellent, will do: unfortunately switch to 2.16 was only thing that fixed
>
>
>
> FAILURE: Build failed with an exception.
>
>
>
> * What went wrong:
>
> Execution failed for task ':sdks:python:setupVirtualenv'.
>
> > A problem occurred starting process 'command 'virtualenv''
>
>
>
>
>
> …if you have suggestions there, I would be most appreciative.
>
> Matt
>
>
>
>
>
> *From: *Kyle Weaver <kc...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 2:25 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> You should probably use 2.15, since 2.16 release artifacts have not been
> published yet. Just follow the instructions that
> say --runner=PortableRunner, not --runner=FlinkRunner, otherwise you'll hit
> that other deserialization bug that was mentioned..
>
>
> Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624479508&sdata=fUSIGXPa9qIVdnADUH47N4AQrD54crNdXVSn2sHkV9E%3D&reserved=0>
> | kcweaver@google.com
>
>
>
>
>
> On Thu, Sep 19, 2019 at 11:19 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Kyle,
>
>
>
> Happily: should I be working from (source) release-2.15.0 or
> release-2.16.0: presumably still specifying flink-1.8.
>
> Matt
>
>
>
> *From: *Kyle Weaver <kc...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 2:16 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> Re Benjamin & Ankur: I don't think `--experiments=beam_fn_api` applies
> here. If I understand correctly, Matthew is just trying to run an
> old-fashioned Beam Java jar, nothing to do with portability/Python, and
> judging by the stack trace provided,
> https://issues.apache.org/jira/browse/BEAM-8037
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-8037&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624479508&sdata=vdg3BVHu7SAdm1Iex4Fs7kC6HCXO5CuCkCnBrAnO4dU%3D&reserved=0> is
> not the issue here.
>
>
>
> Re Matthew: I suspect a Flink version mismatch between the Flink cluster
> and the Beam runner. However, I am not familiar with the Maven commands
> you're using. Since you're planning on running Python eventually anyway,
> could you try following the "portable"/Python instructions on
> https://beam.apache.org/documentation/runners/flink/
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Fdocumentation%2Frunners%2Fflink%2F&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624489505&sdata=63DuYCjr0zL1iJX%2FX%2F2urtSlE4n6BFBdXlFXJLV667A%3D&reserved=0> and
> let us know how that goes?
>
>
> Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624499498&sdata=%2BLZHOLfNX6DLNygN%2FUgKpzWp%2FsOjiqRGEwKnH55n8jA%3D&reserved=0>
> | kcweaver@google.com
>
>
>
>
>
> On Thu, Sep 19, 2019 at 10:52 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Thanks Ankur,
>
>
>
> As one who speaks almost no gradle, is there a preferred way to get gradle
> to respect the conda configured python on `gradlew build` ?
>
>
>
> Matt
>
>
>
> *From: *Ankur Goenka <go...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 1:50 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> We have this bug in 2.15 which is discussed here
> https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624499498&sdata=2p%2BvycaeLCzE9c9dwBkz2PTieuXf%2BchE%2BO9ebCI43iU%3D&reserved=0>
>
>
>
> For now, please add "--experiments=beam_fn_api" to your pipeline to make
> it work.
>
>
>
> On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Hi Ankur,
>
>
>
> Yes, I was using 2.15, but was getting failure to deserialize.
>
>
>
> Thanks,
>
> Matt
>
>
>
> *From: *Ankur Goenka <go...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Wednesday, September 18, 2019 at 9:34 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> Hi Matthew,
>
>
>
> Beam 2.16.0 is not yet released hence you are getting the error. Can you
> try using 2.15.0 version?
>
>
>
> Thanks,
>
> Ankur
>
>
>
> On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Tried
>
> "
> mvn archetype:generate \
> -DarchetypeGroupId=org.apache.beam \
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> -DarchetypeVersion=2.16.0 \
> -DgroupId=org.example \
> -DartifactId=word-count-beam \
> -Dversion="0.1" \
> -Dpackage=org.apache.beam.examples \
> -DinteractiveMode=false
> "
> got
>
> -> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate
> (default-cli) on project standalone-pom: The desired archetype does not
> exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) ->
> [Help 1]
>
>
> On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com> wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
>
> Ah. So maybe try 2.16.0?
>
> On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com> wrote:
> > --experiments=beam_fn_api doesn't apply here, as this is a Java
> pipeline
> > using the non-portable version of the Flink runner.
> >
> > Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624509497&sdata=0xpcUPC1p%2FmylWCZfy446Z7co298RVd1%2BsYEbp4goG0%3D&reserved=0>
> | kcweaver@google.com
> >
> >
> > On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <
> benjamintanweihao@gmail.com>
> > wrote:
> >
> > > Could you try adding "--experiments=beam_fn_api" to the runner
> options?
> > > Kyle did mention that it should be fixed in 2.16.0.
> > >
> > > On 2019/09/17 16:40:41, Matthew Patterson <
> mpatterson@aligntech.com>
> > > wrote:
> > > > Folks,
> > > >
> > > > Am sure I am doing it wrong, but have been beating head against
> wall
> > > long enough I need to ask for help.
> > > >
> > > > My goal is to use the python SDK to run jobs on AWS flink
> cluster, but
> > > very little is working, so I am trying to back down to the very
> simplest
> > > thing that works, and build back up: not going so well.
> > > >
> > > > Now I am running, or trying to, on a minikube link cluster. Now
> running
> > > from just java/mvn, so I go through the following:
> > > >
> > > > 1. bring-up-cluster (see below)
> > > > 2. grab-the-code (see below)
> > > > 3. maven-submit (see below)
> > > >
> > > > Eventual failure is
> > > >
> > > > “[ERROR] Failed to execute goal
> > > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on
> project
> > > word-count-beam: An exception occured while executing the Java
> class.
> > > Pipeline execution failed: Could not retrieve the execution
> result. (JobID:
> > > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph.
> [Failed to
> > > deserialize JobGraph.]”
> > > >
> > > > and log is attached (I hope)
> > > >
> > > > Other details:
> > > >
> > > > on osx
> > > >
> > > >
> > > >
> > > > ➜ minikube version
> > > >
> > > > minikube version: v1.3.1
> > > >
> > > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > > >
> > > >
> > > > ➜ kubectl version
> > > >
> > > > Client Version: version.Info{Major:"1", Minor:"14",
> > > GitVersion:"v1.14.6",
> GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > > >
> > > > Server Version: version.Info{Major:"1", Minor:"15",
> > > GitVersion:"v1.15.2",
> GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > > >
> > > >
> > > > ➜ flink --version
> > > >
> > > > Version: 1.8.1, Commit ID: 7297bac
> > > >
> > > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > > 2.15.0
> > > >
> > > > ➜ curl localhost:8081
> > > > <!--
> > > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > > ~ or more contributor license agreements. See the NOTICE file
> > > > ~ distributed with this work for additional information
> > > > ~ regarding copyright ownership. The ASF licenses this file
> > > > ~ to you under the Apache License, Version 2.0 (the
> > > > ~ "License"); you may not use this file except in compliance
> > > > ~ with the License. You may obtain a copy of the License at
> > > > ~
> > > > ~
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624519492&sdata=aQlyiu5HAJehbzeqWbr0gWE7RPrC1sJ4B8Tf%2Bd366ZU%3D&reserved=0>
> > > > ~
> > > > ~ Unless required by applicable law or agreed to in writing,
> software
> > > > ~ distributed under the License is distributed on an "AS IS"
> BASIS,
> > > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > > implied.
> > > > ~ See the License for the specific language governing
> permissions and
> > > > ~ limitations under the License.
> > > > -->
> > > >
> > > > <!doctype html>
> > > > <html lang="en">
> > > > <head>
> > > > <meta charset="utf-8">
> > > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > > <meta name="viewport" content="width=device-width,
> initial-scale=1">
> > > > <title>Apache Flink Web Dashboard</title>
> > > > <link rel="apple-touch-icon" sizes="180x180"
> > > href="assets/favicon/apple-touch-icon.png">
> > > > <link rel="icon" type="image/png"
> > > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > > <link rel="icon" type="image/png"
> > > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > > <link rel="mask-icon"
> href="assets/favicon/safari-pinned-tab.svg"
> > > color="#aa1919">
> > > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > > <meta name="msapplication-config"
> > > content="assets/favicon/browserconfig.xml">
> > > > <meta name="theme-color" content="#ffffff">
> > > > <base href="./"><link rel="stylesheet"
> > > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > > <body>
> > > > <flink-root></flink-root>
> > > > <script type="text/javascript"
> > > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > > type="text/javascript"
> src="es2015-polyfills.5e343224e81eefb7658e.js"
> > > nomodule></script><script type="text/javascript"
> > > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > > type="text/javascript"
> src="main.dc6e0997d092df051db0.js"></script></body>
> > > > </html>
> > > >
> > > > Thanks for any ideas,
> > > > Matt
> > > >
> > > > >>> bring-up-cluster
> > > >
> > > > # [yamls from here](
> > >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624519492&sdata=mGNxFnzCCtNEpnB%2BPng71Hbfo%2Flj1EZRXnnX6OlvxIg%3D&reserved=0>
> > > )
> > > > minikube start
> > > > minikube ssh 'sudo ip link set docker0 promisc on'
> > > >
> > > > kubectl create -f flink-configuration-configmap.yaml
> > > > kubectl create -f jobmanager-service.yaml
> > > > kubectl create -f jobmanager-deployment.yaml
> > > > kubectl create -f taskmanager-deployment.yaml
> > > > kubectl apply -f jobmanager-rest-service.yaml
> > > >
> > > > >>> grab-the-code
> > > > mvn archetype:generate \
> > > > -DarchetypeGroupId=org.apache.beam \
> > > >
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > > -DarchetypeVersion=2.15.0 \
> > > > -DgroupId=org.example \
> > > > -DartifactId=word-count-beam \
> > > > -Dversion="0.1" \
> > > > -Dpackage=org.apache.beam.examples \
> > > > -DinteractiveMode=false
> > > >
> > > > >>> maven-submit
> > > >
> > > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in
> separate
> > > shell, as blocking
> > > > mvn package exec:java -X
> > > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > > -Dexec.args="--runner=FlinkRunner\
> > > > --flinkMaster="localhost:8081"\
> > > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > > --output=<directory-that-exists>" -Pflink-runner
> > > >
> > > >
> > > >
> > >
> >
>
>
Re: Word-count example
Posted by Matthew Patterson <mp...@aligntech.com>.
Kyle, right: saw that. Grabbing at anything.
The process on the client seems to submit, or at least not bomb, but the cluster does not acknowledge a running job. It is possible I have not waited long enough yet, and images are still being downloaded & launched.
2.16 section on FlinkRunner does not have (that I found) help on gathering logs: might you point me at that as well?
Thanks for the help, Kyle.
Matt
From: Kyle Weaver <kc...@google.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Monday, September 23, 2019 at 3:49 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
"WARNING:root:No unique name set for transform ..." should not affect the pipeline's ability to complete successfully. Is the pipeline failing? If so, could you share more logs?
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624449524&sdata=F5tue28BCAg712rWTOO%2F2Kvbd91lqOYsGR0%2BQ25vcrY%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Mon, Sep 23, 2019 at 11:43 AM Matthew Patterson <mp...@aligntech.com>> wrote:
After setting runner, adding '--environment_type=LOOPBACK', I now get
“
…
WARNING:root:No unique name set for transform fn/read/ref_PCollection_PCollection_17:0
WARNING:root:No unique name set for transform fn/read/ref_PCollection_PCollection_17:0
WARNING:root:No unique name set for transform fn/write/ref_PCollection_PCollection_24:0
…
”
when running wordcount example:
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = runner_options
pipeline_options.view_as(SetupOptions).save_main_session = True
p = beam.Pipeline(options=pipeline_options)
# Read the text file[pattern] into a PCollection.
lines = p | 'read' >> ReadFromText(_input)
# Count the occurrences of each word.
def count_ones(word_ones):
(word, ones) = word_ones
return (word, sum(ones))
counts = (lines
| 'split' >> (beam.ParDo(WordExtractingDoFn()).with_output_types(unicode))
| 'pair_with_one' >> beam.Map(lambda x: (x, 1))
| 'group' >> beam.GroupByKey()
| 'count' >> beam.Map(count_ones))
# Format the counts into a PCollection of strings.
def format_result(word_count):
(word, count) = word_count
return '%s: %d' % (word, count)
output = counts | 'format' >> beam.Map(format_result)
# Write the output using a "Write" transform that has side effects.
# pylint: disable=expression-not-assigned
output | 'write' >> WriteToText(_output)
result = p.run()
result.wait_until_finish()
From: Matthew Patterson <mp...@aligntech.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Monday, September 23, 2019 at 2:22 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
Sorry: needed to add LOOPBACK
From: Matthew Patterson <mp...@aligntech.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Monday, September 23, 2019 at 2:18 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
OK, now using release-2.15.0, calling word-count example with runner, after building & calling
./gradlew :runners:flink:1.5:job-server:runShadow (tried 1.8 as well)
, as
“f'--runner=PortableRunner', f'--job_endpoint=localhost:8099'
”
…and getting
File "/Users/mpatterson/miniconda/envs/aws/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 446, in wait_until_finish
self._job_id, self._state, self._last_error_message()))
RuntimeError: Pipeline BeamApp-mpatterson-0923181154-d9c8a662_737ea859-53e2-4d7e-ba94-2955a1642f94 failed in state FAILED: org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
Ideas? Thanks!
From: Matthew Patterson <mp...@aligntech.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Friday, September 20, 2019 at 9:27 AM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
411:
Although it does not seem correct, adding a symbolic link to `virtualenv` in ` <path-to-beam>/sdks/python` solves the issue of not finding `virtualenv`.
From: Matthew Patterson <mp...@aligntech.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:56 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
Re Kyle:
Just so you may believe me, see below(virtualenv installed, error occurs nonetheless): currently I am trying to figure out if it is a difference between .bash_profile and .bashrc.
Thanks,
Matt
“””
Caused by: net.rubygrapefruit.platform.NativeException: Could not start 'virtualenv'
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
at net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
at org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:97)
at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:70)
... 4 more
Caused by: java.io.IOException: Cannot run program "virtualenv" (in directory "/Users/mpatterson/dev/beam/sdks/python"): error=2, No such file or directory
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
... 8 more
* Get more help at https://help.gradle.org<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhelp.gradle.org&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624459522&sdata=uYZSmBb0wHW6gPiKSWd%2BWEJFBiOPanl5Np%2B9fm7SbAs%3D&reserved=0>
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.gradle.org%2F5.2.1%2Fuserguide%2Fcommand_line_interface.html%23sec%3Acommand_line_warnings&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624459522&sdata=fUTdCyr5oIURuUQOn3K27%2BSnPDotCBqSzfhfmvppE5E%3D&reserved=0>
BUILD FAILED in 0s
5 actionable tasks: 3 executed, 2 up-to-date
bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
MANIFEST.in build.gradle gen_protos.py setup.py tox.ini
OWNERS container scripts test-suites
apache_beam findSupportedPython.groovy setup.cfg test_config.py
bash-3.2$ pip install virtualenv
Requirement already satisfied: virtualenv in /Users/mpatterson/miniconda/lib/python3.7/site-packages (16.7.5)
bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
MANIFEST.in build.gradle gen_protos.py setup.py tox.ini
OWNERS container scripts test-suites
apache_beam findSupportedPython.groovy setup.cfg test_config.py
bash-3.2$
”””
From: Matthew Patterson <mp...@aligntech.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:31 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
Have done: hard to get gradle to respect env (conda? Bash? Fish? Zsh?):
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:29 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I'm guessing you need to install virtualenv: `pip install virtualenv`
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624469516&sdata=L5LJ3FG9He8%2BZAUGrjq6QbGE49FKUCuJawvFbHFfdys%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 11:27 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle,
Excellent, will do: unfortunately switch to 2.16 was only thing that fixed
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:setupVirtualenv'.
> A problem occurred starting process 'command 'virtualenv''
…if you have suggestions there, I would be most appreciative.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:25 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
You should probably use 2.15, since 2.16 release artifacts have not been published yet. Just follow the instructions that say --runner=PortableRunner, not --runner=FlinkRunner, otherwise you'll hit that other deserialization bug that was mentioned..
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624479508&sdata=fUSIGXPa9qIVdnADUH47N4AQrD54crNdXVSn2sHkV9E%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 11:19 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle,
Happily: should I be working from (source) release-2.15.0 or release-2.16.0: presumably still specifying flink-1.8.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:16 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re Benjamin & Ankur: I don't think `--experiments=beam_fn_api` applies here. If I understand correctly, Matthew is just trying to run an old-fashioned Beam Java jar, nothing to do with portability/Python, and judging by the stack trace provided, https://issues.apache.org/jira/browse/BEAM-8037<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-8037&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624479508&sdata=vdg3BVHu7SAdm1Iex4Fs7kC6HCXO5CuCkCnBrAnO4dU%3D&reserved=0> is not the issue here.
Re Matthew: I suspect a Flink version mismatch between the Flink cluster and the Beam runner. However, I am not familiar with the Maven commands you're using. Since you're planning on running Python eventually anyway, could you try following the "portable"/Python instructions on https://beam.apache.org/documentation/runners/flink/<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Fdocumentation%2Frunners%2Fflink%2F&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624489505&sdata=63DuYCjr0zL1iJX%2FX%2F2urtSlE4n6BFBdXlFXJLV667A%3D&reserved=0> and let us know how that goes?
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624499498&sdata=%2BLZHOLfNX6DLNygN%2FUgKpzWp%2FsOjiqRGEwKnH55n8jA%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 10:52 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Thanks Ankur,
As one who speaks almost no gradle, is there a preferred way to get gradle to respect the conda configured python on `gradlew build` ?
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 1:50 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
We have this bug in 2.15 which is discussed here https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624499498&sdata=2p%2BvycaeLCzE9c9dwBkz2PTieuXf%2BchE%2BO9ebCI43iU%3D&reserved=0>
For now, please add "--experiments=beam_fn_api" to your pipeline to make it work.
On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Hi Ankur,
Yes, I was using 2.15, but was getting failure to deserialize.
Thanks,
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Wednesday, September 18, 2019 at 9:34 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Matthew,
Beam 2.16.0 is not yet released hence you are getting the error. Can you try using 2.15.0 version?
Thanks,
Ankur
On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Tried
"
mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
-DarchetypeVersion=2.16.0 \
-DgroupId=org.example \
-DartifactId=word-count-beam \
-Dversion="0.1" \
-Dpackage=org.apache.beam.examples \
-DinteractiveMode=false
"
got
-> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate (default-cli) on project standalone-pom: The desired archetype does not exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) -> [Help 1]
On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com>> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Ah. So maybe try 2.16.0?
On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com>> wrote:
> --experiments=beam_fn_api doesn't apply here, as this is a Java pipeline
> using the non-portable version of the Flink runner.
>
> Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624509497&sdata=0xpcUPC1p%2FmylWCZfy446Z7co298RVd1%2BsYEbp4goG0%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
>
>
> On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <be...@gmail.com>>
> wrote:
>
> > Could you try adding "--experiments=beam_fn_api" to the runner options?
> > Kyle did mention that it should be fixed in 2.16.0.
> >
> > On 2019/09/17 16:40:41, Matthew Patterson <mp...@aligntech.com>>
> > wrote:
> > > Folks,
> > >
> > > Am sure I am doing it wrong, but have been beating head against wall
> > long enough I need to ask for help.
> > >
> > > My goal is to use the python SDK to run jobs on AWS flink cluster, but
> > very little is working, so I am trying to back down to the very simplest
> > thing that works, and build back up: not going so well.
> > >
> > > Now I am running, or trying to, on a minikube link cluster. Now running
> > from just java/mvn, so I go through the following:
> > >
> > > 1. bring-up-cluster (see below)
> > > 2. grab-the-code (see below)
> > > 3. maven-submit (see below)
> > >
> > > Eventual failure is
> > >
> > > “[ERROR] Failed to execute goal
> > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project
> > word-count-beam: An exception occured while executing the Java class.
> > Pipeline execution failed: Could not retrieve the execution result. (JobID:
> > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to
> > deserialize JobGraph.]”
> > >
> > > and log is attached (I hope)
> > >
> > > Other details:
> > >
> > > on osx
> > >
> > >
> > >
> > > ➜ minikube version
> > >
> > > minikube version: v1.3.1
> > >
> > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > >
> > >
> > > ➜ kubectl version
> > >
> > > Client Version: version.Info{Major:"1", Minor:"14",
> > GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > >
> > > Server Version: version.Info{Major:"1", Minor:"15",
> > GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > >
> > >
> > > ➜ flink --version
> > >
> > > Version: 1.8.1, Commit ID: 7297bac
> > >
> > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > 2.15.0
> > >
> > > ➜ curl localhost:8081
> > > <!--
> > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > ~ or more contributor license agreements. See the NOTICE file
> > > ~ distributed with this work for additional information
> > > ~ regarding copyright ownership. The ASF licenses this file
> > > ~ to you under the Apache License, Version 2.0 (the
> > > ~ "License"); you may not use this file except in compliance
> > > ~ with the License. You may obtain a copy of the License at
> > > ~
> > > ~ https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624519492&sdata=aQlyiu5HAJehbzeqWbr0gWE7RPrC1sJ4B8Tf%2Bd366ZU%3D&reserved=0>
> > > ~
> > > ~ Unless required by applicable law or agreed to in writing, software
> > > ~ distributed under the License is distributed on an "AS IS" BASIS,
> > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > implied.
> > > ~ See the License for the specific language governing permissions and
> > > ~ limitations under the License.
> > > -->
> > >
> > > <!doctype html>
> > > <html lang="en">
> > > <head>
> > > <meta charset="utf-8">
> > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > <meta name="viewport" content="width=device-width, initial-scale=1">
> > > <title>Apache Flink Web Dashboard</title>
> > > <link rel="apple-touch-icon" sizes="180x180"
> > href="assets/favicon/apple-touch-icon.png">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > <link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg"
> > color="#aa1919">
> > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > <meta name="msapplication-config"
> > content="assets/favicon/browserconfig.xml">
> > > <meta name="theme-color" content="#ffffff">
> > > <base href="./"><link rel="stylesheet"
> > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > <body>
> > > <flink-root></flink-root>
> > > <script type="text/javascript"
> > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js"
> > nomodule></script><script type="text/javascript"
> > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > type="text/javascript" src="main.dc6e0997d092df051db0.js"></script></body>
> > > </html>
> > >
> > > Thanks for any ideas,
> > > Matt
> > >
> > > >>> bring-up-cluster
> > >
> > > # [yamls from here](
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C8fc0c000d341423ac93408d7405f1ea9%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048649624519492&sdata=mGNxFnzCCtNEpnB%2BPng71Hbfo%2Flj1EZRXnnX6OlvxIg%3D&reserved=0>
> > )
> > > minikube start
> > > minikube ssh 'sudo ip link set docker0 promisc on'
> > >
> > > kubectl create -f flink-configuration-configmap.yaml
> > > kubectl create -f jobmanager-service.yaml
> > > kubectl create -f jobmanager-deployment.yaml
> > > kubectl create -f taskmanager-deployment.yaml
> > > kubectl apply -f jobmanager-rest-service.yaml
> > >
> > > >>> grab-the-code
> > > mvn archetype:generate \
> > > -DarchetypeGroupId=org.apache.beam \
> > > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > -DarchetypeVersion=2.15.0 \
> > > -DgroupId=org.example \
> > > -DartifactId=word-count-beam \
> > > -Dversion="0.1" \
> > > -Dpackage=org.apache.beam.examples \
> > > -DinteractiveMode=false
> > >
> > > >>> maven-submit
> > >
> > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in separate
> > shell, as blocking
> > > mvn package exec:java -X
> > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > -Dexec.args="--runner=FlinkRunner\
> > > --flinkMaster="localhost:8081"\
> > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > --output=<directory-that-exists>" -Pflink-runner
> > >
> > >
> > >
> >
>
Re: Word-count example
Posted by Kyle Weaver <kc...@google.com>.
"WARNING:root:No unique name set for transform ..." should not affect
the pipeline's ability to complete successfully. Is the pipeline
failing? If so, could you share more logs?
Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com
On Mon, Sep 23, 2019 at 11:43 AM Matthew Patterson <mp...@aligntech.com>
wrote:
> After setting runner, adding *'--environment_type=LOOPBACK'*, I now get
>
>
>
> “
>
> …
>
> WARNING:root:No unique name set for transform fn/read/ref_PCollection_PCollection_17:0
>
> WARNING:root:No unique name set for transform fn/read/ref_PCollection_PCollection_17:0
>
> WARNING:root:No unique name set for transform fn/write/ref_PCollection_PCollection_24:0
>
> …
>
> ”
>
> when running wordcount example:
>
>
>
>
>
> *# We use the save_main_session option because one or more DoFn's in this
> # workflow rely on global context (e.g., a module imported at module
> level). *pipeline_options = runner_options
> pipeline_options.view_as(SetupOptions).save_main_session =
>
> *True *p = beam.Pipeline(options=pipeline_options)
>
>
>
> *# Read the text file[pattern] into a PCollection. *lines = p | *'read' *>>
> ReadFromText(_input)
>
>
> *# Count the occurrences of each word. **def *count_ones(word_ones):
> (word, ones) = word_ones
> *return *(word, sum(ones))
>
> counts = (lines
> | *'split' *>>
> (beam.ParDo(WordExtractingDoFn()).with_output_types(unicode))
> | *'pair_with_one' *>> beam.Map(*lambda *x: (x, 1))
> | *'group' *>> beam.GroupByKey()
> | *'count' *>> beam.Map(count_ones))
>
>
> *# Format the counts into a PCollection of strings. **def *
> format_result(word_count):
> (word, count) = word_count
> *return **'%s: %d' *% (word, count)
>
> output = counts | *'format' *>> beam.Map(format_result)
>
>
>
> *# Write the output using a "Write" transform that has side effects. #
> pylint: disable=expression-not-assigned *output | *'write' *>>
> WriteToText(_output)
>
> result = p.run()
> result.wait_until_finish()
>
>
>
>
>
>
>
> *From: *Matthew Patterson <mp...@aligntech.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Monday, September 23, 2019 at 2:22 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> Sorry: needed to add LOOPBACK
>
>
>
> *From: *Matthew Patterson <mp...@aligntech.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Monday, September 23, 2019 at 2:18 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> OK, now using release-2.15.0, calling word-count example with runner,
> after building & calling
>
>
>
> ./gradlew :runners:flink:1.5:job-server:runShadow (tried 1.8 as well)
>
>
>
> , as
>
>
>
> “*f'--runner=**PortableRunner**'*, *f'--job_endpoint=localhost:8099'*
>
> ”
>
> …and getting
>
>
>
> File
> "/Users/mpatterson/miniconda/envs/aws/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py",
> line 446, in wait_until_finish
>
> self._job_id, self._state, self._last_error_message()))
>
> RuntimeError: Pipeline
> BeamApp-mpatterson-0923181154-d9c8a662_737ea859-53e2-4d7e-ba94-2955a1642f94
> failed in state FAILED:
> org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException:
> CANCELLED: cancelled before receiving half close
>
>
>
> Ideas? Thanks!
>
>
>
> *From: *Matthew Patterson <mp...@aligntech.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Friday, September 20, 2019 at 9:27 AM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> 411:
>
>
>
> Although it does not seem correct, adding a symbolic link to `virtualenv`
> in ` <path-to-beam>/sdks/python` solves the issue of not finding
> `virtualenv`.
>
>
>
> *From: *Matthew Patterson <mp...@aligntech.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 2:56 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> Re Kyle:
>
>
>
> Just so you may believe me, see below(virtualenv installed, error occurs
> nonetheless): currently I am trying to figure out if it is a difference
> between .bash_profile and .bashrc.
>
>
>
> Thanks,
>
> Matt
>
>
>
> “””
>
> Caused by: net.rubygrapefruit.platform.NativeException: Could not start
> 'virtualenv'
>
> at
> net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
>
> at
> net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
>
> at
> org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:97)
>
> at
> org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:70)
>
> ... 4 more
>
> Caused by: java.io.IOException: Cannot run program "virtualenv" (in
> directory "/Users/mpatterson/dev/beam/sdks/python"): error=2, No such file
> or directory
>
> at
> net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
>
> ... 7 more
>
> Caused by: java.io.IOException: error=2, No such file or directory
>
> ... 8 more
>
>
>
>
>
> * Get more help at *https://help.gradle.org <https://help.gradle.org>*
>
>
>
> Deprecated Gradle features were used in this build, making it incompatible
> with Gradle 6.0.
>
> Use '--warning-mode all' to show the individual deprecation warnings.
>
> See
> https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
>
>
>
> *BUILD FAILED* in 0s
>
> 5 actionable tasks: 3 executed, 2 up-to-date
>
> bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
>
> MANIFEST.in build.gradle gen_protos.py
> setup.py tox.ini
>
> OWNERS container scripts
> test-suites
>
> apache_beam findSupportedPython.groovy setup.cfg
> test_config.py
>
> bash-3.2$ pip install virtualenv
>
> Requirement already satisfied: virtualenv in
> /Users/mpatterson/miniconda/lib/python3.7/site-packages (16.7.5)
>
> bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
>
> MANIFEST.in build.gradle gen_protos.py
> setup.py tox.ini
>
> OWNERS container scripts
> test-suites
>
> apache_beam findSupportedPython.groovy setup.cfg
> test_config.py
>
> bash-3.2$
>
>
>
> ”””
>
>
>
> *From: *Matthew Patterson <mp...@aligntech.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 2:31 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> Have done: hard to get gradle to respect env (conda? Bash? Fish? Zsh?):
>
>
>
> *From: *Kyle Weaver <kc...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 2:29 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> I'm guessing you need to install virtualenv: `pip install virtualenv`
>
>
> Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cb6d317f8a70940d2ced408d74052f2bf%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048597336750345&sdata=YfCAcNKIdXqeFQ7RyDKv1NZO4vgjTmc0d31xK9wJSX0%3D&reserved=0>
> | kcweaver@google.com
>
>
>
>
>
> On Thu, Sep 19, 2019 at 11:27 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Kyle,
>
>
>
> Excellent, will do: unfortunately switch to 2.16 was only thing that fixed
>
>
>
> FAILURE: Build failed with an exception.
>
>
>
> * What went wrong:
>
> Execution failed for task ':sdks:python:setupVirtualenv'.
>
> > A problem occurred starting process 'command 'virtualenv''
>
>
>
>
>
> …if you have suggestions there, I would be most appreciative.
>
> Matt
>
>
>
>
>
> *From: *Kyle Weaver <kc...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 2:25 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> You should probably use 2.15, since 2.16 release artifacts have not been
> published yet. Just follow the instructions that
> say --runner=PortableRunner, not --runner=FlinkRunner, otherwise you'll hit
> that other deserialization bug that was mentioned..
>
>
> Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cb6d317f8a70940d2ced408d74052f2bf%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048597336760341&sdata=2AnoIOi2rlZnX1TkUNL5BcN8gPGTN7t0bbEtMLE4eCM%3D&reserved=0>
> | kcweaver@google.com
>
>
>
>
>
> On Thu, Sep 19, 2019 at 11:19 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Kyle,
>
>
>
> Happily: should I be working from (source) release-2.15.0 or
> release-2.16.0: presumably still specifying flink-1.8.
>
> Matt
>
>
>
> *From: *Kyle Weaver <kc...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 2:16 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> Re Benjamin & Ankur: I don't think `--experiments=beam_fn_api` applies
> here. If I understand correctly, Matthew is just trying to run an
> old-fashioned Beam Java jar, nothing to do with portability/Python, and
> judging by the stack trace provided,
> https://issues.apache.org/jira/browse/BEAM-8037
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-8037&data=02%7C01%7Cmpatterson%40aligntech.com%7Cb6d317f8a70940d2ced408d74052f2bf%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048597336770339&sdata=mUyefW8aVc%2B6ycry0afaA%2BcRBXctn4aQ1tJJ7IICokA%3D&reserved=0> is
> not the issue here.
>
>
>
> Re Matthew: I suspect a Flink version mismatch between the Flink cluster
> and the Beam runner. However, I am not familiar with the Maven commands
> you're using. Since you're planning on running Python eventually anyway,
> could you try following the "portable"/Python instructions on
> https://beam.apache.org/documentation/runners/flink/
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Fdocumentation%2Frunners%2Fflink%2F&data=02%7C01%7Cmpatterson%40aligntech.com%7Cb6d317f8a70940d2ced408d74052f2bf%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048597336770339&sdata=wcmNi%2FMN0ryq787vQ54%2FKn37GI97D8GcgWxAXOqVvD4%3D&reserved=0> and
> let us know how that goes?
>
>
> Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cb6d317f8a70940d2ced408d74052f2bf%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048597336780334&sdata=XCXdKOj8DiDZu9qNPRl8rl3xqZaczzoLgb%2FH%2FDi8zTc%3D&reserved=0>
> | kcweaver@google.com
>
>
>
>
>
> On Thu, Sep 19, 2019 at 10:52 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Thanks Ankur,
>
>
>
> As one who speaks almost no gradle, is there a preferred way to get gradle
> to respect the conda configured python on `gradlew build` ?
>
>
>
> Matt
>
>
>
> *From: *Ankur Goenka <go...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 1:50 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> We have this bug in 2.15 which is discussed here
> https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7Cb6d317f8a70940d2ced408d74052f2bf%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048597336790329&sdata=TkmwQ4GqRBycmn4s%2BgCkMyG8SO6dsYNKL9NRYIPlfX0%3D&reserved=0>
>
>
>
> For now, please add "--experiments=beam_fn_api" to your pipeline to make
> it work.
>
>
>
> On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Hi Ankur,
>
>
>
> Yes, I was using 2.15, but was getting failure to deserialize.
>
>
>
> Thanks,
>
> Matt
>
>
>
> *From: *Ankur Goenka <go...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Wednesday, September 18, 2019 at 9:34 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> Hi Matthew,
>
>
>
> Beam 2.16.0 is not yet released hence you are getting the error. Can you
> try using 2.15.0 version?
>
>
>
> Thanks,
>
> Ankur
>
>
>
> On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Tried
>
> "
> mvn archetype:generate \
> -DarchetypeGroupId=org.apache.beam \
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> -DarchetypeVersion=2.16.0 \
> -DgroupId=org.example \
> -DartifactId=word-count-beam \
> -Dversion="0.1" \
> -Dpackage=org.apache.beam.examples \
> -DinteractiveMode=false
> "
> got
>
> -> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate
> (default-cli) on project standalone-pom: The desired archetype does not
> exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) ->
> [Help 1]
>
>
> On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com> wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
>
> Ah. So maybe try 2.16.0?
>
> On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com> wrote:
> > --experiments=beam_fn_api doesn't apply here, as this is a Java
> pipeline
> > using the non-portable version of the Flink runner.
> >
> > Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cb6d317f8a70940d2ced408d74052f2bf%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048597336800326&sdata=8mLZgtzoEg9T%2BZmtEMLGUDjIofbeDuaRkBb5kY0HF9U%3D&reserved=0>
> | kcweaver@google.com
> >
> >
> > On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <
> benjamintanweihao@gmail.com>
> > wrote:
> >
> > > Could you try adding "--experiments=beam_fn_api" to the runner
> options?
> > > Kyle did mention that it should be fixed in 2.16.0.
> > >
> > > On 2019/09/17 16:40:41, Matthew Patterson <
> mpatterson@aligntech.com>
> > > wrote:
> > > > Folks,
> > > >
> > > > Am sure I am doing it wrong, but have been beating head against
> wall
> > > long enough I need to ask for help.
> > > >
> > > > My goal is to use the python SDK to run jobs on AWS flink
> cluster, but
> > > very little is working, so I am trying to back down to the very
> simplest
> > > thing that works, and build back up: not going so well.
> > > >
> > > > Now I am running, or trying to, on a minikube link cluster. Now
> running
> > > from just java/mvn, so I go through the following:
> > > >
> > > > 1. bring-up-cluster (see below)
> > > > 2. grab-the-code (see below)
> > > > 3. maven-submit (see below)
> > > >
> > > > Eventual failure is
> > > >
> > > > “[ERROR] Failed to execute goal
> > > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on
> project
> > > word-count-beam: An exception occured while executing the Java
> class.
> > > Pipeline execution failed: Could not retrieve the execution
> result. (JobID:
> > > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph.
> [Failed to
> > > deserialize JobGraph.]”
> > > >
> > > > and log is attached (I hope)
> > > >
> > > > Other details:
> > > >
> > > > on osx
> > > >
> > > >
> > > >
> > > > ➜ minikube version
> > > >
> > > > minikube version: v1.3.1
> > > >
> > > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > > >
> > > >
> > > > ➜ kubectl version
> > > >
> > > > Client Version: version.Info{Major:"1", Minor:"14",
> > > GitVersion:"v1.14.6",
> GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > > >
> > > > Server Version: version.Info{Major:"1", Minor:"15",
> > > GitVersion:"v1.15.2",
> GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > > >
> > > >
> > > > ➜ flink --version
> > > >
> > > > Version: 1.8.1, Commit ID: 7297bac
> > > >
> > > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > > 2.15.0
> > > >
> > > > ➜ curl localhost:8081
> > > > <!--
> > > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > > ~ or more contributor license agreements. See the NOTICE file
> > > > ~ distributed with this work for additional information
> > > > ~ regarding copyright ownership. The ASF licenses this file
> > > > ~ to you under the Apache License, Version 2.0 (the
> > > > ~ "License"); you may not use this file except in compliance
> > > > ~ with the License. You may obtain a copy of the License at
> > > > ~
> > > > ~
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7Cb6d317f8a70940d2ced408d74052f2bf%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048597336800326&sdata=pU44hhkq715BHRd2UaPCdVjmXHrg8%2FVr9LmjiIj8Q24%3D&reserved=0>
> > > > ~
> > > > ~ Unless required by applicable law or agreed to in writing,
> software
> > > > ~ distributed under the License is distributed on an "AS IS"
> BASIS,
> > > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > > implied.
> > > > ~ See the License for the specific language governing
> permissions and
> > > > ~ limitations under the License.
> > > > -->
> > > >
> > > > <!doctype html>
> > > > <html lang="en">
> > > > <head>
> > > > <meta charset="utf-8">
> > > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > > <meta name="viewport" content="width=device-width,
> initial-scale=1">
> > > > <title>Apache Flink Web Dashboard</title>
> > > > <link rel="apple-touch-icon" sizes="180x180"
> > > href="assets/favicon/apple-touch-icon.png">
> > > > <link rel="icon" type="image/png"
> > > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > > <link rel="icon" type="image/png"
> > > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > > <link rel="mask-icon"
> href="assets/favicon/safari-pinned-tab.svg"
> > > color="#aa1919">
> > > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > > <meta name="msapplication-config"
> > > content="assets/favicon/browserconfig.xml">
> > > > <meta name="theme-color" content="#ffffff">
> > > > <base href="./"><link rel="stylesheet"
> > > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > > <body>
> > > > <flink-root></flink-root>
> > > > <script type="text/javascript"
> > > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > > type="text/javascript"
> src="es2015-polyfills.5e343224e81eefb7658e.js"
> > > nomodule></script><script type="text/javascript"
> > > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > > type="text/javascript"
> src="main.dc6e0997d092df051db0.js"></script></body>
> > > > </html>
> > > >
> > > > Thanks for any ideas,
> > > > Matt
> > > >
> > > > >>> bring-up-cluster
> > > >
> > > > # [yamls from here](
> > >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7Cb6d317f8a70940d2ced408d74052f2bf%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048597336810322&sdata=Xj2HftLL5JSL5IKKoFpEix5wuo15ctDbE9k64ti9%2B%2FM%3D&reserved=0>
> > > )
> > > > minikube start
> > > > minikube ssh 'sudo ip link set docker0 promisc on'
> > > >
> > > > kubectl create -f flink-configuration-configmap.yaml
> > > > kubectl create -f jobmanager-service.yaml
> > > > kubectl create -f jobmanager-deployment.yaml
> > > > kubectl create -f taskmanager-deployment.yaml
> > > > kubectl apply -f jobmanager-rest-service.yaml
> > > >
> > > > >>> grab-the-code
> > > > mvn archetype:generate \
> > > > -DarchetypeGroupId=org.apache.beam \
> > > >
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > > -DarchetypeVersion=2.15.0 \
> > > > -DgroupId=org.example \
> > > > -DartifactId=word-count-beam \
> > > > -Dversion="0.1" \
> > > > -Dpackage=org.apache.beam.examples \
> > > > -DinteractiveMode=false
> > > >
> > > > >>> maven-submit
> > > >
> > > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in
> separate
> > > shell, as blocking
> > > > mvn package exec:java -X
> > > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > > -Dexec.args="--runner=FlinkRunner\
> > > > --flinkMaster="localhost:8081"\
> > > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > > --output=<directory-that-exists>" -Pflink-runner
> > > >
> > > >
> > > >
> > >
> >
>
>
Re: Word-count example
Posted by Matthew Patterson <mp...@aligntech.com>.
After setting runner, adding '--environment_type=LOOPBACK', I now get
“
…
WARNING:root:No unique name set for transform fn/read/ref_PCollection_PCollection_17:0
WARNING:root:No unique name set for transform fn/read/ref_PCollection_PCollection_17:0
WARNING:root:No unique name set for transform fn/write/ref_PCollection_PCollection_24:0
…
”
when running wordcount example:
# We use the save_main_session option because one or more DoFn's in this
# workflow rely on global context (e.g., a module imported at module level).
pipeline_options = runner_options
pipeline_options.view_as(SetupOptions).save_main_session = True
p = beam.Pipeline(options=pipeline_options)
# Read the text file[pattern] into a PCollection.
lines = p | 'read' >> ReadFromText(_input)
# Count the occurrences of each word.
def count_ones(word_ones):
(word, ones) = word_ones
return (word, sum(ones))
counts = (lines
| 'split' >> (beam.ParDo(WordExtractingDoFn()).with_output_types(unicode))
| 'pair_with_one' >> beam.Map(lambda x: (x, 1))
| 'group' >> beam.GroupByKey()
| 'count' >> beam.Map(count_ones))
# Format the counts into a PCollection of strings.
def format_result(word_count):
(word, count) = word_count
return '%s: %d' % (word, count)
output = counts | 'format' >> beam.Map(format_result)
# Write the output using a "Write" transform that has side effects.
# pylint: disable=expression-not-assigned
output | 'write' >> WriteToText(_output)
result = p.run()
result.wait_until_finish()
From: Matthew Patterson <mp...@aligntech.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Monday, September 23, 2019 at 2:22 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
Sorry: needed to add LOOPBACK
From: Matthew Patterson <mp...@aligntech.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Monday, September 23, 2019 at 2:18 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
OK, now using release-2.15.0, calling word-count example with runner, after building & calling
./gradlew :runners:flink:1.5:job-server:runShadow (tried 1.8 as well)
, as
“f'--runner=PortableRunner', f'--job_endpoint=localhost:8099'
”
…and getting
File "/Users/mpatterson/miniconda/envs/aws/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 446, in wait_until_finish
self._job_id, self._state, self._last_error_message()))
RuntimeError: Pipeline BeamApp-mpatterson-0923181154-d9c8a662_737ea859-53e2-4d7e-ba94-2955a1642f94 failed in state FAILED: org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
Ideas? Thanks!
From: Matthew Patterson <mp...@aligntech.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Friday, September 20, 2019 at 9:27 AM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
411:
Although it does not seem correct, adding a symbolic link to `virtualenv` in ` <path-to-beam>/sdks/python` solves the issue of not finding `virtualenv`.
From: Matthew Patterson <mp...@aligntech.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Thursday, September 19, 2019 at 2:56 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
Re Kyle:
Just so you may believe me, see below(virtualenv installed, error occurs nonetheless): currently I am trying to figure out if it is a difference between .bash_profile and .bashrc.
Thanks,
Matt
“””
Caused by: net.rubygrapefruit.platform.NativeException: Could not start 'virtualenv'
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
at net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
at org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:97)
at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:70)
... 4 more
Caused by: java.io.IOException: Cannot run program "virtualenv" (in directory "/Users/mpatterson/dev/beam/sdks/python"): error=2, No such file or directory
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
... 8 more
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 0s
5 actionable tasks: 3 executed, 2 up-to-date
bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
MANIFEST.in build.gradle gen_protos.py setup.py tox.ini
OWNERS container scripts test-suites
apache_beam findSupportedPython.groovy setup.cfg test_config.py
bash-3.2$ pip install virtualenv
Requirement already satisfied: virtualenv in /Users/mpatterson/miniconda/lib/python3.7/site-packages (16.7.5)
bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
MANIFEST.in build.gradle gen_protos.py setup.py tox.ini
OWNERS container scripts test-suites
apache_beam findSupportedPython.groovy setup.cfg test_config.py
bash-3.2$
”””
From: Matthew Patterson <mp...@aligntech.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Thursday, September 19, 2019 at 2:31 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
Have done: hard to get gradle to respect env (conda? Bash? Fish? Zsh?):
From: Kyle Weaver <kc...@google.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Thursday, September 19, 2019 at 2:29 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I'm guessing you need to install virtualenv: `pip install virtualenv`
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cb6d317f8a70940d2ced408d74052f2bf%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048597336750345&sdata=YfCAcNKIdXqeFQ7RyDKv1NZO4vgjTmc0d31xK9wJSX0%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 11:27 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle,
Excellent, will do: unfortunately switch to 2.16 was only thing that fixed
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:setupVirtualenv'.
> A problem occurred starting process 'command 'virtualenv''
…if you have suggestions there, I would be most appreciative.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:25 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
You should probably use 2.15, since 2.16 release artifacts have not been published yet. Just follow the instructions that say --runner=PortableRunner, not --runner=FlinkRunner, otherwise you'll hit that other deserialization bug that was mentioned..
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cb6d317f8a70940d2ced408d74052f2bf%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048597336760341&sdata=2AnoIOi2rlZnX1TkUNL5BcN8gPGTN7t0bbEtMLE4eCM%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 11:19 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle,
Happily: should I be working from (source) release-2.15.0 or release-2.16.0: presumably still specifying flink-1.8.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:16 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re Benjamin & Ankur: I don't think `--experiments=beam_fn_api` applies here. If I understand correctly, Matthew is just trying to run an old-fashioned Beam Java jar, nothing to do with portability/Python, and judging by the stack trace provided, https://issues.apache.org/jira/browse/BEAM-8037<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-8037&data=02%7C01%7Cmpatterson%40aligntech.com%7Cb6d317f8a70940d2ced408d74052f2bf%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048597336770339&sdata=mUyefW8aVc%2B6ycry0afaA%2BcRBXctn4aQ1tJJ7IICokA%3D&reserved=0> is not the issue here.
Re Matthew: I suspect a Flink version mismatch between the Flink cluster and the Beam runner. However, I am not familiar with the Maven commands you're using. Since you're planning on running Python eventually anyway, could you try following the "portable"/Python instructions on https://beam.apache.org/documentation/runners/flink/<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Fdocumentation%2Frunners%2Fflink%2F&data=02%7C01%7Cmpatterson%40aligntech.com%7Cb6d317f8a70940d2ced408d74052f2bf%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048597336770339&sdata=wcmNi%2FMN0ryq787vQ54%2FKn37GI97D8GcgWxAXOqVvD4%3D&reserved=0> and let us know how that goes?
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cb6d317f8a70940d2ced408d74052f2bf%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048597336780334&sdata=XCXdKOj8DiDZu9qNPRl8rl3xqZaczzoLgb%2FH%2FDi8zTc%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 10:52 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Thanks Ankur,
As one who speaks almost no gradle, is there a preferred way to get gradle to respect the conda configured python on `gradlew build` ?
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 1:50 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
We have this bug in 2.15 which is discussed here https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7Cb6d317f8a70940d2ced408d74052f2bf%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048597336790329&sdata=TkmwQ4GqRBycmn4s%2BgCkMyG8SO6dsYNKL9NRYIPlfX0%3D&reserved=0>
For now, please add "--experiments=beam_fn_api" to your pipeline to make it work.
On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Hi Ankur,
Yes, I was using 2.15, but was getting failure to deserialize.
Thanks,
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Wednesday, September 18, 2019 at 9:34 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Matthew,
Beam 2.16.0 is not yet released hence you are getting the error. Can you try using 2.15.0 version?
Thanks,
Ankur
On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Tried
"
mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
-DarchetypeVersion=2.16.0 \
-DgroupId=org.example \
-DartifactId=word-count-beam \
-Dversion="0.1" \
-Dpackage=org.apache.beam.examples \
-DinteractiveMode=false
"
got
-> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate (default-cli) on project standalone-pom: The desired archetype does not exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) -> [Help 1]
On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com>> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Ah. So maybe try 2.16.0?
On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com>> wrote:
> --experiments=beam_fn_api doesn't apply here, as this is a Java pipeline
> using the non-portable version of the Flink runner.
>
> Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cb6d317f8a70940d2ced408d74052f2bf%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048597336800326&sdata=8mLZgtzoEg9T%2BZmtEMLGUDjIofbeDuaRkBb5kY0HF9U%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
>
>
> On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <be...@gmail.com>>
> wrote:
>
> > Could you try adding "--experiments=beam_fn_api" to the runner options?
> > Kyle did mention that it should be fixed in 2.16.0.
> >
> > On 2019/09/17 16:40:41, Matthew Patterson <mp...@aligntech.com>>
> > wrote:
> > > Folks,
> > >
> > > Am sure I am doing it wrong, but have been beating head against wall
> > long enough I need to ask for help.
> > >
> > > My goal is to use the python SDK to run jobs on AWS flink cluster, but
> > very little is working, so I am trying to back down to the very simplest
> > thing that works, and build back up: not going so well.
> > >
> > > Now I am running, or trying to, on a minikube link cluster. Now running
> > from just java/mvn, so I go through the following:
> > >
> > > 1. bring-up-cluster (see below)
> > > 2. grab-the-code (see below)
> > > 3. maven-submit (see below)
> > >
> > > Eventual failure is
> > >
> > > “[ERROR] Failed to execute goal
> > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project
> > word-count-beam: An exception occured while executing the Java class.
> > Pipeline execution failed: Could not retrieve the execution result. (JobID:
> > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to
> > deserialize JobGraph.]”
> > >
> > > and log is attached (I hope)
> > >
> > > Other details:
> > >
> > > on osx
> > >
> > >
> > >
> > > ➜ minikube version
> > >
> > > minikube version: v1.3.1
> > >
> > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > >
> > >
> > > ➜ kubectl version
> > >
> > > Client Version: version.Info{Major:"1", Minor:"14",
> > GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > >
> > > Server Version: version.Info{Major:"1", Minor:"15",
> > GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > >
> > >
> > > ➜ flink --version
> > >
> > > Version: 1.8.1, Commit ID: 7297bac
> > >
> > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > 2.15.0
> > >
> > > ➜ curl localhost:8081
> > > <!--
> > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > ~ or more contributor license agreements. See the NOTICE file
> > > ~ distributed with this work for additional information
> > > ~ regarding copyright ownership. The ASF licenses this file
> > > ~ to you under the Apache License, Version 2.0 (the
> > > ~ "License"); you may not use this file except in compliance
> > > ~ with the License. You may obtain a copy of the License at
> > > ~
> > > ~ https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7Cb6d317f8a70940d2ced408d74052f2bf%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048597336800326&sdata=pU44hhkq715BHRd2UaPCdVjmXHrg8%2FVr9LmjiIj8Q24%3D&reserved=0>
> > > ~
> > > ~ Unless required by applicable law or agreed to in writing, software
> > > ~ distributed under the License is distributed on an "AS IS" BASIS,
> > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > implied.
> > > ~ See the License for the specific language governing permissions and
> > > ~ limitations under the License.
> > > -->
> > >
> > > <!doctype html>
> > > <html lang="en">
> > > <head>
> > > <meta charset="utf-8">
> > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > <meta name="viewport" content="width=device-width, initial-scale=1">
> > > <title>Apache Flink Web Dashboard</title>
> > > <link rel="apple-touch-icon" sizes="180x180"
> > href="assets/favicon/apple-touch-icon.png">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > <link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg"
> > color="#aa1919">
> > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > <meta name="msapplication-config"
> > content="assets/favicon/browserconfig.xml">
> > > <meta name="theme-color" content="#ffffff">
> > > <base href="./"><link rel="stylesheet"
> > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > <body>
> > > <flink-root></flink-root>
> > > <script type="text/javascript"
> > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js"
> > nomodule></script><script type="text/javascript"
> > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > type="text/javascript" src="main.dc6e0997d092df051db0.js"></script></body>
> > > </html>
> > >
> > > Thanks for any ideas,
> > > Matt
> > >
> > > >>> bring-up-cluster
> > >
> > > # [yamls from here](
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7Cb6d317f8a70940d2ced408d74052f2bf%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048597336810322&sdata=Xj2HftLL5JSL5IKKoFpEix5wuo15ctDbE9k64ti9%2B%2FM%3D&reserved=0>
> > )
> > > minikube start
> > > minikube ssh 'sudo ip link set docker0 promisc on'
> > >
> > > kubectl create -f flink-configuration-configmap.yaml
> > > kubectl create -f jobmanager-service.yaml
> > > kubectl create -f jobmanager-deployment.yaml
> > > kubectl create -f taskmanager-deployment.yaml
> > > kubectl apply -f jobmanager-rest-service.yaml
> > >
> > > >>> grab-the-code
> > > mvn archetype:generate \
> > > -DarchetypeGroupId=org.apache.beam \
> > > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > -DarchetypeVersion=2.15.0 \
> > > -DgroupId=org.example \
> > > -DartifactId=word-count-beam \
> > > -Dversion="0.1" \
> > > -Dpackage=org.apache.beam.examples \
> > > -DinteractiveMode=false
> > >
> > > >>> maven-submit
> > >
> > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in separate
> > shell, as blocking
> > > mvn package exec:java -X
> > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > -Dexec.args="--runner=FlinkRunner\
> > > --flinkMaster="localhost:8081"\
> > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > --output=<directory-that-exists>" -Pflink-runner
> > >
> > >
> > >
> >
>
Re: Word-count example
Posted by Matthew Patterson <mp...@aligntech.com>.
Sorry: needed to add LOOPBACK
From: Matthew Patterson <mp...@aligntech.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Monday, September 23, 2019 at 2:18 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
OK, now using release-2.15.0, calling word-count example with runner, after building & calling
./gradlew :runners:flink:1.5:job-server:runShadow (tried 1.8 as well)
, as
“f'--runner=PortableRunner', f'--job_endpoint=localhost:8099'
”
…and getting
File "/Users/mpatterson/miniconda/envs/aws/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 446, in wait_until_finish
self._job_id, self._state, self._last_error_message()))
RuntimeError: Pipeline BeamApp-mpatterson-0923181154-d9c8a662_737ea859-53e2-4d7e-ba94-2955a1642f94 failed in state FAILED: org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
Ideas? Thanks!
From: Matthew Patterson <mp...@aligntech.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Friday, September 20, 2019 at 9:27 AM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
411:
Although it does not seem correct, adding a symbolic link to `virtualenv` in ` <path-to-beam>/sdks/python` solves the issue of not finding `virtualenv`.
From: Matthew Patterson <mp...@aligntech.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Thursday, September 19, 2019 at 2:56 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
Re Kyle:
Just so you may believe me, see below(virtualenv installed, error occurs nonetheless): currently I am trying to figure out if it is a difference between .bash_profile and .bashrc.
Thanks,
Matt
“””
Caused by: net.rubygrapefruit.platform.NativeException: Could not start 'virtualenv'
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
at net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
at org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:97)
at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:70)
... 4 more
Caused by: java.io.IOException: Cannot run program "virtualenv" (in directory "/Users/mpatterson/dev/beam/sdks/python"): error=2, No such file or directory
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
... 8 more
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 0s
5 actionable tasks: 3 executed, 2 up-to-date
bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
MANIFEST.in build.gradle gen_protos.py setup.py tox.ini
OWNERS container scripts test-suites
apache_beam findSupportedPython.groovy setup.cfg test_config.py
bash-3.2$ pip install virtualenv
Requirement already satisfied: virtualenv in /Users/mpatterson/miniconda/lib/python3.7/site-packages (16.7.5)
bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
MANIFEST.in build.gradle gen_protos.py setup.py tox.ini
OWNERS container scripts test-suites
apache_beam findSupportedPython.groovy setup.cfg test_config.py
bash-3.2$
”””
From: Matthew Patterson <mp...@aligntech.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Thursday, September 19, 2019 at 2:31 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
Have done: hard to get gradle to respect env (conda? Bash? Fish? Zsh?):
From: Kyle Weaver <kc...@google.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Thursday, September 19, 2019 at 2:29 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I'm guessing you need to install virtualenv: `pip install virtualenv`
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C926073539c4d4d022fb808d7405274f7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048595224889006&sdata=pwoY1IhnKr7BXlaoZ%2B63SK%2FUPLUZ1ymctkHgr81W%2FUI%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 11:27 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle,
Excellent, will do: unfortunately switch to 2.16 was only thing that fixed
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:setupVirtualenv'.
> A problem occurred starting process 'command 'virtualenv''
…if you have suggestions there, I would be most appreciative.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:25 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
You should probably use 2.15, since 2.16 release artifacts have not been published yet. Just follow the instructions that say --runner=PortableRunner, not --runner=FlinkRunner, otherwise you'll hit that other deserialization bug that was mentioned..
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C926073539c4d4d022fb808d7405274f7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048595224899004&sdata=AfIqyAtsTESluja%2BpfB3s2VMM5ztqCrU6erWtNnPhew%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 11:19 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle,
Happily: should I be working from (source) release-2.15.0 or release-2.16.0: presumably still specifying flink-1.8.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:16 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re Benjamin & Ankur: I don't think `--experiments=beam_fn_api` applies here. If I understand correctly, Matthew is just trying to run an old-fashioned Beam Java jar, nothing to do with portability/Python, and judging by the stack trace provided, https://issues.apache.org/jira/browse/BEAM-8037<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-8037&data=02%7C01%7Cmpatterson%40aligntech.com%7C926073539c4d4d022fb808d7405274f7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048595224908995&sdata=7tHvk8KMsn3MPdTJuOwyobzN7wo%2FwasgzcFFP3ZbgfE%3D&reserved=0> is not the issue here.
Re Matthew: I suspect a Flink version mismatch between the Flink cluster and the Beam runner. However, I am not familiar with the Maven commands you're using. Since you're planning on running Python eventually anyway, could you try following the "portable"/Python instructions on https://beam.apache.org/documentation/runners/flink/<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Fdocumentation%2Frunners%2Fflink%2F&data=02%7C01%7Cmpatterson%40aligntech.com%7C926073539c4d4d022fb808d7405274f7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048595224908995&sdata=bxQCAZXfqB8A72xysdBzR2l3IFBG25U0%2Bb6gqLcD298%3D&reserved=0> and let us know how that goes?
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C926073539c4d4d022fb808d7405274f7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048595224918997&sdata=MyC%2BUO3kgB2f33GHqyKinxlxQCPovgPFk0PobGpvfkQ%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 10:52 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Thanks Ankur,
As one who speaks almost no gradle, is there a preferred way to get gradle to respect the conda configured python on `gradlew build` ?
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 1:50 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
We have this bug in 2.15 which is discussed here https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7C926073539c4d4d022fb808d7405274f7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048595224928992&sdata=RPs2zBEjqn9TgnEVWWscatmC9DeV65B6KgSk9RdFcBo%3D&reserved=0>
For now, please add "--experiments=beam_fn_api" to your pipeline to make it work.
On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Hi Ankur,
Yes, I was using 2.15, but was getting failure to deserialize.
Thanks,
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Wednesday, September 18, 2019 at 9:34 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Matthew,
Beam 2.16.0 is not yet released hence you are getting the error. Can you try using 2.15.0 version?
Thanks,
Ankur
On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Tried
"
mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
-DarchetypeVersion=2.16.0 \
-DgroupId=org.example \
-DartifactId=word-count-beam \
-Dversion="0.1" \
-Dpackage=org.apache.beam.examples \
-DinteractiveMode=false
"
got
-> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate (default-cli) on project standalone-pom: The desired archetype does not exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) -> [Help 1]
On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com>> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Ah. So maybe try 2.16.0?
On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com>> wrote:
> --experiments=beam_fn_api doesn't apply here, as this is a Java pipeline
> using the non-portable version of the Flink runner.
>
> Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C926073539c4d4d022fb808d7405274f7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048595224928992&sdata=XCiciM3hHt5MyYuJx4w%2FA53qKy6g9nzYqfsv8VkAbqI%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
>
>
> On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <be...@gmail.com>>
> wrote:
>
> > Could you try adding "--experiments=beam_fn_api" to the runner options?
> > Kyle did mention that it should be fixed in 2.16.0.
> >
> > On 2019/09/17 16:40:41, Matthew Patterson <mp...@aligntech.com>>
> > wrote:
> > > Folks,
> > >
> > > Am sure I am doing it wrong, but have been beating head against wall
> > long enough I need to ask for help.
> > >
> > > My goal is to use the python SDK to run jobs on AWS flink cluster, but
> > very little is working, so I am trying to back down to the very simplest
> > thing that works, and build back up: not going so well.
> > >
> > > Now I am running, or trying to, on a minikube link cluster. Now running
> > from just java/mvn, so I go through the following:
> > >
> > > 1. bring-up-cluster (see below)
> > > 2. grab-the-code (see below)
> > > 3. maven-submit (see below)
> > >
> > > Eventual failure is
> > >
> > > “[ERROR] Failed to execute goal
> > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project
> > word-count-beam: An exception occured while executing the Java class.
> > Pipeline execution failed: Could not retrieve the execution result. (JobID:
> > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to
> > deserialize JobGraph.]”
> > >
> > > and log is attached (I hope)
> > >
> > > Other details:
> > >
> > > on osx
> > >
> > >
> > >
> > > ➜ minikube version
> > >
> > > minikube version: v1.3.1
> > >
> > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > >
> > >
> > > ➜ kubectl version
> > >
> > > Client Version: version.Info{Major:"1", Minor:"14",
> > GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > >
> > > Server Version: version.Info{Major:"1", Minor:"15",
> > GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > >
> > >
> > > ➜ flink --version
> > >
> > > Version: 1.8.1, Commit ID: 7297bac
> > >
> > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > 2.15.0
> > >
> > > ➜ curl localhost:8081
> > > <!--
> > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > ~ or more contributor license agreements. See the NOTICE file
> > > ~ distributed with this work for additional information
> > > ~ regarding copyright ownership. The ASF licenses this file
> > > ~ to you under the Apache License, Version 2.0 (the
> > > ~ "License"); you may not use this file except in compliance
> > > ~ with the License. You may obtain a copy of the License at
> > > ~
> > > ~ https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C926073539c4d4d022fb808d7405274f7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048595224938984&sdata=vb2isCuzcnbVf3w94NtQ3Sws05j15I7BwDX0aQyq9Cs%3D&reserved=0>
> > > ~
> > > ~ Unless required by applicable law or agreed to in writing, software
> > > ~ distributed under the License is distributed on an "AS IS" BASIS,
> > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > implied.
> > > ~ See the License for the specific language governing permissions and
> > > ~ limitations under the License.
> > > -->
> > >
> > > <!doctype html>
> > > <html lang="en">
> > > <head>
> > > <meta charset="utf-8">
> > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > <meta name="viewport" content="width=device-width, initial-scale=1">
> > > <title>Apache Flink Web Dashboard</title>
> > > <link rel="apple-touch-icon" sizes="180x180"
> > href="assets/favicon/apple-touch-icon.png">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > <link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg"
> > color="#aa1919">
> > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > <meta name="msapplication-config"
> > content="assets/favicon/browserconfig.xml">
> > > <meta name="theme-color" content="#ffffff">
> > > <base href="./"><link rel="stylesheet"
> > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > <body>
> > > <flink-root></flink-root>
> > > <script type="text/javascript"
> > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js"
> > nomodule></script><script type="text/javascript"
> > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > type="text/javascript" src="main.dc6e0997d092df051db0.js"></script></body>
> > > </html>
> > >
> > > Thanks for any ideas,
> > > Matt
> > >
> > > >>> bring-up-cluster
> > >
> > > # [yamls from here](
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C926073539c4d4d022fb808d7405274f7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637048595224948978&sdata=2W76N2GeYj1E75KcBPfS1BAFQX5SXdlBQLDiYHLzo8E%3D&reserved=0>
> > )
> > > minikube start
> > > minikube ssh 'sudo ip link set docker0 promisc on'
> > >
> > > kubectl create -f flink-configuration-configmap.yaml
> > > kubectl create -f jobmanager-service.yaml
> > > kubectl create -f jobmanager-deployment.yaml
> > > kubectl create -f taskmanager-deployment.yaml
> > > kubectl apply -f jobmanager-rest-service.yaml
> > >
> > > >>> grab-the-code
> > > mvn archetype:generate \
> > > -DarchetypeGroupId=org.apache.beam \
> > > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > -DarchetypeVersion=2.15.0 \
> > > -DgroupId=org.example \
> > > -DartifactId=word-count-beam \
> > > -Dversion="0.1" \
> > > -Dpackage=org.apache.beam.examples \
> > > -DinteractiveMode=false
> > >
> > > >>> maven-submit
> > >
> > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in separate
> > shell, as blocking
> > > mvn package exec:java -X
> > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > -Dexec.args="--runner=FlinkRunner\
> > > --flinkMaster="localhost:8081"\
> > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > --output=<directory-that-exists>" -Pflink-runner
> > >
> > >
> > >
> >
>
Re: Word-count example
Posted by Matthew Patterson <mp...@aligntech.com>.
OK, now using release-2.15.0, calling word-count example with runner, after building & calling
./gradlew :runners:flink:1.5:job-server:runShadow (tried 1.8 as well)
, as
“f'--runner=PortableRunner', f'--job_endpoint=localhost:8099'
”
…and getting
File "/Users/mpatterson/miniconda/envs/aws/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py", line 446, in wait_until_finish
self._job_id, self._state, self._last_error_message()))
RuntimeError: Pipeline BeamApp-mpatterson-0923181154-d9c8a662_737ea859-53e2-4d7e-ba94-2955a1642f94 failed in state FAILED: org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
Ideas? Thanks!
From: Matthew Patterson <mp...@aligntech.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Friday, September 20, 2019 at 9:27 AM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
411:
Although it does not seem correct, adding a symbolic link to `virtualenv` in ` <path-to-beam>/sdks/python` solves the issue of not finding `virtualenv`.
From: Matthew Patterson <mp...@aligntech.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Thursday, September 19, 2019 at 2:56 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
Re Kyle:
Just so you may believe me, see below(virtualenv installed, error occurs nonetheless): currently I am trying to figure out if it is a difference between .bash_profile and .bashrc.
Thanks,
Matt
“””
Caused by: net.rubygrapefruit.platform.NativeException: Could not start 'virtualenv'
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
at net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
at org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:97)
at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:70)
... 4 more
Caused by: java.io.IOException: Cannot run program "virtualenv" (in directory "/Users/mpatterson/dev/beam/sdks/python"): error=2, No such file or directory
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
... 8 more
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 0s
5 actionable tasks: 3 executed, 2 up-to-date
bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
MANIFEST.in build.gradle gen_protos.py setup.py tox.ini
OWNERS container scripts test-suites
apache_beam findSupportedPython.groovy setup.cfg test_config.py
bash-3.2$ pip install virtualenv
Requirement already satisfied: virtualenv in /Users/mpatterson/miniconda/lib/python3.7/site-packages (16.7.5)
bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
MANIFEST.in build.gradle gen_protos.py setup.py tox.ini
OWNERS container scripts test-suites
apache_beam findSupportedPython.groovy setup.cfg test_config.py
bash-3.2$
”””
From: Matthew Patterson <mp...@aligntech.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Thursday, September 19, 2019 at 2:31 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
Have done: hard to get gradle to respect env (conda? Bash? Fish? Zsh?):
From: Kyle Weaver <kc...@google.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Thursday, September 19, 2019 at 2:29 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I'm guessing you need to install virtualenv: `pip install virtualenv`
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C02e2ba076f944c30e5b008d73dce388f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045828243006791&sdata=d9a%2FydV%2BSh0dgJyCk9ib0rXeKVg27Rma8Rq2QVwnsaQ%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 11:27 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle,
Excellent, will do: unfortunately switch to 2.16 was only thing that fixed
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:setupVirtualenv'.
> A problem occurred starting process 'command 'virtualenv''
…if you have suggestions there, I would be most appreciative.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:25 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
You should probably use 2.15, since 2.16 release artifacts have not been published yet. Just follow the instructions that say --runner=PortableRunner, not --runner=FlinkRunner, otherwise you'll hit that other deserialization bug that was mentioned..
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C02e2ba076f944c30e5b008d73dce388f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045828243006791&sdata=d9a%2FydV%2BSh0dgJyCk9ib0rXeKVg27Rma8Rq2QVwnsaQ%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 11:19 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle,
Happily: should I be working from (source) release-2.15.0 or release-2.16.0: presumably still specifying flink-1.8.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:16 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re Benjamin & Ankur: I don't think `--experiments=beam_fn_api` applies here. If I understand correctly, Matthew is just trying to run an old-fashioned Beam Java jar, nothing to do with portability/Python, and judging by the stack trace provided, https://issues.apache.org/jira/browse/BEAM-8037<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-8037&data=02%7C01%7Cmpatterson%40aligntech.com%7C02e2ba076f944c30e5b008d73dce388f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045828243016787&sdata=L6ySfdkI8xaszyynxatelc6cBA3r%2FPymuZ4Wy8zP8bY%3D&reserved=0> is not the issue here.
Re Matthew: I suspect a Flink version mismatch between the Flink cluster and the Beam runner. However, I am not familiar with the Maven commands you're using. Since you're planning on running Python eventually anyway, could you try following the "portable"/Python instructions on https://beam.apache.org/documentation/runners/flink/<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Fdocumentation%2Frunners%2Fflink%2F&data=02%7C01%7Cmpatterson%40aligntech.com%7C02e2ba076f944c30e5b008d73dce388f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045828243026781&sdata=2voIKobpaYDsw5v8h%2B1LfGR7fLgm%2BZvcIT7k3Yojhuo%3D&reserved=0> and let us know how that goes?
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C02e2ba076f944c30e5b008d73dce388f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045828243026781&sdata=ESmm0Nc%2F7eTmwvvMN5FWfrHacYVjYTFyulVNvU09gfw%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 10:52 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Thanks Ankur,
As one who speaks almost no gradle, is there a preferred way to get gradle to respect the conda configured python on `gradlew build` ?
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 1:50 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
We have this bug in 2.15 which is discussed here https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7C02e2ba076f944c30e5b008d73dce388f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045828243036781&sdata=2zdCDRwSt8FfamXkqRqccLJ5rLbOzf0ZDkpfCG%2B%2FQQw%3D&reserved=0>
For now, please add "--experiments=beam_fn_api" to your pipeline to make it work.
On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Hi Ankur,
Yes, I was using 2.15, but was getting failure to deserialize.
Thanks,
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Wednesday, September 18, 2019 at 9:34 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Matthew,
Beam 2.16.0 is not yet released hence you are getting the error. Can you try using 2.15.0 version?
Thanks,
Ankur
On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Tried
"
mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
-DarchetypeVersion=2.16.0 \
-DgroupId=org.example \
-DartifactId=word-count-beam \
-Dversion="0.1" \
-Dpackage=org.apache.beam.examples \
-DinteractiveMode=false
"
got
-> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate (default-cli) on project standalone-pom: The desired archetype does not exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) -> [Help 1]
On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com>> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Ah. So maybe try 2.16.0?
On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com>> wrote:
> --experiments=beam_fn_api doesn't apply here, as this is a Java pipeline
> using the non-portable version of the Flink runner.
>
> Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C02e2ba076f944c30e5b008d73dce388f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045828243046772&sdata=QgPELvKsS3aqLttqZbmhDGaqqiU1KoXcSr0J19Z8iNo%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
>
>
> On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <be...@gmail.com>>
> wrote:
>
> > Could you try adding "--experiments=beam_fn_api" to the runner options?
> > Kyle did mention that it should be fixed in 2.16.0.
> >
> > On 2019/09/17 16:40:41, Matthew Patterson <mp...@aligntech.com>>
> > wrote:
> > > Folks,
> > >
> > > Am sure I am doing it wrong, but have been beating head against wall
> > long enough I need to ask for help.
> > >
> > > My goal is to use the python SDK to run jobs on AWS flink cluster, but
> > very little is working, so I am trying to back down to the very simplest
> > thing that works, and build back up: not going so well.
> > >
> > > Now I am running, or trying to, on a minikube link cluster. Now running
> > from just java/mvn, so I go through the following:
> > >
> > > 1. bring-up-cluster (see below)
> > > 2. grab-the-code (see below)
> > > 3. maven-submit (see below)
> > >
> > > Eventual failure is
> > >
> > > “[ERROR] Failed to execute goal
> > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project
> > word-count-beam: An exception occured while executing the Java class.
> > Pipeline execution failed: Could not retrieve the execution result. (JobID:
> > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to
> > deserialize JobGraph.]”
> > >
> > > and log is attached (I hope)
> > >
> > > Other details:
> > >
> > > on osx
> > >
> > >
> > >
> > > ➜ minikube version
> > >
> > > minikube version: v1.3.1
> > >
> > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > >
> > >
> > > ➜ kubectl version
> > >
> > > Client Version: version.Info{Major:"1", Minor:"14",
> > GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > >
> > > Server Version: version.Info{Major:"1", Minor:"15",
> > GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > >
> > >
> > > ➜ flink --version
> > >
> > > Version: 1.8.1, Commit ID: 7297bac
> > >
> > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > 2.15.0
> > >
> > > ➜ curl localhost:8081
> > > <!--
> > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > ~ or more contributor license agreements. See the NOTICE file
> > > ~ distributed with this work for additional information
> > > ~ regarding copyright ownership. The ASF licenses this file
> > > ~ to you under the Apache License, Version 2.0 (the
> > > ~ "License"); you may not use this file except in compliance
> > > ~ with the License. You may obtain a copy of the License at
> > > ~
> > > ~ https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C02e2ba076f944c30e5b008d73dce388f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045828243046772&sdata=scrgzMN2aQXxafMoH2OXei2rngNvSBD5oR9CRxsyz5M%3D&reserved=0>
> > > ~
> > > ~ Unless required by applicable law or agreed to in writing, software
> > > ~ distributed under the License is distributed on an "AS IS" BASIS,
> > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > implied.
> > > ~ See the License for the specific language governing permissions and
> > > ~ limitations under the License.
> > > -->
> > >
> > > <!doctype html>
> > > <html lang="en">
> > > <head>
> > > <meta charset="utf-8">
> > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > <meta name="viewport" content="width=device-width, initial-scale=1">
> > > <title>Apache Flink Web Dashboard</title>
> > > <link rel="apple-touch-icon" sizes="180x180"
> > href="assets/favicon/apple-touch-icon.png">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > <link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg"
> > color="#aa1919">
> > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > <meta name="msapplication-config"
> > content="assets/favicon/browserconfig.xml">
> > > <meta name="theme-color" content="#ffffff">
> > > <base href="./"><link rel="stylesheet"
> > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > <body>
> > > <flink-root></flink-root>
> > > <script type="text/javascript"
> > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js"
> > nomodule></script><script type="text/javascript"
> > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > type="text/javascript" src="main.dc6e0997d092df051db0.js"></script></body>
> > > </html>
> > >
> > > Thanks for any ideas,
> > > Matt
> > >
> > > >>> bring-up-cluster
> > >
> > > # [yamls from here](
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C02e2ba076f944c30e5b008d73dce388f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045828243056768&sdata=eMbv%2BCRTjqYJCEEXvZ%2BO9bBFqQf1IJVnzifkmYJZO34%3D&reserved=0>
> > )
> > > minikube start
> > > minikube ssh 'sudo ip link set docker0 promisc on'
> > >
> > > kubectl create -f flink-configuration-configmap.yaml
> > > kubectl create -f jobmanager-service.yaml
> > > kubectl create -f jobmanager-deployment.yaml
> > > kubectl create -f taskmanager-deployment.yaml
> > > kubectl apply -f jobmanager-rest-service.yaml
> > >
> > > >>> grab-the-code
> > > mvn archetype:generate \
> > > -DarchetypeGroupId=org.apache.beam \
> > > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > -DarchetypeVersion=2.15.0 \
> > > -DgroupId=org.example \
> > > -DartifactId=word-count-beam \
> > > -Dversion="0.1" \
> > > -Dpackage=org.apache.beam.examples \
> > > -DinteractiveMode=false
> > >
> > > >>> maven-submit
> > >
> > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in separate
> > shell, as blocking
> > > mvn package exec:java -X
> > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > -Dexec.args="--runner=FlinkRunner\
> > > --flinkMaster="localhost:8081"\
> > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > --output=<directory-that-exists>" -Pflink-runner
> > >
> > >
> > >
> >
>
Re: Word-count example
Posted by Matthew Patterson <mp...@aligntech.com>.
411:
Although it does not seem correct, adding a symbolic link to `virtualenv` in ` <path-to-beam>/sdks/python` solves the issue of not finding `virtualenv`.
From: Matthew Patterson <mp...@aligntech.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Thursday, September 19, 2019 at 2:56 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
Re Kyle:
Just so you may believe me, see below(virtualenv installed, error occurs nonetheless): currently I am trying to figure out if it is a difference between .bash_profile and .bashrc.
Thanks,
Matt
“””
Caused by: net.rubygrapefruit.platform.NativeException: Could not start 'virtualenv'
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
at net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
at org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:97)
at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:70)
... 4 more
Caused by: java.io.IOException: Cannot run program "virtualenv" (in directory "/Users/mpatterson/dev/beam/sdks/python"): error=2, No such file or directory
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
... 8 more
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 0s
5 actionable tasks: 3 executed, 2 up-to-date
bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
MANIFEST.in build.gradle gen_protos.py setup.py tox.ini
OWNERS container scripts test-suites
apache_beam findSupportedPython.groovy setup.cfg test_config.py
bash-3.2$ pip install virtualenv
Requirement already satisfied: virtualenv in /Users/mpatterson/miniconda/lib/python3.7/site-packages (16.7.5)
bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
MANIFEST.in build.gradle gen_protos.py setup.py tox.ini
OWNERS container scripts test-suites
apache_beam findSupportedPython.groovy setup.cfg test_config.py
bash-3.2$
”””
From: Matthew Patterson <mp...@aligntech.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Thursday, September 19, 2019 at 2:31 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
Have done: hard to get gradle to respect env (conda? Bash? Fish? Zsh?):
From: Kyle Weaver <kc...@google.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Thursday, September 19, 2019 at 2:29 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I'm guessing you need to install virtualenv: `pip install virtualenv`
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc1bf0e49f68e45bdeb9b08d73d3301c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045161776630614&sdata=E8TbHgz3s6SkVEfmnIDTRf4HJCPbuirXRqsuq6Qmf5Q%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 11:27 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle,
Excellent, will do: unfortunately switch to 2.16 was only thing that fixed
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:setupVirtualenv'.
> A problem occurred starting process 'command 'virtualenv''
…if you have suggestions there, I would be most appreciative.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:25 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
You should probably use 2.15, since 2.16 release artifacts have not been published yet. Just follow the instructions that say --runner=PortableRunner, not --runner=FlinkRunner, otherwise you'll hit that other deserialization bug that was mentioned..
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc1bf0e49f68e45bdeb9b08d73d3301c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045161776640609&sdata=8Kuk6k%2FGIOfwBorSgtFHyACNsnIE2PaB9I6K71ZRiRs%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 11:19 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle,
Happily: should I be working from (source) release-2.15.0 or release-2.16.0: presumably still specifying flink-1.8.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:16 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re Benjamin & Ankur: I don't think `--experiments=beam_fn_api` applies here. If I understand correctly, Matthew is just trying to run an old-fashioned Beam Java jar, nothing to do with portability/Python, and judging by the stack trace provided, https://issues.apache.org/jira/browse/BEAM-8037<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-8037&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc1bf0e49f68e45bdeb9b08d73d3301c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045161776640609&sdata=smaTUDx9jfxQ4IgwnkV%2BI6v1y9kFhBeH1DrkbJm3ozY%3D&reserved=0> is not the issue here.
Re Matthew: I suspect a Flink version mismatch between the Flink cluster and the Beam runner. However, I am not familiar with the Maven commands you're using. Since you're planning on running Python eventually anyway, could you try following the "portable"/Python instructions on https://beam.apache.org/documentation/runners/flink/<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Fdocumentation%2Frunners%2Fflink%2F&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc1bf0e49f68e45bdeb9b08d73d3301c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045161776650606&sdata=VzYQsfCs4hHHt1BYxLRZmlKnzjuYPS5xyZFJFP5YHKo%3D&reserved=0> and let us know how that goes?
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc1bf0e49f68e45bdeb9b08d73d3301c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045161776660599&sdata=JqWXV%2F2z%2BqTIXLPxygAXsXmYWh0TW3QQAvZxcbsJEqY%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 10:52 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Thanks Ankur,
As one who speaks almost no gradle, is there a preferred way to get gradle to respect the conda configured python on `gradlew build` ?
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 1:50 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
We have this bug in 2.15 which is discussed here https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc1bf0e49f68e45bdeb9b08d73d3301c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045161776660599&sdata=9S71w%2Fe1Bz%2FpxYWVWngrpc9cfIASY0VUPks%2B%2BaKPe7M%3D&reserved=0>
For now, please add "--experiments=beam_fn_api" to your pipeline to make it work.
On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Hi Ankur,
Yes, I was using 2.15, but was getting failure to deserialize.
Thanks,
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Wednesday, September 18, 2019 at 9:34 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Matthew,
Beam 2.16.0 is not yet released hence you are getting the error. Can you try using 2.15.0 version?
Thanks,
Ankur
On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Tried
"
mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
-DarchetypeVersion=2.16.0 \
-DgroupId=org.example \
-DartifactId=word-count-beam \
-Dversion="0.1" \
-Dpackage=org.apache.beam.examples \
-DinteractiveMode=false
"
got
-> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate (default-cli) on project standalone-pom: The desired archetype does not exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) -> [Help 1]
On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com>> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Ah. So maybe try 2.16.0?
On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com>> wrote:
> --experiments=beam_fn_api doesn't apply here, as this is a Java pipeline
> using the non-portable version of the Flink runner.
>
> Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc1bf0e49f68e45bdeb9b08d73d3301c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045161776670597&sdata=lRruQG9xgd7MAnRc7%2FkGbH7bKKAMYaodA9H7%2F9mzQxg%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
>
>
> On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <be...@gmail.com>>
> wrote:
>
> > Could you try adding "--experiments=beam_fn_api" to the runner options?
> > Kyle did mention that it should be fixed in 2.16.0.
> >
> > On 2019/09/17 16:40:41, Matthew Patterson <mp...@aligntech.com>>
> > wrote:
> > > Folks,
> > >
> > > Am sure I am doing it wrong, but have been beating head against wall
> > long enough I need to ask for help.
> > >
> > > My goal is to use the python SDK to run jobs on AWS flink cluster, but
> > very little is working, so I am trying to back down to the very simplest
> > thing that works, and build back up: not going so well.
> > >
> > > Now I am running, or trying to, on a minikube link cluster. Now running
> > from just java/mvn, so I go through the following:
> > >
> > > 1. bring-up-cluster (see below)
> > > 2. grab-the-code (see below)
> > > 3. maven-submit (see below)
> > >
> > > Eventual failure is
> > >
> > > “[ERROR] Failed to execute goal
> > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project
> > word-count-beam: An exception occured while executing the Java class.
> > Pipeline execution failed: Could not retrieve the execution result. (JobID:
> > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to
> > deserialize JobGraph.]”
> > >
> > > and log is attached (I hope)
> > >
> > > Other details:
> > >
> > > on osx
> > >
> > >
> > >
> > > ➜ minikube version
> > >
> > > minikube version: v1.3.1
> > >
> > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > >
> > >
> > > ➜ kubectl version
> > >
> > > Client Version: version.Info{Major:"1", Minor:"14",
> > GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > >
> > > Server Version: version.Info{Major:"1", Minor:"15",
> > GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > >
> > >
> > > ➜ flink --version
> > >
> > > Version: 1.8.1, Commit ID: 7297bac
> > >
> > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > 2.15.0
> > >
> > > ➜ curl localhost:8081
> > > <!--
> > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > ~ or more contributor license agreements. See the NOTICE file
> > > ~ distributed with this work for additional information
> > > ~ regarding copyright ownership. The ASF licenses this file
> > > ~ to you under the Apache License, Version 2.0 (the
> > > ~ "License"); you may not use this file except in compliance
> > > ~ with the License. You may obtain a copy of the License at
> > > ~
> > > ~ https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc1bf0e49f68e45bdeb9b08d73d3301c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045161776680590&sdata=4XUeEb1jalWNXvBrpT91kzAII3uBmBogW8SWVGOttOo%3D&reserved=0>
> > > ~
> > > ~ Unless required by applicable law or agreed to in writing, software
> > > ~ distributed under the License is distributed on an "AS IS" BASIS,
> > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > implied.
> > > ~ See the License for the specific language governing permissions and
> > > ~ limitations under the License.
> > > -->
> > >
> > > <!doctype html>
> > > <html lang="en">
> > > <head>
> > > <meta charset="utf-8">
> > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > <meta name="viewport" content="width=device-width, initial-scale=1">
> > > <title>Apache Flink Web Dashboard</title>
> > > <link rel="apple-touch-icon" sizes="180x180"
> > href="assets/favicon/apple-touch-icon.png">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > <link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg"
> > color="#aa1919">
> > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > <meta name="msapplication-config"
> > content="assets/favicon/browserconfig.xml">
> > > <meta name="theme-color" content="#ffffff">
> > > <base href="./"><link rel="stylesheet"
> > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > <body>
> > > <flink-root></flink-root>
> > > <script type="text/javascript"
> > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js"
> > nomodule></script><script type="text/javascript"
> > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > type="text/javascript" src="main.dc6e0997d092df051db0.js"></script></body>
> > > </html>
> > >
> > > Thanks for any ideas,
> > > Matt
> > >
> > > >>> bring-up-cluster
> > >
> > > # [yamls from here](
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc1bf0e49f68e45bdeb9b08d73d3301c7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045161776690578&sdata=NObCn%2BrUDlHQXP9XUou0UQf6GDFqf0pPsmdd5IZr87E%3D&reserved=0>
> > )
> > > minikube start
> > > minikube ssh 'sudo ip link set docker0 promisc on'
> > >
> > > kubectl create -f flink-configuration-configmap.yaml
> > > kubectl create -f jobmanager-service.yaml
> > > kubectl create -f jobmanager-deployment.yaml
> > > kubectl create -f taskmanager-deployment.yaml
> > > kubectl apply -f jobmanager-rest-service.yaml
> > >
> > > >>> grab-the-code
> > > mvn archetype:generate \
> > > -DarchetypeGroupId=org.apache.beam \
> > > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > -DarchetypeVersion=2.15.0 \
> > > -DgroupId=org.example \
> > > -DartifactId=word-count-beam \
> > > -Dversion="0.1" \
> > > -Dpackage=org.apache.beam.examples \
> > > -DinteractiveMode=false
> > >
> > > >>> maven-submit
> > >
> > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in separate
> > shell, as blocking
> > > mvn package exec:java -X
> > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > -Dexec.args="--runner=FlinkRunner\
> > > --flinkMaster="localhost:8081"\
> > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > --output=<directory-that-exists>" -Pflink-runner
> > >
> > >
> > >
> >
>
Re: Word-count example
Posted by Matthew Patterson <mp...@aligntech.com>.
Re Kyle:
Just so you may believe me, see below(virtualenv installed, error occurs nonetheless): currently I am trying to figure out if it is a difference between .bash_profile and .bashrc.
Thanks,
Matt
“””
Caused by: net.rubygrapefruit.platform.NativeException: Could not start 'virtualenv'
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:27)
at net.rubygrapefruit.platform.internal.WrapperProcessLauncher.start(WrapperProcessLauncher.java:36)
at org.gradle.process.internal.ExecHandleRunner.startProcess(ExecHandleRunner.java:97)
at org.gradle.process.internal.ExecHandleRunner.run(ExecHandleRunner.java:70)
... 4 more
Caused by: java.io.IOException: Cannot run program "virtualenv" (in directory "/Users/mpatterson/dev/beam/sdks/python"): error=2, No such file or directory
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
... 8 more
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 0s
5 actionable tasks: 3 executed, 2 up-to-date
bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
MANIFEST.in build.gradle gen_protos.py setup.py tox.ini
OWNERS container scripts test-suites
apache_beam findSupportedPython.groovy setup.cfg test_config.py
bash-3.2$ pip install virtualenv
Requirement already satisfied: virtualenv in /Users/mpatterson/miniconda/lib/python3.7/site-packages (16.7.5)
bash-3.2$ ls /Users/mpatterson/dev/beam/sdks/python
MANIFEST.in build.gradle gen_protos.py setup.py tox.ini
OWNERS container scripts test-suites
apache_beam findSupportedPython.groovy setup.cfg test_config.py
bash-3.2$
”””
From: Matthew Patterson <mp...@aligntech.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Thursday, September 19, 2019 at 2:31 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
Have done: hard to get gradle to respect env (conda? Bash? Fish? Zsh?):
From: Kyle Weaver <kc...@google.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Thursday, September 19, 2019 at 2:29 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I'm guessing you need to install virtualenv: `pip install virtualenv`
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cd02587a174254890f5ad08d73d2f8ea7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045146795023575&sdata=hQ1mLx78c2clBiZaPPsS5U53OOKr1qeGK6hHe2%2BQrfw%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 11:27 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle,
Excellent, will do: unfortunately switch to 2.16 was only thing that fixed
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:setupVirtualenv'.
> A problem occurred starting process 'command 'virtualenv''
…if you have suggestions there, I would be most appreciative.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:25 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
You should probably use 2.15, since 2.16 release artifacts have not been published yet. Just follow the instructions that say --runner=PortableRunner, not --runner=FlinkRunner, otherwise you'll hit that other deserialization bug that was mentioned..
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cd02587a174254890f5ad08d73d2f8ea7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045146795033569&sdata=xMgPVHC0MVLH%2Fv9dXJp7d8gGlkoSS83TtX979%2BlrK70%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 11:19 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle,
Happily: should I be working from (source) release-2.15.0 or release-2.16.0: presumably still specifying flink-1.8.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:16 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re Benjamin & Ankur: I don't think `--experiments=beam_fn_api` applies here. If I understand correctly, Matthew is just trying to run an old-fashioned Beam Java jar, nothing to do with portability/Python, and judging by the stack trace provided, https://issues.apache.org/jira/browse/BEAM-8037<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-8037&data=02%7C01%7Cmpatterson%40aligntech.com%7Cd02587a174254890f5ad08d73d2f8ea7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045146795043571&sdata=D5yBeer%2F%2Fl%2Fht40%2Fq8R1SoarQs8JWlT7zeCh9Lv5KL8%3D&reserved=0> is not the issue here.
Re Matthew: I suspect a Flink version mismatch between the Flink cluster and the Beam runner. However, I am not familiar with the Maven commands you're using. Since you're planning on running Python eventually anyway, could you try following the "portable"/Python instructions on https://beam.apache.org/documentation/runners/flink/<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Fdocumentation%2Frunners%2Fflink%2F&data=02%7C01%7Cmpatterson%40aligntech.com%7Cd02587a174254890f5ad08d73d2f8ea7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045146795053565&sdata=1OIEYeRNcoM7nhsrnmVqUi5StSdxQsn106ApagPqmBg%3D&reserved=0> and let us know how that goes?
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cd02587a174254890f5ad08d73d2f8ea7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045146795053565&sdata=OmDIVim%2BHIuNcszDN7acYH6qNc5HmX%2FAvmJ1jjHxX%2Fo%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 10:52 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Thanks Ankur,
As one who speaks almost no gradle, is there a preferred way to get gradle to respect the conda configured python on `gradlew build` ?
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 1:50 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
We have this bug in 2.15 which is discussed here https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7Cd02587a174254890f5ad08d73d2f8ea7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045146795063559&sdata=ugUxc0oRruWNR20RIatkJssdm3v9UH63IiT9ZhdF3XM%3D&reserved=0>
For now, please add "--experiments=beam_fn_api" to your pipeline to make it work.
On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Hi Ankur,
Yes, I was using 2.15, but was getting failure to deserialize.
Thanks,
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Wednesday, September 18, 2019 at 9:34 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Matthew,
Beam 2.16.0 is not yet released hence you are getting the error. Can you try using 2.15.0 version?
Thanks,
Ankur
On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Tried
"
mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
-DarchetypeVersion=2.16.0 \
-DgroupId=org.example \
-DartifactId=word-count-beam \
-Dversion="0.1" \
-Dpackage=org.apache.beam.examples \
-DinteractiveMode=false
"
got
-> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate (default-cli) on project standalone-pom: The desired archetype does not exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) -> [Help 1]
On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com>> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Ah. So maybe try 2.16.0?
On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com>> wrote:
> --experiments=beam_fn_api doesn't apply here, as this is a Java pipeline
> using the non-portable version of the Flink runner.
>
> Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cd02587a174254890f5ad08d73d2f8ea7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045146795073554&sdata=6ooeA74idWHCvCEPjaBuFMOZyDpqDCGj3iGfzl3gGPc%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
>
>
> On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <be...@gmail.com>>
> wrote:
>
> > Could you try adding "--experiments=beam_fn_api" to the runner options?
> > Kyle did mention that it should be fixed in 2.16.0.
> >
> > On 2019/09/17 16:40:41, Matthew Patterson <mp...@aligntech.com>>
> > wrote:
> > > Folks,
> > >
> > > Am sure I am doing it wrong, but have been beating head against wall
> > long enough I need to ask for help.
> > >
> > > My goal is to use the python SDK to run jobs on AWS flink cluster, but
> > very little is working, so I am trying to back down to the very simplest
> > thing that works, and build back up: not going so well.
> > >
> > > Now I am running, or trying to, on a minikube link cluster. Now running
> > from just java/mvn, so I go through the following:
> > >
> > > 1. bring-up-cluster (see below)
> > > 2. grab-the-code (see below)
> > > 3. maven-submit (see below)
> > >
> > > Eventual failure is
> > >
> > > “[ERROR] Failed to execute goal
> > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project
> > word-count-beam: An exception occured while executing the Java class.
> > Pipeline execution failed: Could not retrieve the execution result. (JobID:
> > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to
> > deserialize JobGraph.]”
> > >
> > > and log is attached (I hope)
> > >
> > > Other details:
> > >
> > > on osx
> > >
> > >
> > >
> > > ➜ minikube version
> > >
> > > minikube version: v1.3.1
> > >
> > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > >
> > >
> > > ➜ kubectl version
> > >
> > > Client Version: version.Info{Major:"1", Minor:"14",
> > GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > >
> > > Server Version: version.Info{Major:"1", Minor:"15",
> > GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > >
> > >
> > > ➜ flink --version
> > >
> > > Version: 1.8.1, Commit ID: 7297bac
> > >
> > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > 2.15.0
> > >
> > > ➜ curl localhost:8081
> > > <!--
> > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > ~ or more contributor license agreements. See the NOTICE file
> > > ~ distributed with this work for additional information
> > > ~ regarding copyright ownership. The ASF licenses this file
> > > ~ to you under the Apache License, Version 2.0 (the
> > > ~ "License"); you may not use this file except in compliance
> > > ~ with the License. You may obtain a copy of the License at
> > > ~
> > > ~ https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7Cd02587a174254890f5ad08d73d2f8ea7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045146795073554&sdata=Ba%2FDJz8q2yXjCzkLggGePhyM2PFNUwy5d7ESOhyhBEQ%3D&reserved=0>
> > > ~
> > > ~ Unless required by applicable law or agreed to in writing, software
> > > ~ distributed under the License is distributed on an "AS IS" BASIS,
> > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > implied.
> > > ~ See the License for the specific language governing permissions and
> > > ~ limitations under the License.
> > > -->
> > >
> > > <!doctype html>
> > > <html lang="en">
> > > <head>
> > > <meta charset="utf-8">
> > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > <meta name="viewport" content="width=device-width, initial-scale=1">
> > > <title>Apache Flink Web Dashboard</title>
> > > <link rel="apple-touch-icon" sizes="180x180"
> > href="assets/favicon/apple-touch-icon.png">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > <link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg"
> > color="#aa1919">
> > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > <meta name="msapplication-config"
> > content="assets/favicon/browserconfig.xml">
> > > <meta name="theme-color" content="#ffffff">
> > > <base href="./"><link rel="stylesheet"
> > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > <body>
> > > <flink-root></flink-root>
> > > <script type="text/javascript"
> > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js"
> > nomodule></script><script type="text/javascript"
> > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > type="text/javascript" src="main.dc6e0997d092df051db0.js"></script></body>
> > > </html>
> > >
> > > Thanks for any ideas,
> > > Matt
> > >
> > > >>> bring-up-cluster
> > >
> > > # [yamls from here](
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7Cd02587a174254890f5ad08d73d2f8ea7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045146795083548&sdata=FvbhjO%2BBCI%2B8A%2FOYC6I1TlS5uyjXx0v7fK4HygniI4U%3D&reserved=0>
> > )
> > > minikube start
> > > minikube ssh 'sudo ip link set docker0 promisc on'
> > >
> > > kubectl create -f flink-configuration-configmap.yaml
> > > kubectl create -f jobmanager-service.yaml
> > > kubectl create -f jobmanager-deployment.yaml
> > > kubectl create -f taskmanager-deployment.yaml
> > > kubectl apply -f jobmanager-rest-service.yaml
> > >
> > > >>> grab-the-code
> > > mvn archetype:generate \
> > > -DarchetypeGroupId=org.apache.beam \
> > > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > -DarchetypeVersion=2.15.0 \
> > > -DgroupId=org.example \
> > > -DartifactId=word-count-beam \
> > > -Dversion="0.1" \
> > > -Dpackage=org.apache.beam.examples \
> > > -DinteractiveMode=false
> > >
> > > >>> maven-submit
> > >
> > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in separate
> > shell, as blocking
> > > mvn package exec:java -X
> > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > -Dexec.args="--runner=FlinkRunner\
> > > --flinkMaster="localhost:8081"\
> > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > --output=<directory-that-exists>" -Pflink-runner
> > >
> > >
> > >
> >
>
Re: Word-count example
Posted by Matthew Patterson <mp...@aligntech.com>.
Have done: hard to get gradle to respect env (conda? Bash? Fish? Zsh?):
From: Kyle Weaver <kc...@google.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Thursday, September 19, 2019 at 2:29 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I'm guessing you need to install virtualenv: `pip install virtualenv`
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C5b778446bf674b50481308d73d2f44e7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045145545724874&sdata=fbHPNWEgizfii8bbOhxk%2BbXL00G5F0SvbozABooIcmo%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 11:27 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle,
Excellent, will do: unfortunately switch to 2.16 was only thing that fixed
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:setupVirtualenv'.
> A problem occurred starting process 'command 'virtualenv''
…if you have suggestions there, I would be most appreciative.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:25 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
You should probably use 2.15, since 2.16 release artifacts have not been published yet. Just follow the instructions that say --runner=PortableRunner, not --runner=FlinkRunner, otherwise you'll hit that other deserialization bug that was mentioned..
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C5b778446bf674b50481308d73d2f44e7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045145545734870&sdata=bA5zcoEiQsMGo6R1GzMSCeGE6d%2BIjNSIFR9n25q3Avc%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 11:19 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle,
Happily: should I be working from (source) release-2.15.0 or release-2.16.0: presumably still specifying flink-1.8.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:16 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re Benjamin & Ankur: I don't think `--experiments=beam_fn_api` applies here. If I understand correctly, Matthew is just trying to run an old-fashioned Beam Java jar, nothing to do with portability/Python, and judging by the stack trace provided, https://issues.apache.org/jira/browse/BEAM-8037<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-8037&data=02%7C01%7Cmpatterson%40aligntech.com%7C5b778446bf674b50481308d73d2f44e7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045145545734870&sdata=JSQgeJOpiGhxhLW%2FABpIB07aREc0N%2FThMs%2FT9D%2BO46w%3D&reserved=0> is not the issue here.
Re Matthew: I suspect a Flink version mismatch between the Flink cluster and the Beam runner. However, I am not familiar with the Maven commands you're using. Since you're planning on running Python eventually anyway, could you try following the "portable"/Python instructions on https://beam.apache.org/documentation/runners/flink/<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Fdocumentation%2Frunners%2Fflink%2F&data=02%7C01%7Cmpatterson%40aligntech.com%7C5b778446bf674b50481308d73d2f44e7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045145545744866&sdata=DtAPRxTr%2F3IFHcY%2BjAajqhe%2BCq0dk1QV0vD8%2B55tUtw%3D&reserved=0> and let us know how that goes?
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C5b778446bf674b50481308d73d2f44e7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045145545754860&sdata=CKHY43ZYlM0dpF9Ol0vU%2B%2Fu6OnI4QrbW2S7q8YTaraQ%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 10:52 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Thanks Ankur,
As one who speaks almost no gradle, is there a preferred way to get gradle to respect the conda configured python on `gradlew build` ?
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 1:50 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
We have this bug in 2.15 which is discussed here https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7C5b778446bf674b50481308d73d2f44e7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045145545754860&sdata=UM7B048MXhC2TeHQ4%2FU1GKMkNFDxaPIuyOye3tTdMMw%3D&reserved=0>
For now, please add "--experiments=beam_fn_api" to your pipeline to make it work.
On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Hi Ankur,
Yes, I was using 2.15, but was getting failure to deserialize.
Thanks,
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Wednesday, September 18, 2019 at 9:34 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Matthew,
Beam 2.16.0 is not yet released hence you are getting the error. Can you try using 2.15.0 version?
Thanks,
Ankur
On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Tried
"
mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
-DarchetypeVersion=2.16.0 \
-DgroupId=org.example \
-DartifactId=word-count-beam \
-Dversion="0.1" \
-Dpackage=org.apache.beam.examples \
-DinteractiveMode=false
"
got
-> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate (default-cli) on project standalone-pom: The desired archetype does not exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) -> [Help 1]
On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com>> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Ah. So maybe try 2.16.0?
On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com>> wrote:
> --experiments=beam_fn_api doesn't apply here, as this is a Java pipeline
> using the non-portable version of the Flink runner.
>
> Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C5b778446bf674b50481308d73d2f44e7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045145545764859&sdata=WMCbBm77j4V2L1GInUkprvrH98l8oemMfQv52fWHlhs%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
>
>
> On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <be...@gmail.com>>
> wrote:
>
> > Could you try adding "--experiments=beam_fn_api" to the runner options?
> > Kyle did mention that it should be fixed in 2.16.0.
> >
> > On 2019/09/17 16:40:41, Matthew Patterson <mp...@aligntech.com>>
> > wrote:
> > > Folks,
> > >
> > > Am sure I am doing it wrong, but have been beating head against wall
> > long enough I need to ask for help.
> > >
> > > My goal is to use the python SDK to run jobs on AWS flink cluster, but
> > very little is working, so I am trying to back down to the very simplest
> > thing that works, and build back up: not going so well.
> > >
> > > Now I am running, or trying to, on a minikube link cluster. Now running
> > from just java/mvn, so I go through the following:
> > >
> > > 1. bring-up-cluster (see below)
> > > 2. grab-the-code (see below)
> > > 3. maven-submit (see below)
> > >
> > > Eventual failure is
> > >
> > > “[ERROR] Failed to execute goal
> > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project
> > word-count-beam: An exception occured while executing the Java class.
> > Pipeline execution failed: Could not retrieve the execution result. (JobID:
> > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to
> > deserialize JobGraph.]”
> > >
> > > and log is attached (I hope)
> > >
> > > Other details:
> > >
> > > on osx
> > >
> > >
> > >
> > > ➜ minikube version
> > >
> > > minikube version: v1.3.1
> > >
> > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > >
> > >
> > > ➜ kubectl version
> > >
> > > Client Version: version.Info{Major:"1", Minor:"14",
> > GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > >
> > > Server Version: version.Info{Major:"1", Minor:"15",
> > GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > >
> > >
> > > ➜ flink --version
> > >
> > > Version: 1.8.1, Commit ID: 7297bac
> > >
> > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > 2.15.0
> > >
> > > ➜ curl localhost:8081
> > > <!--
> > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > ~ or more contributor license agreements. See the NOTICE file
> > > ~ distributed with this work for additional information
> > > ~ regarding copyright ownership. The ASF licenses this file
> > > ~ to you under the Apache License, Version 2.0 (the
> > > ~ "License"); you may not use this file except in compliance
> > > ~ with the License. You may obtain a copy of the License at
> > > ~
> > > ~ https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C5b778446bf674b50481308d73d2f44e7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045145545764859&sdata=ZZ5kp8ly%2BiH0AkywwAhBslEKPNQqdRMXnqlJp7FaZ%2FM%3D&reserved=0>
> > > ~
> > > ~ Unless required by applicable law or agreed to in writing, software
> > > ~ distributed under the License is distributed on an "AS IS" BASIS,
> > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > implied.
> > > ~ See the License for the specific language governing permissions and
> > > ~ limitations under the License.
> > > -->
> > >
> > > <!doctype html>
> > > <html lang="en">
> > > <head>
> > > <meta charset="utf-8">
> > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > <meta name="viewport" content="width=device-width, initial-scale=1">
> > > <title>Apache Flink Web Dashboard</title>
> > > <link rel="apple-touch-icon" sizes="180x180"
> > href="assets/favicon/apple-touch-icon.png">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > <link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg"
> > color="#aa1919">
> > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > <meta name="msapplication-config"
> > content="assets/favicon/browserconfig.xml">
> > > <meta name="theme-color" content="#ffffff">
> > > <base href="./"><link rel="stylesheet"
> > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > <body>
> > > <flink-root></flink-root>
> > > <script type="text/javascript"
> > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js"
> > nomodule></script><script type="text/javascript"
> > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > type="text/javascript" src="main.dc6e0997d092df051db0.js"></script></body>
> > > </html>
> > >
> > > Thanks for any ideas,
> > > Matt
> > >
> > > >>> bring-up-cluster
> > >
> > > # [yamls from here](
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C5b778446bf674b50481308d73d2f44e7%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045145545774851&sdata=M5Gp1MlGwBsd9Cz9gZ8UjxzZ1jwJMUS1BMEtcSW0Hh0%3D&reserved=0>
> > )
> > > minikube start
> > > minikube ssh 'sudo ip link set docker0 promisc on'
> > >
> > > kubectl create -f flink-configuration-configmap.yaml
> > > kubectl create -f jobmanager-service.yaml
> > > kubectl create -f jobmanager-deployment.yaml
> > > kubectl create -f taskmanager-deployment.yaml
> > > kubectl apply -f jobmanager-rest-service.yaml
> > >
> > > >>> grab-the-code
> > > mvn archetype:generate \
> > > -DarchetypeGroupId=org.apache.beam \
> > > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > -DarchetypeVersion=2.15.0 \
> > > -DgroupId=org.example \
> > > -DartifactId=word-count-beam \
> > > -Dversion="0.1" \
> > > -Dpackage=org.apache.beam.examples \
> > > -DinteractiveMode=false
> > >
> > > >>> maven-submit
> > >
> > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in separate
> > shell, as blocking
> > > mvn package exec:java -X
> > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > -Dexec.args="--runner=FlinkRunner\
> > > --flinkMaster="localhost:8081"\
> > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > --output=<directory-that-exists>" -Pflink-runner
> > >
> > >
> > >
> >
>
Re: Word-count example
Posted by Kyle Weaver <kc...@google.com>.
I'm guessing you need to install virtualenv: `pip install virtualenv`
Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com
On Thu, Sep 19, 2019 at 11:27 AM Matthew Patterson <mp...@aligntech.com>
wrote:
> Kyle,
>
>
>
> Excellent, will do: unfortunately switch to 2.16 was only thing that fixed
>
>
>
> FAILURE: Build failed with an exception.
>
>
>
> * What went wrong:
>
> Execution failed for task ':sdks:python:setupVirtualenv'.
>
> > A problem occurred starting process 'command 'virtualenv''
>
>
>
>
>
> …if you have suggestions there, I would be most appreciative.
>
> Matt
>
>
>
>
>
> *From: *Kyle Weaver <kc...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 2:25 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> You should probably use 2.15, since 2.16 release artifacts have not been
> published yet. Just follow the instructions that
> say --runner=PortableRunner, not --runner=FlinkRunner, otherwise you'll hit
> that other deserialization bug that was mentioned..
>
>
> Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466096491&sdata=K%2F35WYwuy5EMBvpFX2uFmlJnoQSX94FuZNg%2Bj5RNelg%3D&reserved=0>
> | kcweaver@google.com
>
>
>
>
>
> On Thu, Sep 19, 2019 at 11:19 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Kyle,
>
>
>
> Happily: should I be working from (source) release-2.15.0 or
> release-2.16.0: presumably still specifying flink-1.8.
>
> Matt
>
>
>
> *From: *Kyle Weaver <kc...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 2:16 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> Re Benjamin & Ankur: I don't think `--experiments=beam_fn_api` applies
> here. If I understand correctly, Matthew is just trying to run an
> old-fashioned Beam Java jar, nothing to do with portability/Python, and
> judging by the stack trace provided,
> https://issues.apache.org/jira/browse/BEAM-8037
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-8037&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466106484&sdata=U0r4Nx4dJuEy2IHe6o1ozKpe6uwCo6F%2F3%2Bn4mK9pge0%3D&reserved=0> is
> not the issue here.
>
>
>
> Re Matthew: I suspect a Flink version mismatch between the Flink cluster
> and the Beam runner. However, I am not familiar with the Maven commands
> you're using. Since you're planning on running Python eventually anyway,
> could you try following the "portable"/Python instructions on
> https://beam.apache.org/documentation/runners/flink/
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Fdocumentation%2Frunners%2Fflink%2F&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466106484&sdata=I9WBW9BkLx3RMsBSD4NPZUJHugmyh0UpxGWzHdSeLbU%3D&reserved=0> and
> let us know how that goes?
>
>
> Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466116480&sdata=fDrnP5FRiHYt0ceL98BywtKPtHMAA4uKlF40Xp5gi9k%3D&reserved=0>
> | kcweaver@google.com
>
>
>
>
>
> On Thu, Sep 19, 2019 at 10:52 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Thanks Ankur,
>
>
>
> As one who speaks almost no gradle, is there a preferred way to get gradle
> to respect the conda configured python on `gradlew build` ?
>
>
>
> Matt
>
>
>
> *From: *Ankur Goenka <go...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 1:50 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> We have this bug in 2.15 which is discussed here
> https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466126476&sdata=aMWF9Sbh2lKEupW45eddMclF2fZ5on9kJHgtQfAJIAQ%3D&reserved=0>
>
>
>
> For now, please add "--experiments=beam_fn_api" to your pipeline to make
> it work.
>
>
>
> On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Hi Ankur,
>
>
>
> Yes, I was using 2.15, but was getting failure to deserialize.
>
>
>
> Thanks,
>
> Matt
>
>
>
> *From: *Ankur Goenka <go...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Wednesday, September 18, 2019 at 9:34 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> Hi Matthew,
>
>
>
> Beam 2.16.0 is not yet released hence you are getting the error. Can you
> try using 2.15.0 version?
>
>
>
> Thanks,
>
> Ankur
>
>
>
> On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Tried
>
> "
> mvn archetype:generate \
> -DarchetypeGroupId=org.apache.beam \
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> -DarchetypeVersion=2.16.0 \
> -DgroupId=org.example \
> -DartifactId=word-count-beam \
> -Dversion="0.1" \
> -Dpackage=org.apache.beam.examples \
> -DinteractiveMode=false
> "
> got
>
> -> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate
> (default-cli) on project standalone-pom: The desired archetype does not
> exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) ->
> [Help 1]
>
>
> On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com> wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
>
> Ah. So maybe try 2.16.0?
>
> On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com> wrote:
> > --experiments=beam_fn_api doesn't apply here, as this is a Java
> pipeline
> > using the non-portable version of the Flink runner.
> >
> > Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466126476&sdata=t17K7Pf1pA4GQd1gbEEEaPiO%2ByyeggXTheF6szt5e4c%3D&reserved=0>
> | kcweaver@google.com
> >
> >
> > On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <
> benjamintanweihao@gmail.com>
> > wrote:
> >
> > > Could you try adding "--experiments=beam_fn_api" to the runner
> options?
> > > Kyle did mention that it should be fixed in 2.16.0.
> > >
> > > On 2019/09/17 16:40:41, Matthew Patterson <
> mpatterson@aligntech.com>
> > > wrote:
> > > > Folks,
> > > >
> > > > Am sure I am doing it wrong, but have been beating head against
> wall
> > > long enough I need to ask for help.
> > > >
> > > > My goal is to use the python SDK to run jobs on AWS flink
> cluster, but
> > > very little is working, so I am trying to back down to the very
> simplest
> > > thing that works, and build back up: not going so well.
> > > >
> > > > Now I am running, or trying to, on a minikube link cluster. Now
> running
> > > from just java/mvn, so I go through the following:
> > > >
> > > > 1. bring-up-cluster (see below)
> > > > 2. grab-the-code (see below)
> > > > 3. maven-submit (see below)
> > > >
> > > > Eventual failure is
> > > >
> > > > “[ERROR] Failed to execute goal
> > > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on
> project
> > > word-count-beam: An exception occured while executing the Java
> class.
> > > Pipeline execution failed: Could not retrieve the execution
> result. (JobID:
> > > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph.
> [Failed to
> > > deserialize JobGraph.]”
> > > >
> > > > and log is attached (I hope)
> > > >
> > > > Other details:
> > > >
> > > > on osx
> > > >
> > > >
> > > >
> > > > ➜ minikube version
> > > >
> > > > minikube version: v1.3.1
> > > >
> > > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > > >
> > > >
> > > > ➜ kubectl version
> > > >
> > > > Client Version: version.Info{Major:"1", Minor:"14",
> > > GitVersion:"v1.14.6",
> GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > > >
> > > > Server Version: version.Info{Major:"1", Minor:"15",
> > > GitVersion:"v1.15.2",
> GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > > >
> > > >
> > > > ➜ flink --version
> > > >
> > > > Version: 1.8.1, Commit ID: 7297bac
> > > >
> > > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > > 2.15.0
> > > >
> > > > ➜ curl localhost:8081
> > > > <!--
> > > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > > ~ or more contributor license agreements. See the NOTICE file
> > > > ~ distributed with this work for additional information
> > > > ~ regarding copyright ownership. The ASF licenses this file
> > > > ~ to you under the Apache License, Version 2.0 (the
> > > > ~ "License"); you may not use this file except in compliance
> > > > ~ with the License. You may obtain a copy of the License at
> > > > ~
> > > > ~
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466136471&sdata=eUFzyBye%2F13md7qkKI0jHnAT1suszfti6PgOPADLDno%3D&reserved=0>
> > > > ~
> > > > ~ Unless required by applicable law or agreed to in writing,
> software
> > > > ~ distributed under the License is distributed on an "AS IS"
> BASIS,
> > > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > > implied.
> > > > ~ See the License for the specific language governing
> permissions and
> > > > ~ limitations under the License.
> > > > -->
> > > >
> > > > <!doctype html>
> > > > <html lang="en">
> > > > <head>
> > > > <meta charset="utf-8">
> > > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > > <meta name="viewport" content="width=device-width,
> initial-scale=1">
> > > > <title>Apache Flink Web Dashboard</title>
> > > > <link rel="apple-touch-icon" sizes="180x180"
> > > href="assets/favicon/apple-touch-icon.png">
> > > > <link rel="icon" type="image/png"
> > > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > > <link rel="icon" type="image/png"
> > > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > > <link rel="mask-icon"
> href="assets/favicon/safari-pinned-tab.svg"
> > > color="#aa1919">
> > > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > > <meta name="msapplication-config"
> > > content="assets/favicon/browserconfig.xml">
> > > > <meta name="theme-color" content="#ffffff">
> > > > <base href="./"><link rel="stylesheet"
> > > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > > <body>
> > > > <flink-root></flink-root>
> > > > <script type="text/javascript"
> > > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > > type="text/javascript"
> src="es2015-polyfills.5e343224e81eefb7658e.js"
> > > nomodule></script><script type="text/javascript"
> > > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > > type="text/javascript"
> src="main.dc6e0997d092df051db0.js"></script></body>
> > > > </html>
> > > >
> > > > Thanks for any ideas,
> > > > Matt
> > > >
> > > > >>> bring-up-cluster
> > > >
> > > > # [yamls from here](
> > >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466146466&sdata=UtKVyOEu6bFQd6g06WHtBv%2FZQE7Dj2umyr5aPhuAKlg%3D&reserved=0>
> > > )
> > > > minikube start
> > > > minikube ssh 'sudo ip link set docker0 promisc on'
> > > >
> > > > kubectl create -f flink-configuration-configmap.yaml
> > > > kubectl create -f jobmanager-service.yaml
> > > > kubectl create -f jobmanager-deployment.yaml
> > > > kubectl create -f taskmanager-deployment.yaml
> > > > kubectl apply -f jobmanager-rest-service.yaml
> > > >
> > > > >>> grab-the-code
> > > > mvn archetype:generate \
> > > > -DarchetypeGroupId=org.apache.beam \
> > > >
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > > -DarchetypeVersion=2.15.0 \
> > > > -DgroupId=org.example \
> > > > -DartifactId=word-count-beam \
> > > > -Dversion="0.1" \
> > > > -Dpackage=org.apache.beam.examples \
> > > > -DinteractiveMode=false
> > > >
> > > > >>> maven-submit
> > > >
> > > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in
> separate
> > > shell, as blocking
> > > > mvn package exec:java -X
> > > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > > -Dexec.args="--runner=FlinkRunner\
> > > > --flinkMaster="localhost:8081"\
> > > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > > --output=<directory-that-exists>" -Pflink-runner
> > > >
> > > >
> > > >
> > >
> >
>
>
Re: Word-count example
Posted by Matthew Patterson <mp...@aligntech.com>.
Kyle,
Excellent, will do: unfortunately switch to 2.16 was only thing that fixed
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:setupVirtualenv'.
> A problem occurred starting process 'command 'virtualenv''
…if you have suggestions there, I would be most appreciative.
Matt
From: Kyle Weaver <kc...@google.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Thursday, September 19, 2019 at 2:25 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
You should probably use 2.15, since 2.16 release artifacts have not been published yet. Just follow the instructions that say --runner=PortableRunner, not --runner=FlinkRunner, otherwise you'll hit that other deserialization bug that was mentioned..
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466096491&sdata=K%2F35WYwuy5EMBvpFX2uFmlJnoQSX94FuZNg%2Bj5RNelg%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 11:19 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Kyle,
Happily: should I be working from (source) release-2.15.0 or release-2.16.0: presumably still specifying flink-1.8.
Matt
From: Kyle Weaver <kc...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 2:16 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re Benjamin & Ankur: I don't think `--experiments=beam_fn_api` applies here. If I understand correctly, Matthew is just trying to run an old-fashioned Beam Java jar, nothing to do with portability/Python, and judging by the stack trace provided, https://issues.apache.org/jira/browse/BEAM-8037<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-8037&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466106484&sdata=U0r4Nx4dJuEy2IHe6o1ozKpe6uwCo6F%2F3%2Bn4mK9pge0%3D&reserved=0> is not the issue here.
Re Matthew: I suspect a Flink version mismatch between the Flink cluster and the Beam runner. However, I am not familiar with the Maven commands you're using. Since you're planning on running Python eventually anyway, could you try following the "portable"/Python instructions on https://beam.apache.org/documentation/runners/flink/<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Fdocumentation%2Frunners%2Fflink%2F&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466106484&sdata=I9WBW9BkLx3RMsBSD4NPZUJHugmyh0UpxGWzHdSeLbU%3D&reserved=0> and let us know how that goes?
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466116480&sdata=fDrnP5FRiHYt0ceL98BywtKPtHMAA4uKlF40Xp5gi9k%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 10:52 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Thanks Ankur,
As one who speaks almost no gradle, is there a preferred way to get gradle to respect the conda configured python on `gradlew build` ?
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 1:50 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
We have this bug in 2.15 which is discussed here https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466126476&sdata=aMWF9Sbh2lKEupW45eddMclF2fZ5on9kJHgtQfAJIAQ%3D&reserved=0>
For now, please add "--experiments=beam_fn_api" to your pipeline to make it work.
On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Hi Ankur,
Yes, I was using 2.15, but was getting failure to deserialize.
Thanks,
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Wednesday, September 18, 2019 at 9:34 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Matthew,
Beam 2.16.0 is not yet released hence you are getting the error. Can you try using 2.15.0 version?
Thanks,
Ankur
On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Tried
"
mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
-DarchetypeVersion=2.16.0 \
-DgroupId=org.example \
-DartifactId=word-count-beam \
-Dversion="0.1" \
-Dpackage=org.apache.beam.examples \
-DinteractiveMode=false
"
got
-> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate (default-cli) on project standalone-pom: The desired archetype does not exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) -> [Help 1]
On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com>> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Ah. So maybe try 2.16.0?
On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com>> wrote:
> --experiments=beam_fn_api doesn't apply here, as this is a Java pipeline
> using the non-portable version of the Flink runner.
>
> Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466126476&sdata=t17K7Pf1pA4GQd1gbEEEaPiO%2ByyeggXTheF6szt5e4c%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
>
>
> On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <be...@gmail.com>>
> wrote:
>
> > Could you try adding "--experiments=beam_fn_api" to the runner options?
> > Kyle did mention that it should be fixed in 2.16.0.
> >
> > On 2019/09/17 16:40:41, Matthew Patterson <mp...@aligntech.com>>
> > wrote:
> > > Folks,
> > >
> > > Am sure I am doing it wrong, but have been beating head against wall
> > long enough I need to ask for help.
> > >
> > > My goal is to use the python SDK to run jobs on AWS flink cluster, but
> > very little is working, so I am trying to back down to the very simplest
> > thing that works, and build back up: not going so well.
> > >
> > > Now I am running, or trying to, on a minikube link cluster. Now running
> > from just java/mvn, so I go through the following:
> > >
> > > 1. bring-up-cluster (see below)
> > > 2. grab-the-code (see below)
> > > 3. maven-submit (see below)
> > >
> > > Eventual failure is
> > >
> > > “[ERROR] Failed to execute goal
> > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project
> > word-count-beam: An exception occured while executing the Java class.
> > Pipeline execution failed: Could not retrieve the execution result. (JobID:
> > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to
> > deserialize JobGraph.]”
> > >
> > > and log is attached (I hope)
> > >
> > > Other details:
> > >
> > > on osx
> > >
> > >
> > >
> > > ➜ minikube version
> > >
> > > minikube version: v1.3.1
> > >
> > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > >
> > >
> > > ➜ kubectl version
> > >
> > > Client Version: version.Info{Major:"1", Minor:"14",
> > GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > >
> > > Server Version: version.Info{Major:"1", Minor:"15",
> > GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > >
> > >
> > > ➜ flink --version
> > >
> > > Version: 1.8.1, Commit ID: 7297bac
> > >
> > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > 2.15.0
> > >
> > > ➜ curl localhost:8081
> > > <!--
> > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > ~ or more contributor license agreements. See the NOTICE file
> > > ~ distributed with this work for additional information
> > > ~ regarding copyright ownership. The ASF licenses this file
> > > ~ to you under the Apache License, Version 2.0 (the
> > > ~ "License"); you may not use this file except in compliance
> > > ~ with the License. You may obtain a copy of the License at
> > > ~
> > > ~ https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466136471&sdata=eUFzyBye%2F13md7qkKI0jHnAT1suszfti6PgOPADLDno%3D&reserved=0>
> > > ~
> > > ~ Unless required by applicable law or agreed to in writing, software
> > > ~ distributed under the License is distributed on an "AS IS" BASIS,
> > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > implied.
> > > ~ See the License for the specific language governing permissions and
> > > ~ limitations under the License.
> > > -->
> > >
> > > <!doctype html>
> > > <html lang="en">
> > > <head>
> > > <meta charset="utf-8">
> > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > <meta name="viewport" content="width=device-width, initial-scale=1">
> > > <title>Apache Flink Web Dashboard</title>
> > > <link rel="apple-touch-icon" sizes="180x180"
> > href="assets/favicon/apple-touch-icon.png">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > <link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg"
> > color="#aa1919">
> > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > <meta name="msapplication-config"
> > content="assets/favicon/browserconfig.xml">
> > > <meta name="theme-color" content="#ffffff">
> > > <base href="./"><link rel="stylesheet"
> > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > <body>
> > > <flink-root></flink-root>
> > > <script type="text/javascript"
> > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js"
> > nomodule></script><script type="text/javascript"
> > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > type="text/javascript" src="main.dc6e0997d092df051db0.js"></script></body>
> > > </html>
> > >
> > > Thanks for any ideas,
> > > Matt
> > >
> > > >>> bring-up-cluster
> > >
> > > # [yamls from here](
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7Ccc6ec10a47b64712880508d73d2ec84e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045143466146466&sdata=UtKVyOEu6bFQd6g06WHtBv%2FZQE7Dj2umyr5aPhuAKlg%3D&reserved=0>
> > )
> > > minikube start
> > > minikube ssh 'sudo ip link set docker0 promisc on'
> > >
> > > kubectl create -f flink-configuration-configmap.yaml
> > > kubectl create -f jobmanager-service.yaml
> > > kubectl create -f jobmanager-deployment.yaml
> > > kubectl create -f taskmanager-deployment.yaml
> > > kubectl apply -f jobmanager-rest-service.yaml
> > >
> > > >>> grab-the-code
> > > mvn archetype:generate \
> > > -DarchetypeGroupId=org.apache.beam \
> > > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > -DarchetypeVersion=2.15.0 \
> > > -DgroupId=org.example \
> > > -DartifactId=word-count-beam \
> > > -Dversion="0.1" \
> > > -Dpackage=org.apache.beam.examples \
> > > -DinteractiveMode=false
> > >
> > > >>> maven-submit
> > >
> > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in separate
> > shell, as blocking
> > > mvn package exec:java -X
> > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > -Dexec.args="--runner=FlinkRunner\
> > > --flinkMaster="localhost:8081"\
> > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > --output=<directory-that-exists>" -Pflink-runner
> > >
> > >
> > >
> >
>
Re: Word-count example
Posted by Kyle Weaver <kc...@google.com>.
You should probably use 2.15, since 2.16 release artifacts have not been
published yet. Just follow the instructions that
say --runner=PortableRunner, not --runner=FlinkRunner, otherwise you'll hit
that other deserialization bug that was mentioned..
Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com
On Thu, Sep 19, 2019 at 11:19 AM Matthew Patterson <mp...@aligntech.com>
wrote:
> Kyle,
>
>
>
> Happily: should I be working from (source) release-2.15.0 or
> release-2.16.0: presumably still specifying flink-1.8.
>
> Matt
>
>
>
> *From: *Kyle Weaver <kc...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 2:16 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> Re Benjamin & Ankur: I don't think `--experiments=beam_fn_api` applies
> here. If I understand correctly, Matthew is just trying to run an
> old-fashioned Beam Java jar, nothing to do with portability/Python, and
> judging by the stack trace provided,
> https://issues.apache.org/jira/browse/BEAM-8037
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-8037&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc61a77b9f07d4e86528b08d73d2d807e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045137955374002&sdata=VQBpPQCV6uXS3rSZZx2iDyRHYRlG28PvLaSgF8IieOw%3D&reserved=0> is
> not the issue here.
>
>
>
> Re Matthew: I suspect a Flink version mismatch between the Flink cluster
> and the Beam runner. However, I am not familiar with the Maven commands
> you're using. Since you're planning on running Python eventually anyway,
> could you try following the "portable"/Python instructions on
> https://beam.apache.org/documentation/runners/flink/
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Fdocumentation%2Frunners%2Fflink%2F&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc61a77b9f07d4e86528b08d73d2d807e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045137955383997&sdata=%2FOGRiuJ08RVVWeTjjsh0%2FrmIFDpIM6ZRpP5N4uwvBZ8%3D&reserved=0> and
> let us know how that goes?
>
>
> Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc61a77b9f07d4e86528b08d73d2d807e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045137955383997&sdata=zo%2B0ARoOFjh2q%2BcKqGj3NKgGvFIp5AERywepuT2iEK0%3D&reserved=0>
> | kcweaver@google.com
>
>
>
>
>
> On Thu, Sep 19, 2019 at 10:52 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Thanks Ankur,
>
>
>
> As one who speaks almost no gradle, is there a preferred way to get gradle
> to respect the conda configured python on `gradlew build` ?
>
>
>
> Matt
>
>
>
> *From: *Ankur Goenka <go...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 1:50 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> We have this bug in 2.15 which is discussed here
> https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc61a77b9f07d4e86528b08d73d2d807e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045137955393995&sdata=fBomzWr4SBaYG30miMdpDgnp3o3naOC1N%2BeGs1HeOWw%3D&reserved=0>
>
>
>
> For now, please add "--experiments=beam_fn_api" to your pipeline to make
> it work.
>
>
>
> On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Hi Ankur,
>
>
>
> Yes, I was using 2.15, but was getting failure to deserialize.
>
>
>
> Thanks,
>
> Matt
>
>
>
> *From: *Ankur Goenka <go...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Wednesday, September 18, 2019 at 9:34 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> Hi Matthew,
>
>
>
> Beam 2.16.0 is not yet released hence you are getting the error. Can you
> try using 2.15.0 version?
>
>
>
> Thanks,
>
> Ankur
>
>
>
> On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Tried
>
> "
> mvn archetype:generate \
> -DarchetypeGroupId=org.apache.beam \
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> -DarchetypeVersion=2.16.0 \
> -DgroupId=org.example \
> -DartifactId=word-count-beam \
> -Dversion="0.1" \
> -Dpackage=org.apache.beam.examples \
> -DinteractiveMode=false
> "
> got
>
> -> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate
> (default-cli) on project standalone-pom: The desired archetype does not
> exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) ->
> [Help 1]
>
>
> On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com> wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
>
> Ah. So maybe try 2.16.0?
>
> On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com> wrote:
> > --experiments=beam_fn_api doesn't apply here, as this is a Java
> pipeline
> > using the non-portable version of the Flink runner.
> >
> > Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc61a77b9f07d4e86528b08d73d2d807e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045137955403987&sdata=ZyCTCOdBvXchFr2ckjnBRIyDsdoWkZty3eaQe8bQWAo%3D&reserved=0>
> | kcweaver@google.com
> >
> >
> > On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <
> benjamintanweihao@gmail.com>
> > wrote:
> >
> > > Could you try adding "--experiments=beam_fn_api" to the runner
> options?
> > > Kyle did mention that it should be fixed in 2.16.0.
> > >
> > > On 2019/09/17 16:40:41, Matthew Patterson <
> mpatterson@aligntech.com>
> > > wrote:
> > > > Folks,
> > > >
> > > > Am sure I am doing it wrong, but have been beating head against
> wall
> > > long enough I need to ask for help.
> > > >
> > > > My goal is to use the python SDK to run jobs on AWS flink
> cluster, but
> > > very little is working, so I am trying to back down to the very
> simplest
> > > thing that works, and build back up: not going so well.
> > > >
> > > > Now I am running, or trying to, on a minikube link cluster. Now
> running
> > > from just java/mvn, so I go through the following:
> > > >
> > > > 1. bring-up-cluster (see below)
> > > > 2. grab-the-code (see below)
> > > > 3. maven-submit (see below)
> > > >
> > > > Eventual failure is
> > > >
> > > > “[ERROR] Failed to execute goal
> > > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on
> project
> > > word-count-beam: An exception occured while executing the Java
> class.
> > > Pipeline execution failed: Could not retrieve the execution
> result. (JobID:
> > > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph.
> [Failed to
> > > deserialize JobGraph.]”
> > > >
> > > > and log is attached (I hope)
> > > >
> > > > Other details:
> > > >
> > > > on osx
> > > >
> > > >
> > > >
> > > > ➜ minikube version
> > > >
> > > > minikube version: v1.3.1
> > > >
> > > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > > >
> > > >
> > > > ➜ kubectl version
> > > >
> > > > Client Version: version.Info{Major:"1", Minor:"14",
> > > GitVersion:"v1.14.6",
> GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > > >
> > > > Server Version: version.Info{Major:"1", Minor:"15",
> > > GitVersion:"v1.15.2",
> GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > > >
> > > >
> > > > ➜ flink --version
> > > >
> > > > Version: 1.8.1, Commit ID: 7297bac
> > > >
> > > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > > 2.15.0
> > > >
> > > > ➜ curl localhost:8081
> > > > <!--
> > > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > > ~ or more contributor license agreements. See the NOTICE file
> > > > ~ distributed with this work for additional information
> > > > ~ regarding copyright ownership. The ASF licenses this file
> > > > ~ to you under the Apache License, Version 2.0 (the
> > > > ~ "License"); you may not use this file except in compliance
> > > > ~ with the License. You may obtain a copy of the License at
> > > > ~
> > > > ~
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc61a77b9f07d4e86528b08d73d2d807e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045137955403987&sdata=PUe7CC5PgzOTqbexzl0ypmr5v2P8GSmikf0Y6qJIA5g%3D&reserved=0>
> > > > ~
> > > > ~ Unless required by applicable law or agreed to in writing,
> software
> > > > ~ distributed under the License is distributed on an "AS IS"
> BASIS,
> > > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > > implied.
> > > > ~ See the License for the specific language governing
> permissions and
> > > > ~ limitations under the License.
> > > > -->
> > > >
> > > > <!doctype html>
> > > > <html lang="en">
> > > > <head>
> > > > <meta charset="utf-8">
> > > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > > <meta name="viewport" content="width=device-width,
> initial-scale=1">
> > > > <title>Apache Flink Web Dashboard</title>
> > > > <link rel="apple-touch-icon" sizes="180x180"
> > > href="assets/favicon/apple-touch-icon.png">
> > > > <link rel="icon" type="image/png"
> > > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > > <link rel="icon" type="image/png"
> > > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > > <link rel="mask-icon"
> href="assets/favicon/safari-pinned-tab.svg"
> > > color="#aa1919">
> > > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > > <meta name="msapplication-config"
> > > content="assets/favicon/browserconfig.xml">
> > > > <meta name="theme-color" content="#ffffff">
> > > > <base href="./"><link rel="stylesheet"
> > > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > > <body>
> > > > <flink-root></flink-root>
> > > > <script type="text/javascript"
> > > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > > type="text/javascript"
> src="es2015-polyfills.5e343224e81eefb7658e.js"
> > > nomodule></script><script type="text/javascript"
> > > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > > type="text/javascript"
> src="main.dc6e0997d092df051db0.js"></script></body>
> > > > </html>
> > > >
> > > > Thanks for any ideas,
> > > > Matt
> > > >
> > > > >>> bring-up-cluster
> > > >
> > > > # [yamls from here](
> > >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc61a77b9f07d4e86528b08d73d2d807e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045137955413989&sdata=wvqhyAJbgR7284uq78M7rFs1s4CdnHM9odZ6GTrtO7g%3D&reserved=0>
> > > )
> > > > minikube start
> > > > minikube ssh 'sudo ip link set docker0 promisc on'
> > > >
> > > > kubectl create -f flink-configuration-configmap.yaml
> > > > kubectl create -f jobmanager-service.yaml
> > > > kubectl create -f jobmanager-deployment.yaml
> > > > kubectl create -f taskmanager-deployment.yaml
> > > > kubectl apply -f jobmanager-rest-service.yaml
> > > >
> > > > >>> grab-the-code
> > > > mvn archetype:generate \
> > > > -DarchetypeGroupId=org.apache.beam \
> > > >
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > > -DarchetypeVersion=2.15.0 \
> > > > -DgroupId=org.example \
> > > > -DartifactId=word-count-beam \
> > > > -Dversion="0.1" \
> > > > -Dpackage=org.apache.beam.examples \
> > > > -DinteractiveMode=false
> > > >
> > > > >>> maven-submit
> > > >
> > > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in
> separate
> > > shell, as blocking
> > > > mvn package exec:java -X
> > > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > > -Dexec.args="--runner=FlinkRunner\
> > > > --flinkMaster="localhost:8081"\
> > > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > > --output=<directory-that-exists>" -Pflink-runner
> > > >
> > > >
> > > >
> > >
> >
>
>
Re: Word-count example
Posted by Matthew Patterson <mp...@aligntech.com>.
Kyle,
Happily: should I be working from (source) release-2.15.0 or release-2.16.0: presumably still specifying flink-1.8.
Matt
From: Kyle Weaver <kc...@google.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Thursday, September 19, 2019 at 2:16 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re Benjamin & Ankur: I don't think `--experiments=beam_fn_api` applies here. If I understand correctly, Matthew is just trying to run an old-fashioned Beam Java jar, nothing to do with portability/Python, and judging by the stack trace provided, https://issues.apache.org/jira/browse/BEAM-8037<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FBEAM-8037&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc61a77b9f07d4e86528b08d73d2d807e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045137955374002&sdata=VQBpPQCV6uXS3rSZZx2iDyRHYRlG28PvLaSgF8IieOw%3D&reserved=0> is not the issue here.
Re Matthew: I suspect a Flink version mismatch between the Flink cluster and the Beam runner. However, I am not familiar with the Maven commands you're using. Since you're planning on running Python eventually anyway, could you try following the "portable"/Python instructions on https://beam.apache.org/documentation/runners/flink/<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Fdocumentation%2Frunners%2Fflink%2F&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc61a77b9f07d4e86528b08d73d2d807e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045137955383997&sdata=%2FOGRiuJ08RVVWeTjjsh0%2FrmIFDpIM6ZRpP5N4uwvBZ8%3D&reserved=0> and let us know how that goes?
Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc61a77b9f07d4e86528b08d73d2d807e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045137955383997&sdata=zo%2B0ARoOFjh2q%2BcKqGj3NKgGvFIp5AERywepuT2iEK0%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
On Thu, Sep 19, 2019 at 10:52 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Thanks Ankur,
As one who speaks almost no gradle, is there a preferred way to get gradle to respect the conda configured python on `gradlew build` ?
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Thursday, September 19, 2019 at 1:50 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
We have this bug in 2.15 which is discussed here https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc61a77b9f07d4e86528b08d73d2d807e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045137955393995&sdata=fBomzWr4SBaYG30miMdpDgnp3o3naOC1N%2BeGs1HeOWw%3D&reserved=0>
For now, please add "--experiments=beam_fn_api" to your pipeline to make it work.
On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Hi Ankur,
Yes, I was using 2.15, but was getting failure to deserialize.
Thanks,
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Wednesday, September 18, 2019 at 9:34 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Matthew,
Beam 2.16.0 is not yet released hence you are getting the error. Can you try using 2.15.0 version?
Thanks,
Ankur
On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Tried
"
mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
-DarchetypeVersion=2.16.0 \
-DgroupId=org.example \
-DartifactId=word-count-beam \
-Dversion="0.1" \
-Dpackage=org.apache.beam.examples \
-DinteractiveMode=false
"
got
-> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate (default-cli) on project standalone-pom: The desired archetype does not exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) -> [Help 1]
On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com>> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Ah. So maybe try 2.16.0?
On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com>> wrote:
> --experiments=beam_fn_api doesn't apply here, as this is a Java pipeline
> using the non-portable version of the Flink runner.
>
> Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc61a77b9f07d4e86528b08d73d2d807e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045137955403987&sdata=ZyCTCOdBvXchFr2ckjnBRIyDsdoWkZty3eaQe8bQWAo%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
>
>
> On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <be...@gmail.com>>
> wrote:
>
> > Could you try adding "--experiments=beam_fn_api" to the runner options?
> > Kyle did mention that it should be fixed in 2.16.0.
> >
> > On 2019/09/17 16:40:41, Matthew Patterson <mp...@aligntech.com>>
> > wrote:
> > > Folks,
> > >
> > > Am sure I am doing it wrong, but have been beating head against wall
> > long enough I need to ask for help.
> > >
> > > My goal is to use the python SDK to run jobs on AWS flink cluster, but
> > very little is working, so I am trying to back down to the very simplest
> > thing that works, and build back up: not going so well.
> > >
> > > Now I am running, or trying to, on a minikube link cluster. Now running
> > from just java/mvn, so I go through the following:
> > >
> > > 1. bring-up-cluster (see below)
> > > 2. grab-the-code (see below)
> > > 3. maven-submit (see below)
> > >
> > > Eventual failure is
> > >
> > > “[ERROR] Failed to execute goal
> > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project
> > word-count-beam: An exception occured while executing the Java class.
> > Pipeline execution failed: Could not retrieve the execution result. (JobID:
> > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to
> > deserialize JobGraph.]”
> > >
> > > and log is attached (I hope)
> > >
> > > Other details:
> > >
> > > on osx
> > >
> > >
> > >
> > > ➜ minikube version
> > >
> > > minikube version: v1.3.1
> > >
> > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > >
> > >
> > > ➜ kubectl version
> > >
> > > Client Version: version.Info{Major:"1", Minor:"14",
> > GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > >
> > > Server Version: version.Info{Major:"1", Minor:"15",
> > GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > >
> > >
> > > ➜ flink --version
> > >
> > > Version: 1.8.1, Commit ID: 7297bac
> > >
> > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > 2.15.0
> > >
> > > ➜ curl localhost:8081
> > > <!--
> > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > ~ or more contributor license agreements. See the NOTICE file
> > > ~ distributed with this work for additional information
> > > ~ regarding copyright ownership. The ASF licenses this file
> > > ~ to you under the Apache License, Version 2.0 (the
> > > ~ "License"); you may not use this file except in compliance
> > > ~ with the License. You may obtain a copy of the License at
> > > ~
> > > ~ https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc61a77b9f07d4e86528b08d73d2d807e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045137955403987&sdata=PUe7CC5PgzOTqbexzl0ypmr5v2P8GSmikf0Y6qJIA5g%3D&reserved=0>
> > > ~
> > > ~ Unless required by applicable law or agreed to in writing, software
> > > ~ distributed under the License is distributed on an "AS IS" BASIS,
> > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > implied.
> > > ~ See the License for the specific language governing permissions and
> > > ~ limitations under the License.
> > > -->
> > >
> > > <!doctype html>
> > > <html lang="en">
> > > <head>
> > > <meta charset="utf-8">
> > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > <meta name="viewport" content="width=device-width, initial-scale=1">
> > > <title>Apache Flink Web Dashboard</title>
> > > <link rel="apple-touch-icon" sizes="180x180"
> > href="assets/favicon/apple-touch-icon.png">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > <link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg"
> > color="#aa1919">
> > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > <meta name="msapplication-config"
> > content="assets/favicon/browserconfig.xml">
> > > <meta name="theme-color" content="#ffffff">
> > > <base href="./"><link rel="stylesheet"
> > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > <body>
> > > <flink-root></flink-root>
> > > <script type="text/javascript"
> > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js"
> > nomodule></script><script type="text/javascript"
> > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > type="text/javascript" src="main.dc6e0997d092df051db0.js"></script></body>
> > > </html>
> > >
> > > Thanks for any ideas,
> > > Matt
> > >
> > > >>> bring-up-cluster
> > >
> > > # [yamls from here](
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7Cc61a77b9f07d4e86528b08d73d2d807e%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045137955413989&sdata=wvqhyAJbgR7284uq78M7rFs1s4CdnHM9odZ6GTrtO7g%3D&reserved=0>
> > )
> > > minikube start
> > > minikube ssh 'sudo ip link set docker0 promisc on'
> > >
> > > kubectl create -f flink-configuration-configmap.yaml
> > > kubectl create -f jobmanager-service.yaml
> > > kubectl create -f jobmanager-deployment.yaml
> > > kubectl create -f taskmanager-deployment.yaml
> > > kubectl apply -f jobmanager-rest-service.yaml
> > >
> > > >>> grab-the-code
> > > mvn archetype:generate \
> > > -DarchetypeGroupId=org.apache.beam \
> > > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > -DarchetypeVersion=2.15.0 \
> > > -DgroupId=org.example \
> > > -DartifactId=word-count-beam \
> > > -Dversion="0.1" \
> > > -Dpackage=org.apache.beam.examples \
> > > -DinteractiveMode=false
> > >
> > > >>> maven-submit
> > >
> > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in separate
> > shell, as blocking
> > > mvn package exec:java -X
> > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > -Dexec.args="--runner=FlinkRunner\
> > > --flinkMaster="localhost:8081"\
> > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > --output=<directory-that-exists>" -Pflink-runner
> > >
> > >
> > >
> >
>
Re: Word-count example
Posted by Kyle Weaver <kc...@google.com>.
Re Benjamin & Ankur: I don't think `--experiments=beam_fn_api` applies
here. If I understand correctly, Matthew is just trying to run an
old-fashioned Beam Java jar, nothing to do with portability/Python, and
judging by the stack trace provided,
https://issues.apache.org/jira/browse/BEAM-8037 is not the issue here.
Re Matthew: I suspect a Flink version mismatch between the Flink cluster
and the Beam runner. However, I am not familiar with the Maven commands
you're using. Since you're planning on running Python eventually anyway,
could you try following the "portable"/Python instructions on
https://beam.apache.org/documentation/runners/flink/ and let us know how
that goes?
Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com
On Thu, Sep 19, 2019 at 10:52 AM Matthew Patterson <mp...@aligntech.com>
wrote:
> Thanks Ankur,
>
>
>
> As one who speaks almost no gradle, is there a preferred way to get gradle
> to respect the conda configured python on `gradlew build` ?
>
>
>
> Matt
>
>
>
> *From: *Ankur Goenka <go...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Thursday, September 19, 2019 at 1:50 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> We have this bug in 2.15 which is discussed here
> https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7C92c66b429d3e4b2a89cf08d73d29ce61%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045122081208279&sdata=mUbuMEq0UHke7igqGwC0Lv4p7CLpS3tG7zDEHoS9B%2B4%3D&reserved=0>
>
>
>
> For now, please add "--experiments=beam_fn_api" to your pipeline to make
> it work.
>
>
>
> On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Hi Ankur,
>
>
>
> Yes, I was using 2.15, but was getting failure to deserialize.
>
>
>
> Thanks,
>
> Matt
>
>
>
> *From: *Ankur Goenka <go...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Wednesday, September 18, 2019 at 9:34 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> Hi Matthew,
>
>
>
> Beam 2.16.0 is not yet released hence you are getting the error. Can you
> try using 2.15.0 version?
>
>
>
> Thanks,
>
> Ankur
>
>
>
> On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Tried
>
> "
> mvn archetype:generate \
> -DarchetypeGroupId=org.apache.beam \
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> -DarchetypeVersion=2.16.0 \
> -DgroupId=org.example \
> -DartifactId=word-count-beam \
> -Dversion="0.1" \
> -Dpackage=org.apache.beam.examples \
> -DinteractiveMode=false
> "
> got
>
> -> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate
> (default-cli) on project standalone-pom: The desired archetype does not
> exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) ->
> [Help 1]
>
>
> On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com> wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
>
> Ah. So maybe try 2.16.0?
>
> On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com> wrote:
> > --experiments=beam_fn_api doesn't apply here, as this is a Java
> pipeline
> > using the non-portable version of the Flink runner.
> >
> > Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C92c66b429d3e4b2a89cf08d73d29ce61%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045122081218275&sdata=lFcrLDfXJ86ZICr70bvdnBa4gyKiukFhWYolPrBHZO4%3D&reserved=0>
> | kcweaver@google.com
> >
> >
> > On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <
> benjamintanweihao@gmail.com>
> > wrote:
> >
> > > Could you try adding "--experiments=beam_fn_api" to the runner
> options?
> > > Kyle did mention that it should be fixed in 2.16.0.
> > >
> > > On 2019/09/17 16:40:41, Matthew Patterson <
> mpatterson@aligntech.com>
> > > wrote:
> > > > Folks,
> > > >
> > > > Am sure I am doing it wrong, but have been beating head against
> wall
> > > long enough I need to ask for help.
> > > >
> > > > My goal is to use the python SDK to run jobs on AWS flink
> cluster, but
> > > very little is working, so I am trying to back down to the very
> simplest
> > > thing that works, and build back up: not going so well.
> > > >
> > > > Now I am running, or trying to, on a minikube link cluster. Now
> running
> > > from just java/mvn, so I go through the following:
> > > >
> > > > 1. bring-up-cluster (see below)
> > > > 2. grab-the-code (see below)
> > > > 3. maven-submit (see below)
> > > >
> > > > Eventual failure is
> > > >
> > > > “[ERROR] Failed to execute goal
> > > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on
> project
> > > word-count-beam: An exception occured while executing the Java
> class.
> > > Pipeline execution failed: Could not retrieve the execution
> result. (JobID:
> > > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph.
> [Failed to
> > > deserialize JobGraph.]”
> > > >
> > > > and log is attached (I hope)
> > > >
> > > > Other details:
> > > >
> > > > on osx
> > > >
> > > >
> > > >
> > > > ➜ minikube version
> > > >
> > > > minikube version: v1.3.1
> > > >
> > > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > > >
> > > >
> > > > ➜ kubectl version
> > > >
> > > > Client Version: version.Info{Major:"1", Minor:"14",
> > > GitVersion:"v1.14.6",
> GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > > >
> > > > Server Version: version.Info{Major:"1", Minor:"15",
> > > GitVersion:"v1.15.2",
> GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > > >
> > > >
> > > > ➜ flink --version
> > > >
> > > > Version: 1.8.1, Commit ID: 7297bac
> > > >
> > > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > > 2.15.0
> > > >
> > > > ➜ curl localhost:8081
> > > > <!--
> > > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > > ~ or more contributor license agreements. See the NOTICE file
> > > > ~ distributed with this work for additional information
> > > > ~ regarding copyright ownership. The ASF licenses this file
> > > > ~ to you under the Apache License, Version 2.0 (the
> > > > ~ "License"); you may not use this file except in compliance
> > > > ~ with the License. You may obtain a copy of the License at
> > > > ~
> > > > ~
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C92c66b429d3e4b2a89cf08d73d29ce61%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045122081228269&sdata=Zhg43%2FegQVw%2BX9h0ht4bw%2FoQnT7Vbclgx6Wg%2BI16jOg%3D&reserved=0>
> > > > ~
> > > > ~ Unless required by applicable law or agreed to in writing,
> software
> > > > ~ distributed under the License is distributed on an "AS IS"
> BASIS,
> > > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > > implied.
> > > > ~ See the License for the specific language governing
> permissions and
> > > > ~ limitations under the License.
> > > > -->
> > > >
> > > > <!doctype html>
> > > > <html lang="en">
> > > > <head>
> > > > <meta charset="utf-8">
> > > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > > <meta name="viewport" content="width=device-width,
> initial-scale=1">
> > > > <title>Apache Flink Web Dashboard</title>
> > > > <link rel="apple-touch-icon" sizes="180x180"
> > > href="assets/favicon/apple-touch-icon.png">
> > > > <link rel="icon" type="image/png"
> > > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > > <link rel="icon" type="image/png"
> > > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > > <link rel="mask-icon"
> href="assets/favicon/safari-pinned-tab.svg"
> > > color="#aa1919">
> > > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > > <meta name="msapplication-config"
> > > content="assets/favicon/browserconfig.xml">
> > > > <meta name="theme-color" content="#ffffff">
> > > > <base href="./"><link rel="stylesheet"
> > > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > > <body>
> > > > <flink-root></flink-root>
> > > > <script type="text/javascript"
> > > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > > type="text/javascript"
> src="es2015-polyfills.5e343224e81eefb7658e.js"
> > > nomodule></script><script type="text/javascript"
> > > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > > type="text/javascript"
> src="main.dc6e0997d092df051db0.js"></script></body>
> > > > </html>
> > > >
> > > > Thanks for any ideas,
> > > > Matt
> > > >
> > > > >>> bring-up-cluster
> > > >
> > > > # [yamls from here](
> > >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C92c66b429d3e4b2a89cf08d73d29ce61%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045122081228269&sdata=eegtSWsLzx1MtI6b%2Bvnm0H58VSRhOhph1Zh86CJL0iI%3D&reserved=0>
> > > )
> > > > minikube start
> > > > minikube ssh 'sudo ip link set docker0 promisc on'
> > > >
> > > > kubectl create -f flink-configuration-configmap.yaml
> > > > kubectl create -f jobmanager-service.yaml
> > > > kubectl create -f jobmanager-deployment.yaml
> > > > kubectl create -f taskmanager-deployment.yaml
> > > > kubectl apply -f jobmanager-rest-service.yaml
> > > >
> > > > >>> grab-the-code
> > > > mvn archetype:generate \
> > > > -DarchetypeGroupId=org.apache.beam \
> > > >
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > > -DarchetypeVersion=2.15.0 \
> > > > -DgroupId=org.example \
> > > > -DartifactId=word-count-beam \
> > > > -Dversion="0.1" \
> > > > -Dpackage=org.apache.beam.examples \
> > > > -DinteractiveMode=false
> > > >
> > > > >>> maven-submit
> > > >
> > > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in
> separate
> > > shell, as blocking
> > > > mvn package exec:java -X
> > > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > > -Dexec.args="--runner=FlinkRunner\
> > > > --flinkMaster="localhost:8081"\
> > > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > > --output=<directory-that-exists>" -Pflink-runner
> > > >
> > > >
> > > >
> > >
> >
>
>
Re: Word-count example
Posted by Matthew Patterson <mp...@aligntech.com>.
Thanks Ankur,
As one who speaks almost no gradle, is there a preferred way to get gradle to respect the conda configured python on `gradlew build` ?
Matt
From: Ankur Goenka <go...@google.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Thursday, September 19, 2019 at 1:50 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
We have this bug in 2.15 which is discussed here https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2F76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b%40%253Cuser.beam.apache.org%253E&data=02%7C01%7Cmpatterson%40aligntech.com%7C92c66b429d3e4b2a89cf08d73d29ce61%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045122081208279&sdata=mUbuMEq0UHke7igqGwC0Lv4p7CLpS3tG7zDEHoS9B%2B4%3D&reserved=0>
For now, please add "--experiments=beam_fn_api" to your pipeline to make it work.
On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Hi Ankur,
Yes, I was using 2.15, but was getting failure to deserialize.
Thanks,
Matt
From: Ankur Goenka <go...@google.com>>
Reply-To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Date: Wednesday, September 18, 2019 at 9:34 PM
To: "user@beam.apache.org<ma...@beam.apache.org>" <us...@beam.apache.org>>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Matthew,
Beam 2.16.0 is not yet released hence you are getting the error. Can you try using 2.15.0 version?
Thanks,
Ankur
On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Tried
"
mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
-DarchetypeVersion=2.16.0 \
-DgroupId=org.example \
-DartifactId=word-count-beam \
-Dversion="0.1" \
-Dpackage=org.apache.beam.examples \
-DinteractiveMode=false
"
got
-> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate (default-cli) on project standalone-pom: The desired archetype does not exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) -> [Help 1]
On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com>> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Ah. So maybe try 2.16.0?
On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com>> wrote:
> --experiments=beam_fn_api doesn't apply here, as this is a Java pipeline
> using the non-portable version of the Flink runner.
>
> Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7C92c66b429d3e4b2a89cf08d73d29ce61%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045122081218275&sdata=lFcrLDfXJ86ZICr70bvdnBa4gyKiukFhWYolPrBHZO4%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
>
>
> On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <be...@gmail.com>>
> wrote:
>
> > Could you try adding "--experiments=beam_fn_api" to the runner options?
> > Kyle did mention that it should be fixed in 2.16.0.
> >
> > On 2019/09/17 16:40:41, Matthew Patterson <mp...@aligntech.com>>
> > wrote:
> > > Folks,
> > >
> > > Am sure I am doing it wrong, but have been beating head against wall
> > long enough I need to ask for help.
> > >
> > > My goal is to use the python SDK to run jobs on AWS flink cluster, but
> > very little is working, so I am trying to back down to the very simplest
> > thing that works, and build back up: not going so well.
> > >
> > > Now I am running, or trying to, on a minikube link cluster. Now running
> > from just java/mvn, so I go through the following:
> > >
> > > 1. bring-up-cluster (see below)
> > > 2. grab-the-code (see below)
> > > 3. maven-submit (see below)
> > >
> > > Eventual failure is
> > >
> > > “[ERROR] Failed to execute goal
> > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project
> > word-count-beam: An exception occured while executing the Java class.
> > Pipeline execution failed: Could not retrieve the execution result. (JobID:
> > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to
> > deserialize JobGraph.]”
> > >
> > > and log is attached (I hope)
> > >
> > > Other details:
> > >
> > > on osx
> > >
> > >
> > >
> > > ➜ minikube version
> > >
> > > minikube version: v1.3.1
> > >
> > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > >
> > >
> > > ➜ kubectl version
> > >
> > > Client Version: version.Info{Major:"1", Minor:"14",
> > GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > >
> > > Server Version: version.Info{Major:"1", Minor:"15",
> > GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > >
> > >
> > > ➜ flink --version
> > >
> > > Version: 1.8.1, Commit ID: 7297bac
> > >
> > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > 2.15.0
> > >
> > > ➜ curl localhost:8081
> > > <!--
> > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > ~ or more contributor license agreements. See the NOTICE file
> > > ~ distributed with this work for additional information
> > > ~ regarding copyright ownership. The ASF licenses this file
> > > ~ to you under the Apache License, Version 2.0 (the
> > > ~ "License"); you may not use this file except in compliance
> > > ~ with the License. You may obtain a copy of the License at
> > > ~
> > > ~ https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C92c66b429d3e4b2a89cf08d73d29ce61%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045122081228269&sdata=Zhg43%2FegQVw%2BX9h0ht4bw%2FoQnT7Vbclgx6Wg%2BI16jOg%3D&reserved=0>
> > > ~
> > > ~ Unless required by applicable law or agreed to in writing, software
> > > ~ distributed under the License is distributed on an "AS IS" BASIS,
> > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > implied.
> > > ~ See the License for the specific language governing permissions and
> > > ~ limitations under the License.
> > > -->
> > >
> > > <!doctype html>
> > > <html lang="en">
> > > <head>
> > > <meta charset="utf-8">
> > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > <meta name="viewport" content="width=device-width, initial-scale=1">
> > > <title>Apache Flink Web Dashboard</title>
> > > <link rel="apple-touch-icon" sizes="180x180"
> > href="assets/favicon/apple-touch-icon.png">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > <link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg"
> > color="#aa1919">
> > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > <meta name="msapplication-config"
> > content="assets/favicon/browserconfig.xml">
> > > <meta name="theme-color" content="#ffffff">
> > > <base href="./"><link rel="stylesheet"
> > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > <body>
> > > <flink-root></flink-root>
> > > <script type="text/javascript"
> > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js"
> > nomodule></script><script type="text/javascript"
> > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > type="text/javascript" src="main.dc6e0997d092df051db0.js"></script></body>
> > > </html>
> > >
> > > Thanks for any ideas,
> > > Matt
> > >
> > > >>> bring-up-cluster
> > >
> > > # [yamls from here](
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C92c66b429d3e4b2a89cf08d73d29ce61%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637045122081228269&sdata=eegtSWsLzx1MtI6b%2Bvnm0H58VSRhOhph1Zh86CJL0iI%3D&reserved=0>
> > )
> > > minikube start
> > > minikube ssh 'sudo ip link set docker0 promisc on'
> > >
> > > kubectl create -f flink-configuration-configmap.yaml
> > > kubectl create -f jobmanager-service.yaml
> > > kubectl create -f jobmanager-deployment.yaml
> > > kubectl create -f taskmanager-deployment.yaml
> > > kubectl apply -f jobmanager-rest-service.yaml
> > >
> > > >>> grab-the-code
> > > mvn archetype:generate \
> > > -DarchetypeGroupId=org.apache.beam \
> > > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > -DarchetypeVersion=2.15.0 \
> > > -DgroupId=org.example \
> > > -DartifactId=word-count-beam \
> > > -Dversion="0.1" \
> > > -Dpackage=org.apache.beam.examples \
> > > -DinteractiveMode=false
> > >
> > > >>> maven-submit
> > >
> > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in separate
> > shell, as blocking
> > > mvn package exec:java -X
> > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > -Dexec.args="--runner=FlinkRunner\
> > > --flinkMaster="localhost:8081"\
> > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > --output=<directory-that-exists>" -Pflink-runner
> > >
> > >
> > >
> >
>
Re: Word-count example
Posted by Ankur Goenka <go...@google.com>.
We have this bug in 2.15 which is discussed here
https://lists.apache.org/thread.html/76150a1ffca859bae7af0c6fb91724dc405dc55efc51c3b515f0520b@%3Cuser.beam.apache.org%3E
For now, please add "--experiments=beam_fn_api" to your pipeline to make it
work.
On Thu, Sep 19, 2019 at 6:03 AM Matthew Patterson <mp...@aligntech.com>
wrote:
> Hi Ankur,
>
>
>
> Yes, I was using 2.15, but was getting failure to deserialize.
>
>
>
> Thanks,
>
> Matt
>
>
>
> *From: *Ankur Goenka <go...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Wednesday, September 18, 2019 at 9:34 PM
> *To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Subject: *Re: Word-count example
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> Hi Matthew,
>
>
>
> Beam 2.16.0 is not yet released hence you are getting the error. Can you
> try using 2.15.0 version?
>
>
>
> Thanks,
>
> Ankur
>
>
>
> On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <
> mpatterson@aligntech.com> wrote:
>
> Tried
>
> "
> mvn archetype:generate \
> -DarchetypeGroupId=org.apache.beam \
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> -DarchetypeVersion=2.16.0 \
> -DgroupId=org.example \
> -DartifactId=word-count-beam \
> -Dversion="0.1" \
> -Dpackage=org.apache.beam.examples \
> -DinteractiveMode=false
> "
> got
>
> -> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate
> (default-cli) on project standalone-pom: The desired archetype does not
> exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) ->
> [Help 1]
>
>
> On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com> wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
>
> Ah. So maybe try 2.16.0?
>
> On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com> wrote:
> > --experiments=beam_fn_api doesn't apply here, as this is a Java
> pipeline
> > using the non-portable version of the Flink runner.
> >
> > Kyle Weaver | Software Engineer | github.com/ibzib
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Ce73cc973ae1642c5015d08d73ca1742f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637044536460058010&sdata=xut5961XZFOBoo0DG1%2F5qgD%2FAOasLXNAbzfETnu09kg%3D&reserved=0>
> | kcweaver@google.com
> >
> >
> > On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <
> benjamintanweihao@gmail.com>
> > wrote:
> >
> > > Could you try adding "--experiments=beam_fn_api" to the runner
> options?
> > > Kyle did mention that it should be fixed in 2.16.0.
> > >
> > > On 2019/09/17 16:40:41, Matthew Patterson <
> mpatterson@aligntech.com>
> > > wrote:
> > > > Folks,
> > > >
> > > > Am sure I am doing it wrong, but have been beating head against
> wall
> > > long enough I need to ask for help.
> > > >
> > > > My goal is to use the python SDK to run jobs on AWS flink
> cluster, but
> > > very little is working, so I am trying to back down to the very
> simplest
> > > thing that works, and build back up: not going so well.
> > > >
> > > > Now I am running, or trying to, on a minikube link cluster. Now
> running
> > > from just java/mvn, so I go through the following:
> > > >
> > > > 1. bring-up-cluster (see below)
> > > > 2. grab-the-code (see below)
> > > > 3. maven-submit (see below)
> > > >
> > > > Eventual failure is
> > > >
> > > > “[ERROR] Failed to execute goal
> > > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on
> project
> > > word-count-beam: An exception occured while executing the Java
> class.
> > > Pipeline execution failed: Could not retrieve the execution
> result. (JobID:
> > > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph.
> [Failed to
> > > deserialize JobGraph.]”
> > > >
> > > > and log is attached (I hope)
> > > >
> > > > Other details:
> > > >
> > > > on osx
> > > >
> > > >
> > > >
> > > > ➜ minikube version
> > > >
> > > > minikube version: v1.3.1
> > > >
> > > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > > >
> > > >
> > > > ➜ kubectl version
> > > >
> > > > Client Version: version.Info{Major:"1", Minor:"14",
> > > GitVersion:"v1.14.6",
> GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > > >
> > > > Server Version: version.Info{Major:"1", Minor:"15",
> > > GitVersion:"v1.15.2",
> GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > > >
> > > >
> > > > ➜ flink --version
> > > >
> > > > Version: 1.8.1, Commit ID: 7297bac
> > > >
> > > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > > 2.15.0
> > > >
> > > > ➜ curl localhost:8081
> > > > <!--
> > > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > > ~ or more contributor license agreements. See the NOTICE file
> > > > ~ distributed with this work for additional information
> > > > ~ regarding copyright ownership. The ASF licenses this file
> > > > ~ to you under the Apache License, Version 2.0 (the
> > > > ~ "License"); you may not use this file except in compliance
> > > > ~ with the License. You may obtain a copy of the License at
> > > > ~
> > > > ~
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0
> <https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7Ce73cc973ae1642c5015d08d73ca1742f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637044536460068006&sdata=1HzYabQkaVH2ob6RL5%2B3L8Mo2hzq69jgLebHSaNzCVI%3D&reserved=0>
> > > > ~
> > > > ~ Unless required by applicable law or agreed to in writing,
> software
> > > > ~ distributed under the License is distributed on an "AS IS"
> BASIS,
> > > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > > implied.
> > > > ~ See the License for the specific language governing
> permissions and
> > > > ~ limitations under the License.
> > > > -->
> > > >
> > > > <!doctype html>
> > > > <html lang="en">
> > > > <head>
> > > > <meta charset="utf-8">
> > > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > > <meta name="viewport" content="width=device-width,
> initial-scale=1">
> > > > <title>Apache Flink Web Dashboard</title>
> > > > <link rel="apple-touch-icon" sizes="180x180"
> > > href="assets/favicon/apple-touch-icon.png">
> > > > <link rel="icon" type="image/png"
> > > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > > <link rel="icon" type="image/png"
> > > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > > <link rel="mask-icon"
> href="assets/favicon/safari-pinned-tab.svg"
> > > color="#aa1919">
> > > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > > <meta name="msapplication-config"
> > > content="assets/favicon/browserconfig.xml">
> > > > <meta name="theme-color" content="#ffffff">
> > > > <base href="./"><link rel="stylesheet"
> > > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > > <body>
> > > > <flink-root></flink-root>
> > > > <script type="text/javascript"
> > > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > > type="text/javascript"
> src="es2015-polyfills.5e343224e81eefb7658e.js"
> > > nomodule></script><script type="text/javascript"
> > > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > > type="text/javascript"
> src="main.dc6e0997d092df051db0.js"></script></body>
> > > > </html>
> > > >
> > > > Thanks for any ideas,
> > > > Matt
> > > >
> > > > >>> bring-up-cluster
> > > >
> > > > # [yamls from here](
> > >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0
> <https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7Ce73cc973ae1642c5015d08d73ca1742f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637044536460068006&sdata=NAc1LtB9D33cwZZoSNq9vFud6EPDzVegOHrKtDxF4IQ%3D&reserved=0>
> > > )
> > > > minikube start
> > > > minikube ssh 'sudo ip link set docker0 promisc on'
> > > >
> > > > kubectl create -f flink-configuration-configmap.yaml
> > > > kubectl create -f jobmanager-service.yaml
> > > > kubectl create -f jobmanager-deployment.yaml
> > > > kubectl create -f taskmanager-deployment.yaml
> > > > kubectl apply -f jobmanager-rest-service.yaml
> > > >
> > > > >>> grab-the-code
> > > > mvn archetype:generate \
> > > > -DarchetypeGroupId=org.apache.beam \
> > > >
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > > -DarchetypeVersion=2.15.0 \
> > > > -DgroupId=org.example \
> > > > -DartifactId=word-count-beam \
> > > > -Dversion="0.1" \
> > > > -Dpackage=org.apache.beam.examples \
> > > > -DinteractiveMode=false
> > > >
> > > > >>> maven-submit
> > > >
> > > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in
> separate
> > > shell, as blocking
> > > > mvn package exec:java -X
> > > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > > -Dexec.args="--runner=FlinkRunner\
> > > > --flinkMaster="localhost:8081"\
> > > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > > --output=<directory-that-exists>" -Pflink-runner
> > > >
> > > >
> > > >
> > >
> >
>
>
Re: Word-count example
Posted by Matthew Patterson <mp...@aligntech.com>.
Hi Ankur,
Yes, I was using 2.15, but was getting failure to deserialize.
Thanks,
Matt
From: Ankur Goenka <go...@google.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Wednesday, September 18, 2019 at 9:34 PM
To: "user@beam.apache.org" <us...@beam.apache.org>
Subject: Re: Word-count example
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi Matthew,
Beam 2.16.0 is not yet released hence you are getting the error. Can you try using 2.15.0 version?
Thanks,
Ankur
On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <mp...@aligntech.com>> wrote:
Tried
"
mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
-DarchetypeVersion=2.16.0 \
-DgroupId=org.example \
-DartifactId=word-count-beam \
-Dversion="0.1" \
-Dpackage=org.apache.beam.examples \
-DinteractiveMode=false
"
got
-> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate (default-cli) on project standalone-pom: The desired archetype does not exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) -> [Help 1]
On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com>> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Ah. So maybe try 2.16.0?
On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com>> wrote:
> --experiments=beam_fn_api doesn't apply here, as this is a Java pipeline
> using the non-portable version of the Flink runner.
>
> Kyle Weaver | Software Engineer | github.com/ibzib<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgithub.com%2Fibzib&data=02%7C01%7Cmpatterson%40aligntech.com%7Ce73cc973ae1642c5015d08d73ca1742f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637044536460058010&sdata=xut5961XZFOBoo0DG1%2F5qgD%2FAOasLXNAbzfETnu09kg%3D&reserved=0> | kcweaver@google.com<ma...@google.com>
>
>
> On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <be...@gmail.com>>
> wrote:
>
> > Could you try adding "--experiments=beam_fn_api" to the runner options?
> > Kyle did mention that it should be fixed in 2.16.0.
> >
> > On 2019/09/17 16:40:41, Matthew Patterson <mp...@aligntech.com>>
> > wrote:
> > > Folks,
> > >
> > > Am sure I am doing it wrong, but have been beating head against wall
> > long enough I need to ask for help.
> > >
> > > My goal is to use the python SDK to run jobs on AWS flink cluster, but
> > very little is working, so I am trying to back down to the very simplest
> > thing that works, and build back up: not going so well.
> > >
> > > Now I am running, or trying to, on a minikube link cluster. Now running
> > from just java/mvn, so I go through the following:
> > >
> > > 1. bring-up-cluster (see below)
> > > 2. grab-the-code (see below)
> > > 3. maven-submit (see below)
> > >
> > > Eventual failure is
> > >
> > > “[ERROR] Failed to execute goal
> > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project
> > word-count-beam: An exception occured while executing the Java class.
> > Pipeline execution failed: Could not retrieve the execution result. (JobID:
> > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to
> > deserialize JobGraph.]”
> > >
> > > and log is attached (I hope)
> > >
> > > Other details:
> > >
> > > on osx
> > >
> > >
> > >
> > > ➜ minikube version
> > >
> > > minikube version: v1.3.1
> > >
> > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > >
> > >
> > > ➜ kubectl version
> > >
> > > Client Version: version.Info{Major:"1", Minor:"14",
> > GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > >
> > > Server Version: version.Info{Major:"1", Minor:"15",
> > GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > >
> > >
> > > ➜ flink --version
> > >
> > > Version: 1.8.1, Commit ID: 7297bac
> > >
> > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > 2.15.0
> > >
> > > ➜ curl localhost:8081
> > > <!--
> > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > ~ or more contributor license agreements. See the NOTICE file
> > > ~ distributed with this work for additional information
> > > ~ regarding copyright ownership. The ASF licenses this file
> > > ~ to you under the Apache License, Version 2.0 (the
> > > ~ "License"); you may not use this file except in compliance
> > > ~ with the License. You may obtain a copy of the License at
> > > ~
> > > ~ https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7Ce73cc973ae1642c5015d08d73ca1742f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637044536460068006&sdata=1HzYabQkaVH2ob6RL5%2B3L8Mo2hzq69jgLebHSaNzCVI%3D&reserved=0>
> > > ~
> > > ~ Unless required by applicable law or agreed to in writing, software
> > > ~ distributed under the License is distributed on an "AS IS" BASIS,
> > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > implied.
> > > ~ See the License for the specific language governing permissions and
> > > ~ limitations under the License.
> > > -->
> > >
> > > <!doctype html>
> > > <html lang="en">
> > > <head>
> > > <meta charset="utf-8">
> > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > <meta name="viewport" content="width=device-width, initial-scale=1">
> > > <title>Apache Flink Web Dashboard</title>
> > > <link rel="apple-touch-icon" sizes="180x180"
> > href="assets/favicon/apple-touch-icon.png">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > <link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg"
> > color="#aa1919">
> > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > <meta name="msapplication-config"
> > content="assets/favicon/browserconfig.xml">
> > > <meta name="theme-color" content="#ffffff">
> > > <base href="./"><link rel="stylesheet"
> > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > <body>
> > > <flink-root></flink-root>
> > > <script type="text/javascript"
> > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js"
> > nomodule></script><script type="text/javascript"
> > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > type="text/javascript" src="main.dc6e0997d092df051db0.js"></script></body>
> > > </html>
> > >
> > > Thanks for any ideas,
> > > Matt
> > >
> > > >>> bring-up-cluster
> > >
> > > # [yamls from here](
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7Ce73cc973ae1642c5015d08d73ca1742f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637044536460068006&sdata=NAc1LtB9D33cwZZoSNq9vFud6EPDzVegOHrKtDxF4IQ%3D&reserved=0>
> > )
> > > minikube start
> > > minikube ssh 'sudo ip link set docker0 promisc on'
> > >
> > > kubectl create -f flink-configuration-configmap.yaml
> > > kubectl create -f jobmanager-service.yaml
> > > kubectl create -f jobmanager-deployment.yaml
> > > kubectl create -f taskmanager-deployment.yaml
> > > kubectl apply -f jobmanager-rest-service.yaml
> > >
> > > >>> grab-the-code
> > > mvn archetype:generate \
> > > -DarchetypeGroupId=org.apache.beam \
> > > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > -DarchetypeVersion=2.15.0 \
> > > -DgroupId=org.example \
> > > -DartifactId=word-count-beam \
> > > -Dversion="0.1" \
> > > -Dpackage=org.apache.beam.examples \
> > > -DinteractiveMode=false
> > >
> > > >>> maven-submit
> > >
> > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in separate
> > shell, as blocking
> > > mvn package exec:java -X
> > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > -Dexec.args="--runner=FlinkRunner\
> > > --flinkMaster="localhost:8081"\
> > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > --output=<directory-that-exists>" -Pflink-runner
> > >
> > >
> > >
> >
>
Re: Word-count example
Posted by Ankur Goenka <go...@google.com>.
Hi Matthew,
Beam 2.16.0 is not yet released hence you are getting the error. Can you
try using 2.15.0 version?
Thanks,
Ankur
On Wed, Sep 18, 2019 at 6:59 AM Matthew Patterson <mp...@aligntech.com>
wrote:
> Tried
>
> "
> mvn archetype:generate \
> -DarchetypeGroupId=org.apache.beam \
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> -DarchetypeVersion=2.16.0 \
> -DgroupId=org.example \
> -DartifactId=word-count-beam \
> -Dversion="0.1" \
> -Dpackage=org.apache.beam.examples \
> -DinteractiveMode=false
> "
> got
>
> -> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate
> (default-cli) on project standalone-pom: The desired archetype does not
> exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) ->
> [Help 1]
>
>
> On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com> wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
>
> Ah. So maybe try 2.16.0?
>
> On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com> wrote:
> > --experiments=beam_fn_api doesn't apply here, as this is a Java
> pipeline
> > using the non-portable version of the Flink runner.
> >
> > Kyle Weaver | Software Engineer | github.com/ibzib |
> kcweaver@google.com
> >
> >
> > On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <
> benjamintanweihao@gmail.com>
> > wrote:
> >
> > > Could you try adding "--experiments=beam_fn_api" to the runner
> options?
> > > Kyle did mention that it should be fixed in 2.16.0.
> > >
> > > On 2019/09/17 16:40:41, Matthew Patterson <
> mpatterson@aligntech.com>
> > > wrote:
> > > > Folks,
> > > >
> > > > Am sure I am doing it wrong, but have been beating head against
> wall
> > > long enough I need to ask for help.
> > > >
> > > > My goal is to use the python SDK to run jobs on AWS flink
> cluster, but
> > > very little is working, so I am trying to back down to the very
> simplest
> > > thing that works, and build back up: not going so well.
> > > >
> > > > Now I am running, or trying to, on a minikube link cluster. Now
> running
> > > from just java/mvn, so I go through the following:
> > > >
> > > > 1. bring-up-cluster (see below)
> > > > 2. grab-the-code (see below)
> > > > 3. maven-submit (see below)
> > > >
> > > > Eventual failure is
> > > >
> > > > “[ERROR] Failed to execute goal
> > > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on
> project
> > > word-count-beam: An exception occured while executing the Java
> class.
> > > Pipeline execution failed: Could not retrieve the execution
> result. (JobID:
> > > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph.
> [Failed to
> > > deserialize JobGraph.]”
> > > >
> > > > and log is attached (I hope)
> > > >
> > > > Other details:
> > > >
> > > > on osx
> > > >
> > > >
> > > >
> > > > ➜ minikube version
> > > >
> > > > minikube version: v1.3.1
> > > >
> > > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > > >
> > > >
> > > > ➜ kubectl version
> > > >
> > > > Client Version: version.Info{Major:"1", Minor:"14",
> > > GitVersion:"v1.14.6",
> GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > > >
> > > > Server Version: version.Info{Major:"1", Minor:"15",
> > > GitVersion:"v1.15.2",
> GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > > >
> > > >
> > > > ➜ flink --version
> > > >
> > > > Version: 1.8.1, Commit ID: 7297bac
> > > >
> > > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > > 2.15.0
> > > >
> > > > ➜ curl localhost:8081
> > > > <!--
> > > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > > ~ or more contributor license agreements. See the NOTICE file
> > > > ~ distributed with this work for additional information
> > > > ~ regarding copyright ownership. The ASF licenses this file
> > > > ~ to you under the Apache License, Version 2.0 (the
> > > > ~ "License"); you may not use this file except in compliance
> > > > ~ with the License. You may obtain a copy of the License at
> > > > ~
> > > > ~
> https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0
> > > > ~
> > > > ~ Unless required by applicable law or agreed to in writing,
> software
> > > > ~ distributed under the License is distributed on an "AS IS"
> BASIS,
> > > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> or
> > > implied.
> > > > ~ See the License for the specific language governing
> permissions and
> > > > ~ limitations under the License.
> > > > -->
> > > >
> > > > <!doctype html>
> > > > <html lang="en">
> > > > <head>
> > > > <meta charset="utf-8">
> > > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > > <meta name="viewport" content="width=device-width,
> initial-scale=1">
> > > > <title>Apache Flink Web Dashboard</title>
> > > > <link rel="apple-touch-icon" sizes="180x180"
> > > href="assets/favicon/apple-touch-icon.png">
> > > > <link rel="icon" type="image/png"
> > > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > > <link rel="icon" type="image/png"
> > > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > > <link rel="mask-icon"
> href="assets/favicon/safari-pinned-tab.svg"
> > > color="#aa1919">
> > > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > > <meta name="msapplication-config"
> > > content="assets/favicon/browserconfig.xml">
> > > > <meta name="theme-color" content="#ffffff">
> > > > <base href="./"><link rel="stylesheet"
> > > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > > <body>
> > > > <flink-root></flink-root>
> > > > <script type="text/javascript"
> > > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > > type="text/javascript"
> src="es2015-polyfills.5e343224e81eefb7658e.js"
> > > nomodule></script><script type="text/javascript"
> > > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > > type="text/javascript"
> src="main.dc6e0997d092df051db0.js"></script></body>
> > > > </html>
> > > >
> > > > Thanks for any ideas,
> > > > Matt
> > > >
> > > > >>> bring-up-cluster
> > > >
> > > > # [yamls from here](
> > >
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0
> > > )
> > > > minikube start
> > > > minikube ssh 'sudo ip link set docker0 promisc on'
> > > >
> > > > kubectl create -f flink-configuration-configmap.yaml
> > > > kubectl create -f jobmanager-service.yaml
> > > > kubectl create -f jobmanager-deployment.yaml
> > > > kubectl create -f taskmanager-deployment.yaml
> > > > kubectl apply -f jobmanager-rest-service.yaml
> > > >
> > > > >>> grab-the-code
> > > > mvn archetype:generate \
> > > > -DarchetypeGroupId=org.apache.beam \
> > > >
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > > -DarchetypeVersion=2.15.0 \
> > > > -DgroupId=org.example \
> > > > -DartifactId=word-count-beam \
> > > > -Dversion="0.1" \
> > > > -Dpackage=org.apache.beam.examples \
> > > > -DinteractiveMode=false
> > > >
> > > > >>> maven-submit
> > > >
> > > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in
> separate
> > > shell, as blocking
> > > > mvn package exec:java -X
> > > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > > -Dexec.args="--runner=FlinkRunner\
> > > > --flinkMaster="localhost:8081"\
> > > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > > --output=<directory-that-exists>" -Pflink-runner
> > > >
> > > >
> > > >
> > >
> >
>
>
>
Re: Word-count example
Posted by Matthew Patterson <mp...@aligntech.com>.
Tried
"
mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
-DarchetypeVersion=2.16.0 \
-DgroupId=org.example \
-DartifactId=word-count-beam \
-Dversion="0.1" \
-Dpackage=org.apache.beam.examples \
-DinteractiveMode=false
"
got
-> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate (default-cli) on project standalone-pom: The desired archetype does not exist (org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.16.0) -> [Help 1]
On 9/17/19, 7:51 PM, "Benjamin Tan" <be...@gmail.com> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Ah. So maybe try 2.16.0?
On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com> wrote:
> --experiments=beam_fn_api doesn't apply here, as this is a Java pipeline
> using the non-portable version of the Flink runner.
>
> Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com
>
>
> On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <be...@gmail.com>
> wrote:
>
> > Could you try adding "--experiments=beam_fn_api" to the runner options?
> > Kyle did mention that it should be fixed in 2.16.0.
> >
> > On 2019/09/17 16:40:41, Matthew Patterson <mp...@aligntech.com>
> > wrote:
> > > Folks,
> > >
> > > Am sure I am doing it wrong, but have been beating head against wall
> > long enough I need to ask for help.
> > >
> > > My goal is to use the python SDK to run jobs on AWS flink cluster, but
> > very little is working, so I am trying to back down to the very simplest
> > thing that works, and build back up: not going so well.
> > >
> > > Now I am running, or trying to, on a minikube link cluster. Now running
> > from just java/mvn, so I go through the following:
> > >
> > > 1. bring-up-cluster (see below)
> > > 2. grab-the-code (see below)
> > > 3. maven-submit (see below)
> > >
> > > Eventual failure is
> > >
> > > “[ERROR] Failed to execute goal
> > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project
> > word-count-beam: An exception occured while executing the Java class.
> > Pipeline execution failed: Could not retrieve the execution result. (JobID:
> > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to
> > deserialize JobGraph.]”
> > >
> > > and log is attached (I hope)
> > >
> > > Other details:
> > >
> > > on osx
> > >
> > >
> > >
> > > ➜ minikube version
> > >
> > > minikube version: v1.3.1
> > >
> > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > >
> > >
> > > ➜ kubectl version
> > >
> > > Client Version: version.Info{Major:"1", Minor:"14",
> > GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > >
> > > Server Version: version.Info{Major:"1", Minor:"15",
> > GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > >
> > >
> > > ➜ flink --version
> > >
> > > Version: 1.8.1, Commit ID: 7297bac
> > >
> > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > 2.15.0
> > >
> > > ➜ curl localhost:8081
> > > <!--
> > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > ~ or more contributor license agreements. See the NOTICE file
> > > ~ distributed with this work for additional information
> > > ~ regarding copyright ownership. The ASF licenses this file
> > > ~ to you under the Apache License, Version 2.0 (the
> > > ~ "License"); you may not use this file except in compliance
> > > ~ with the License. You may obtain a copy of the License at
> > > ~
> > > ~ https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=4WV%2Bk3Wao0qrbwKJEJTYxG0KixNNGEN%2FW%2FQLz2MrbDk%3D&reserved=0
> > > ~
> > > ~ Unless required by applicable law or agreed to in writing, software
> > > ~ distributed under the License is distributed on an "AS IS" BASIS,
> > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > implied.
> > > ~ See the License for the specific language governing permissions and
> > > ~ limitations under the License.
> > > -->
> > >
> > > <!doctype html>
> > > <html lang="en">
> > > <head>
> > > <meta charset="utf-8">
> > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > <meta name="viewport" content="width=device-width, initial-scale=1">
> > > <title>Apache Flink Web Dashboard</title>
> > > <link rel="apple-touch-icon" sizes="180x180"
> > href="assets/favicon/apple-touch-icon.png">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > <link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg"
> > color="#aa1919">
> > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > <meta name="msapplication-config"
> > content="assets/favicon/browserconfig.xml">
> > > <meta name="theme-color" content="#ffffff">
> > > <base href="./"><link rel="stylesheet"
> > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > <body>
> > > <flink-root></flink-root>
> > > <script type="text/javascript"
> > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js"
> > nomodule></script><script type="text/javascript"
> > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > type="text/javascript" src="main.dc6e0997d092df051db0.js"></script></body>
> > > </html>
> > >
> > > Thanks for any ideas,
> > > Matt
> > >
> > > >>> bring-up-cluster
> > >
> > > # [yamls from here](
> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-stable%2Fops%2Fdeployment%2Fkubernetes.html&data=02%7C01%7Cmpatterson%40aligntech.com%7C118675de36e745219ee508d73bc9fb6f%7C9ac44c96980a481bae23d8f56b82c605%7C0%7C0%7C637043611035577083&sdata=CISNOH4Yw5oYzkQnq9fc%2B64svTpuvMHVXl9j5Qxv2FU%3D&reserved=0
> > )
> > > minikube start
> > > minikube ssh 'sudo ip link set docker0 promisc on'
> > >
> > > kubectl create -f flink-configuration-configmap.yaml
> > > kubectl create -f jobmanager-service.yaml
> > > kubectl create -f jobmanager-deployment.yaml
> > > kubectl create -f taskmanager-deployment.yaml
> > > kubectl apply -f jobmanager-rest-service.yaml
> > >
> > > >>> grab-the-code
> > > mvn archetype:generate \
> > > -DarchetypeGroupId=org.apache.beam \
> > > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > -DarchetypeVersion=2.15.0 \
> > > -DgroupId=org.example \
> > > -DartifactId=word-count-beam \
> > > -Dversion="0.1" \
> > > -Dpackage=org.apache.beam.examples \
> > > -DinteractiveMode=false
> > >
> > > >>> maven-submit
> > >
> > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in separate
> > shell, as blocking
> > > mvn package exec:java -X
> > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > -Dexec.args="--runner=FlinkRunner\
> > > --flinkMaster="localhost:8081"\
> > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > --output=<directory-that-exists>" -Pflink-runner
> > >
> > >
> > >
> >
>
Re: Word-count example
Posted by Benjamin Tan <be...@gmail.com>.
Ah. So maybe try 2.16.0?
On 2019/09/17 23:47:16, Kyle Weaver <kc...@google.com> wrote:
> --experiments=beam_fn_api doesn't apply here, as this is a Java pipeline
> using the non-portable version of the Flink runner.
>
> Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com
>
>
> On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <be...@gmail.com>
> wrote:
>
> > Could you try adding "--experiments=beam_fn_api" to the runner options?
> > Kyle did mention that it should be fixed in 2.16.0.
> >
> > On 2019/09/17 16:40:41, Matthew Patterson <mp...@aligntech.com>
> > wrote:
> > > Folks,
> > >
> > > Am sure I am doing it wrong, but have been beating head against wall
> > long enough I need to ask for help.
> > >
> > > My goal is to use the python SDK to run jobs on AWS flink cluster, but
> > very little is working, so I am trying to back down to the very simplest
> > thing that works, and build back up: not going so well.
> > >
> > > Now I am running, or trying to, on a minikube link cluster. Now running
> > from just java/mvn, so I go through the following:
> > >
> > > 1. bring-up-cluster (see below)
> > > 2. grab-the-code (see below)
> > > 3. maven-submit (see below)
> > >
> > > Eventual failure is
> > >
> > > “[ERROR] Failed to execute goal
> > org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project
> > word-count-beam: An exception occured while executing the Java class.
> > Pipeline execution failed: Could not retrieve the execution result. (JobID:
> > 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to
> > deserialize JobGraph.]”
> > >
> > > and log is attached (I hope)
> > >
> > > Other details:
> > >
> > > on osx
> > >
> > >
> > >
> > > ➜ minikube version
> > >
> > > minikube version: v1.3.1
> > >
> > > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> > >
> > >
> > > ➜ kubectl version
> > >
> > > Client Version: version.Info{Major:"1", Minor:"14",
> > GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> > GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> > GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> > >
> > > Server Version: version.Info{Major:"1", Minor:"15",
> > GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> > GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> > GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> > >
> > >
> > > ➜ flink --version
> > >
> > > Version: 1.8.1, Commit ID: 7297bac
> > >
> > > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > > 2.15.0
> > >
> > > ➜ curl localhost:8081
> > > <!--
> > > ~ Licensed to the Apache Software Foundation (ASF) under one
> > > ~ or more contributor license agreements. See the NOTICE file
> > > ~ distributed with this work for additional information
> > > ~ regarding copyright ownership. The ASF licenses this file
> > > ~ to you under the Apache License, Version 2.0 (the
> > > ~ "License"); you may not use this file except in compliance
> > > ~ with the License. You may obtain a copy of the License at
> > > ~
> > > ~ http://www.apache.org/licenses/LICENSE-2.0
> > > ~
> > > ~ Unless required by applicable law or agreed to in writing, software
> > > ~ distributed under the License is distributed on an "AS IS" BASIS,
> > > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > implied.
> > > ~ See the License for the specific language governing permissions and
> > > ~ limitations under the License.
> > > -->
> > >
> > > <!doctype html>
> > > <html lang="en">
> > > <head>
> > > <meta charset="utf-8">
> > > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > > <meta name="viewport" content="width=device-width, initial-scale=1">
> > > <title>Apache Flink Web Dashboard</title>
> > > <link rel="apple-touch-icon" sizes="180x180"
> > href="assets/favicon/apple-touch-icon.png">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > > <link rel="icon" type="image/png"
> > href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > > <link rel="manifest" href="assets/favicon/manifest.json">
> > > <link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg"
> > color="#aa1919">
> > > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > > <meta name="msapplication-config"
> > content="assets/favicon/browserconfig.xml">
> > > <meta name="theme-color" content="#ffffff">
> > > <base href="./"><link rel="stylesheet"
> > href="styles.30d0912c1ece284d8d9a.css"></head>
> > > <body>
> > > <flink-root></flink-root>
> > > <script type="text/javascript"
> > src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> > type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js"
> > nomodule></script><script type="text/javascript"
> > src="polyfills.b37850e8279bc3caafc9.js"></script><script
> > type="text/javascript" src="main.dc6e0997d092df051db0.js"></script></body>
> > > </html>
> > >
> > > Thanks for any ideas,
> > > Matt
> > >
> > > >>> bring-up-cluster
> > >
> > > # [yamls from here](
> > https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/kubernetes.html
> > )
> > > minikube start
> > > minikube ssh 'sudo ip link set docker0 promisc on'
> > >
> > > kubectl create -f flink-configuration-configmap.yaml
> > > kubectl create -f jobmanager-service.yaml
> > > kubectl create -f jobmanager-deployment.yaml
> > > kubectl create -f taskmanager-deployment.yaml
> > > kubectl apply -f jobmanager-rest-service.yaml
> > >
> > > >>> grab-the-code
> > > mvn archetype:generate \
> > > -DarchetypeGroupId=org.apache.beam \
> > > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > > -DarchetypeVersion=2.15.0 \
> > > -DgroupId=org.example \
> > > -DartifactId=word-count-beam \
> > > -Dversion="0.1" \
> > > -Dpackage=org.apache.beam.examples \
> > > -DinteractiveMode=false
> > >
> > > >>> maven-submit
> > >
> > > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in separate
> > shell, as blocking
> > > mvn package exec:java -X
> > -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > > -Dexec.args="--runner=FlinkRunner\
> > > --flinkMaster="localhost:8081"\
> > > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > > --output=<directory-that-exists>" -Pflink-runner
> > >
> > >
> > >
> >
>
Re: Word-count example
Posted by Kyle Weaver <kc...@google.com>.
--experiments=beam_fn_api doesn't apply here, as this is a Java pipeline
using the non-portable version of the Flink runner.
Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com
On Tue, Sep 17, 2019 at 4:41 PM Benjamin Tan <be...@gmail.com>
wrote:
> Could you try adding "--experiments=beam_fn_api" to the runner options?
> Kyle did mention that it should be fixed in 2.16.0.
>
> On 2019/09/17 16:40:41, Matthew Patterson <mp...@aligntech.com>
> wrote:
> > Folks,
> >
> > Am sure I am doing it wrong, but have been beating head against wall
> long enough I need to ask for help.
> >
> > My goal is to use the python SDK to run jobs on AWS flink cluster, but
> very little is working, so I am trying to back down to the very simplest
> thing that works, and build back up: not going so well.
> >
> > Now I am running, or trying to, on a minikube link cluster. Now running
> from just java/mvn, so I go through the following:
> >
> > 1. bring-up-cluster (see below)
> > 2. grab-the-code (see below)
> > 3. maven-submit (see below)
> >
> > Eventual failure is
> >
> > “[ERROR] Failed to execute goal
> org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project
> word-count-beam: An exception occured while executing the Java class.
> Pipeline execution failed: Could not retrieve the execution result. (JobID:
> 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to
> deserialize JobGraph.]”
> >
> > and log is attached (I hope)
> >
> > Other details:
> >
> > on osx
> >
> >
> >
> > ➜ minikube version
> >
> > minikube version: v1.3.1
> >
> > commit: ca60a424ce69a4d79f502650199ca2b52f29e631
> >
> >
> > ➜ kubectl version
> >
> > Client Version: version.Info{Major:"1", Minor:"14",
> GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc",
> GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z",
> GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
> >
> > Server Version: version.Info{Major:"1", Minor:"15",
> GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568",
> GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z",
> GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
> >
> >
> > ➜ flink --version
> >
> > Version: 1.8.1, Commit ID: 7297bac
> >
> > ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> > 2.15.0
> >
> > ➜ curl localhost:8081
> > <!--
> > ~ Licensed to the Apache Software Foundation (ASF) under one
> > ~ or more contributor license agreements. See the NOTICE file
> > ~ distributed with this work for additional information
> > ~ regarding copyright ownership. The ASF licenses this file
> > ~ to you under the Apache License, Version 2.0 (the
> > ~ "License"); you may not use this file except in compliance
> > ~ with the License. You may obtain a copy of the License at
> > ~
> > ~ http://www.apache.org/licenses/LICENSE-2.0
> > ~
> > ~ Unless required by applicable law or agreed to in writing, software
> > ~ distributed under the License is distributed on an "AS IS" BASIS,
> > ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> implied.
> > ~ See the License for the specific language governing permissions and
> > ~ limitations under the License.
> > -->
> >
> > <!doctype html>
> > <html lang="en">
> > <head>
> > <meta charset="utf-8">
> > <meta http-equiv="X-UA-Compatible" content="IE=edge">
> > <meta name="viewport" content="width=device-width, initial-scale=1">
> > <title>Apache Flink Web Dashboard</title>
> > <link rel="apple-touch-icon" sizes="180x180"
> href="assets/favicon/apple-touch-icon.png">
> > <link rel="icon" type="image/png"
> href="assets/favicon/favicon-32x32.png" sizes="32x32">
> > <link rel="icon" type="image/png"
> href="assets/favicon/favicon-16x16.png" sizes="16x16">
> > <link rel="manifest" href="assets/favicon/manifest.json">
> > <link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg"
> color="#aa1919">
> > <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> > <meta name="msapplication-config"
> content="assets/favicon/browserconfig.xml">
> > <meta name="theme-color" content="#ffffff">
> > <base href="./"><link rel="stylesheet"
> href="styles.30d0912c1ece284d8d9a.css"></head>
> > <body>
> > <flink-root></flink-root>
> > <script type="text/javascript"
> src="runtime.ebbfed06bb8e06aaa889.js"></script><script
> type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js"
> nomodule></script><script type="text/javascript"
> src="polyfills.b37850e8279bc3caafc9.js"></script><script
> type="text/javascript" src="main.dc6e0997d092df051db0.js"></script></body>
> > </html>
> >
> > Thanks for any ideas,
> > Matt
> >
> > >>> bring-up-cluster
> >
> > # [yamls from here](
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/kubernetes.html
> )
> > minikube start
> > minikube ssh 'sudo ip link set docker0 promisc on'
> >
> > kubectl create -f flink-configuration-configmap.yaml
> > kubectl create -f jobmanager-service.yaml
> > kubectl create -f jobmanager-deployment.yaml
> > kubectl create -f taskmanager-deployment.yaml
> > kubectl apply -f jobmanager-rest-service.yaml
> >
> > >>> grab-the-code
> > mvn archetype:generate \
> > -DarchetypeGroupId=org.apache.beam \
> > -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> > -DarchetypeVersion=2.15.0 \
> > -DgroupId=org.example \
> > -DartifactId=word-count-beam \
> > -Dversion="0.1" \
> > -Dpackage=org.apache.beam.examples \
> > -DinteractiveMode=false
> >
> > >>> maven-submit
> >
> > # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in separate
> shell, as blocking
> > mvn package exec:java -X
> -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> > -Dexec.args="--runner=FlinkRunner\
> > --flinkMaster="localhost:8081"\
> > --filesToStage=target/word-count-beam-bundled-0.1.jar\
> > --output=<directory-that-exists>" -Pflink-runner
> >
> >
> >
>
Re: Word-count example
Posted by Benjamin Tan <be...@gmail.com>.
Could you try adding "--experiments=beam_fn_api" to the runner options? Kyle did mention that it should be fixed in 2.16.0.
On 2019/09/17 16:40:41, Matthew Patterson <mp...@aligntech.com> wrote:
> Folks,
>
> Am sure I am doing it wrong, but have been beating head against wall long enough I need to ask for help.
>
> My goal is to use the python SDK to run jobs on AWS flink cluster, but very little is working, so I am trying to back down to the very simplest thing that works, and build back up: not going so well.
>
> Now I am running, or trying to, on a minikube link cluster. Now running from just java/mvn, so I go through the following:
>
> 1. bring-up-cluster (see below)
> 2. grab-the-code (see below)
> 3. maven-submit (see below)
>
> Eventual failure is
>
> “[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project word-count-beam: An exception occured while executing the Java class. Pipeline execution failed: Could not retrieve the execution result. (JobID: 6edad04faa43fbf22bdea24171ba9d59): Failed to submit JobGraph. [Failed to deserialize JobGraph.]”
>
> and log is attached (I hope)
>
> Other details:
>
> on osx
>
>
>
> ➜ minikube version
>
> minikube version: v1.3.1
>
> commit: ca60a424ce69a4d79f502650199ca2b52f29e631
>
>
> ➜ kubectl version
>
> Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
>
> Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
>
>
> ➜ flink --version
>
> Version: 1.8.1, Commit ID: 7297bac
>
> ➜ python -c"import apache_beam; print(apache_beam.__version__)"
> 2.15.0
>
> ➜ curl localhost:8081
> <!--
> ~ Licensed to the Apache Software Foundation (ASF) under one
> ~ or more contributor license agreements. See the NOTICE file
> ~ distributed with this work for additional information
> ~ regarding copyright ownership. The ASF licenses this file
> ~ to you under the Apache License, Version 2.0 (the
> ~ "License"); you may not use this file except in compliance
> ~ with the License. You may obtain a copy of the License at
> ~
> ~ http://www.apache.org/licenses/LICENSE-2.0
> ~
> ~ Unless required by applicable law or agreed to in writing, software
> ~ distributed under the License is distributed on an "AS IS" BASIS,
> ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> ~ See the License for the specific language governing permissions and
> ~ limitations under the License.
> -->
>
> <!doctype html>
> <html lang="en">
> <head>
> <meta charset="utf-8">
> <meta http-equiv="X-UA-Compatible" content="IE=edge">
> <meta name="viewport" content="width=device-width, initial-scale=1">
> <title>Apache Flink Web Dashboard</title>
> <link rel="apple-touch-icon" sizes="180x180" href="assets/favicon/apple-touch-icon.png">
> <link rel="icon" type="image/png" href="assets/favicon/favicon-32x32.png" sizes="32x32">
> <link rel="icon" type="image/png" href="assets/favicon/favicon-16x16.png" sizes="16x16">
> <link rel="manifest" href="assets/favicon/manifest.json">
> <link rel="mask-icon" href="assets/favicon/safari-pinned-tab.svg" color="#aa1919">
> <link rel="shortcut icon" href="assets/favicon/favicon.ico">
> <meta name="msapplication-config" content="assets/favicon/browserconfig.xml">
> <meta name="theme-color" content="#ffffff">
> <base href="./"><link rel="stylesheet" href="styles.30d0912c1ece284d8d9a.css"></head>
> <body>
> <flink-root></flink-root>
> <script type="text/javascript" src="runtime.ebbfed06bb8e06aaa889.js"></script><script type="text/javascript" src="es2015-polyfills.5e343224e81eefb7658e.js" nomodule></script><script type="text/javascript" src="polyfills.b37850e8279bc3caafc9.js"></script><script type="text/javascript" src="main.dc6e0997d092df051db0.js"></script></body>
> </html>
>
> Thanks for any ideas,
> Matt
>
> >>> bring-up-cluster
>
> # [yamls from here](https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/kubernetes.html)
> minikube start
> minikube ssh 'sudo ip link set docker0 promisc on'
>
> kubectl create -f flink-configuration-configmap.yaml
> kubectl create -f jobmanager-service.yaml
> kubectl create -f jobmanager-deployment.yaml
> kubectl create -f taskmanager-deployment.yaml
> kubectl apply -f jobmanager-rest-service.yaml
>
> >>> grab-the-code
> mvn archetype:generate \
> -DarchetypeGroupId=org.apache.beam \
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> -DarchetypeVersion=2.15.0 \
> -DgroupId=org.example \
> -DartifactId=word-count-beam \
> -Dversion="0.1" \
> -Dpackage=org.apache.beam.examples \
> -DinteractiveMode=false
>
> >>> maven-submit
>
> # kubectl port-forward svc/flink-jobmanager-rest 8081:8081 # in separate shell, as blocking
> mvn package exec:java -X -Dexec.mainClass=org.apache.beam.examples.DebuggingWordCount \
> -Dexec.args="--runner=FlinkRunner\
> --flinkMaster="localhost:8081"\
> --filesToStage=target/word-count-beam-bundled-0.1.jar\
> --output=<directory-that-exists>" -Pflink-runner
>
>
>