You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/08/07 05:43:00 UTC
[jira] [Commented] (BEAM-2083) Develop a Go SDK for Beam
[ https://issues.apache.org/jira/browse/BEAM-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116077#comment-16116077 ]
ASF GitHub Bot commented on BEAM-2083:
--------------------------------------
GitHub user c0b opened a pull request:
https://github.com/apache/beam/pull/3694
could you allow github issues here? [dummy pr for issue comment only]
_I don't understand why do you require jira ticket instead of github issues here; here I'd only want to comment on the tickets but creating an account on https://issues.apache.org for comment is a broken user experience (compared to github issues)_
- https://issues.apache.org/jira/browse/BEAM-2083 for Go SDK
- https://issues.apache.org/jira/browse/BEAM-1754 for NodeJS SDK
- https://issues.apache.org/jira/browse/BEAM-14 for a generic declarative DSL for any language SDK writers can use
from the https://beam.apache.org/documentation/runners/capability-matrix/ I did my first test run is to see how many runners supported by existing languages (Java & Python); I did test the example wordcount with both Java and Python, from this error from Python seems like it does not have most other runners, and Python so far only support Direct and DataflowRunner, still lack important features like triggers?
ValueError: Unexpected pipeline runner: ApexRunner. Valid values are DirectRunner, EagerRunner, DataflowRunner, TestDataflowRunner or the fully qualified name of a PipelineRunner subclass.
so focus at DataflowRunner with Python or try another programming language:
dataflow is providing REST API calls, but the difficulty here for another programming language is how to provide the job create request body? especially, how define and encode the job steps ?
https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.jobs#Job.Step
from two test runs of wordcount examples, so far I found the clues:
1. with jobs list api with `view=JOB_VIEW_ALL` I can see java and python uses a different **workerHarnessContainerImage**, so I do docker pull these images to locally to look into, but where are the source code for each? are these open sourced? what is the default entrypoint `/opt/google/dataflow/boot` ?
"workerHarnessContainerImage": "dataflow.gcr.io/v1beta3/beam-java-batch:beam-2.0.0"
"workerHarnessContainerImage": "dataflow.gcr.io/v1beta3/python:2.0.0"
```console
$ docker images --filter='reference=dataflow.gcr.io/v1beta3/*:*'
REPOSITORY TAG IMAGE ID CREATED SIZE
dataflow.gcr.io/v1beta3/python 2.0.0 2a1e69afbef9 2 months ago 1.3GB
dataflow.gcr.io/v1beta3/beam-java-batch beam-2.0.0 2686ad94cb93 5 months ago 393MB
$ docker run -it --rm --entrypoint=/bin/bash dataflow.gcr.io/v1beta3/python:2.0.0
...
root@ddfe741352d6:/# \du -sh /usr/local/gcloud/google-cloud-sdk /usr/local/lib/python2.7/dist-packages/tensorflow /usr/local/lib/python2.7/dist-packages/scipy /usr/local/lib/python2.7/dist-packages/sklearn /opt/google/dataflow
226M /usr/local/gcloud/google-cloud-sdk
167M /usr/local/lib/python2.7/dist-packages/tensorflow
155M /usr/local/lib/python2.7/dist-packages/scipy
72M /usr/local/lib/python2.7/dist-packages/sklearn
26M /opt/google/dataflow
root@ddfe741352d6:/# ls -lih /opt/google/dataflow
total 26M
19005540 -r-xr-xr-x 1 root root 43K Jan 1 1970 NOTICES.shuffle
19005538 -r-xr-xr-x 1 root root 14M Jan 1 1970 boot
19005539 -r-xr-xr-x 1 root root 680K Jan 1 1970 dataflow_python_worker.tar
19005541 -r-xr-xr-x 1 root root 12M Jan 1 1970 shuffle_client.so
```
2. the REST API only defined each step requires a `kind`, `name` and `properties`; but what's the internal structure of `properties` ? for the python one I spent some time figured out the `serialized_fn` is base64 encoded of zlib compressed of a pickle serialized object code of the python function, and Java version's serialized_fn is using another way of serialization of a function (looks like snappy compression of java byte code?),
so the question is here: does this mean the `properties` is complete up to SDK Writers? if somebody is going to do Go or NodeJS, since different language has very different way of serialization of a function's code; all these will look like duplicating a lot of effort, then BEAM-14 could be a better approach?
but generally, could you share more necessary documentation for SDK writters? so far I feel these are necessary: **1) define a function serialization protocol, to be used in the `steps / properties`** **2) a language specific docker image, to be used as `workerHarnessContainerImage`, this image will need to interpret the serialization protocol from `steps / properties`**
```json
{
"kind": "ParallelDo",
"name": "s2",
"properties": {
"serialized_fn": {
"value": "eNq1VvtX3EQUnuwuLGRBhWKtrdbYShukZLU+sFTbIvTl2m3dMmIfGCfJ7E4gm+xNJgUq0VYO1T/UP8Q72WzpntPHT27OLjPf3Pvdm/sanpRM1/ODwFI/uu3GnEluy90e14EMHQUR84oDjY6t4UJ9oXQA5RZU6PR6FHtXd2TMXOmHndXoWggjLusxV3Db4axr4VGYtKO4m1huFHNdiegwegDVDMZMOmnbSRDJkHV5YtswvgE61W27G3lpwBGo0THcMT/E9QSt5a7Ywg9lApPDdvAgxy2PoyEmozjRb95W3t5QsA5vzT2Ft5sZvGPSmh/2UpmTJTDVpBNRKg+B6Wb6DI44tNqLI5cnCcy8JFjtNMRXjvBd3jUFhmYl8vqhOXoA77XgmNkoNUYalVsrV+jBnkY2CdknJCuRvRJJZskeIZsIloinkX2NaCEhskw2K8TLJfZLJCuTnetkr0xWNy6TrJJrjCgNpNHUZlRtZJXsVRSDIlHoGNkcH0bhEllHmrvwPv23xWUah4nBQsOXPA+SET3isSEFN7YxlYkRtXHjJwYPeJeH0tJ1Az9reF4gBh4yI/BDnsvyHWkZxs12TpGjeO4ELNw6Z4SR5Igzec6QUVRQLcedZClfGQPKpVx5wO9wrCSjiD33+lqF5wNF5c9ziUNf1REcb2i09mB54T5bePzw7MY8nPgbPjDpSCJjvwcf0mne7cldW/lqu1EaYiDgJC37oQsf0VLMwaDVth96LAjgYzqZh+W54Ck6owA74GFHikP8NC0jBJ/QqaFjz08kzNLRtOdh0cAZCWdNWkl40AaTVgu/YY6OqzjmLsGndCQ3CfNU24ZzEhbo/fojFtdF1OV1j8d8y8KqDzt1K4hcFtQD36n3dqWIwvPWYj3BxC5gX2yxDk/qL3RIne+wbi9AUNHnflu9XbDETMOi06Q2oY1qR7RpbRKfo9pxDepzcxI+a8Hnrm07qR9I1YT6oBt1MdPEZjov4YsWfEmrtu1FKAhf0bk7LE4wnWj5eZnk/ZYXCy5l1K81C75W3e2HvkTFRYGNhG2j4VNWbbMtsSk0VfjYDkXh95uk3wuqdUpqbY+SIbw0wKs5PjbAywN8fFi+MsB14hFslG/MppinEy8mEi70C0Ht+0ldorXDOkrgooRvVY2lPSyG78SIWKTVWxwrzk3gEq0OyuQyHbdtN2CJGnZXxClxmk4outh3UjVNYFnMipMSvhfmAaz8r4lfFYuNM3SCYNKP4TOKqYerKuXXWnD9FSlfVCm/IeFmC35IJTRa8CMO1VsZNE0xKdSAvY0Cd0wx1RT5FP3JEbNDXB0umZSxDq2h6d3th6r4qxeR0+EuPbqKXdJh+e3yYqDWngFtwc9ocj2DX2itr2mriwTuvYbbYQkv+Jsoq8N9pHiQwUOTjue3EGpy2KAnBm9tveSK+5VWckO2WEqdxME8tsRdOnXo6kqR79+eAWuBgybcDDxxTyhrPIO2Kd5goiNsMZ+TnxRKX2TgF/qbGWy9UT9A/Yu5/ulcv5tBWOhHGfTeqA+of0Hpp84GxBkkGyBfe7Gv48iMtlFdhxSNPMpg26Rj2znaDmHnVcp9Cf16EDks6JNg4neR4nEGvzs4GmO/0+Excuy9iqMQ0Vd5m6WBXCu2kCHLHxn86dBp6WNuJTYCjuyugz0bw5MGoVPMddNuGjBVVOrfDg5PEa7ZPg7vPhv89U/qSNi3/gNiHxCp",
"@type": "http://schema.org/Text"
},
"display_data": [
{
"value": {
"@type": "http://schema.org/Text",
"value": "__main__.WordExtractingDoFn"
},
[...]
```
3. I see the Python API uses a lot of operator overloading like `|` and `>>`, but is that a very thoughtful decision, for reading input to use `p | 'label' >> beam.ReadFrom...()` I don't feel very intuitive, why not use `<<` to mean read from ? are there any good writtings from engineers behind?
it's similar for other languages which have many other kinds of syntax sugar, will be more interesting if can program in other languages than Java or Python; but will it become true? or will google will dedicate some more effort? or answer is **never**; could you do first fill parity from Python to have all features of Java API? the missing feature of triggers is an important one
I don't see BigData processing area (Streaming or Batch, the competence lead by Spark vs. Apex vs. Flink vs. Gearpump vs. DataFlow) for any other programming language than Java is mature so far, to do any serious work on BigData processing, I feel choices are still limited to Java only, at least for this year 2017. would you say more programming language can do BigData, more SDKs coming next year
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/c0b/beam patch-1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3694.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3694
----
commit e4f6d0502e9d85f0760dd657314e890909d9cca3
Author: c0b <de...@gmail.com>
Date: 2017-08-06T20:03:31Z
could you allow github issues here?
----
> Develop a Go SDK for Beam
> -------------------------
>
> Key: BEAM-2083
> URL: https://issues.apache.org/jira/browse/BEAM-2083
> Project: Beam
> Issue Type: New Feature
> Components: sdk-ideas
> Reporter: Bill Neubauer
> Assignee: Bill Neubauer
> Priority: Minor
>
> Allow users of the Go programming language (https://golang.org/) to write Beam pipelines in this language.
> The effort is focusing on full-fledged SDK that leverages the Beam Fn API to bootstrap a native Go experience.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)