[jira] [Created] (BEAM-5350) Running autocomplete.go on dataflow fails

["Tomas Roos (JIRA)" <ji...@apache.org>]

Tomas Roos created BEAM-5350:
--------------------------------

             Summary: Running autocomplete.go on dataflow fails
                 Key: BEAM-5350
                 URL: https://issues.apache.org/jira/browse/BEAM-5350
             Project: Beam
          Issue Type: Bug
          Components: sdk-go
    Affects Versions: Not applicable
            Reporter: Tomas Roos
            Assignee: Henning Rohde


I'm in the process as a external developer make sure that all examples are runnable on both direct and the dataflow runner as its crucial for people onboarding this project.

 

I've visted the projects before and some are runnable, some probably where previously, and some are def not runnable.

 

So I started top down today, in order to make autocomplete.go run on dataflow as well as the direct runner i changed the input in order to make it platform independent instead of pointing to a local file.The reading of the source from the public cloud storage went fine but it fails to run the top.Largest anonymous less function (ran on id: go-job-1-1536575613531078735) failed with

 

RESP: instruction_id: "-205" error: "Invalid bundle desc: decode: bad userfn: bad struct encoding: failed to decode data: decode: failed to find symbol main.main.func1: main.main.func1 not found. Use runtime.RegisterFunction in unit tests" register: < >

 

So in order to fix this I introduced the local func called lessFn and registered in the init process. This though now instead when running

 

go run autocomplete.go --project fair-app-213019 --runner dataflow --staging_location=gs://fair-app-213019/staging-test2 --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515

 

fails with

 

2018/09/10 13:37:10 Running autocomplete
Unable to encode combiner for lifting: failed to encode custom coder: bad underlying type: bad field type: bad element: unencodable type: interface {}2018/09/10 13:37:10 Using running binary as worker binary: '/tmp/go-build157286122/b001/exe/autocomplete'
2018/09/10 13:37:10 Staging worker binary: /tmp/go-build157286122/b001/exe/autocomplete

 

And I know this is when invoking the top.Largest since I've removed the piece of code and then the job runs fine, could you please point me in the right direction why my local func is not encoable as a interface {} and I will of course happily send a PR when this is working on direct and the dataflow direct so I can move on to the other examples

 

 

 

(All changes can be seen here) [https://github.com/apache/beam/compare/master...ptomasroos:make-autocomplete-dataflowable?expand=1]

 

 

https://github.com/apache/beam/blob/master/sdks/go/examples/complete/autocomplete/autocomplete.go#L63

 

2018/09/10 13:37:10 Running autocomplete
Unable to encode combiner for lifting: failed to encode custom coder: bad underlying type: bad field type: bad element: unencodable type: interface {}2018/09/10 13:37:10 Using running binary as worker binary: '/tmp/go-build157286122/b001/exe/autocomplete'
2018/09/10 13:37:10 Staging worker binary: /tmp/go-build157286122/b001/exe/autocomplete



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)