You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Yu Watanabe <yu...@gmail.com> on 2022/08/28 11:49:54 UTC
How to run expansion service using go sdk in local development environment ?
Hello.
I would like to ask a question about expansion service. I'm currently
testing my expansion service in my local development environment.
I have read notes about kafka in advance,
https://github.com/apache/beam/blob/master/sdks/go/examples/kafka/taxi.go#L93
I have prepared sdk containers .
[ywatanabe@laptop-archlinux Development]$ docker image ls | grep apache
apache/beam_java8_sdk 2.42.0.dev
f7e9d38b01fe 11 days ago 643MB
apache/beam_go_sdk latest
8a87ea45255b 11 days ago 149MB
However, when I run the code in my local environment, I get an error.
[ywatanabe@laptop-archlinux go]$ go run ./examples/elasticsearch/sample.go \
--runner direct \
--sdk_harness_container_image_override
".*java.*,apache/beam_java8_sdk:2.42.0.dev"
Hello world.
2022/08/28 20:39:01 Executing pipeline with the direct runner.
2022/08/28 20:39:01 Pipeline:
2022/08/28 20:39:01 Nodes: {1: []uint8/bytes GLO}
{2: string/string GLO}
{3: []uint8/bytes GLO}
{4: []uint8/bytes GLO}
Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T ->
{2: string/string GLO}]
3: External [In(Main): string <- {2: string/string GLO}] -> [Out:
[]uint8 -> {3: []uint8/bytes GLO} Out: []uint8 -> {4: []uint8/bytes
GLO}]
Pipeline failed: translation failed
caused by:
external transforms like 3: External [In(Main): string <- {2:
string/string GLO}] -> [Out: []uint8 -> {3: []uint8/bytes GLO} Out:
[]uint8 -> {4: []uint8/bytes GLO}] are not supported in the Go direct
runner, please execute your pipel[ywatanabe@laptop-archlinux go]$
Am I missing something ?
My main and io code can be found below.
https://gist.github.com/yuwtennis/dec3bf3bfc0c4fa54d9d3565c98d008e
Thanks,
Yu
--
Yu Watanabe
linkedin: www.linkedin.com/in/yuwatanabe1/
twitter: twitter.com/yuwtennis
Re: How to run expansion service using go sdk in local development environment ?
Posted by Yu Watanabe <yu...@gmail.com>.
Hello Danny.
Ah . I see . Thank you for your advice.
Thanks,
Yu Watanabe
On Mon, Aug 29, 2022 at 9:26 AM Danny McCormick via user
<us...@beam.apache.org> wrote:
>
> Hey Yu, as the error you posted suggests, the Go direct runner which you're using in your local development environment doesn't support external transforms using an expansion service. If you're going to do a x-lang transform using an expansion service you should use a different runner like Dataflow, Flink, Spark, or one of the other runners listed here - https://beam.apache.org/documentation/runners/capability-matrix/
>
> Thanks,
> Danny
>
> On Sun, Aug 28, 2022 at 7:50 AM Yu Watanabe <yu...@gmail.com> wrote:
>>
>> Hello.
>>
>> I would like to ask a question about expansion service. I'm currently
>> testing my expansion service in my local development environment.
>> I have read notes about kafka in advance,
>>
>> https://github.com/apache/beam/blob/master/sdks/go/examples/kafka/taxi.go#L93
>>
>> I have prepared sdk containers .
>>
>> [ywatanabe@laptop-archlinux Development]$ docker image ls | grep apache
>> apache/beam_java8_sdk 2.42.0.dev
>> f7e9d38b01fe 11 days ago 643MB
>> apache/beam_go_sdk latest
>> 8a87ea45255b 11 days ago 149MB
>>
>> However, when I run the code in my local environment, I get an error.
>>
>> [ywatanabe@laptop-archlinux go]$ go run ./examples/elasticsearch/sample.go \
>> --runner direct \
>> --sdk_harness_container_image_override
>> ".*java.*,apache/beam_java8_sdk:2.42.0.dev"
>> Hello world.
>> 2022/08/28 20:39:01 Executing pipeline with the direct runner.
>> 2022/08/28 20:39:01 Pipeline:
>> 2022/08/28 20:39:01 Nodes: {1: []uint8/bytes GLO}
>> {2: string/string GLO}
>> {3: []uint8/bytes GLO}
>> {4: []uint8/bytes GLO}
>> Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
>> 2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T ->
>> {2: string/string GLO}]
>> 3: External [In(Main): string <- {2: string/string GLO}] -> [Out:
>> []uint8 -> {3: []uint8/bytes GLO} Out: []uint8 -> {4: []uint8/bytes
>> GLO}]
>> Pipeline failed: translation failed
>> caused by:
>> external transforms like 3: External [In(Main): string <- {2:
>> string/string GLO}] -> [Out: []uint8 -> {3: []uint8/bytes GLO} Out:
>> []uint8 -> {4: []uint8/bytes GLO}] are not supported in the Go direct
>> runner, please execute your pipel[ywatanabe@laptop-archlinux go]$
>>
>> Am I missing something ?
>>
>> My main and io code can be found below.
>>
>> https://gist.github.com/yuwtennis/dec3bf3bfc0c4fa54d9d3565c98d008e
>>
>> Thanks,
>> Yu
>>
>>
>> --
>> Yu Watanabe
>>
>> linkedin: www.linkedin.com/in/yuwatanabe1/
>> twitter: twitter.com/yuwtennis
--
Yu Watanabe
linkedin: www.linkedin.com/in/yuwatanabe1/
twitter: twitter.com/yuwtennis
Re: How to run expansion service using go sdk in local development environment ?
Posted by Danny McCormick via user <us...@beam.apache.org>.
Hey Yu, as the error you posted suggests, the Go direct runner which you're
using in your local development environment doesn't support external
transforms using an expansion service. If you're going to do a x-lang
transform using an expansion service you should use a different runner like
Dataflow, Flink, Spark, or one of the other runners listed here -
https://beam.apache.org/documentation/runners/capability-matrix/
Thanks,
Danny
On Sun, Aug 28, 2022 at 7:50 AM Yu Watanabe <yu...@gmail.com> wrote:
> Hello.
>
> I would like to ask a question about expansion service. I'm currently
> testing my expansion service in my local development environment.
> I have read notes about kafka in advance,
>
>
> https://github.com/apache/beam/blob/master/sdks/go/examples/kafka/taxi.go#L93
>
> I have prepared sdk containers .
>
> [ywatanabe@laptop-archlinux Development]$ docker image ls | grep apache
> apache/beam_java8_sdk 2.42.0.dev
> f7e9d38b01fe 11 days ago 643MB
> apache/beam_go_sdk latest
> 8a87ea45255b 11 days ago 149MB
>
> However, when I run the code in my local environment, I get an error.
>
> [ywatanabe@laptop-archlinux go]$ go run
> ./examples/elasticsearch/sample.go \
> --runner direct \
> --sdk_harness_container_image_override
> ".*java.*,apache/beam_java8_sdk:2.42.0.dev"
> Hello world.
> 2022/08/28 20:39:01 Executing pipeline with the direct runner.
> 2022/08/28 20:39:01 Pipeline:
> 2022/08/28 20:39:01 Nodes: {1: []uint8/bytes GLO}
> {2: string/string GLO}
> {3: []uint8/bytes GLO}
> {4: []uint8/bytes GLO}
> Edges: 1: Impulse [] -> [Out: []uint8 -> {1: []uint8/bytes GLO}]
> 2: ParDo [In(Main): []uint8 <- {1: []uint8/bytes GLO}] -> [Out: T ->
> {2: string/string GLO}]
> 3: External [In(Main): string <- {2: string/string GLO}] -> [Out:
> []uint8 -> {3: []uint8/bytes GLO} Out: []uint8 -> {4: []uint8/bytes
> GLO}]
> Pipeline failed: translation failed
> caused by:
> external transforms like 3: External [In(Main): string <- {2:
> string/string GLO}] -> [Out: []uint8 -> {3: []uint8/bytes GLO} Out:
> []uint8 -> {4: []uint8/bytes GLO}] are not supported in the Go direct
> runner, please execute your pipel[ywatanabe@laptop-archlinux go]$
>
> Am I missing something ?
>
> My main and io code can be found below.
>
> https://gist.github.com/yuwtennis/dec3bf3bfc0c4fa54d9d3565c98d008e
>
> Thanks,
> Yu
>
>
> --
> Yu Watanabe
>
> linkedin: www.linkedin.com/in/yuwatanabe1/
> twitter: twitter.com/yuwtennis
>