You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Yu Watanabe <yu...@gmail.com> on 2022/07/08 07:51:05 UTC
Any guideline for building golang connector ?
Hello .
Is there any guideline for building a go sdk connector ?
I was reviewing the document but I could not find one for golang.
https://beam.apache.org/documentation/io/developing-io-overview/
I was thinking of building one for elasticsearch.
Thanks,
Yu Watanabe
--
Yu Watanabe
linkedin: www.linkedin.com/in/yuwatanabe1/
twitter: twitter.com/yuwtennis
Re: Any guideline for building golang connector ?
Posted by Chamikara Jayalath via user <us...@beam.apache.org>.
Strong +1 for using x-lang instead if re-implementing the ElasticSearch
connector in Go.
Thanks,
Cham
On Fri, Jul 8, 2022 at 5:16 AM Yu Watanabe <yu...@gmail.com> wrote:
> Hello Danny.
>
> Thank you for the details. I appreciate your message.
>
> I am a newbie around building io . So I will look into the links and first
> build my knowledge.
>
> Thanks,
> Yu Watanabe
>
> On Fri, Jul 8, 2022 at 8:28 PM Danny McCormick via user <
> user@beam.apache.org> wrote:
>
>> Hey Yu,
>>
>> The guidance on that page should generally apply for Go as well, though
>> we are missing an example transform; I filed
>> https://github.com/apache/beam/issues/22194 to fix this, but a couple
>> examples are our textio implementation
>> <https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/io/textio/textio.go>
>> and this native streaming example
>> <https://github.com/apache/beam/blob/master/sdks/go/examples/native_wordcap/nativepubsubio/native.go>
>> (non-productionized, but it shows a lot of good concepts for handling
>> streaming sources).
>>
>> Another option would be to use Java's elasticsearch implementation
>> <https://github.com/apache/beam/blob/master/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java>
>> with a cross language transform. If you're thinking of contributing this
>> back to the main beam code base, I'd probably recommend that approach. In
>> general, we're currently investing most heavily in cross language
>> transforms because it's a much lower burden to build/support since it
>> reuses the main components of the original transform. There are several
>> examples of wrapped cross-language transforms in
>> https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/io/xlang. If
>> you have specific questions about authoring IOs that aren't answered in the
>> docs/by example, feel free to drop them in this thread as well!
>>
>> Thanks,
>> Danny
>>
>> On Fri, Jul 8, 2022 at 3:51 AM Yu Watanabe <yu...@gmail.com> wrote:
>>
>>> Hello .
>>>
>>> Is there any guideline for building a go sdk connector ?
>>> I was reviewing the document but I could not find one for golang.
>>>
>>> https://beam.apache.org/documentation/io/developing-io-overview/
>>>
>>> I was thinking of building one for elasticsearch.
>>>
>>> Thanks,
>>> Yu Watanabe
>>>
>>> --
>>> Yu Watanabe
>>>
>>> linkedin: www.linkedin.com/in/yuwatanabe1/
>>> twitter: twitter.com/yuwtennis
>>>
>>>
>>
>
> --
> Yu Watanabe
>
> linkedin: www.linkedin.com/in/yuwatanabe1/
> twitter: twitter.com/yuwtennis
>
>
Re: Any guideline for building golang connector ?
Posted by Yu Watanabe <yu...@gmail.com>.
Hello Danny.
Thank you for the details. I appreciate your message.
I am a newbie around building io . So I will look into the links and first
build my knowledge.
Thanks,
Yu Watanabe
On Fri, Jul 8, 2022 at 8:28 PM Danny McCormick via user <
user@beam.apache.org> wrote:
> Hey Yu,
>
> The guidance on that page should generally apply for Go as well, though we
> are missing an example transform; I filed
> https://github.com/apache/beam/issues/22194 to fix this, but a couple
> examples are our textio implementation
> <https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/io/textio/textio.go>
> and this native streaming example
> <https://github.com/apache/beam/blob/master/sdks/go/examples/native_wordcap/nativepubsubio/native.go>
> (non-productionized, but it shows a lot of good concepts for handling
> streaming sources).
>
> Another option would be to use Java's elasticsearch implementation
> <https://github.com/apache/beam/blob/master/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java>
> with a cross language transform. If you're thinking of contributing this
> back to the main beam code base, I'd probably recommend that approach. In
> general, we're currently investing most heavily in cross language
> transforms because it's a much lower burden to build/support since it
> reuses the main components of the original transform. There are several
> examples of wrapped cross-language transforms in
> https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/io/xlang. If
> you have specific questions about authoring IOs that aren't answered in the
> docs/by example, feel free to drop them in this thread as well!
>
> Thanks,
> Danny
>
> On Fri, Jul 8, 2022 at 3:51 AM Yu Watanabe <yu...@gmail.com> wrote:
>
>> Hello .
>>
>> Is there any guideline for building a go sdk connector ?
>> I was reviewing the document but I could not find one for golang.
>>
>> https://beam.apache.org/documentation/io/developing-io-overview/
>>
>> I was thinking of building one for elasticsearch.
>>
>> Thanks,
>> Yu Watanabe
>>
>> --
>> Yu Watanabe
>>
>> linkedin: www.linkedin.com/in/yuwatanabe1/
>> twitter: twitter.com/yuwtennis
>>
>>
>
--
Yu Watanabe
linkedin: www.linkedin.com/in/yuwatanabe1/
twitter: twitter.com/yuwtennis
Re: Any guideline for building golang connector ?
Posted by Danny McCormick via user <us...@beam.apache.org>.
Hey Yu,
The guidance on that page should generally apply for Go as well, though we
are missing an example transform; I filed
https://github.com/apache/beam/issues/22194 to fix this, but a couple
examples are our textio implementation
<https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/io/textio/textio.go>
and this native streaming example
<https://github.com/apache/beam/blob/master/sdks/go/examples/native_wordcap/nativepubsubio/native.go>
(non-productionized, but it shows a lot of good concepts for handling
streaming sources).
Another option would be to use Java's elasticsearch implementation
<https://github.com/apache/beam/blob/master/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java>
with a cross language transform. If you're thinking of contributing this
back to the main beam code base, I'd probably recommend that approach. In
general, we're currently investing most heavily in cross language
transforms because it's a much lower burden to build/support since it
reuses the main components of the original transform. There are several
examples of wrapped cross-language transforms in
https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/io/xlang. If
you have specific questions about authoring IOs that aren't answered in the
docs/by example, feel free to drop them in this thread as well!
Thanks,
Danny
On Fri, Jul 8, 2022 at 3:51 AM Yu Watanabe <yu...@gmail.com> wrote:
> Hello .
>
> Is there any guideline for building a go sdk connector ?
> I was reviewing the document but I could not find one for golang.
>
> https://beam.apache.org/documentation/io/developing-io-overview/
>
> I was thinking of building one for elasticsearch.
>
> Thanks,
> Yu Watanabe
>
> --
> Yu Watanabe
>
> linkedin: www.linkedin.com/in/yuwatanabe1/
> twitter: twitter.com/yuwtennis
>
>