Go IOs ?

[Austin Bennett <wh...@gmail.com>]

Hi All,

https://beam.apache.org/documentation/io/built-in/

Only includes Ios for Java and Python.  I have heard talk of a Go SDK :-)
 Are there working IOs?  If yes, should they also be on that page?

Thanks,
Austin

Re: Go IOs ?

[Chamikara Jayalath <ch...@google.com>]

On Mon, Oct 28, 2019 at 9:15 AM Robert Burke <ro...@frantil.com> wrote:

> There are IOs that work [1], but they haven't been vetted for production
> use (performance, overhead etc) just yet. As such, I don't recommend
> putting them on the site at this time. Of course, folks a free to discuss
> and disagree with me on that :).
>

Makes sense.


>
> The Go SDK doesn't yet have a way of scaling transforms without putting in
> a reshards/GBK, which single threads reads and slows things down
> dramatically. The portable API to do so is usually called Splittable DoFn,
> and is presently being worked on for the Go SDK [2].
>

I think scaling through re-shards should be enough for many cases. We have
some transforms in Java/Python, that depend on reshards for scaling and
successfully used by my folks in production. For example,
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java#L224
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java#L424

It would be good if we can create JIRAs to capture work needed to make
existing Go IO connectors production ready.
For example,
(1) Add reshards to parallelize reading.
(2) Add performance tests to detect anomalies

Thanks,
Cham



> A slightly faster path to performant IOs is to leverage the Cross Language
> Transform work so the Go SDK can use the existing Java and Python IOs. IIRC
> the trick would be to integrate the client side work to communicate with
> the expansion server that Python uses to make that work.
>

+1

>
> [1] https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/io
> [2] https://s.apache.org/beam-go-sdf
>
>
> On Sun, Oct 27, 2019, 10:10 AM Austin Bennett <wh...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> https://beam.apache.org/documentation/io/built-in/
>>
>> Only includes Ios for Java and Python.  I have heard talk of a Go SDK :-)
>>  Are there working IOs?  If yes, should they also be on that page?
>>
>> Thanks,
>> Austin
>>
>>
>>

Re: Go IOs ?

[Robert Burke <ro...@frantil.com>]

There are IOs that work [1], but they haven't been vetted for production
use (performance, overhead etc) just yet. As such, I don't recommend
putting them on the site at this time. Of course, folks a free to discuss
and disagree with me on that :).

The Go SDK doesn't yet have a way of scaling transforms without putting in
a reshards/GBK, which single threads reads and slows things down
dramatically. The portable API to do so is usually called Splittable DoFn,
and is presently being worked on for the Go SDK [2].

A slightly faster path to performant IOs is to leverage the Cross Language
Transform work so the Go SDK can use the existing Java and Python IOs. IIRC
the trick would be to integrate the client side work to communicate with
the expansion server that Python uses to make that work.

[1] https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/io
[2] https://s.apache.org/beam-go-sdf


On Sun, Oct 27, 2019, 10:10 AM Austin Bennett <wh...@gmail.com>
wrote:

> Hi All,
>
> https://beam.apache.org/documentation/io/built-in/
>
> Only includes Ios for Java and Python.  I have heard talk of a Go SDK :-)
>  Are there working IOs?  If yes, should they also be on that page?
>
> Thanks,
> Austin
>
>
>

Re: Go IOs ?

[Chamikara Jayalath <ch...@google.com>]

Seems like we have several file-based IO connectors and datastore here:
https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/io
We should definitely add these to the list.

I guess bulk of IOs will be supported through cross-language transforms
framework when we have that for Go SDK.

Thanks,
Cham

On Sun, Oct 27, 2019 at 10:10 AM Austin Bennett <wh...@gmail.com>
wrote:

> Hi All,
>
> https://beam.apache.org/documentation/io/built-in/
>
> Only includes Ios for Java and Python.  I have heard talk of a Go SDK :-)
>  Are there working IOs?  If yes, should they also be on that page?
>
> Thanks,
> Austin
>
>
>