SBE Beam Schema

[Zachary Houfek <zh...@google.com>]

Hi, everyone,

This is a follow-up to my original proposal for adding SBE message support
in Beam. A couple things came out in that review:

1. A coder-focused design for working with SBE types directly can cause a
lot of confusion.
2. Since the primary use case will be IO support, it would be best to focus
on `PayloadSerializer` support. In other words, Beam schemas should be the
primary focus, not future work.

Based on that, I wrote up another doc for how to map SBE schemas to Beam
schemas. I tried looking over some existing schemas (mostly proto ones) to
make sure I wasn't doing anything  too weird, but I would love some
feedback:

 SBE Schema in Beam
<https://docs.google.com/document/d/1q2KyBxPQ6V6Gfm50Z74YWofATgVauYu066F0t1JPxRU/edit?usp=drive_web>

Thanks,
Zach

-- 

Zachary Houfek

Software Engineer

DataPLS PLAT

zhoufek@google.com

Re: SBE Beam Schema

[Kenneth Knowles <ke...@apache.org>]

Thanks for the super clear summary. Makes sense!

On Tue, Sep 28, 2021 at 9:57 AM Zachary Houfek <zh...@google.com> wrote:

> Hi, everyone,
>
> This is a follow-up to my original proposal for adding SBE message support
> in Beam. A couple things came out in that review:
>
> 1. A coder-focused design for working with SBE types directly can cause a
> lot of confusion.
> 2. Since the primary use case will be IO support, it would be best to
> focus on `PayloadSerializer` support. In other words, Beam schemas should
> be the primary focus, not future work.
>
> Based on that, I wrote up another doc for how to map SBE schemas to Beam
> schemas. I tried looking over some existing schemas (mostly proto ones) to
> make sure I wasn't doing anything  too weird, but I would love some
> feedback:
>
>  SBE Schema in Beam
> <https://docs.google.com/document/d/1q2KyBxPQ6V6Gfm50Z74YWofATgVauYu066F0t1JPxRU/edit?usp=drive_web>
>
> Thanks,
> Zach
>
> --
>
> Zachary Houfek
>
> Software Engineer
>
> DataPLS PLAT
>
> zhoufek@google.com
>