You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Pablo Estrada <pa...@google.com> on 2022/01/24 16:00:00 UTC
[RFC][design] Standardizing Beam IO connectors
Hi all!
A few of us have started putting together a proposal to try and standardize
IO connectors for Beam. What does this mean?
Most Beam connectors have been created by separate individuals, and they've
evolved over time with different features and API choices. This is fine,
but it makes Beam more difficult to learn as a framework, because for every
new IO connector, a user needs to study it and learn before picking it up.
The goal for this doc is to simplify the development, review and *usage* of
IO connectors for Beam. Would you please take a look and add your comments
/ opinions / ideas? We'd love everyone's feedback (dev@ and user@ alike!)
so that we can, in the future, strive for connectors that are easier to
understand and develop.
*PTAL: *https://s.apache.org/beam-io-api-standard
Best
-P.
Re: [RFC][design] Standardizing Beam IO connectors
Posted by Pablo Estrada <pa...@google.com>.
Hi all,
thanks for the great comments. I am finding them very useful, so I think
we'll make a few more modifications to the doc based on your comments.
I am thinking that we can distill some of this doc into a sort of checklist
to put up on the website. LMK what you think.
Best
-P.
On Tue, Jan 25, 2022 at 8:44 AM Alexey Romanenko <ar...@gmail.com>
wrote:
> Thanks Pablo and others for creating such doc, great initiative! I left my
> comments.
>
> After all corrections and discussions, I suppose, it should be added to
> Beam Contribution Guide.
>
> —
> Alexey
>
> On 24 Jan 2022, at 17:00, Pablo Estrada <pa...@google.com> wrote:
>
> Hi all!
>
> A few of us have started putting together a proposal to try and
> standardize IO connectors for Beam. What does this mean?
>
> Most Beam connectors have been created by separate individuals, and
> they've evolved over time with different features and API choices. This is
> fine, but it makes Beam more difficult to learn as a framework, because for
> every new IO connector, a user needs to study it and learn before picking
> it up.
>
> The goal for this doc is to simplify the development, review and *usage* of
> IO connectors for Beam. Would you please take a look and add your comments
> / opinions / ideas? We'd love everyone's feedback (dev@ and user@ alike!)
> so that we can, in the future, strive for connectors that are easier to
> understand and develop.
>
> *PTAL: *https://s.apache.org/beam-io-api-standard
>
> Best
> -P.
>
>
>
Re: [RFC][design] Standardizing Beam IO connectors
Posted by Pablo Estrada <pa...@google.com>.
Hi all,
thanks for the great comments. I am finding them very useful, so I think
we'll make a few more modifications to the doc based on your comments.
I am thinking that we can distill some of this doc into a sort of checklist
to put up on the website. LMK what you think.
Best
-P.
On Tue, Jan 25, 2022 at 8:44 AM Alexey Romanenko <ar...@gmail.com>
wrote:
> Thanks Pablo and others for creating such doc, great initiative! I left my
> comments.
>
> After all corrections and discussions, I suppose, it should be added to
> Beam Contribution Guide.
>
> —
> Alexey
>
> On 24 Jan 2022, at 17:00, Pablo Estrada <pa...@google.com> wrote:
>
> Hi all!
>
> A few of us have started putting together a proposal to try and
> standardize IO connectors for Beam. What does this mean?
>
> Most Beam connectors have been created by separate individuals, and
> they've evolved over time with different features and API choices. This is
> fine, but it makes Beam more difficult to learn as a framework, because for
> every new IO connector, a user needs to study it and learn before picking
> it up.
>
> The goal for this doc is to simplify the development, review and *usage* of
> IO connectors for Beam. Would you please take a look and add your comments
> / opinions / ideas? We'd love everyone's feedback (dev@ and user@ alike!)
> so that we can, in the future, strive for connectors that are easier to
> understand and develop.
>
> *PTAL: *https://s.apache.org/beam-io-api-standard
>
> Best
> -P.
>
>
>
Re: [RFC][design] Standardizing Beam IO connectors
Posted by Alexey Romanenko <ar...@gmail.com>.
Thanks Pablo and others for creating such doc, great initiative! I left my comments.
After all corrections and discussions, I suppose, it should be added to Beam Contribution Guide.
—
Alexey
> On 24 Jan 2022, at 17:00, Pablo Estrada <pa...@google.com> wrote:
>
> Hi all!
>
> A few of us have started putting together a proposal to try and standardize IO connectors for Beam. What does this mean?
>
> Most Beam connectors have been created by separate individuals, and they've evolved over time with different features and API choices. This is fine, but it makes Beam more difficult to learn as a framework, because for every new IO connector, a user needs to study it and learn before picking it up.
>
> The goal for this doc is to simplify the development, review and usage of IO connectors for Beam. Would you please take a look and add your comments / opinions / ideas? We'd love everyone's feedback (dev@ and user@ alike!) so that we can, in the future, strive for connectors that are easier to understand and develop.
>
> PTAL: https://s.apache.org/beam-io-api-standard <https://s.apache.org/beam-io-api-standard>
>
> Best
> -P.
Re: [RFC][design] Standardizing Beam IO connectors
Posted by Alexey Romanenko <ar...@gmail.com>.
Thanks Pablo and others for creating such doc, great initiative! I left my comments.
After all corrections and discussions, I suppose, it should be added to Beam Contribution Guide.
—
Alexey
> On 24 Jan 2022, at 17:00, Pablo Estrada <pa...@google.com> wrote:
>
> Hi all!
>
> A few of us have started putting together a proposal to try and standardize IO connectors for Beam. What does this mean?
>
> Most Beam connectors have been created by separate individuals, and they've evolved over time with different features and API choices. This is fine, but it makes Beam more difficult to learn as a framework, because for every new IO connector, a user needs to study it and learn before picking it up.
>
> The goal for this doc is to simplify the development, review and usage of IO connectors for Beam. Would you please take a look and add your comments / opinions / ideas? We'd love everyone's feedback (dev@ and user@ alike!) so that we can, in the future, strive for connectors that are easier to understand and develop.
>
> PTAL: https://s.apache.org/beam-io-api-standard <https://s.apache.org/beam-io-api-standard>
>
> Best
> -P.