You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Reza Rokni <re...@google.com> on 2022/03/31 22:47:24 UTC
[RFC][design] Standardizing Beam IO connectors
Hi Everyone,
This is a follow up on Pablo's recent email on a proposal to try and
standardize IO connectors for Beam.
Most Beam connectors have been created by separate individuals, and they've
evolved over time with different features and API choices. This is fine,
but it makes Beam more difficult to learn as a framework, because for every
new IO connector, a user needs to study it and learn before picking it up.
The goal for these doc is to simplify the development, review and usage of
IO connectors for Beam.
We would also like to propose the following standard around documentation
for I/O's, Would you please take a look and add your comments / opinions /
ideas? We'd love everyone's feedback (dev@ and user@ alike!) so that we
can, in the future, strive for connectors that are easier to understand and
develop.
https://s.apache.org/beam-io-api-standard-documentation
Again thanks everyone for your comments on the first document which for
context is found here:
https://s.apache.org/beam-io-api-standard
Regards
Reza
Re: [RFC][design] Standardizing Beam IO connectors
Posted by Pablo Estrada <pa...@google.com>.
Yes, once we can wrap up the docs being discussed, I plan to add the
API+Syntax doc to Beam's site docs. Reza had thought of having Java/Pydoc
templates for new IOs, which I think is excellent.
On Fri, Apr 1, 2022 at 10:47 AM Sachin Agarwal <sa...@google.com> wrote:
> Great doc, and appreciate it. Is the intention that these two Google docs
> end up being a part of the documentation at
> https://beam.apache.org/documentation/io/developing-io-overview/ and
> other places?
>
> On Fri, Apr 1, 2022 at 10:41 AM Reza Rokni <re...@google.com> wrote:
>
>> Hi,
>>
>> Yes, great point with Relational items we are in a little bit of a
>> chicken and egg situation, I think its useful to have a placeholder, but
>> maybe it needs to avoid being specific for now. Do you mind putting a
>> comment into the doc please, so we can keep a history of feedback in a
>> single location? We can then figure out best approach in the doc.
>>
>> Thanx
>>
>> Reza
>>
>> On Fri, Apr 1, 2022 at 10:21 AM Andrew Pilloud <ap...@google.com>
>> wrote:
>>
>>> Thanks for sending this out. I'm just getting around to reviewing the
>>> original doc as well! How do you plan to map the documentation back to the
>>> standards? For example, I see you've defined documentation for a bunch of
>>> relational things (F5) that aren't in the standards doc.
>>>
>>> Andrew
>>>
>>> On Thu, Mar 31, 2022 at 3:47 PM Reza Rokni <re...@google.com> wrote:
>>>
>>>> Hi Everyone,
>>>>
>>>> This is a follow up on Pablo's recent email on a proposal to try and
>>>> standardize IO connectors for Beam.
>>>>
>>>> Most Beam connectors have been created by separate individuals, and
>>>> they've evolved over time with different features and API choices. This is
>>>> fine, but it makes Beam more difficult to learn as a framework, because for
>>>> every new IO connector, a user needs to study it and learn before picking
>>>> it up. The goal for these doc is to simplify the development, review
>>>> and usage of IO connectors for Beam.
>>>>
>>>> We would also like to propose the following standard around
>>>> documentation for I/O's, Would you please take a look and add your comments
>>>> / opinions / ideas? We'd love everyone's feedback (dev@ and user@
>>>> alike!) so that we can, in the future, strive for connectors that are
>>>> easier to understand and develop.
>>>>
>>>> https://s.apache.org/beam-io-api-standard-documentation
>>>>
>>>> Again thanks everyone for your comments on the first document which for
>>>> context is found here:
>>>>
>>>> https://s.apache.org/beam-io-api-standard
>>>>
>>>> Regards
>>>>
>>>> Reza
>>>>
>>>>
>>>>
Re: [RFC][design] Standardizing Beam IO connectors
Posted by Pablo Estrada <pa...@google.com>.
Yes, once we can wrap up the docs being discussed, I plan to add the
API+Syntax doc to Beam's site docs. Reza had thought of having Java/Pydoc
templates for new IOs, which I think is excellent.
On Fri, Apr 1, 2022 at 10:47 AM Sachin Agarwal <sa...@google.com> wrote:
> Great doc, and appreciate it. Is the intention that these two Google docs
> end up being a part of the documentation at
> https://beam.apache.org/documentation/io/developing-io-overview/ and
> other places?
>
> On Fri, Apr 1, 2022 at 10:41 AM Reza Rokni <re...@google.com> wrote:
>
>> Hi,
>>
>> Yes, great point with Relational items we are in a little bit of a
>> chicken and egg situation, I think its useful to have a placeholder, but
>> maybe it needs to avoid being specific for now. Do you mind putting a
>> comment into the doc please, so we can keep a history of feedback in a
>> single location? We can then figure out best approach in the doc.
>>
>> Thanx
>>
>> Reza
>>
>> On Fri, Apr 1, 2022 at 10:21 AM Andrew Pilloud <ap...@google.com>
>> wrote:
>>
>>> Thanks for sending this out. I'm just getting around to reviewing the
>>> original doc as well! How do you plan to map the documentation back to the
>>> standards? For example, I see you've defined documentation for a bunch of
>>> relational things (F5) that aren't in the standards doc.
>>>
>>> Andrew
>>>
>>> On Thu, Mar 31, 2022 at 3:47 PM Reza Rokni <re...@google.com> wrote:
>>>
>>>> Hi Everyone,
>>>>
>>>> This is a follow up on Pablo's recent email on a proposal to try and
>>>> standardize IO connectors for Beam.
>>>>
>>>> Most Beam connectors have been created by separate individuals, and
>>>> they've evolved over time with different features and API choices. This is
>>>> fine, but it makes Beam more difficult to learn as a framework, because for
>>>> every new IO connector, a user needs to study it and learn before picking
>>>> it up. The goal for these doc is to simplify the development, review
>>>> and usage of IO connectors for Beam.
>>>>
>>>> We would also like to propose the following standard around
>>>> documentation for I/O's, Would you please take a look and add your comments
>>>> / opinions / ideas? We'd love everyone's feedback (dev@ and user@
>>>> alike!) so that we can, in the future, strive for connectors that are
>>>> easier to understand and develop.
>>>>
>>>> https://s.apache.org/beam-io-api-standard-documentation
>>>>
>>>> Again thanks everyone for your comments on the first document which for
>>>> context is found here:
>>>>
>>>> https://s.apache.org/beam-io-api-standard
>>>>
>>>> Regards
>>>>
>>>> Reza
>>>>
>>>>
>>>>
Re: [RFC][design] Standardizing Beam IO connectors
Posted by Sachin Agarwal <sa...@google.com>.
Great doc, and appreciate it. Is the intention that these two Google docs
end up being a part of the documentation at
https://beam.apache.org/documentation/io/developing-io-overview/ and other
places?
On Fri, Apr 1, 2022 at 10:41 AM Reza Rokni <re...@google.com> wrote:
> Hi,
>
> Yes, great point with Relational items we are in a little bit of a chicken
> and egg situation, I think its useful to have a placeholder, but maybe it
> needs to avoid being specific for now. Do you mind putting a comment into
> the doc please, so we can keep a history of feedback in a single location?
> We can then figure out best approach in the doc.
>
> Thanx
>
> Reza
>
> On Fri, Apr 1, 2022 at 10:21 AM Andrew Pilloud <ap...@google.com>
> wrote:
>
>> Thanks for sending this out. I'm just getting around to reviewing the
>> original doc as well! How do you plan to map the documentation back to the
>> standards? For example, I see you've defined documentation for a bunch of
>> relational things (F5) that aren't in the standards doc.
>>
>> Andrew
>>
>> On Thu, Mar 31, 2022 at 3:47 PM Reza Rokni <re...@google.com> wrote:
>>
>>> Hi Everyone,
>>>
>>> This is a follow up on Pablo's recent email on a proposal to try and
>>> standardize IO connectors for Beam.
>>>
>>> Most Beam connectors have been created by separate individuals, and
>>> they've evolved over time with different features and API choices. This is
>>> fine, but it makes Beam more difficult to learn as a framework, because for
>>> every new IO connector, a user needs to study it and learn before picking
>>> it up. The goal for these doc is to simplify the development, review
>>> and usage of IO connectors for Beam.
>>>
>>> We would also like to propose the following standard around
>>> documentation for I/O's, Would you please take a look and add your comments
>>> / opinions / ideas? We'd love everyone's feedback (dev@ and user@
>>> alike!) so that we can, in the future, strive for connectors that are
>>> easier to understand and develop.
>>>
>>> https://s.apache.org/beam-io-api-standard-documentation
>>>
>>> Again thanks everyone for your comments on the first document which for
>>> context is found here:
>>>
>>> https://s.apache.org/beam-io-api-standard
>>>
>>> Regards
>>>
>>> Reza
>>>
>>>
>>>
Re: [RFC][design] Standardizing Beam IO connectors
Posted by Reza Rokni <re...@google.com>.
Hi,
Yes, great point with Relational items we are in a little bit of a chicken
and egg situation, I think its useful to have a placeholder, but maybe it
needs to avoid being specific for now. Do you mind putting a comment into
the doc please, so we can keep a history of feedback in a single location?
We can then figure out best approach in the doc.
Thanx
Reza
On Fri, Apr 1, 2022 at 10:21 AM Andrew Pilloud <ap...@google.com> wrote:
> Thanks for sending this out. I'm just getting around to reviewing the
> original doc as well! How do you plan to map the documentation back to the
> standards? For example, I see you've defined documentation for a bunch of
> relational things (F5) that aren't in the standards doc.
>
> Andrew
>
> On Thu, Mar 31, 2022 at 3:47 PM Reza Rokni <re...@google.com> wrote:
>
>> Hi Everyone,
>>
>> This is a follow up on Pablo's recent email on a proposal to try and
>> standardize IO connectors for Beam.
>>
>> Most Beam connectors have been created by separate individuals, and
>> they've evolved over time with different features and API choices. This is
>> fine, but it makes Beam more difficult to learn as a framework, because for
>> every new IO connector, a user needs to study it and learn before picking
>> it up. The goal for these doc is to simplify the development, review
>> and usage of IO connectors for Beam.
>>
>> We would also like to propose the following standard around documentation
>> for I/O's, Would you please take a look and add your comments / opinions /
>> ideas? We'd love everyone's feedback (dev@ and user@ alike!) so that we
>> can, in the future, strive for connectors that are easier to understand and
>> develop.
>>
>> https://s.apache.org/beam-io-api-standard-documentation
>>
>> Again thanks everyone for your comments on the first document which for
>> context is found here:
>>
>> https://s.apache.org/beam-io-api-standard
>>
>> Regards
>>
>> Reza
>>
>>
>>
Re: [RFC][design] Standardizing Beam IO connectors
Posted by Reza Rokni <re...@google.com>.
Hi,
Yes, great point with Relational items we are in a little bit of a chicken
and egg situation, I think its useful to have a placeholder, but maybe it
needs to avoid being specific for now. Do you mind putting a comment into
the doc please, so we can keep a history of feedback in a single location?
We can then figure out best approach in the doc.
Thanx
Reza
On Fri, Apr 1, 2022 at 10:21 AM Andrew Pilloud <ap...@google.com> wrote:
> Thanks for sending this out. I'm just getting around to reviewing the
> original doc as well! How do you plan to map the documentation back to the
> standards? For example, I see you've defined documentation for a bunch of
> relational things (F5) that aren't in the standards doc.
>
> Andrew
>
> On Thu, Mar 31, 2022 at 3:47 PM Reza Rokni <re...@google.com> wrote:
>
>> Hi Everyone,
>>
>> This is a follow up on Pablo's recent email on a proposal to try and
>> standardize IO connectors for Beam.
>>
>> Most Beam connectors have been created by separate individuals, and
>> they've evolved over time with different features and API choices. This is
>> fine, but it makes Beam more difficult to learn as a framework, because for
>> every new IO connector, a user needs to study it and learn before picking
>> it up. The goal for these doc is to simplify the development, review
>> and usage of IO connectors for Beam.
>>
>> We would also like to propose the following standard around documentation
>> for I/O's, Would you please take a look and add your comments / opinions /
>> ideas? We'd love everyone's feedback (dev@ and user@ alike!) so that we
>> can, in the future, strive for connectors that are easier to understand and
>> develop.
>>
>> https://s.apache.org/beam-io-api-standard-documentation
>>
>> Again thanks everyone for your comments on the first document which for
>> context is found here:
>>
>> https://s.apache.org/beam-io-api-standard
>>
>> Regards
>>
>> Reza
>>
>>
>>
Re: [RFC][design] Standardizing Beam IO connectors
Posted by Andrew Pilloud <ap...@google.com>.
Thanks for sending this out. I'm just getting around to reviewing the
original doc as well! How do you plan to map the documentation back to the
standards? For example, I see you've defined documentation for a bunch of
relational things (F5) that aren't in the standards doc.
Andrew
On Thu, Mar 31, 2022 at 3:47 PM Reza Rokni <re...@google.com> wrote:
> Hi Everyone,
>
> This is a follow up on Pablo's recent email on a proposal to try and
> standardize IO connectors for Beam.
>
> Most Beam connectors have been created by separate individuals, and
> they've evolved over time with different features and API choices. This is
> fine, but it makes Beam more difficult to learn as a framework, because for
> every new IO connector, a user needs to study it and learn before picking
> it up. The goal for these doc is to simplify the development, review
> and usage of IO connectors for Beam.
>
> We would also like to propose the following standard around documentation
> for I/O's, Would you please take a look and add your comments / opinions /
> ideas? We'd love everyone's feedback (dev@ and user@ alike!) so that we
> can, in the future, strive for connectors that are easier to understand and
> develop.
>
> https://s.apache.org/beam-io-api-standard-documentation
>
> Again thanks everyone for your comments on the first document which for
> context is found here:
>
> https://s.apache.org/beam-io-api-standard
>
> Regards
>
> Reza
>
>
>
Re: [RFC][design] Standardizing Beam IO connectors
Posted by Andrew Pilloud <ap...@google.com>.
Thanks for sending this out. I'm just getting around to reviewing the
original doc as well! How do you plan to map the documentation back to the
standards? For example, I see you've defined documentation for a bunch of
relational things (F5) that aren't in the standards doc.
Andrew
On Thu, Mar 31, 2022 at 3:47 PM Reza Rokni <re...@google.com> wrote:
> Hi Everyone,
>
> This is a follow up on Pablo's recent email on a proposal to try and
> standardize IO connectors for Beam.
>
> Most Beam connectors have been created by separate individuals, and
> they've evolved over time with different features and API choices. This is
> fine, but it makes Beam more difficult to learn as a framework, because for
> every new IO connector, a user needs to study it and learn before picking
> it up. The goal for these doc is to simplify the development, review
> and usage of IO connectors for Beam.
>
> We would also like to propose the following standard around documentation
> for I/O's, Would you please take a look and add your comments / opinions /
> ideas? We'd love everyone's feedback (dev@ and user@ alike!) so that we
> can, in the future, strive for connectors that are easier to understand and
> develop.
>
> https://s.apache.org/beam-io-api-standard-documentation
>
> Again thanks everyone for your comments on the first document which for
> context is found here:
>
> https://s.apache.org/beam-io-api-standard
>
> Regards
>
> Reza
>
>
>