You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Chamikara Jayalath <ch...@google.com> on 2018/10/26 02:39:40 UTC

Roadmap section on IO related features

(Forwarding to dev list from the PR on roadmap.)

Given that we now have a main roadmap Webpage with some content, we were
wondering if there should be a section on IO related efforts. We already
have following Webpage on current and upcoming IO transforms.
https://beam.apache.org/documentation/io/built-in/

IO roadmap page can be an expanded version of above where we can include
following among other things.
(1) More details about some of the upcoming IO transforms that we expect to
be very popular.
(2) Major IO related efforts such as cross-platform IO and SDF (to
complement what will be available in portability roadmap on these features).
(3) Other IO related efforts (for example, documentation, performance
testing).

I think we have three options.

(1) Add a top level IO roadmap.
(2) Add IO roadmap sub-sections under each SDK instead of at top level.
(3) We don't need a IO roadmap since we already have
https://beam.apache.org/documentation/io/built-in/

WDYT ?

Thanks,
Cham


---------- Forwarded message ---------
From: Chamikara Jayalath <no...@github.com>
Date: Thu, Oct 25, 2018 at 7:07 PM
Subject: Re: [apache/beam] [Website] Add roadmap at top level (#6718)
To: apache/beam <be...@noreply.github.com>
Cc: Chamikara Jayalath <ch...@google.com>, Your activity <
your_activity@noreply.github.com>


Ok. Makes sense. Kenn and others, WDYT ? We can start writing down a
roadmap for IO if there's no objection. It can include more details about
some of the proposed IO mentioned in
https://beam.apache.org/documentation/io/built-in/ as well as information
on upcoming major IO related efforts such as cross-language IO support and
SDF (in addition to what will be available in portability roadmap for these
features).

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<https://github.com/apache/beam/pull/6718#issuecomment-433262537>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AKUZEUcsbL6bhjOfeRiNZmqtab3PDWBjks5uom5KgaJpZM4Xj9s1>
.

Re: Roadmap section on IO related features

Posted by Chamikara Jayalath <ch...@google.com>.
I created https://github.com/apache/beam/pull/7003 that adds a first
version.

I included planned connectors mentioned in [1] that currently has an owner
assigned to this section. Did not include connectors that have already been
completed and connectors for which corresponding JIRAs do not have an owner
assigned.

Thanks,
Cham

[1] https://beam.apache.org/documentation/io/built-in/

On Fri, Oct 26, 2018 at 7:52 AM Chamikara Jayalath <ch...@google.com>
wrote:

> +1 for using the term connectors.
>
> JB, thanks for agreeing to add content to this section.
>
> - Cham
>
> On Thu, Oct 25, 2018 at 8:48 PM Ahmet Altay <al...@google.com> wrote:
>
>> I like this idea and also the first option. I agree with making it clear
>> that things about IO are language specific.
>>
>> And +1 to calling it connectors or anything else that will resonate with
>> end users.
>>
>> On Thu, Oct 25, 2018 at 7:59 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>
>>> Agree, I think connector is a more meaning name for users.
>>>
>>> IO is more the Beam "internal" wording.
>>>
>>> I will update this section as I have new connectors ( :) ) on the fly.
>>>
>>> Regards
>>> JB
>>>
>>> On 26/10/2018 04:49, Kenneth Knowles wrote:
>>> > My $0.02
>>> >
>>> > "IO" has an established meaning in Beam dev argot but I think on the
>>> web
>>> > page I would use the word "connector" or something more universal.
>>> >
>>> > On Thu, Oct 25, 2018 at 7:39 PM Chamikara Jayalath <
>>> chamikara@google.com
>>> > <ma...@google.com>> wrote:
>>> >
>>> >
>>> >     (1) Add a top level IO roadmap.
>>> >
>>> >
>>> > I like this, but it is important on the roadmap to be very clear about
>>> > language / SDK.
>>> >
>>> >
>>> >     (2) Add IO roadmap sub-sections under each SDK instead of at top
>>> level.
>>> >
>>> >
>>> > This seems OK to me too. If you are a user and you have some data you
>>> > just want to see if it is going to be accessible to you soon. You
>>> > probably already committed to a language.
>>> >
>>> >
>>> >     (3) We don't need a IO roadmap since we already have
>>> >     https://beam.apache.org/documentation/io/built-in/
>>> >
>>> >
>>> > I think the roadmap / in-progress part should move to the new Roadmap
>>> > and/or the wiki.
>>> >
>>> > Kenn
>>> >
>>> >
>>> >
>>> >     WDYT ?
>>> >
>>> >     Thanks,
>>> >     Cham
>>> >
>>> >
>>> >     ---------- Forwarded message ---------
>>> >     From: *Chamikara Jayalath* <notifications@github.com
>>> >     <ma...@github.com>>
>>> >     Date: Thu, Oct 25, 2018 at 7:07 PM
>>> >     Subject: Re: [apache/beam] [Website] Add roadmap at top level
>>> (#6718)
>>> >     To: apache/beam <beam@noreply.github.com
>>> >     <ma...@noreply.github.com>>
>>> >     Cc: Chamikara Jayalath <chamikara@google.com
>>> >     <ma...@google.com>>, Your activity
>>> >     <your_activity@noreply.github.com
>>> >     <ma...@noreply.github.com>>
>>> >
>>> >
>>> >     Ok. Makes sense. Kenn and others, WDYT ? We can start writing down
>>> a
>>> >     roadmap for IO if there's no objection. It can include more details
>>> >     about some of the proposed IO mentioned in
>>> >     https://beam.apache.org/documentation/io/built-in/ as well as
>>> >     information on upcoming major IO related efforts such as
>>> >     cross-language IO support and SDF (in addition to what will be
>>> >     available in portability roadmap for these features).
>>> >
>>> >     —
>>> >     You are receiving this because you are subscribed to this thread.
>>> >     Reply to this email directly, view it on GitHub
>>> >     <https://github.com/apache/beam/pull/6718#issuecomment-433262537>,
>>> >     or mute the thread
>>> >     <
>>> https://github.com/notifications/unsubscribe-auth/AKUZEUcsbL6bhjOfeRiNZmqtab3PDWBjks5uom5KgaJpZM4Xj9s1
>>> >.
>>> >
>>>
>>
>>

Re: Roadmap section on IO related features

Posted by Chamikara Jayalath <ch...@google.com>.
+1 for using the term connectors.

JB, thanks for agreeing to add content to this section.

- Cham

On Thu, Oct 25, 2018 at 8:48 PM Ahmet Altay <al...@google.com> wrote:

> I like this idea and also the first option. I agree with making it clear
> that things about IO are language specific.
>
> And +1 to calling it connectors or anything else that will resonate with
> end users.
>
> On Thu, Oct 25, 2018 at 7:59 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Agree, I think connector is a more meaning name for users.
>>
>> IO is more the Beam "internal" wording.
>>
>> I will update this section as I have new connectors ( :) ) on the fly.
>>
>> Regards
>> JB
>>
>> On 26/10/2018 04:49, Kenneth Knowles wrote:
>> > My $0.02
>> >
>> > "IO" has an established meaning in Beam dev argot but I think on the web
>> > page I would use the word "connector" or something more universal.
>> >
>> > On Thu, Oct 25, 2018 at 7:39 PM Chamikara Jayalath <
>> chamikara@google.com
>> > <ma...@google.com>> wrote:
>> >
>> >
>> >     (1) Add a top level IO roadmap.
>> >
>> >
>> > I like this, but it is important on the roadmap to be very clear about
>> > language / SDK.
>> >
>> >
>> >     (2) Add IO roadmap sub-sections under each SDK instead of at top
>> level.
>> >
>> >
>> > This seems OK to me too. If you are a user and you have some data you
>> > just want to see if it is going to be accessible to you soon. You
>> > probably already committed to a language.
>> >
>> >
>> >     (3) We don't need a IO roadmap since we already have
>> >     https://beam.apache.org/documentation/io/built-in/
>> >
>> >
>> > I think the roadmap / in-progress part should move to the new Roadmap
>> > and/or the wiki.
>> >
>> > Kenn
>> >
>> >
>> >
>> >     WDYT ?
>> >
>> >     Thanks,
>> >     Cham
>> >
>> >
>> >     ---------- Forwarded message ---------
>> >     From: *Chamikara Jayalath* <notifications@github.com
>> >     <ma...@github.com>>
>> >     Date: Thu, Oct 25, 2018 at 7:07 PM
>> >     Subject: Re: [apache/beam] [Website] Add roadmap at top level
>> (#6718)
>> >     To: apache/beam <beam@noreply.github.com
>> >     <ma...@noreply.github.com>>
>> >     Cc: Chamikara Jayalath <chamikara@google.com
>> >     <ma...@google.com>>, Your activity
>> >     <your_activity@noreply.github.com
>> >     <ma...@noreply.github.com>>
>> >
>> >
>> >     Ok. Makes sense. Kenn and others, WDYT ? We can start writing down a
>> >     roadmap for IO if there's no objection. It can include more details
>> >     about some of the proposed IO mentioned in
>> >     https://beam.apache.org/documentation/io/built-in/ as well as
>> >     information on upcoming major IO related efforts such as
>> >     cross-language IO support and SDF (in addition to what will be
>> >     available in portability roadmap for these features).
>> >
>> >     —
>> >     You are receiving this because you are subscribed to this thread.
>> >     Reply to this email directly, view it on GitHub
>> >     <https://github.com/apache/beam/pull/6718#issuecomment-433262537>,
>> >     or mute the thread
>> >     <
>> https://github.com/notifications/unsubscribe-auth/AKUZEUcsbL6bhjOfeRiNZmqtab3PDWBjks5uom5KgaJpZM4Xj9s1
>> >.
>> >
>>
>
>

Re: Roadmap section on IO related features

Posted by Ahmet Altay <al...@google.com>.
I like this idea and also the first option. I agree with making it clear
that things about IO are language specific.

And +1 to calling it connectors or anything else that will resonate with
end users.

On Thu, Oct 25, 2018 at 7:59 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:

> Agree, I think connector is a more meaning name for users.
>
> IO is more the Beam "internal" wording.
>
> I will update this section as I have new connectors ( :) ) on the fly.
>
> Regards
> JB
>
> On 26/10/2018 04:49, Kenneth Knowles wrote:
> > My $0.02
> >
> > "IO" has an established meaning in Beam dev argot but I think on the web
> > page I would use the word "connector" or something more universal.
> >
> > On Thu, Oct 25, 2018 at 7:39 PM Chamikara Jayalath <chamikara@google.com
> > <ma...@google.com>> wrote:
> >
> >
> >     (1) Add a top level IO roadmap.
> >
> >
> > I like this, but it is important on the roadmap to be very clear about
> > language / SDK.
> >
> >
> >     (2) Add IO roadmap sub-sections under each SDK instead of at top
> level.
> >
> >
> > This seems OK to me too. If you are a user and you have some data you
> > just want to see if it is going to be accessible to you soon. You
> > probably already committed to a language.
> >
> >
> >     (3) We don't need a IO roadmap since we already have
> >     https://beam.apache.org/documentation/io/built-in/
> >
> >
> > I think the roadmap / in-progress part should move to the new Roadmap
> > and/or the wiki.
> >
> > Kenn
> >
> >
> >
> >     WDYT ?
> >
> >     Thanks,
> >     Cham
> >
> >
> >     ---------- Forwarded message ---------
> >     From: *Chamikara Jayalath* <notifications@github.com
> >     <ma...@github.com>>
> >     Date: Thu, Oct 25, 2018 at 7:07 PM
> >     Subject: Re: [apache/beam] [Website] Add roadmap at top level (#6718)
> >     To: apache/beam <beam@noreply.github.com
> >     <ma...@noreply.github.com>>
> >     Cc: Chamikara Jayalath <chamikara@google.com
> >     <ma...@google.com>>, Your activity
> >     <your_activity@noreply.github.com
> >     <ma...@noreply.github.com>>
> >
> >
> >     Ok. Makes sense. Kenn and others, WDYT ? We can start writing down a
> >     roadmap for IO if there's no objection. It can include more details
> >     about some of the proposed IO mentioned in
> >     https://beam.apache.org/documentation/io/built-in/ as well as
> >     information on upcoming major IO related efforts such as
> >     cross-language IO support and SDF (in addition to what will be
> >     available in portability roadmap for these features).
> >
> >     —
> >     You are receiving this because you are subscribed to this thread.
> >     Reply to this email directly, view it on GitHub
> >     <https://github.com/apache/beam/pull/6718#issuecomment-433262537>,
> >     or mute the thread
> >     <https://github.com/notifications/unsubscribe-auth/
> AKUZEUcsbL6bhjOfeRiNZmqtab3PDWBjks5uom5KgaJpZM4Xj9s1>.
> >
>

Re: Roadmap section on IO related features

Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Agree, I think connector is a more meaning name for users.

IO is more the Beam "internal" wording.

I will update this section as I have new connectors ( :) ) on the fly.

Regards
JB

On 26/10/2018 04:49, Kenneth Knowles wrote:
> My $0.02
> 
> "IO" has an established meaning in Beam dev argot but I think on the web
> page I would use the word "connector" or something more universal.
> 
> On Thu, Oct 25, 2018 at 7:39 PM Chamikara Jayalath <chamikara@google.com
> <ma...@google.com>> wrote:
> 
> 
>     (1) Add a top level IO roadmap.
> 
> 
> I like this, but it is important on the roadmap to be very clear about
> language / SDK.
>  
> 
>     (2) Add IO roadmap sub-sections under each SDK instead of at top level.
> 
> 
> This seems OK to me too. If you are a user and you have some data you
> just want to see if it is going to be accessible to you soon. You
> probably already committed to a language.
>  
> 
>     (3) We don't need a IO roadmap since we already have
>     https://beam.apache.org/documentation/io/built-in/
> 
> 
> I think the roadmap / in-progress part should move to the new Roadmap
> and/or the wiki.
>  
> Kenn
> 
>  
> 
>     WDYT ?
> 
>     Thanks,
>     Cham
> 
> 
>     ---------- Forwarded message ---------
>     From: *Chamikara Jayalath* <notifications@github.com
>     <ma...@github.com>>
>     Date: Thu, Oct 25, 2018 at 7:07 PM
>     Subject: Re: [apache/beam] [Website] Add roadmap at top level (#6718)
>     To: apache/beam <beam@noreply.github.com
>     <ma...@noreply.github.com>>
>     Cc: Chamikara Jayalath <chamikara@google.com
>     <ma...@google.com>>, Your activity
>     <your_activity@noreply.github.com
>     <ma...@noreply.github.com>>
> 
> 
>     Ok. Makes sense. Kenn and others, WDYT ? We can start writing down a
>     roadmap for IO if there's no objection. It can include more details
>     about some of the proposed IO mentioned in
>     https://beam.apache.org/documentation/io/built-in/ as well as
>     information on upcoming major IO related efforts such as
>     cross-language IO support and SDF (in addition to what will be
>     available in portability roadmap for these features).
> 
>     —
>     You are receiving this because you are subscribed to this thread.
>     Reply to this email directly, view it on GitHub
>     <https://github.com/apache/beam/pull/6718#issuecomment-433262537>,
>     or mute the thread
>     <https://github.com/notifications/unsubscribe-auth/AKUZEUcsbL6bhjOfeRiNZmqtab3PDWBjks5uom5KgaJpZM4Xj9s1>.
> 

Re: Roadmap section on IO related features

Posted by Kenneth Knowles <ke...@apache.org>.
My $0.02

"IO" has an established meaning in Beam dev argot but I think on the web
page I would use the word "connector" or something more universal.

On Thu, Oct 25, 2018 at 7:39 PM Chamikara Jayalath <ch...@google.com>
wrote:

>
> (1) Add a top level IO roadmap.
>

I like this, but it is important on the roadmap to be very clear about
language / SDK.


> (2) Add IO roadmap sub-sections under each SDK instead of at top level.
>

This seems OK to me too. If you are a user and you have some data you just
want to see if it is going to be accessible to you soon. You probably
already committed to a language.


> (3) We don't need a IO roadmap since we already have
> https://beam.apache.org/documentation/io/built-in/
>

I think the roadmap / in-progress part should move to the new Roadmap
and/or the wiki.

Kenn



> WDYT ?
>
> Thanks,
> Cham
>
>
> ---------- Forwarded message ---------
> From: Chamikara Jayalath <no...@github.com>
> Date: Thu, Oct 25, 2018 at 7:07 PM
> Subject: Re: [apache/beam] [Website] Add roadmap at top level (#6718)
> To: apache/beam <be...@noreply.github.com>
> Cc: Chamikara Jayalath <ch...@google.com>, Your activity <
> your_activity@noreply.github.com>
>
>
> Ok. Makes sense. Kenn and others, WDYT ? We can start writing down a
> roadmap for IO if there's no objection. It can include more details about
> some of the proposed IO mentioned in
> https://beam.apache.org/documentation/io/built-in/ as well as information
> on upcoming major IO related efforts such as cross-language IO support and
> SDF (in addition to what will be available in portability roadmap for these
> features).
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/pull/6718#issuecomment-433262537>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AKUZEUcsbL6bhjOfeRiNZmqtab3PDWBjks5uom5KgaJpZM4Xj9s1>
> .
>