design docs that get deleted, etc

[Kenneth Knowles <ke...@apache.org>]

Hi all,

Yesterday someone asked me about the design doc linked from
https://github.com/apache/beam/issues/18297 because it is now a 404.

There are plenty of reasons a Google Doc might no longer be accessible.
They exist outside the project's control. This is part of why ASF projects
emphasize having discussions on the dev@ list and often put all their
designs directly onto some ASF-hosted infrastructure, such as a Wiki
(Example:
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals).
In the early days we had a project-owned shared folder but it fell into
disuse.

In my opinion, Google Docs are still the best place for design docs to get
feedback and be revised, but the lack of durability is a downside to stay
aware of. I've also gotten lots of complaints of lack of discoverability
and lack of systematization of design docs, neither of which would be
addressed by a shared folder.

I don't have a proposal or suggestion. I don't think this is super urgent,
certainly not my personal highest priority, but I thought I'd just share
this as food for thought.

Kenn

Re: design docs that get deleted, etc

[Niel Markwick via dev <de...@beam.apache.org>]

> google docs -> markdown converters

https://workspace.google.com/marketplace/app/docs_to_markdown/700168918607

is a pretty good gdocs addon that converts to markdown

There are some issues which may need some manual cleanup, but provided the
Gdoc is relatively 'simple', it works well.




-- 
<https://cloud.google.com>
* •  **Niel Markwick*
* •  *Cloud Solutions Architect <https://cloud.google.com/docs/tutorials>
* •  *Google Belgium
* •  *nielm@google.com
* •  *+32 2 894 6771

Google Belgium NV/SA, Steenweg op Etterbeek 180, 1040 Brussel, Belgie.
RPR: 0878.065.378

If you have received this communication by mistake, please don't forward it
to anyone else (it may contain confidential or privileged information),
please erase all copies of it, including all attachments, and please let
the sender know it went to the wrong person. Thanks


On Mon, 3 Jun 2024 at 16:09, Kenneth Knowles <ke...@apache.org> wrote:

> I like the idea of a "final" step of exporting the Google Doc to markdown
> and checking it in, if there is any tool that does an OK job (my experience
> with such tools over my lifetime has been underwhelming, but I'm always
> open to being pleasantly surprised). PDF would be a close second.
>
> Kenn
>
> On Thu, May 30, 2024 at 5:41 AM Jan Lukavský <je...@seznam.cz> wrote:
>
>> Yes, I think that the discoverability and durability would be part of
>> some "process". There could be "informal" part, where the design can
>> take place virtually anywhere and the document can be owned by anyone
>> (as we do today), then it would be copied to a durable store (asf wiki?)
>> and a [DISCUSS] thread could be started on ML, followed by [VOTE] (if
>> needed?) after that, then the document would be marked as accepted and
>> will be turned into issues. I believe this is how the FLIPs and KIPs are
>> processed.
>>
>>   Jan
>>
>> On 5/29/24 17:56, Robert Bradshaw via dev wrote:
>> > While I'm not opposed to having a more formal process for proposals to
>> > go from idea to consensus to implementation, I'm not sure how much
>> > this would solve the primary issues we face (discoverability and
>> > durability). But maybe that could be built into the process? At the
>> > very least we could have an "index" which would give identifiers (and
>> > hopefully good titles) to all the proposals, and maybe have an offline
>> > process to snapshot such docs (even just periodically pulling the
>> > content to a repo like I do with
>> > https://github.com/cython/cython-issues ). I have yet to find a medium
>> > (not even wikis) that facilitates conversation/collaborative editing
>> > to the extent that docs does, but I agree with the downside that
>> > ownership by random individuals can pose a problem.
>> >
>> > On Wed, May 29, 2024 at 7:07 AM Jan Lukavský <je...@seznam.cz> wrote:
>> >> Hi,
>> >>
>> >> regarding changing the way we document past (and more importantly
>> >> future) changes, I've always been a big fan of the FLIP analogy [1]. I
>> >> would love if we could make this work for Beam as well, while
>> preserving
>> >> the 'informal' part that I believe all of us want to keep. On the other
>> >> hand, this could make the design decisions more searchable, transparent
>> >> and get more people involved in the process. Having design documents
>> >> durable is of course a highly important part of it.
>> >>
>> >>    Jan
>> >>
>> >> [1] https://lists.apache.org/thread/whfy3706w2d0q6rdk4kwyrzvhfd4b5kg
>> >>
>> >> On 5/29/24 15:04, Kenneth Knowles wrote:
>> >>> Hi all,
>> >>>
>> >>> Yesterday someone asked me about the design doc linked from
>> >>> https://github.com/apache/beam/issues/18297 because it is now a 404.
>> >>>
>> >>> There are plenty of reasons a Google Doc might no longer be
>> >>> accessible. They exist outside the project's control. This is part of
>> >>> why ASF projects emphasize having discussions on the dev@ list and
>> >>> often put all their designs directly onto some ASF-hosted
>> >>> infrastructure, such as a Wiki (Example:
>> >>>
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>> ).
>> >>> In the early days we had a project-owned shared folder but it fell
>> >>> into disuse.
>> >>>
>> >>> In my opinion, Google Docs are still the best place for design docs to
>> >>> get feedback and be revised, but the lack of durability is a downside
>> >>> to stay aware of. I've also gotten lots of complaints of lack of
>> >>> discoverability and lack of systematization of design docs, neither of
>> >>> which would be addressed by a shared folder.
>> >>>
>> >>> I don't have a proposal or suggestion. I don't think this is super
>> >>> urgent, certainly not my personal highest priority, but I thought I'd
>> >>> just share this as food for thought.
>> >>>
>> >>> Kenn
>>
>

Re: design docs that get deleted, etc

[Kenneth Knowles <ke...@apache.org>]

I like the idea of a "final" step of exporting the Google Doc to markdown
and checking it in, if there is any tool that does an OK job (my experience
with such tools over my lifetime has been underwhelming, but I'm always
open to being pleasantly surprised). PDF would be a close second.

Kenn

On Thu, May 30, 2024 at 5:41 AM Jan Lukavský <je...@seznam.cz> wrote:

> Yes, I think that the discoverability and durability would be part of
> some "process". There could be "informal" part, where the design can
> take place virtually anywhere and the document can be owned by anyone
> (as we do today), then it would be copied to a durable store (asf wiki?)
> and a [DISCUSS] thread could be started on ML, followed by [VOTE] (if
> needed?) after that, then the document would be marked as accepted and
> will be turned into issues. I believe this is how the FLIPs and KIPs are
> processed.
>
>   Jan
>
> On 5/29/24 17:56, Robert Bradshaw via dev wrote:
> > While I'm not opposed to having a more formal process for proposals to
> > go from idea to consensus to implementation, I'm not sure how much
> > this would solve the primary issues we face (discoverability and
> > durability). But maybe that could be built into the process? At the
> > very least we could have an "index" which would give identifiers (and
> > hopefully good titles) to all the proposals, and maybe have an offline
> > process to snapshot such docs (even just periodically pulling the
> > content to a repo like I do with
> > https://github.com/cython/cython-issues ). I have yet to find a medium
> > (not even wikis) that facilitates conversation/collaborative editing
> > to the extent that docs does, but I agree with the downside that
> > ownership by random individuals can pose a problem.
> >
> > On Wed, May 29, 2024 at 7:07 AM Jan Lukavský <je...@seznam.cz> wrote:
> >> Hi,
> >>
> >> regarding changing the way we document past (and more importantly
> >> future) changes, I've always been a big fan of the FLIP analogy [1]. I
> >> would love if we could make this work for Beam as well, while preserving
> >> the 'informal' part that I believe all of us want to keep. On the other
> >> hand, this could make the design decisions more searchable, transparent
> >> and get more people involved in the process. Having design documents
> >> durable is of course a highly important part of it.
> >>
> >>    Jan
> >>
> >> [1] https://lists.apache.org/thread/whfy3706w2d0q6rdk4kwyrzvhfd4b5kg
> >>
> >> On 5/29/24 15:04, Kenneth Knowles wrote:
> >>> Hi all,
> >>>
> >>> Yesterday someone asked me about the design doc linked from
> >>> https://github.com/apache/beam/issues/18297 because it is now a 404.
> >>>
> >>> There are plenty of reasons a Google Doc might no longer be
> >>> accessible. They exist outside the project's control. This is part of
> >>> why ASF projects emphasize having discussions on the dev@ list and
> >>> often put all their designs directly onto some ASF-hosted
> >>> infrastructure, such as a Wiki (Example:
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> ).
> >>> In the early days we had a project-owned shared folder but it fell
> >>> into disuse.
> >>>
> >>> In my opinion, Google Docs are still the best place for design docs to
> >>> get feedback and be revised, but the lack of durability is a downside
> >>> to stay aware of. I've also gotten lots of complaints of lack of
> >>> discoverability and lack of systematization of design docs, neither of
> >>> which would be addressed by a shared folder.
> >>>
> >>> I don't have a proposal or suggestion. I don't think this is super
> >>> urgent, certainly not my personal highest priority, but I thought I'd
> >>> just share this as food for thought.
> >>>
> >>> Kenn
>

Re: design docs that get deleted, etc

[Jan Lukavský <je...@seznam.cz>]

Yes, I think that the discoverability and durability would be part of 
some "process". There could be "informal" part, where the design can 
take place virtually anywhere and the document can be owned by anyone 
(as we do today), then it would be copied to a durable store (asf wiki?) 
and a [DISCUSS] thread could be started on ML, followed by [VOTE] (if 
needed?) after that, then the document would be marked as accepted and 
will be turned into issues. I believe this is how the FLIPs and KIPs are 
processed.

  Jan

On 5/29/24 17:56, Robert Bradshaw via dev wrote:
> While I'm not opposed to having a more formal process for proposals to
> go from idea to consensus to implementation, I'm not sure how much
> this would solve the primary issues we face (discoverability and
> durability). But maybe that could be built into the process? At the
> very least we could have an "index" which would give identifiers (and
> hopefully good titles) to all the proposals, and maybe have an offline
> process to snapshot such docs (even just periodically pulling the
> content to a repo like I do with
> https://github.com/cython/cython-issues ). I have yet to find a medium
> (not even wikis) that facilitates conversation/collaborative editing
> to the extent that docs does, but I agree with the downside that
> ownership by random individuals can pose a problem.
>
> On Wed, May 29, 2024 at 7:07 AM Jan Lukavský <je...@seznam.cz> wrote:
>> Hi,
>>
>> regarding changing the way we document past (and more importantly
>> future) changes, I've always been a big fan of the FLIP analogy [1]. I
>> would love if we could make this work for Beam as well, while preserving
>> the 'informal' part that I believe all of us want to keep. On the other
>> hand, this could make the design decisions more searchable, transparent
>> and get more people involved in the process. Having design documents
>> durable is of course a highly important part of it.
>>
>>    Jan
>>
>> [1] https://lists.apache.org/thread/whfy3706w2d0q6rdk4kwyrzvhfd4b5kg
>>
>> On 5/29/24 15:04, Kenneth Knowles wrote:
>>> Hi all,
>>>
>>> Yesterday someone asked me about the design doc linked from
>>> https://github.com/apache/beam/issues/18297 because it is now a 404.
>>>
>>> There are plenty of reasons a Google Doc might no longer be
>>> accessible. They exist outside the project's control. This is part of
>>> why ASF projects emphasize having discussions on the dev@ list and
>>> often put all their designs directly onto some ASF-hosted
>>> infrastructure, such as a Wiki (Example:
>>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals).
>>> In the early days we had a project-owned shared folder but it fell
>>> into disuse.
>>>
>>> In my opinion, Google Docs are still the best place for design docs to
>>> get feedback and be revised, but the lack of durability is a downside
>>> to stay aware of. I've also gotten lots of complaints of lack of
>>> discoverability and lack of systematization of design docs, neither of
>>> which would be addressed by a shared folder.
>>>
>>> I don't have a proposal or suggestion. I don't think this is super
>>> urgent, certainly not my personal highest priority, but I thought I'd
>>> just share this as food for thought.
>>>
>>> Kenn

Re: design docs that get deleted, etc

[Robert Bradshaw via dev <de...@beam.apache.org>]

On Wed, May 29, 2024 at 1:01 PM Robert Burke <ro...@frantil.com> wrote:
>
> Honestly, it's less about how we propose things and more to do with actually making "completing" the documentation work to some permanent place.
>
> That can be as simple as "migrate the final implemented design to a markdown file in the GitHub repo at *foo* or published on the beam website at *bar*.
>
> The big issue is backfilling our existing set of proposals.

I actually don't think there's a big difference between past proposals
and future ones--unless we can make the process really easy (aka
essentially automated) I think the motivation to manually migrate will
be low in the future too (and note that the doc remains a living doc
generally until a short while after its implementation). The "two
sources of truth" doesn't work well until it's static and ready to be
archived, at which point additional work is harder to motivate.

> I do like the google docs for commentary and dispersed discussions, but it's not a long term solution (per the initial message in this thread).

I'm a big fan of markdown in a repo for long-term storage. Anyone know
of any good google docs -> markdown converters? That might be harder
to do if there are images (and of course comment threads, etc. would
be less likely to be preserved).

Even snapshotting them as PDF or HTML might not be that bad, if the
primary purpose is for archiving.


> On Wed, May 29, 2024, 11:56 AM Robert Bradshaw via dev <de...@beam.apache.org> wrote:
>>
>> While I'm not opposed to having a more formal process for proposals to
>> go from idea to consensus to implementation, I'm not sure how much
>> this would solve the primary issues we face (discoverability and
>> durability). But maybe that could be built into the process? At the
>> very least we could have an "index" which would give identifiers (and
>> hopefully good titles) to all the proposals, and maybe have an offline
>> process to snapshot such docs (even just periodically pulling the
>> content to a repo like I do with
>> https://github.com/cython/cython-issues ). I have yet to find a medium
>> (not even wikis) that facilitates conversation/collaborative editing
>> to the extent that docs does, but I agree with the downside that
>> ownership by random individuals can pose a problem.
>>
>> On Wed, May 29, 2024 at 7:07 AM Jan Lukavský <je...@seznam.cz> wrote:
>> >
>> > Hi,
>> >
>> > regarding changing the way we document past (and more importantly
>> > future) changes, I've always been a big fan of the FLIP analogy [1]. I
>> > would love if we could make this work for Beam as well, while preserving
>> > the 'informal' part that I believe all of us want to keep. On the other
>> > hand, this could make the design decisions more searchable, transparent
>> > and get more people involved in the process. Having design documents
>> > durable is of course a highly important part of it.
>> >
>> >   Jan
>> >
>> > [1] https://lists.apache.org/thread/whfy3706w2d0q6rdk4kwyrzvhfd4b5kg
>> >
>> > On 5/29/24 15:04, Kenneth Knowles wrote:
>> > > Hi all,
>> > >
>> > > Yesterday someone asked me about the design doc linked from
>> > > https://github.com/apache/beam/issues/18297 because it is now a 404.
>> > >
>> > > There are plenty of reasons a Google Doc might no longer be
>> > > accessible. They exist outside the project's control. This is part of
>> > > why ASF projects emphasize having discussions on the dev@ list and
>> > > often put all their designs directly onto some ASF-hosted
>> > > infrastructure, such as a Wiki (Example:
>> > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals).
>> > > In the early days we had a project-owned shared folder but it fell
>> > > into disuse.
>> > >
>> > > In my opinion, Google Docs are still the best place for design docs to
>> > > get feedback and be revised, but the lack of durability is a downside
>> > > to stay aware of. I've also gotten lots of complaints of lack of
>> > > discoverability and lack of systematization of design docs, neither of
>> > > which would be addressed by a shared folder.
>> > >
>> > > I don't have a proposal or suggestion. I don't think this is super
>> > > urgent, certainly not my personal highest priority, but I thought I'd
>> > > just share this as food for thought.
>> > >
>> > > Kenn

Re: design docs that get deleted, etc

[Robert Burke <ro...@frantil.com>]

Honestly, it's less about how we propose things and more to do with
actually making "completing" the documentation work to some permanent place.

That can be as simple as "migrate the final implemented design to a
markdown file in the GitHub repo at *foo* or published on the beam website
at *bar*.

The big issue is backfilling our existing set of proposals.

I do like the google docs for commentary and dispersed discussions, but
it's not a long term solution (per the initial message in this thread).

On Wed, May 29, 2024, 11:56 AM Robert Bradshaw via dev <de...@beam.apache.org>
wrote:

> While I'm not opposed to having a more formal process for proposals to
> go from idea to consensus to implementation, I'm not sure how much
> this would solve the primary issues we face (discoverability and
> durability). But maybe that could be built into the process? At the
> very least we could have an "index" which would give identifiers (and
> hopefully good titles) to all the proposals, and maybe have an offline
> process to snapshot such docs (even just periodically pulling the
> content to a repo like I do with
> https://github.com/cython/cython-issues ). I have yet to find a medium
> (not even wikis) that facilitates conversation/collaborative editing
> to the extent that docs does, but I agree with the downside that
> ownership by random individuals can pose a problem.
>
> On Wed, May 29, 2024 at 7:07 AM Jan Lukavský <je...@seznam.cz> wrote:
> >
> > Hi,
> >
> > regarding changing the way we document past (and more importantly
> > future) changes, I've always been a big fan of the FLIP analogy [1]. I
> > would love if we could make this work for Beam as well, while preserving
> > the 'informal' part that I believe all of us want to keep. On the other
> > hand, this could make the design decisions more searchable, transparent
> > and get more people involved in the process. Having design documents
> > durable is of course a highly important part of it.
> >
> >   Jan
> >
> > [1] https://lists.apache.org/thread/whfy3706w2d0q6rdk4kwyrzvhfd4b5kg
> >
> > On 5/29/24 15:04, Kenneth Knowles wrote:
> > > Hi all,
> > >
> > > Yesterday someone asked me about the design doc linked from
> > > https://github.com/apache/beam/issues/18297 because it is now a 404.
> > >
> > > There are plenty of reasons a Google Doc might no longer be
> > > accessible. They exist outside the project's control. This is part of
> > > why ASF projects emphasize having discussions on the dev@ list and
> > > often put all their designs directly onto some ASF-hosted
> > > infrastructure, such as a Wiki (Example:
> > >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> ).
> > > In the early days we had a project-owned shared folder but it fell
> > > into disuse.
> > >
> > > In my opinion, Google Docs are still the best place for design docs to
> > > get feedback and be revised, but the lack of durability is a downside
> > > to stay aware of. I've also gotten lots of complaints of lack of
> > > discoverability and lack of systematization of design docs, neither of
> > > which would be addressed by a shared folder.
> > >
> > > I don't have a proposal or suggestion. I don't think this is super
> > > urgent, certainly not my personal highest priority, but I thought I'd
> > > just share this as food for thought.
> > >
> > > Kenn
>

Re: design docs that get deleted, etc

[Robert Bradshaw via dev <de...@beam.apache.org>]

While I'm not opposed to having a more formal process for proposals to
go from idea to consensus to implementation, I'm not sure how much
this would solve the primary issues we face (discoverability and
durability). But maybe that could be built into the process? At the
very least we could have an "index" which would give identifiers (and
hopefully good titles) to all the proposals, and maybe have an offline
process to snapshot such docs (even just periodically pulling the
content to a repo like I do with
https://github.com/cython/cython-issues ). I have yet to find a medium
(not even wikis) that facilitates conversation/collaborative editing
to the extent that docs does, but I agree with the downside that
ownership by random individuals can pose a problem.

On Wed, May 29, 2024 at 7:07 AM Jan Lukavský <je...@seznam.cz> wrote:
>
> Hi,
>
> regarding changing the way we document past (and more importantly
> future) changes, I've always been a big fan of the FLIP analogy [1]. I
> would love if we could make this work for Beam as well, while preserving
> the 'informal' part that I believe all of us want to keep. On the other
> hand, this could make the design decisions more searchable, transparent
> and get more people involved in the process. Having design documents
> durable is of course a highly important part of it.
>
>   Jan
>
> [1] https://lists.apache.org/thread/whfy3706w2d0q6rdk4kwyrzvhfd4b5kg
>
> On 5/29/24 15:04, Kenneth Knowles wrote:
> > Hi all,
> >
> > Yesterday someone asked me about the design doc linked from
> > https://github.com/apache/beam/issues/18297 because it is now a 404.
> >
> > There are plenty of reasons a Google Doc might no longer be
> > accessible. They exist outside the project's control. This is part of
> > why ASF projects emphasize having discussions on the dev@ list and
> > often put all their designs directly onto some ASF-hosted
> > infrastructure, such as a Wiki (Example:
> > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals).
> > In the early days we had a project-owned shared folder but it fell
> > into disuse.
> >
> > In my opinion, Google Docs are still the best place for design docs to
> > get feedback and be revised, but the lack of durability is a downside
> > to stay aware of. I've also gotten lots of complaints of lack of
> > discoverability and lack of systematization of design docs, neither of
> > which would be addressed by a shared folder.
> >
> > I don't have a proposal or suggestion. I don't think this is super
> > urgent, certainly not my personal highest priority, but I thought I'd
> > just share this as food for thought.
> >
> > Kenn

Re: design docs that get deleted, etc

[Jan Lukavský <je...@seznam.cz>]

Hi,

regarding changing the way we document past (and more importantly 
future) changes, I've always been a big fan of the FLIP analogy [1]. I 
would love if we could make this work for Beam as well, while preserving 
the 'informal' part that I believe all of us want to keep. On the other 
hand, this could make the design decisions more searchable, transparent 
and get more people involved in the process. Having design documents 
durable is of course a highly important part of it.

  Jan

[1] https://lists.apache.org/thread/whfy3706w2d0q6rdk4kwyrzvhfd4b5kg

On 5/29/24 15:04, Kenneth Knowles wrote:
> Hi all,
>
> Yesterday someone asked me about the design doc linked from 
> https://github.com/apache/beam/issues/18297 because it is now a 404.
>
> There are plenty of reasons a Google Doc might no longer be 
> accessible. They exist outside the project's control. This is part of 
> why ASF projects emphasize having discussions on the dev@ list and 
> often put all their designs directly onto some ASF-hosted 
> infrastructure, such as a Wiki (Example: 
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals). 
> In the early days we had a project-owned shared folder but it fell 
> into disuse.
>
> In my opinion, Google Docs are still the best place for design docs to 
> get feedback and be revised, but the lack of durability is a downside 
> to stay aware of. I've also gotten lots of complaints of lack of 
> discoverability and lack of systematization of design docs, neither of 
> which would be addressed by a shared folder.
>
> I don't have a proposal or suggestion. I don't think this is super 
> urgent, certainly not my personal highest priority, but I thought I'd 
> just share this as food for thought.
>
> Kenn