You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by tison <wa...@gmail.com> on 2022/12/15 23:07:47 UTC

[PROPOSAL] Website precommit and move the source of docs to the site repo

Hi,

After several works around the build flow of our official website[1][2][3],
the content sync and site build flow is debuggable and reproducible now.

However, compared to other Apache projects' websites' project layouts and
workflow, we still meet two challenges on the Pulsar site:

1. We don't have a pre-commit workflow for any website-related changes.
Thus, we don't detect broken links or syntax errors when reviewing new
patches[4][5][6].
2. The website's content is two-level down in `site2/website-next` for
historical reasons, which is confusing for contributors.

To overcome these two shortcomings, I propose the following:

1. Move the website's content to the root level, then we have a first-class
Docu&yarn-based JS project layout. It's more convenient and familiar to
related developers.
2. Host the source of docs in the site repo (apache/pulsar-site) instead of
under `site2` folder in the main repo and do content sync.

Below are the pros and cons:

Pros

1. Obviously, we have the pre-commit workflow now. And since we host the
source of docs in one repo, we don't have to run the pre-commit workflow in
two places, which can be quite cumbersome to implement.
2. The size of the source release of the main repo can be reduced a lot.
Currently, 63MB out of 140MB of the sources are taken by the site2 folder,
which we can remove totally. In addition, we carry out full-versioned docs
every release.
3. We can clean up a large portion of "integration" to debug the site
brittlely on the main repo[7]  (etc.) and redundant contribution guide[8].
This way, when updating docs, we can preview the result in one repo instead
of actually doing the sync on the fly. In addition, this integration blocks
we move the website content to the top level since it makes strong
assumptions about the relative layout.

Cons

The most significant con is that we cannot update the code and docs in one
patch against apache/pulsar now. You must open a new pull request to
apache/pulsar-site, cross-reference each other and manage the merge order
(synchronization).

Alternatives:

To resolve the versioned docs issue, an alternative is to host only the
user docs along with each version, like Flink does[9]. But it both detaches
from the Docu framework and requires significant development efforts.

Since it can explicitly change the development flow (that is, you should
now update docs separately), I am starting this discussion here to reach
for your feedback.

Welcome to leave your comments!

Best,
tison.

[1] https://pulsar.apache.org/
[2] https://github.com/apache/pulsar-site
[3] https://github.com/apache/pulsar/issues/18014
[4] https://github.com/apache/pulsar/issues/17599
[5] https://github.com/apache/pulsar/pull/17863#discussion_r990174850
[6] https://github.com/apache/pulsar/pull/17853#discussion_r991803704
[7]
https://github.com/apache/pulsar/blob/b1f9e351fa4d5aba197d33cfc0c536516b55b61f/site2/website/start.sh
[8]
https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
[9] https://github.com/apache/flink/tree/master/docs

Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

Posted by Li Li <ll...@streamnative.io.INVALID>.
+1, That’s a good idea.

Thanks,
Li Li

> On Dec 16, 2022, at 07:07, tison <wa...@gmail.com> wrote:
> 
> Hi,
> 
> After several works around the build flow of our official website[1][2][3],
> the content sync and site build flow is debuggable and reproducible now.
> 
> However, compared to other Apache projects' websites' project layouts and
> workflow, we still meet two challenges on the Pulsar site:
> 
> 1. We don't have a pre-commit workflow for any website-related changes.
> Thus, we don't detect broken links or syntax errors when reviewing new
> patches[4][5][6].
> 2. The website's content is two-level down in `site2/website-next` for
> historical reasons, which is confusing for contributors.
> 
> To overcome these two shortcomings, I propose the following:
> 
> 1. Move the website's content to the root level, then we have a first-class
> Docu&yarn-based JS project layout. It's more convenient and familiar to
> related developers.
> 2. Host the source of docs in the site repo (apache/pulsar-site) instead of
> under `site2` folder in the main repo and do content sync.
> 
> Below are the pros and cons:
> 
> Pros
> 
> 1. Obviously, we have the pre-commit workflow now. And since we host the
> source of docs in one repo, we don't have to run the pre-commit workflow in
> two places, which can be quite cumbersome to implement.
> 2. The size of the source release of the main repo can be reduced a lot.
> Currently, 63MB out of 140MB of the sources are taken by the site2 folder,
> which we can remove totally. In addition, we carry out full-versioned docs
> every release.
> 3. We can clean up a large portion of "integration" to debug the site
> brittlely on the main repo[7]  (etc.) and redundant contribution guide[8].
> This way, when updating docs, we can preview the result in one repo instead
> of actually doing the sync on the fly. In addition, this integration blocks
> we move the website content to the top level since it makes strong
> assumptions about the relative layout.
> 
> Cons
> 
> The most significant con is that we cannot update the code and docs in one
> patch against apache/pulsar now. You must open a new pull request to
> apache/pulsar-site, cross-reference each other and manage the merge order
> (synchronization).
> 
> Alternatives:
> 
> To resolve the versioned docs issue, an alternative is to host only the
> user docs along with each version, like Flink does[9]. But it both detaches
> from the Docu framework and requires significant development efforts.
> 
> Since it can explicitly change the development flow (that is, you should
> now update docs separately), I am starting this discussion here to reach
> for your feedback.
> 
> Welcome to leave your comments!
> 
> Best,
> tison.
> 
> [1] https://pulsar.apache.org/
> [2] https://github.com/apache/pulsar-site
> [3] https://github.com/apache/pulsar/issues/18014
> [4] https://github.com/apache/pulsar/issues/17599
> [5] https://github.com/apache/pulsar/pull/17863#discussion_r990174850
> [6] https://github.com/apache/pulsar/pull/17853#discussion_r991803704
> [7]
> https://github.com/apache/pulsar/blob/b1f9e351fa4d5aba197d33cfc0c536516b55b61f/site2/website/start.sh
> [8]
> https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
> [9] https://github.com/apache/flink/tree/master/docs


Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

Posted by tison <wa...@gmail.com>.
Mail: https://lists.apache.org/thread/ff659j0y46j94c6jc6zlwotc78pq2n40
Issue: https://github.com/apache/pulsar/issues/19111

Best,
tison.


tison <wa...@gmail.com> 于2022年12月30日周五 10:46写道:

> Hi Yu,
>
> Thanks for your recommendation! Yes, I'm going to do this.
>
> Best,
> tison.
>
>
> Yu <li...@apache.org> 于2022年12月30日周五 09:20写道:
>
>> Thanks tison!
>>
>> It's better to
>> a) update the contribution processes of docs, website content, and release
>> notes [1] [2] [3] [4] [5]
>> b) inform the community about this breaking change officially in another
>> email (reminders for doc contributions)
>>
>> ~~~~~~~~
>>
>> docs:
>> [1] https://pulsar.apache.org/contribute/document-contribution/
>> [2]
>>
>> https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
>> [3]
>>
>> https://pulsar.apache.org/contribute/document-preview/#preview-website-changes
>>
>> release notes:
>> [4] https://pulsar.apache.org/contribute/release-process/#update-the-site
>> [5] https://pulsar.apache.org/contribute/release-note-guide/
>>
>> On Thu, Dec 29, 2022 at 9:07 PM tison <wa...@gmail.com> wrote:
>>
>> > The new repo is reduced by 100MB:
>> >
>> > $ gh repo clone apache/pulsar -- --single-branch --depth=1
>> > $ du -sh pulsar | sort -rh
>> >  53M pulsar
>> >
>> > Best,
>> > tison.
>> >
>> >
>> > tison <wa...@gmail.com> 于2022年12月29日周四 21:01写道:
>> >
>> > > Landed.
>> > >
>> > > Best,
>> > > tison.
>> > >
>> > >
>> > > tison <wa...@gmail.com> 于2022年12月29日周四 17:51写道:
>> > >
>> > >> Here are the related PRs:
>> > >>
>> > >> * https://github.com/apache/pulsar/pull/19100
>> > >> * https://github.com/apache/pulsar-site/pull/348
>> > >>
>> > >> Best,
>> > >> tison.
>> > >>
>> > >>
>> > >> tison <wa...@gmail.com> 于2022年12月26日周一 21:45写道:
>> > >>
>> > >>> FYI tracking issue has been created:
>> > >>> https://github.com/apache/pulsar/issues/19064
>> > >>>
>> > >>> I plan to finish it by the end of next month.
>> > >>>
>> > >>> Best,
>> > >>> tison.
>> > >>>
>> > >>>
>> > >>> tison <wa...@gmail.com> 于2022年12月21日周三 11:33写道:
>> > >>>
>> > >>>> Thanks for your feedback!
>> > >>>>
>> > >>>> @Yu
>> > >>>>
>> > >>>> Thanks for sharing the previous thread. I looped in
>> @michaeljmarshall
>> > >>>> here.
>> > >>>>
>> > >>>> @Jun
>> > >>>>
>> > >>>> It's possible but causes a new shortcoming: Now you should tell the
>> > >>>> contributor that the versioned docs are different from the NEXT
>> > version
>> > >>>> docs, lol.
>> > >>>>
>> > >>>> If our developers don't complain about these separated sources.
>> Like
>> > @Asaf
>> > >>>> comment:
>> > >>>>
>> > >>>> > We can take, let's say, five features and see if they were
>> actually
>> > >>>> done in
>> > >>>> > the same PR or separate PR. I guess that most documentation is
>> > >>>> actually
>> > >>>> > updated separately. Thus, from that perspective, maybe it’s not a
>> > con.
>> > >>>>
>> > >>>> Then we can do this refactor thoroughgoing.
>> > >>>>
>> > >>>> Also, if we keep, somehow several sources in the main repo. We
>> still
>> > >>>> have shortcomings:
>> > >>>>
>> > >>>> 1. Duplicated CI workflows.
>> > >>>> 2. Cumbersome preview scaffolding in the main repo.
>> > >>>>
>> > >>>> ... which is the original purpose I'd like to overcome.
>> > >>>>
>> > >>>> Best,
>> > >>>> tison.
>> > >>>>
>> > >>>>
>> > >>>> Jun Ma <mo...@hotmail.com> 于2022年12月21日周三 11:19写道:
>> > >>>>
>> > >>>>> Is it possible to come up with a compromised solution that has the
>> > >>>>> pros of both sides but minimizes the side effect? I'm thinking
>> maybe
>> > it's
>> > >>>>> not necessary to sacrifice the current contribution process, as
>> long
>> > as it
>> > >>>>> can greatly reduce the load of back-end actions and source size.
>> For
>> > >>>>> example, if we only move out the versioned docs to the site repo
>> but
>> > keep
>> > >>>>> the source of the NEXT docs in the pulsar repo, does this help to
>> > win a
>> > >>>>> large proportion of those pros when people can still contribute as
>> > usual?
>> > >>>>>
>> > >>>>> ________________________________
>> > >>>>> From: Jiaqi Shen <gl...@gmail.com>
>> > >>>>> Sent: Tuesday, December 20, 2022 17:15
>> > >>>>> To: dev@pulsar.apache.org <de...@pulsar.apache.org>
>> > >>>>> Subject: Re: [PROPOSAL] Website precommit and move the source of
>> docs
>> > >>>>> to the site repo
>> > >>>>>
>> > >>>>> +1, it makes sense to me.
>> > >>>>>
>> > >>>>> Thanks,
>> > >>>>> Jiaqi Shen
>> > >>>>>
>> > >>>>>
>> > >>>>> Yu <li...@apache.org> 于2022年12月19日周一 20:57写道:
>> > >>>>>
>> > >>>>> > Hi tison,
>> > >>>>> >
>> > >>>>> > Thanks for raising this up!
>> > >>>>> >
>> > >>>>> > Our community had a similar discussion previously and chose to
>> > >>>>> "keep" the
>> > >>>>> > doc repo stay in the Pulsar main repo at that time.
>> > >>>>> >
>> > >>>>> > [1] lists the pros and cons of "keep" and "not keep" solutions.
>> > >>>>> >
>> > >>>>> > I'm +0 on this proposal because I think the total scores of
>> these
>> > two
>> > >>>>> > solutions are almost equal after weighing the pros and cons.
>> > >>>>> >
>> > >>>>> > ~~~~~~~~~~~~~~~~~~~~
>> > >>>>> >
>> > >>>>> > [1]
>> > https://lists.apache.org/thread/mf2xwntfgn84dq78ksqv22jk3drq6xb3
>> > >>>>> >
>> > >>>>> >
>> > >>>>> > On Mon, Dec 19, 2022 at 5:40 PM tison <wa...@gmail.com>
>> > wrote:
>> > >>>>> >
>> > >>>>> > > Thanks for your feedback!
>> > >>>>> > >
>> > >>>>> > > @Asaf
>> > >>>>> > >
>> > >>>>> > > > pre-commit
>> > >>>>> > >
>> > >>>>> > > I mean CI checks before merging a patch. Currently, we don't
>> run
>> > >>>>> checks
>> > >>>>> > for
>> > >>>>> > > the content before merging them. This causes a series of
>> syntax
>> > >>>>> errors
>> > >>>>> > and
>> > >>>>> > > broken links issues. If we hold docs under site2 folder in the
>> > >>>>> main repo
>> > >>>>> > > and then copied to the site repo, we have two places to build
>> > such
>> > >>>>> CI
>> > >>>>> > > checks. What's worse, the checks for the main repo will be
>> quite
>> > >>>>> > > cumbersome (that you do some if-else logic in the whole
>> Pulsar CI
>> > >>>>> > > workflows, and do the sync sequentially in that workflow).
>> > >>>>> > >
>> > >>>>> > > If we hold the source of docs only in the site repo, we can
>> > extend
>> > >>>>> the
>> > >>>>> > > "precommit" workflow[1] I added recently to check for syntax
>> > >>>>> errors and
>> > >>>>> > > broken links also.
>> > >>>>> > >
>> > >>>>> > > > What does the apache/pulsar-site repo contain today?
>> > >>>>> > >
>> > >>>>> > > It should be covered by the documentation guide page[2]. It
>> holds
>> > >>>>> the
>> > >>>>> > > source of the official website and the user docs are synced
>> from
>> > >>>>> the main
>> > >>>>> > > repo.
>> > >>>>> > >
>> > >>>>> > > > What content do we have today in the pulsar repo related to
>> the
>> > >>>>> site?
>> > >>>>> > >
>> > >>>>> > > After issue-18014[3] is done, we host only user docs and some
>> > JSON
>> > >>>>> > metadata
>> > >>>>> > > in the main repo, which is synced by site_syncer.py[4].
>> > >>>>> > >
>> > >>>>> > > > Can you explain that better? Are you saying pulsar source
>> JARs
>> > >>>>> contain
>> > >>>>> > > the documentation?
>> > >>>>> > >
>> > >>>>> > > No. Source JARs contain only the Java files and necessary
>> > >>>>> copyrights
>> > >>>>> > info.
>> > >>>>> > > The source release is, for example,
>> > >>>>> > >
>> > >>>>> > >
>> > >>>>> >
>> > >>>>>
>> >
>> https://archive.apache.org/dist/pulsar/pulsar-2.10.2/apache-pulsar-2.10.2-src.tar.gz
>> > >>>>> > > ,
>> > >>>>> > > which is extracted to 173M where 129M is occupied by the site2
>> > >>>>> folder.
>> > >>>>> > >
>> > >>>>> > > This also affects when developers do git clone to clone the
>> repo.
>> > >>>>> > >
>> > >>>>> > > > I mean, if you wish to document a bug fix in 2.9.x, for
>> > example,
>> > >>>>> would
>> > >>>>> > > you do it in the 2.9.x branch under site2/docs or
>> > >>>>> > > site2/website/versioned_docs/2.9.5?
>> > >>>>> > >
>> > >>>>> > > This is another question. Ideally, we should have hosted
>> > versioned
>> > >>>>> docs
>> > >>>>> > > associated with the specific version to that branch, like
>> Apache
>> > >>>>> Flink
>> > >>>>> > does
>> > >>>>> > > as I mentioned[5]. But we do not, and actually the situation
>> is
>> > we
>> > >>>>> update
>> > >>>>> > > the versioned docs under the master branch and thus, the docs
>> can
>> > >>>>> be
>> > >>>>> > synced
>> > >>>>> > > properly.
>> > >>>>> > >
>> > >>>>> > > See also the "Alternatives" section in the original email.
>> > >>>>> > >
>> > >>>>> > > @All
>> > >>>>> > >
>> > >>>>> > > Since we don't have objections to the possible cons listed
>> above
>> > >>>>> or any
>> > >>>>> > new
>> > >>>>> > > ones, I'm going to create a tracking issue later this week and
>> > >>>>> show what
>> > >>>>> > > will be changed in PRs for further review.
>> > >>>>> > >
>> > >>>>> > > Best,
>> > >>>>> > > tison.
>> > >>>>> > >
>> > >>>>> > > [1]
>> > >>>>> > >
>> > >>>>> > >
>> > >>>>> >
>> > >>>>>
>> >
>> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/.github/workflows/ci-precommit.yml
>> > >>>>> > > [2]
>> > >>>>> > >
>> > >>>>> > >
>> > >>>>> >
>> > >>>>>
>> >
>> https://pulsar.apache.org/contribute/document-contribution/#source-repositories
>> > >>>>> > > [3] https://github.com/apache/pulsar/issues/18014
>> > >>>>> > > [4]
>> > >>>>> > >
>> > >>>>> > >
>> > >>>>> >
>> > >>>>>
>> >
>> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/tools/pytools/lib/execute/site_syncer.py
>> > >>>>> > > [5] https://github.com/apache/flink/tree/master/docs
>> > >>>>> > >
>> > >>>>> > >
>> > >>>>> > > PengHui Li <pe...@apache.org> 于2022年12月19日周一 16:26写道:
>> > >>>>> > >
>> > >>>>> > > > +1
>> > >>>>> > > >
>> > >>>>> > > > I support moving them to the website repo.
>> > >>>>> > > >
>> > >>>>> > > > Thanks,
>> > >>>>> > > > Penghui
>> > >>>>> > > >
>> > >>>>> > > > On Mon, Dec 19, 2022 at 12:04 PM Yunze Xu
>> > >>>>> <yzxu@streamnative.io.invalid
>> > >>>>> > >
>> > >>>>> > > > wrote:
>> > >>>>> > > >
>> > >>>>> > > > > +1. The most significant point to me is that we can
>> preview
>> > >>>>> all the
>> > >>>>> > > > > content of the website without synchronizing contents from
>> > the
>> > >>>>> > > > > apache/pulsar repo.
>> > >>>>> > > > >
>> > >>>>> > > > > Thanks,
>> > >>>>> > > > > Yunze
>> > >>>>> > > > >
>> > >>>>> > > > > On Mon, Dec 19, 2022 at 9:53 AM Li Li <ur...@apache.org>
>> > >>>>> wrote:
>> > >>>>> > > > > >
>> > >>>>> > > > > > +1, That’s a good idea.
>> > >>>>> > > > > >
>> > >>>>> > > > > > > On Dec 16, 2022, at 07:07, tison <
>> wander4096@gmail.com>
>> > >>>>> wrote:
>> > >>>>> > > > > > >
>> > >>>>> > > > > > > Hi,
>> > >>>>> > > > > > >
>> > >>>>> > > > > > > After several works around the build flow of our
>> official
>> > >>>>> > > > > website[1][2][3],
>> > >>>>> > > > > > > the content sync and site build flow is debuggable and
>> > >>>>> > reproducible
>> > >>>>> > > > > now.
>> > >>>>> > > > > > >
>> > >>>>> > > > > > > However, compared to other Apache projects' websites'
>> > >>>>> project
>> > >>>>> > > layouts
>> > >>>>> > > > > and
>> > >>>>> > > > > > > workflow, we still meet two challenges on the Pulsar
>> > site:
>> > >>>>> > > > > > >
>> > >>>>> > > > > > > 1. We don't have a pre-commit workflow for any
>> > >>>>> website-related
>> > >>>>> > > > changes.
>> > >>>>> > > > > > > Thus, we don't detect broken links or syntax errors
>> when
>> > >>>>> > reviewing
>> > >>>>> > > > new
>> > >>>>> > > > > > > patches[4][5][6].
>> > >>>>> > > > > > > 2. The website's content is two-level down in
>> > >>>>> > `site2/website-next`
>> > >>>>> > > > for
>> > >>>>> > > > > > > historical reasons, which is confusing for
>> contributors.
>> > >>>>> > > > > > >
>> > >>>>> > > > > > > To overcome these two shortcomings, I propose the
>> > >>>>> following:
>> > >>>>> > > > > > >
>> > >>>>> > > > > > > 1. Move the website's content to the root level, then
>> we
>> > >>>>> have a
>> > >>>>> > > > > first-class
>> > >>>>> > > > > > > Docu&yarn-based JS project layout. It's more
>> convenient
>> > and
>> > >>>>> > > familiar
>> > >>>>> > > > to
>> > >>>>> > > > > > > related developers.
>> > >>>>> > > > > > > 2. Host the source of docs in the site repo
>> > >>>>> (apache/pulsar-site)
>> > >>>>> > > > > instead of
>> > >>>>> > > > > > > under `site2` folder in the main repo and do content
>> > sync.
>> > >>>>> > > > > > >
>> > >>>>> > > > > > > Below are the pros and cons:
>> > >>>>> > > > > > >
>> > >>>>> > > > > > > Pros
>> > >>>>> > > > > > >
>> > >>>>> > > > > > > 1. Obviously, we have the pre-commit workflow now. And
>> > >>>>> since we
>> > >>>>> > > host
>> > >>>>> > > > > the
>> > >>>>> > > > > > > source of docs in one repo, we don't have to run the
>> > >>>>> pre-commit
>> > >>>>> > > > > workflow in
>> > >>>>> > > > > > > two places, which can be quite cumbersome to
>> implement.
>> > >>>>> > > > > > > 2. The size of the source release of the main repo
>> can be
>> > >>>>> > reduced a
>> > >>>>> > > > > lot.
>> > >>>>> > > > > > > Currently, 63MB out of 140MB of the sources are taken
>> by
>> > >>>>> the
>> > >>>>> > site2
>> > >>>>> > > > > folder,
>> > >>>>> > > > > > > which we can remove totally. In addition, we carry out
>> > >>>>> > > full-versioned
>> > >>>>> > > > > docs
>> > >>>>> > > > > > > every release.
>> > >>>>> > > > > > > 3. We can clean up a large portion of "integration" to
>> > >>>>> debug the
>> > >>>>> > > site
>> > >>>>> > > > > > > brittlely on the main repo[7]  (etc.) and redundant
>> > >>>>> contribution
>> > >>>>> > > > > guide[8].
>> > >>>>> > > > > > > This way, when updating docs, we can preview the
>> result
>> > in
>> > >>>>> one
>> > >>>>> > repo
>> > >>>>> > > > > instead
>> > >>>>> > > > > > > of actually doing the sync on the fly. In addition,
>> this
>> > >>>>> > > integration
>> > >>>>> > > > > blocks
>> > >>>>> > > > > > > we move the website content to the top level since it
>> > makes
>> > >>>>> > strong
>> > >>>>> > > > > > > assumptions about the relative layout.
>> > >>>>> > > > > > >
>> > >>>>> > > > > > > Cons
>> > >>>>> > > > > > >
>> > >>>>> > > > > > > The most significant con is that we cannot update the
>> > code
>> > >>>>> and
>> > >>>>> > docs
>> > >>>>> > > > in
>> > >>>>> > > > > one
>> > >>>>> > > > > > > patch against apache/pulsar now. You must open a new
>> pull
>> > >>>>> request
>> > >>>>> > > to
>> > >>>>> > > > > > > apache/pulsar-site, cross-reference each other and
>> manage
>> > >>>>> the
>> > >>>>> > merge
>> > >>>>> > > > > order
>> > >>>>> > > > > > > (synchronization).
>> > >>>>> > > > > > >
>> > >>>>> > > > > > > Alternatives:
>> > >>>>> > > > > > >
>> > >>>>> > > > > > > To resolve the versioned docs issue, an alternative
>> is to
>> > >>>>> host
>> > >>>>> > only
>> > >>>>> > > > the
>> > >>>>> > > > > > > user docs along with each version, like Flink does[9].
>> > But
>> > >>>>> it
>> > >>>>> > both
>> > >>>>> > > > > detaches
>> > >>>>> > > > > > > from the Docu framework and requires significant
>> > >>>>> development
>> > >>>>> > > efforts.
>> > >>>>> > > > > > >
>> > >>>>> > > > > > > Since it can explicitly change the development flow
>> (that
>> > >>>>> is, you
>> > >>>>> > > > > should
>> > >>>>> > > > > > > now update docs separately), I am starting this
>> > discussion
>> > >>>>> here
>> > >>>>> > to
>> > >>>>> > > > > reach
>> > >>>>> > > > > > > for your feedback.
>> > >>>>> > > > > > >
>> > >>>>> > > > > > > Welcome to leave your comments!
>> > >>>>> > > > > > >
>> > >>>>> > > > > > > Best,
>> > >>>>> > > > > > > tison.
>> > >>>>> > > > > > >
>> > >>>>> > > > > > > [1] https://pulsar.apache.org/
>> > >>>>> > > > > > > [2] https://github.com/apache/pulsar-site
>> > >>>>> > > > > > > [3] https://github.com/apache/pulsar/issues/18014
>> > >>>>> > > > > > > [4] https://github.com/apache/pulsar/issues/17599
>> > >>>>> > > > > > > [5]
>> > >>>>> > > >
>> > >>>>> https://github.com/apache/pulsar/pull/17863#discussion_r990174850
>> > >>>>> > > > > > > [6]
>> > >>>>> > > >
>> > >>>>> https://github.com/apache/pulsar/pull/17853#discussion_r991803704
>> > >>>>> > > > > > > [7]
>> > >>>>> > > > > > >
>> > >>>>> > > > >
>> > >>>>> > > >
>> > >>>>> > >
>> > >>>>> >
>> > >>>>>
>> >
>> https://github.com/apache/pulsar/blob/b1f9e351fa4d5aba197d33cfc0c536516b55b61f/site2/website/start.sh
>> > >>>>> > > > > > > [8]
>> > >>>>> > > > > > >
>> > >>>>> > > > >
>> > >>>>> > > >
>> > >>>>> > >
>> > >>>>> >
>> > >>>>>
>> >
>> https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
>> > >>>>> > > > > > > [9] https://github.com/apache/flink/tree/master/docs
>> > >>>>> > > > > >
>> > >>>>> > > > >
>> > >>>>> > > >
>> > >>>>> > >
>> > >>>>> >
>> > >>>>>
>> > >>>>
>> >
>>
>

Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

Posted by tison <wa...@gmail.com>.
Hi Yu,

Thanks for your recommendation! Yes, I'm going to do this.

Best,
tison.


Yu <li...@apache.org> 于2022年12月30日周五 09:20写道:

> Thanks tison!
>
> It's better to
> a) update the contribution processes of docs, website content, and release
> notes [1] [2] [3] [4] [5]
> b) inform the community about this breaking change officially in another
> email (reminders for doc contributions)
>
> ~~~~~~~~
>
> docs:
> [1] https://pulsar.apache.org/contribute/document-contribution/
> [2]
>
> https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
> [3]
>
> https://pulsar.apache.org/contribute/document-preview/#preview-website-changes
>
> release notes:
> [4] https://pulsar.apache.org/contribute/release-process/#update-the-site
> [5] https://pulsar.apache.org/contribute/release-note-guide/
>
> On Thu, Dec 29, 2022 at 9:07 PM tison <wa...@gmail.com> wrote:
>
> > The new repo is reduced by 100MB:
> >
> > $ gh repo clone apache/pulsar -- --single-branch --depth=1
> > $ du -sh pulsar | sort -rh
> >  53M pulsar
> >
> > Best,
> > tison.
> >
> >
> > tison <wa...@gmail.com> 于2022年12月29日周四 21:01写道:
> >
> > > Landed.
> > >
> > > Best,
> > > tison.
> > >
> > >
> > > tison <wa...@gmail.com> 于2022年12月29日周四 17:51写道:
> > >
> > >> Here are the related PRs:
> > >>
> > >> * https://github.com/apache/pulsar/pull/19100
> > >> * https://github.com/apache/pulsar-site/pull/348
> > >>
> > >> Best,
> > >> tison.
> > >>
> > >>
> > >> tison <wa...@gmail.com> 于2022年12月26日周一 21:45写道:
> > >>
> > >>> FYI tracking issue has been created:
> > >>> https://github.com/apache/pulsar/issues/19064
> > >>>
> > >>> I plan to finish it by the end of next month.
> > >>>
> > >>> Best,
> > >>> tison.
> > >>>
> > >>>
> > >>> tison <wa...@gmail.com> 于2022年12月21日周三 11:33写道:
> > >>>
> > >>>> Thanks for your feedback!
> > >>>>
> > >>>> @Yu
> > >>>>
> > >>>> Thanks for sharing the previous thread. I looped in
> @michaeljmarshall
> > >>>> here.
> > >>>>
> > >>>> @Jun
> > >>>>
> > >>>> It's possible but causes a new shortcoming: Now you should tell the
> > >>>> contributor that the versioned docs are different from the NEXT
> > version
> > >>>> docs, lol.
> > >>>>
> > >>>> If our developers don't complain about these separated sources. Like
> > @Asaf
> > >>>> comment:
> > >>>>
> > >>>> > We can take, let's say, five features and see if they were
> actually
> > >>>> done in
> > >>>> > the same PR or separate PR. I guess that most documentation is
> > >>>> actually
> > >>>> > updated separately. Thus, from that perspective, maybe it’s not a
> > con.
> > >>>>
> > >>>> Then we can do this refactor thoroughgoing.
> > >>>>
> > >>>> Also, if we keep, somehow several sources in the main repo. We still
> > >>>> have shortcomings:
> > >>>>
> > >>>> 1. Duplicated CI workflows.
> > >>>> 2. Cumbersome preview scaffolding in the main repo.
> > >>>>
> > >>>> ... which is the original purpose I'd like to overcome.
> > >>>>
> > >>>> Best,
> > >>>> tison.
> > >>>>
> > >>>>
> > >>>> Jun Ma <mo...@hotmail.com> 于2022年12月21日周三 11:19写道:
> > >>>>
> > >>>>> Is it possible to come up with a compromised solution that has the
> > >>>>> pros of both sides but minimizes the side effect? I'm thinking
> maybe
> > it's
> > >>>>> not necessary to sacrifice the current contribution process, as
> long
> > as it
> > >>>>> can greatly reduce the load of back-end actions and source size.
> For
> > >>>>> example, if we only move out the versioned docs to the site repo
> but
> > keep
> > >>>>> the source of the NEXT docs in the pulsar repo, does this help to
> > win a
> > >>>>> large proportion of those pros when people can still contribute as
> > usual?
> > >>>>>
> > >>>>> ________________________________
> > >>>>> From: Jiaqi Shen <gl...@gmail.com>
> > >>>>> Sent: Tuesday, December 20, 2022 17:15
> > >>>>> To: dev@pulsar.apache.org <de...@pulsar.apache.org>
> > >>>>> Subject: Re: [PROPOSAL] Website precommit and move the source of
> docs
> > >>>>> to the site repo
> > >>>>>
> > >>>>> +1, it makes sense to me.
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Jiaqi Shen
> > >>>>>
> > >>>>>
> > >>>>> Yu <li...@apache.org> 于2022年12月19日周一 20:57写道:
> > >>>>>
> > >>>>> > Hi tison,
> > >>>>> >
> > >>>>> > Thanks for raising this up!
> > >>>>> >
> > >>>>> > Our community had a similar discussion previously and chose to
> > >>>>> "keep" the
> > >>>>> > doc repo stay in the Pulsar main repo at that time.
> > >>>>> >
> > >>>>> > [1] lists the pros and cons of "keep" and "not keep" solutions.
> > >>>>> >
> > >>>>> > I'm +0 on this proposal because I think the total scores of these
> > two
> > >>>>> > solutions are almost equal after weighing the pros and cons.
> > >>>>> >
> > >>>>> > ~~~~~~~~~~~~~~~~~~~~
> > >>>>> >
> > >>>>> > [1]
> > https://lists.apache.org/thread/mf2xwntfgn84dq78ksqv22jk3drq6xb3
> > >>>>> >
> > >>>>> >
> > >>>>> > On Mon, Dec 19, 2022 at 5:40 PM tison <wa...@gmail.com>
> > wrote:
> > >>>>> >
> > >>>>> > > Thanks for your feedback!
> > >>>>> > >
> > >>>>> > > @Asaf
> > >>>>> > >
> > >>>>> > > > pre-commit
> > >>>>> > >
> > >>>>> > > I mean CI checks before merging a patch. Currently, we don't
> run
> > >>>>> checks
> > >>>>> > for
> > >>>>> > > the content before merging them. This causes a series of syntax
> > >>>>> errors
> > >>>>> > and
> > >>>>> > > broken links issues. If we hold docs under site2 folder in the
> > >>>>> main repo
> > >>>>> > > and then copied to the site repo, we have two places to build
> > such
> > >>>>> CI
> > >>>>> > > checks. What's worse, the checks for the main repo will be
> quite
> > >>>>> > > cumbersome (that you do some if-else logic in the whole Pulsar
> CI
> > >>>>> > > workflows, and do the sync sequentially in that workflow).
> > >>>>> > >
> > >>>>> > > If we hold the source of docs only in the site repo, we can
> > extend
> > >>>>> the
> > >>>>> > > "precommit" workflow[1] I added recently to check for syntax
> > >>>>> errors and
> > >>>>> > > broken links also.
> > >>>>> > >
> > >>>>> > > > What does the apache/pulsar-site repo contain today?
> > >>>>> > >
> > >>>>> > > It should be covered by the documentation guide page[2]. It
> holds
> > >>>>> the
> > >>>>> > > source of the official website and the user docs are synced
> from
> > >>>>> the main
> > >>>>> > > repo.
> > >>>>> > >
> > >>>>> > > > What content do we have today in the pulsar repo related to
> the
> > >>>>> site?
> > >>>>> > >
> > >>>>> > > After issue-18014[3] is done, we host only user docs and some
> > JSON
> > >>>>> > metadata
> > >>>>> > > in the main repo, which is synced by site_syncer.py[4].
> > >>>>> > >
> > >>>>> > > > Can you explain that better? Are you saying pulsar source
> JARs
> > >>>>> contain
> > >>>>> > > the documentation?
> > >>>>> > >
> > >>>>> > > No. Source JARs contain only the Java files and necessary
> > >>>>> copyrights
> > >>>>> > info.
> > >>>>> > > The source release is, for example,
> > >>>>> > >
> > >>>>> > >
> > >>>>> >
> > >>>>>
> >
> https://archive.apache.org/dist/pulsar/pulsar-2.10.2/apache-pulsar-2.10.2-src.tar.gz
> > >>>>> > > ,
> > >>>>> > > which is extracted to 173M where 129M is occupied by the site2
> > >>>>> folder.
> > >>>>> > >
> > >>>>> > > This also affects when developers do git clone to clone the
> repo.
> > >>>>> > >
> > >>>>> > > > I mean, if you wish to document a bug fix in 2.9.x, for
> > example,
> > >>>>> would
> > >>>>> > > you do it in the 2.9.x branch under site2/docs or
> > >>>>> > > site2/website/versioned_docs/2.9.5?
> > >>>>> > >
> > >>>>> > > This is another question. Ideally, we should have hosted
> > versioned
> > >>>>> docs
> > >>>>> > > associated with the specific version to that branch, like
> Apache
> > >>>>> Flink
> > >>>>> > does
> > >>>>> > > as I mentioned[5]. But we do not, and actually the situation is
> > we
> > >>>>> update
> > >>>>> > > the versioned docs under the master branch and thus, the docs
> can
> > >>>>> be
> > >>>>> > synced
> > >>>>> > > properly.
> > >>>>> > >
> > >>>>> > > See also the "Alternatives" section in the original email.
> > >>>>> > >
> > >>>>> > > @All
> > >>>>> > >
> > >>>>> > > Since we don't have objections to the possible cons listed
> above
> > >>>>> or any
> > >>>>> > new
> > >>>>> > > ones, I'm going to create a tracking issue later this week and
> > >>>>> show what
> > >>>>> > > will be changed in PRs for further review.
> > >>>>> > >
> > >>>>> > > Best,
> > >>>>> > > tison.
> > >>>>> > >
> > >>>>> > > [1]
> > >>>>> > >
> > >>>>> > >
> > >>>>> >
> > >>>>>
> >
> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/.github/workflows/ci-precommit.yml
> > >>>>> > > [2]
> > >>>>> > >
> > >>>>> > >
> > >>>>> >
> > >>>>>
> >
> https://pulsar.apache.org/contribute/document-contribution/#source-repositories
> > >>>>> > > [3] https://github.com/apache/pulsar/issues/18014
> > >>>>> > > [4]
> > >>>>> > >
> > >>>>> > >
> > >>>>> >
> > >>>>>
> >
> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/tools/pytools/lib/execute/site_syncer.py
> > >>>>> > > [5] https://github.com/apache/flink/tree/master/docs
> > >>>>> > >
> > >>>>> > >
> > >>>>> > > PengHui Li <pe...@apache.org> 于2022年12月19日周一 16:26写道:
> > >>>>> > >
> > >>>>> > > > +1
> > >>>>> > > >
> > >>>>> > > > I support moving them to the website repo.
> > >>>>> > > >
> > >>>>> > > > Thanks,
> > >>>>> > > > Penghui
> > >>>>> > > >
> > >>>>> > > > On Mon, Dec 19, 2022 at 12:04 PM Yunze Xu
> > >>>>> <yzxu@streamnative.io.invalid
> > >>>>> > >
> > >>>>> > > > wrote:
> > >>>>> > > >
> > >>>>> > > > > +1. The most significant point to me is that we can preview
> > >>>>> all the
> > >>>>> > > > > content of the website without synchronizing contents from
> > the
> > >>>>> > > > > apache/pulsar repo.
> > >>>>> > > > >
> > >>>>> > > > > Thanks,
> > >>>>> > > > > Yunze
> > >>>>> > > > >
> > >>>>> > > > > On Mon, Dec 19, 2022 at 9:53 AM Li Li <ur...@apache.org>
> > >>>>> wrote:
> > >>>>> > > > > >
> > >>>>> > > > > > +1, That’s a good idea.
> > >>>>> > > > > >
> > >>>>> > > > > > > On Dec 16, 2022, at 07:07, tison <wander4096@gmail.com
> >
> > >>>>> wrote:
> > >>>>> > > > > > >
> > >>>>> > > > > > > Hi,
> > >>>>> > > > > > >
> > >>>>> > > > > > > After several works around the build flow of our
> official
> > >>>>> > > > > website[1][2][3],
> > >>>>> > > > > > > the content sync and site build flow is debuggable and
> > >>>>> > reproducible
> > >>>>> > > > > now.
> > >>>>> > > > > > >
> > >>>>> > > > > > > However, compared to other Apache projects' websites'
> > >>>>> project
> > >>>>> > > layouts
> > >>>>> > > > > and
> > >>>>> > > > > > > workflow, we still meet two challenges on the Pulsar
> > site:
> > >>>>> > > > > > >
> > >>>>> > > > > > > 1. We don't have a pre-commit workflow for any
> > >>>>> website-related
> > >>>>> > > > changes.
> > >>>>> > > > > > > Thus, we don't detect broken links or syntax errors
> when
> > >>>>> > reviewing
> > >>>>> > > > new
> > >>>>> > > > > > > patches[4][5][6].
> > >>>>> > > > > > > 2. The website's content is two-level down in
> > >>>>> > `site2/website-next`
> > >>>>> > > > for
> > >>>>> > > > > > > historical reasons, which is confusing for
> contributors.
> > >>>>> > > > > > >
> > >>>>> > > > > > > To overcome these two shortcomings, I propose the
> > >>>>> following:
> > >>>>> > > > > > >
> > >>>>> > > > > > > 1. Move the website's content to the root level, then
> we
> > >>>>> have a
> > >>>>> > > > > first-class
> > >>>>> > > > > > > Docu&yarn-based JS project layout. It's more convenient
> > and
> > >>>>> > > familiar
> > >>>>> > > > to
> > >>>>> > > > > > > related developers.
> > >>>>> > > > > > > 2. Host the source of docs in the site repo
> > >>>>> (apache/pulsar-site)
> > >>>>> > > > > instead of
> > >>>>> > > > > > > under `site2` folder in the main repo and do content
> > sync.
> > >>>>> > > > > > >
> > >>>>> > > > > > > Below are the pros and cons:
> > >>>>> > > > > > >
> > >>>>> > > > > > > Pros
> > >>>>> > > > > > >
> > >>>>> > > > > > > 1. Obviously, we have the pre-commit workflow now. And
> > >>>>> since we
> > >>>>> > > host
> > >>>>> > > > > the
> > >>>>> > > > > > > source of docs in one repo, we don't have to run the
> > >>>>> pre-commit
> > >>>>> > > > > workflow in
> > >>>>> > > > > > > two places, which can be quite cumbersome to implement.
> > >>>>> > > > > > > 2. The size of the source release of the main repo can
> be
> > >>>>> > reduced a
> > >>>>> > > > > lot.
> > >>>>> > > > > > > Currently, 63MB out of 140MB of the sources are taken
> by
> > >>>>> the
> > >>>>> > site2
> > >>>>> > > > > folder,
> > >>>>> > > > > > > which we can remove totally. In addition, we carry out
> > >>>>> > > full-versioned
> > >>>>> > > > > docs
> > >>>>> > > > > > > every release.
> > >>>>> > > > > > > 3. We can clean up a large portion of "integration" to
> > >>>>> debug the
> > >>>>> > > site
> > >>>>> > > > > > > brittlely on the main repo[7]  (etc.) and redundant
> > >>>>> contribution
> > >>>>> > > > > guide[8].
> > >>>>> > > > > > > This way, when updating docs, we can preview the result
> > in
> > >>>>> one
> > >>>>> > repo
> > >>>>> > > > > instead
> > >>>>> > > > > > > of actually doing the sync on the fly. In addition,
> this
> > >>>>> > > integration
> > >>>>> > > > > blocks
> > >>>>> > > > > > > we move the website content to the top level since it
> > makes
> > >>>>> > strong
> > >>>>> > > > > > > assumptions about the relative layout.
> > >>>>> > > > > > >
> > >>>>> > > > > > > Cons
> > >>>>> > > > > > >
> > >>>>> > > > > > > The most significant con is that we cannot update the
> > code
> > >>>>> and
> > >>>>> > docs
> > >>>>> > > > in
> > >>>>> > > > > one
> > >>>>> > > > > > > patch against apache/pulsar now. You must open a new
> pull
> > >>>>> request
> > >>>>> > > to
> > >>>>> > > > > > > apache/pulsar-site, cross-reference each other and
> manage
> > >>>>> the
> > >>>>> > merge
> > >>>>> > > > > order
> > >>>>> > > > > > > (synchronization).
> > >>>>> > > > > > >
> > >>>>> > > > > > > Alternatives:
> > >>>>> > > > > > >
> > >>>>> > > > > > > To resolve the versioned docs issue, an alternative is
> to
> > >>>>> host
> > >>>>> > only
> > >>>>> > > > the
> > >>>>> > > > > > > user docs along with each version, like Flink does[9].
> > But
> > >>>>> it
> > >>>>> > both
> > >>>>> > > > > detaches
> > >>>>> > > > > > > from the Docu framework and requires significant
> > >>>>> development
> > >>>>> > > efforts.
> > >>>>> > > > > > >
> > >>>>> > > > > > > Since it can explicitly change the development flow
> (that
> > >>>>> is, you
> > >>>>> > > > > should
> > >>>>> > > > > > > now update docs separately), I am starting this
> > discussion
> > >>>>> here
> > >>>>> > to
> > >>>>> > > > > reach
> > >>>>> > > > > > > for your feedback.
> > >>>>> > > > > > >
> > >>>>> > > > > > > Welcome to leave your comments!
> > >>>>> > > > > > >
> > >>>>> > > > > > > Best,
> > >>>>> > > > > > > tison.
> > >>>>> > > > > > >
> > >>>>> > > > > > > [1] https://pulsar.apache.org/
> > >>>>> > > > > > > [2] https://github.com/apache/pulsar-site
> > >>>>> > > > > > > [3] https://github.com/apache/pulsar/issues/18014
> > >>>>> > > > > > > [4] https://github.com/apache/pulsar/issues/17599
> > >>>>> > > > > > > [5]
> > >>>>> > > >
> > >>>>> https://github.com/apache/pulsar/pull/17863#discussion_r990174850
> > >>>>> > > > > > > [6]
> > >>>>> > > >
> > >>>>> https://github.com/apache/pulsar/pull/17853#discussion_r991803704
> > >>>>> > > > > > > [7]
> > >>>>> > > > > > >
> > >>>>> > > > >
> > >>>>> > > >
> > >>>>> > >
> > >>>>> >
> > >>>>>
> >
> https://github.com/apache/pulsar/blob/b1f9e351fa4d5aba197d33cfc0c536516b55b61f/site2/website/start.sh
> > >>>>> > > > > > > [8]
> > >>>>> > > > > > >
> > >>>>> > > > >
> > >>>>> > > >
> > >>>>> > >
> > >>>>> >
> > >>>>>
> >
> https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
> > >>>>> > > > > > > [9] https://github.com/apache/flink/tree/master/docs
> > >>>>> > > > > >
> > >>>>> > > > >
> > >>>>> > > >
> > >>>>> > >
> > >>>>> >
> > >>>>>
> > >>>>
> >
>

Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

Posted by Yu <li...@apache.org>.
Thanks tison!

It's better to
a) update the contribution processes of docs, website content, and release
notes [1] [2] [3] [4] [5]
b) inform the community about this breaking change officially in another
email (reminders for doc contributions)

~~~~~~~~

docs:
[1] https://pulsar.apache.org/contribute/document-contribution/
[2]
https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
[3]
https://pulsar.apache.org/contribute/document-preview/#preview-website-changes

release notes:
[4] https://pulsar.apache.org/contribute/release-process/#update-the-site
[5] https://pulsar.apache.org/contribute/release-note-guide/

On Thu, Dec 29, 2022 at 9:07 PM tison <wa...@gmail.com> wrote:

> The new repo is reduced by 100MB:
>
> $ gh repo clone apache/pulsar -- --single-branch --depth=1
> $ du -sh pulsar | sort -rh
>  53M pulsar
>
> Best,
> tison.
>
>
> tison <wa...@gmail.com> 于2022年12月29日周四 21:01写道:
>
> > Landed.
> >
> > Best,
> > tison.
> >
> >
> > tison <wa...@gmail.com> 于2022年12月29日周四 17:51写道:
> >
> >> Here are the related PRs:
> >>
> >> * https://github.com/apache/pulsar/pull/19100
> >> * https://github.com/apache/pulsar-site/pull/348
> >>
> >> Best,
> >> tison.
> >>
> >>
> >> tison <wa...@gmail.com> 于2022年12月26日周一 21:45写道:
> >>
> >>> FYI tracking issue has been created:
> >>> https://github.com/apache/pulsar/issues/19064
> >>>
> >>> I plan to finish it by the end of next month.
> >>>
> >>> Best,
> >>> tison.
> >>>
> >>>
> >>> tison <wa...@gmail.com> 于2022年12月21日周三 11:33写道:
> >>>
> >>>> Thanks for your feedback!
> >>>>
> >>>> @Yu
> >>>>
> >>>> Thanks for sharing the previous thread. I looped in @michaeljmarshall
> >>>> here.
> >>>>
> >>>> @Jun
> >>>>
> >>>> It's possible but causes a new shortcoming: Now you should tell the
> >>>> contributor that the versioned docs are different from the NEXT
> version
> >>>> docs, lol.
> >>>>
> >>>> If our developers don't complain about these separated sources. Like
> @Asaf
> >>>> comment:
> >>>>
> >>>> > We can take, let's say, five features and see if they were actually
> >>>> done in
> >>>> > the same PR or separate PR. I guess that most documentation is
> >>>> actually
> >>>> > updated separately. Thus, from that perspective, maybe it’s not a
> con.
> >>>>
> >>>> Then we can do this refactor thoroughgoing.
> >>>>
> >>>> Also, if we keep, somehow several sources in the main repo. We still
> >>>> have shortcomings:
> >>>>
> >>>> 1. Duplicated CI workflows.
> >>>> 2. Cumbersome preview scaffolding in the main repo.
> >>>>
> >>>> ... which is the original purpose I'd like to overcome.
> >>>>
> >>>> Best,
> >>>> tison.
> >>>>
> >>>>
> >>>> Jun Ma <mo...@hotmail.com> 于2022年12月21日周三 11:19写道:
> >>>>
> >>>>> Is it possible to come up with a compromised solution that has the
> >>>>> pros of both sides but minimizes the side effect? I'm thinking maybe
> it's
> >>>>> not necessary to sacrifice the current contribution process, as long
> as it
> >>>>> can greatly reduce the load of back-end actions and source size. For
> >>>>> example, if we only move out the versioned docs to the site repo but
> keep
> >>>>> the source of the NEXT docs in the pulsar repo, does this help to
> win a
> >>>>> large proportion of those pros when people can still contribute as
> usual?
> >>>>>
> >>>>> ________________________________
> >>>>> From: Jiaqi Shen <gl...@gmail.com>
> >>>>> Sent: Tuesday, December 20, 2022 17:15
> >>>>> To: dev@pulsar.apache.org <de...@pulsar.apache.org>
> >>>>> Subject: Re: [PROPOSAL] Website precommit and move the source of docs
> >>>>> to the site repo
> >>>>>
> >>>>> +1, it makes sense to me.
> >>>>>
> >>>>> Thanks,
> >>>>> Jiaqi Shen
> >>>>>
> >>>>>
> >>>>> Yu <li...@apache.org> 于2022年12月19日周一 20:57写道:
> >>>>>
> >>>>> > Hi tison,
> >>>>> >
> >>>>> > Thanks for raising this up!
> >>>>> >
> >>>>> > Our community had a similar discussion previously and chose to
> >>>>> "keep" the
> >>>>> > doc repo stay in the Pulsar main repo at that time.
> >>>>> >
> >>>>> > [1] lists the pros and cons of "keep" and "not keep" solutions.
> >>>>> >
> >>>>> > I'm +0 on this proposal because I think the total scores of these
> two
> >>>>> > solutions are almost equal after weighing the pros and cons.
> >>>>> >
> >>>>> > ~~~~~~~~~~~~~~~~~~~~
> >>>>> >
> >>>>> > [1]
> https://lists.apache.org/thread/mf2xwntfgn84dq78ksqv22jk3drq6xb3
> >>>>> >
> >>>>> >
> >>>>> > On Mon, Dec 19, 2022 at 5:40 PM tison <wa...@gmail.com>
> wrote:
> >>>>> >
> >>>>> > > Thanks for your feedback!
> >>>>> > >
> >>>>> > > @Asaf
> >>>>> > >
> >>>>> > > > pre-commit
> >>>>> > >
> >>>>> > > I mean CI checks before merging a patch. Currently, we don't run
> >>>>> checks
> >>>>> > for
> >>>>> > > the content before merging them. This causes a series of syntax
> >>>>> errors
> >>>>> > and
> >>>>> > > broken links issues. If we hold docs under site2 folder in the
> >>>>> main repo
> >>>>> > > and then copied to the site repo, we have two places to build
> such
> >>>>> CI
> >>>>> > > checks. What's worse, the checks for the main repo will be quite
> >>>>> > > cumbersome (that you do some if-else logic in the whole Pulsar CI
> >>>>> > > workflows, and do the sync sequentially in that workflow).
> >>>>> > >
> >>>>> > > If we hold the source of docs only in the site repo, we can
> extend
> >>>>> the
> >>>>> > > "precommit" workflow[1] I added recently to check for syntax
> >>>>> errors and
> >>>>> > > broken links also.
> >>>>> > >
> >>>>> > > > What does the apache/pulsar-site repo contain today?
> >>>>> > >
> >>>>> > > It should be covered by the documentation guide page[2]. It holds
> >>>>> the
> >>>>> > > source of the official website and the user docs are synced from
> >>>>> the main
> >>>>> > > repo.
> >>>>> > >
> >>>>> > > > What content do we have today in the pulsar repo related to the
> >>>>> site?
> >>>>> > >
> >>>>> > > After issue-18014[3] is done, we host only user docs and some
> JSON
> >>>>> > metadata
> >>>>> > > in the main repo, which is synced by site_syncer.py[4].
> >>>>> > >
> >>>>> > > > Can you explain that better? Are you saying pulsar source JARs
> >>>>> contain
> >>>>> > > the documentation?
> >>>>> > >
> >>>>> > > No. Source JARs contain only the Java files and necessary
> >>>>> copyrights
> >>>>> > info.
> >>>>> > > The source release is, for example,
> >>>>> > >
> >>>>> > >
> >>>>> >
> >>>>>
> https://archive.apache.org/dist/pulsar/pulsar-2.10.2/apache-pulsar-2.10.2-src.tar.gz
> >>>>> > > ,
> >>>>> > > which is extracted to 173M where 129M is occupied by the site2
> >>>>> folder.
> >>>>> > >
> >>>>> > > This also affects when developers do git clone to clone the repo.
> >>>>> > >
> >>>>> > > > I mean, if you wish to document a bug fix in 2.9.x, for
> example,
> >>>>> would
> >>>>> > > you do it in the 2.9.x branch under site2/docs or
> >>>>> > > site2/website/versioned_docs/2.9.5?
> >>>>> > >
> >>>>> > > This is another question. Ideally, we should have hosted
> versioned
> >>>>> docs
> >>>>> > > associated with the specific version to that branch, like Apache
> >>>>> Flink
> >>>>> > does
> >>>>> > > as I mentioned[5]. But we do not, and actually the situation is
> we
> >>>>> update
> >>>>> > > the versioned docs under the master branch and thus, the docs can
> >>>>> be
> >>>>> > synced
> >>>>> > > properly.
> >>>>> > >
> >>>>> > > See also the "Alternatives" section in the original email.
> >>>>> > >
> >>>>> > > @All
> >>>>> > >
> >>>>> > > Since we don't have objections to the possible cons listed above
> >>>>> or any
> >>>>> > new
> >>>>> > > ones, I'm going to create a tracking issue later this week and
> >>>>> show what
> >>>>> > > will be changed in PRs for further review.
> >>>>> > >
> >>>>> > > Best,
> >>>>> > > tison.
> >>>>> > >
> >>>>> > > [1]
> >>>>> > >
> >>>>> > >
> >>>>> >
> >>>>>
> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/.github/workflows/ci-precommit.yml
> >>>>> > > [2]
> >>>>> > >
> >>>>> > >
> >>>>> >
> >>>>>
> https://pulsar.apache.org/contribute/document-contribution/#source-repositories
> >>>>> > > [3] https://github.com/apache/pulsar/issues/18014
> >>>>> > > [4]
> >>>>> > >
> >>>>> > >
> >>>>> >
> >>>>>
> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/tools/pytools/lib/execute/site_syncer.py
> >>>>> > > [5] https://github.com/apache/flink/tree/master/docs
> >>>>> > >
> >>>>> > >
> >>>>> > > PengHui Li <pe...@apache.org> 于2022年12月19日周一 16:26写道:
> >>>>> > >
> >>>>> > > > +1
> >>>>> > > >
> >>>>> > > > I support moving them to the website repo.
> >>>>> > > >
> >>>>> > > > Thanks,
> >>>>> > > > Penghui
> >>>>> > > >
> >>>>> > > > On Mon, Dec 19, 2022 at 12:04 PM Yunze Xu
> >>>>> <yzxu@streamnative.io.invalid
> >>>>> > >
> >>>>> > > > wrote:
> >>>>> > > >
> >>>>> > > > > +1. The most significant point to me is that we can preview
> >>>>> all the
> >>>>> > > > > content of the website without synchronizing contents from
> the
> >>>>> > > > > apache/pulsar repo.
> >>>>> > > > >
> >>>>> > > > > Thanks,
> >>>>> > > > > Yunze
> >>>>> > > > >
> >>>>> > > > > On Mon, Dec 19, 2022 at 9:53 AM Li Li <ur...@apache.org>
> >>>>> wrote:
> >>>>> > > > > >
> >>>>> > > > > > +1, That’s a good idea.
> >>>>> > > > > >
> >>>>> > > > > > > On Dec 16, 2022, at 07:07, tison <wa...@gmail.com>
> >>>>> wrote:
> >>>>> > > > > > >
> >>>>> > > > > > > Hi,
> >>>>> > > > > > >
> >>>>> > > > > > > After several works around the build flow of our official
> >>>>> > > > > website[1][2][3],
> >>>>> > > > > > > the content sync and site build flow is debuggable and
> >>>>> > reproducible
> >>>>> > > > > now.
> >>>>> > > > > > >
> >>>>> > > > > > > However, compared to other Apache projects' websites'
> >>>>> project
> >>>>> > > layouts
> >>>>> > > > > and
> >>>>> > > > > > > workflow, we still meet two challenges on the Pulsar
> site:
> >>>>> > > > > > >
> >>>>> > > > > > > 1. We don't have a pre-commit workflow for any
> >>>>> website-related
> >>>>> > > > changes.
> >>>>> > > > > > > Thus, we don't detect broken links or syntax errors when
> >>>>> > reviewing
> >>>>> > > > new
> >>>>> > > > > > > patches[4][5][6].
> >>>>> > > > > > > 2. The website's content is two-level down in
> >>>>> > `site2/website-next`
> >>>>> > > > for
> >>>>> > > > > > > historical reasons, which is confusing for contributors.
> >>>>> > > > > > >
> >>>>> > > > > > > To overcome these two shortcomings, I propose the
> >>>>> following:
> >>>>> > > > > > >
> >>>>> > > > > > > 1. Move the website's content to the root level, then we
> >>>>> have a
> >>>>> > > > > first-class
> >>>>> > > > > > > Docu&yarn-based JS project layout. It's more convenient
> and
> >>>>> > > familiar
> >>>>> > > > to
> >>>>> > > > > > > related developers.
> >>>>> > > > > > > 2. Host the source of docs in the site repo
> >>>>> (apache/pulsar-site)
> >>>>> > > > > instead of
> >>>>> > > > > > > under `site2` folder in the main repo and do content
> sync.
> >>>>> > > > > > >
> >>>>> > > > > > > Below are the pros and cons:
> >>>>> > > > > > >
> >>>>> > > > > > > Pros
> >>>>> > > > > > >
> >>>>> > > > > > > 1. Obviously, we have the pre-commit workflow now. And
> >>>>> since we
> >>>>> > > host
> >>>>> > > > > the
> >>>>> > > > > > > source of docs in one repo, we don't have to run the
> >>>>> pre-commit
> >>>>> > > > > workflow in
> >>>>> > > > > > > two places, which can be quite cumbersome to implement.
> >>>>> > > > > > > 2. The size of the source release of the main repo can be
> >>>>> > reduced a
> >>>>> > > > > lot.
> >>>>> > > > > > > Currently, 63MB out of 140MB of the sources are taken by
> >>>>> the
> >>>>> > site2
> >>>>> > > > > folder,
> >>>>> > > > > > > which we can remove totally. In addition, we carry out
> >>>>> > > full-versioned
> >>>>> > > > > docs
> >>>>> > > > > > > every release.
> >>>>> > > > > > > 3. We can clean up a large portion of "integration" to
> >>>>> debug the
> >>>>> > > site
> >>>>> > > > > > > brittlely on the main repo[7]  (etc.) and redundant
> >>>>> contribution
> >>>>> > > > > guide[8].
> >>>>> > > > > > > This way, when updating docs, we can preview the result
> in
> >>>>> one
> >>>>> > repo
> >>>>> > > > > instead
> >>>>> > > > > > > of actually doing the sync on the fly. In addition, this
> >>>>> > > integration
> >>>>> > > > > blocks
> >>>>> > > > > > > we move the website content to the top level since it
> makes
> >>>>> > strong
> >>>>> > > > > > > assumptions about the relative layout.
> >>>>> > > > > > >
> >>>>> > > > > > > Cons
> >>>>> > > > > > >
> >>>>> > > > > > > The most significant con is that we cannot update the
> code
> >>>>> and
> >>>>> > docs
> >>>>> > > > in
> >>>>> > > > > one
> >>>>> > > > > > > patch against apache/pulsar now. You must open a new pull
> >>>>> request
> >>>>> > > to
> >>>>> > > > > > > apache/pulsar-site, cross-reference each other and manage
> >>>>> the
> >>>>> > merge
> >>>>> > > > > order
> >>>>> > > > > > > (synchronization).
> >>>>> > > > > > >
> >>>>> > > > > > > Alternatives:
> >>>>> > > > > > >
> >>>>> > > > > > > To resolve the versioned docs issue, an alternative is to
> >>>>> host
> >>>>> > only
> >>>>> > > > the
> >>>>> > > > > > > user docs along with each version, like Flink does[9].
> But
> >>>>> it
> >>>>> > both
> >>>>> > > > > detaches
> >>>>> > > > > > > from the Docu framework and requires significant
> >>>>> development
> >>>>> > > efforts.
> >>>>> > > > > > >
> >>>>> > > > > > > Since it can explicitly change the development flow (that
> >>>>> is, you
> >>>>> > > > > should
> >>>>> > > > > > > now update docs separately), I am starting this
> discussion
> >>>>> here
> >>>>> > to
> >>>>> > > > > reach
> >>>>> > > > > > > for your feedback.
> >>>>> > > > > > >
> >>>>> > > > > > > Welcome to leave your comments!
> >>>>> > > > > > >
> >>>>> > > > > > > Best,
> >>>>> > > > > > > tison.
> >>>>> > > > > > >
> >>>>> > > > > > > [1] https://pulsar.apache.org/
> >>>>> > > > > > > [2] https://github.com/apache/pulsar-site
> >>>>> > > > > > > [3] https://github.com/apache/pulsar/issues/18014
> >>>>> > > > > > > [4] https://github.com/apache/pulsar/issues/17599
> >>>>> > > > > > > [5]
> >>>>> > > >
> >>>>> https://github.com/apache/pulsar/pull/17863#discussion_r990174850
> >>>>> > > > > > > [6]
> >>>>> > > >
> >>>>> https://github.com/apache/pulsar/pull/17853#discussion_r991803704
> >>>>> > > > > > > [7]
> >>>>> > > > > > >
> >>>>> > > > >
> >>>>> > > >
> >>>>> > >
> >>>>> >
> >>>>>
> https://github.com/apache/pulsar/blob/b1f9e351fa4d5aba197d33cfc0c536516b55b61f/site2/website/start.sh
> >>>>> > > > > > > [8]
> >>>>> > > > > > >
> >>>>> > > > >
> >>>>> > > >
> >>>>> > >
> >>>>> >
> >>>>>
> https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
> >>>>> > > > > > > [9] https://github.com/apache/flink/tree/master/docs
> >>>>> > > > > >
> >>>>> > > > >
> >>>>> > > >
> >>>>> > >
> >>>>> >
> >>>>>
> >>>>
>

Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

Posted by tison <wa...@gmail.com>.
The new repo is reduced by 100MB:

$ gh repo clone apache/pulsar -- --single-branch --depth=1
$ du -sh pulsar | sort -rh
 53M pulsar

Best,
tison.


tison <wa...@gmail.com> 于2022年12月29日周四 21:01写道:

> Landed.
>
> Best,
> tison.
>
>
> tison <wa...@gmail.com> 于2022年12月29日周四 17:51写道:
>
>> Here are the related PRs:
>>
>> * https://github.com/apache/pulsar/pull/19100
>> * https://github.com/apache/pulsar-site/pull/348
>>
>> Best,
>> tison.
>>
>>
>> tison <wa...@gmail.com> 于2022年12月26日周一 21:45写道:
>>
>>> FYI tracking issue has been created:
>>> https://github.com/apache/pulsar/issues/19064
>>>
>>> I plan to finish it by the end of next month.
>>>
>>> Best,
>>> tison.
>>>
>>>
>>> tison <wa...@gmail.com> 于2022年12月21日周三 11:33写道:
>>>
>>>> Thanks for your feedback!
>>>>
>>>> @Yu
>>>>
>>>> Thanks for sharing the previous thread. I looped in @michaeljmarshall
>>>> here.
>>>>
>>>> @Jun
>>>>
>>>> It's possible but causes a new shortcoming: Now you should tell the
>>>> contributor that the versioned docs are different from the NEXT version
>>>> docs, lol.
>>>>
>>>> If our developers don't complain about these separated sources. Like @Asaf
>>>> comment:
>>>>
>>>> > We can take, let's say, five features and see if they were actually
>>>> done in
>>>> > the same PR or separate PR. I guess that most documentation is
>>>> actually
>>>> > updated separately. Thus, from that perspective, maybe it’s not a con.
>>>>
>>>> Then we can do this refactor thoroughgoing.
>>>>
>>>> Also, if we keep, somehow several sources in the main repo. We still
>>>> have shortcomings:
>>>>
>>>> 1. Duplicated CI workflows.
>>>> 2. Cumbersome preview scaffolding in the main repo.
>>>>
>>>> ... which is the original purpose I'd like to overcome.
>>>>
>>>> Best,
>>>> tison.
>>>>
>>>>
>>>> Jun Ma <mo...@hotmail.com> 于2022年12月21日周三 11:19写道:
>>>>
>>>>> Is it possible to come up with a compromised solution that has the
>>>>> pros of both sides but minimizes the side effect? I'm thinking maybe it's
>>>>> not necessary to sacrifice the current contribution process, as long as it
>>>>> can greatly reduce the load of back-end actions and source size. For
>>>>> example, if we only move out the versioned docs to the site repo but keep
>>>>> the source of the NEXT docs in the pulsar repo, does this help to win a
>>>>> large proportion of those pros when people can still contribute as usual?
>>>>>
>>>>> ________________________________
>>>>> From: Jiaqi Shen <gl...@gmail.com>
>>>>> Sent: Tuesday, December 20, 2022 17:15
>>>>> To: dev@pulsar.apache.org <de...@pulsar.apache.org>
>>>>> Subject: Re: [PROPOSAL] Website precommit and move the source of docs
>>>>> to the site repo
>>>>>
>>>>> +1, it makes sense to me.
>>>>>
>>>>> Thanks,
>>>>> Jiaqi Shen
>>>>>
>>>>>
>>>>> Yu <li...@apache.org> 于2022年12月19日周一 20:57写道:
>>>>>
>>>>> > Hi tison,
>>>>> >
>>>>> > Thanks for raising this up!
>>>>> >
>>>>> > Our community had a similar discussion previously and chose to
>>>>> "keep" the
>>>>> > doc repo stay in the Pulsar main repo at that time.
>>>>> >
>>>>> > [1] lists the pros and cons of "keep" and "not keep" solutions.
>>>>> >
>>>>> > I'm +0 on this proposal because I think the total scores of these two
>>>>> > solutions are almost equal after weighing the pros and cons.
>>>>> >
>>>>> > ~~~~~~~~~~~~~~~~~~~~
>>>>> >
>>>>> > [1] https://lists.apache.org/thread/mf2xwntfgn84dq78ksqv22jk3drq6xb3
>>>>> >
>>>>> >
>>>>> > On Mon, Dec 19, 2022 at 5:40 PM tison <wa...@gmail.com> wrote:
>>>>> >
>>>>> > > Thanks for your feedback!
>>>>> > >
>>>>> > > @Asaf
>>>>> > >
>>>>> > > > pre-commit
>>>>> > >
>>>>> > > I mean CI checks before merging a patch. Currently, we don't run
>>>>> checks
>>>>> > for
>>>>> > > the content before merging them. This causes a series of syntax
>>>>> errors
>>>>> > and
>>>>> > > broken links issues. If we hold docs under site2 folder in the
>>>>> main repo
>>>>> > > and then copied to the site repo, we have two places to build such
>>>>> CI
>>>>> > > checks. What's worse, the checks for the main repo will be quite
>>>>> > > cumbersome (that you do some if-else logic in the whole Pulsar CI
>>>>> > > workflows, and do the sync sequentially in that workflow).
>>>>> > >
>>>>> > > If we hold the source of docs only in the site repo, we can extend
>>>>> the
>>>>> > > "precommit" workflow[1] I added recently to check for syntax
>>>>> errors and
>>>>> > > broken links also.
>>>>> > >
>>>>> > > > What does the apache/pulsar-site repo contain today?
>>>>> > >
>>>>> > > It should be covered by the documentation guide page[2]. It holds
>>>>> the
>>>>> > > source of the official website and the user docs are synced from
>>>>> the main
>>>>> > > repo.
>>>>> > >
>>>>> > > > What content do we have today in the pulsar repo related to the
>>>>> site?
>>>>> > >
>>>>> > > After issue-18014[3] is done, we host only user docs and some JSON
>>>>> > metadata
>>>>> > > in the main repo, which is synced by site_syncer.py[4].
>>>>> > >
>>>>> > > > Can you explain that better? Are you saying pulsar source JARs
>>>>> contain
>>>>> > > the documentation?
>>>>> > >
>>>>> > > No. Source JARs contain only the Java files and necessary
>>>>> copyrights
>>>>> > info.
>>>>> > > The source release is, for example,
>>>>> > >
>>>>> > >
>>>>> >
>>>>> https://archive.apache.org/dist/pulsar/pulsar-2.10.2/apache-pulsar-2.10.2-src.tar.gz
>>>>> > > ,
>>>>> > > which is extracted to 173M where 129M is occupied by the site2
>>>>> folder.
>>>>> > >
>>>>> > > This also affects when developers do git clone to clone the repo.
>>>>> > >
>>>>> > > > I mean, if you wish to document a bug fix in 2.9.x, for example,
>>>>> would
>>>>> > > you do it in the 2.9.x branch under site2/docs or
>>>>> > > site2/website/versioned_docs/2.9.5?
>>>>> > >
>>>>> > > This is another question. Ideally, we should have hosted versioned
>>>>> docs
>>>>> > > associated with the specific version to that branch, like Apache
>>>>> Flink
>>>>> > does
>>>>> > > as I mentioned[5]. But we do not, and actually the situation is we
>>>>> update
>>>>> > > the versioned docs under the master branch and thus, the docs can
>>>>> be
>>>>> > synced
>>>>> > > properly.
>>>>> > >
>>>>> > > See also the "Alternatives" section in the original email.
>>>>> > >
>>>>> > > @All
>>>>> > >
>>>>> > > Since we don't have objections to the possible cons listed above
>>>>> or any
>>>>> > new
>>>>> > > ones, I'm going to create a tracking issue later this week and
>>>>> show what
>>>>> > > will be changed in PRs for further review.
>>>>> > >
>>>>> > > Best,
>>>>> > > tison.
>>>>> > >
>>>>> > > [1]
>>>>> > >
>>>>> > >
>>>>> >
>>>>> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/.github/workflows/ci-precommit.yml
>>>>> > > [2]
>>>>> > >
>>>>> > >
>>>>> >
>>>>> https://pulsar.apache.org/contribute/document-contribution/#source-repositories
>>>>> > > [3] https://github.com/apache/pulsar/issues/18014
>>>>> > > [4]
>>>>> > >
>>>>> > >
>>>>> >
>>>>> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/tools/pytools/lib/execute/site_syncer.py
>>>>> > > [5] https://github.com/apache/flink/tree/master/docs
>>>>> > >
>>>>> > >
>>>>> > > PengHui Li <pe...@apache.org> 于2022年12月19日周一 16:26写道:
>>>>> > >
>>>>> > > > +1
>>>>> > > >
>>>>> > > > I support moving them to the website repo.
>>>>> > > >
>>>>> > > > Thanks,
>>>>> > > > Penghui
>>>>> > > >
>>>>> > > > On Mon, Dec 19, 2022 at 12:04 PM Yunze Xu
>>>>> <yzxu@streamnative.io.invalid
>>>>> > >
>>>>> > > > wrote:
>>>>> > > >
>>>>> > > > > +1. The most significant point to me is that we can preview
>>>>> all the
>>>>> > > > > content of the website without synchronizing contents from the
>>>>> > > > > apache/pulsar repo.
>>>>> > > > >
>>>>> > > > > Thanks,
>>>>> > > > > Yunze
>>>>> > > > >
>>>>> > > > > On Mon, Dec 19, 2022 at 9:53 AM Li Li <ur...@apache.org>
>>>>> wrote:
>>>>> > > > > >
>>>>> > > > > > +1, That’s a good idea.
>>>>> > > > > >
>>>>> > > > > > > On Dec 16, 2022, at 07:07, tison <wa...@gmail.com>
>>>>> wrote:
>>>>> > > > > > >
>>>>> > > > > > > Hi,
>>>>> > > > > > >
>>>>> > > > > > > After several works around the build flow of our official
>>>>> > > > > website[1][2][3],
>>>>> > > > > > > the content sync and site build flow is debuggable and
>>>>> > reproducible
>>>>> > > > > now.
>>>>> > > > > > >
>>>>> > > > > > > However, compared to other Apache projects' websites'
>>>>> project
>>>>> > > layouts
>>>>> > > > > and
>>>>> > > > > > > workflow, we still meet two challenges on the Pulsar site:
>>>>> > > > > > >
>>>>> > > > > > > 1. We don't have a pre-commit workflow for any
>>>>> website-related
>>>>> > > > changes.
>>>>> > > > > > > Thus, we don't detect broken links or syntax errors when
>>>>> > reviewing
>>>>> > > > new
>>>>> > > > > > > patches[4][5][6].
>>>>> > > > > > > 2. The website's content is two-level down in
>>>>> > `site2/website-next`
>>>>> > > > for
>>>>> > > > > > > historical reasons, which is confusing for contributors.
>>>>> > > > > > >
>>>>> > > > > > > To overcome these two shortcomings, I propose the
>>>>> following:
>>>>> > > > > > >
>>>>> > > > > > > 1. Move the website's content to the root level, then we
>>>>> have a
>>>>> > > > > first-class
>>>>> > > > > > > Docu&yarn-based JS project layout. It's more convenient and
>>>>> > > familiar
>>>>> > > > to
>>>>> > > > > > > related developers.
>>>>> > > > > > > 2. Host the source of docs in the site repo
>>>>> (apache/pulsar-site)
>>>>> > > > > instead of
>>>>> > > > > > > under `site2` folder in the main repo and do content sync.
>>>>> > > > > > >
>>>>> > > > > > > Below are the pros and cons:
>>>>> > > > > > >
>>>>> > > > > > > Pros
>>>>> > > > > > >
>>>>> > > > > > > 1. Obviously, we have the pre-commit workflow now. And
>>>>> since we
>>>>> > > host
>>>>> > > > > the
>>>>> > > > > > > source of docs in one repo, we don't have to run the
>>>>> pre-commit
>>>>> > > > > workflow in
>>>>> > > > > > > two places, which can be quite cumbersome to implement.
>>>>> > > > > > > 2. The size of the source release of the main repo can be
>>>>> > reduced a
>>>>> > > > > lot.
>>>>> > > > > > > Currently, 63MB out of 140MB of the sources are taken by
>>>>> the
>>>>> > site2
>>>>> > > > > folder,
>>>>> > > > > > > which we can remove totally. In addition, we carry out
>>>>> > > full-versioned
>>>>> > > > > docs
>>>>> > > > > > > every release.
>>>>> > > > > > > 3. We can clean up a large portion of "integration" to
>>>>> debug the
>>>>> > > site
>>>>> > > > > > > brittlely on the main repo[7]  (etc.) and redundant
>>>>> contribution
>>>>> > > > > guide[8].
>>>>> > > > > > > This way, when updating docs, we can preview the result in
>>>>> one
>>>>> > repo
>>>>> > > > > instead
>>>>> > > > > > > of actually doing the sync on the fly. In addition, this
>>>>> > > integration
>>>>> > > > > blocks
>>>>> > > > > > > we move the website content to the top level since it makes
>>>>> > strong
>>>>> > > > > > > assumptions about the relative layout.
>>>>> > > > > > >
>>>>> > > > > > > Cons
>>>>> > > > > > >
>>>>> > > > > > > The most significant con is that we cannot update the code
>>>>> and
>>>>> > docs
>>>>> > > > in
>>>>> > > > > one
>>>>> > > > > > > patch against apache/pulsar now. You must open a new pull
>>>>> request
>>>>> > > to
>>>>> > > > > > > apache/pulsar-site, cross-reference each other and manage
>>>>> the
>>>>> > merge
>>>>> > > > > order
>>>>> > > > > > > (synchronization).
>>>>> > > > > > >
>>>>> > > > > > > Alternatives:
>>>>> > > > > > >
>>>>> > > > > > > To resolve the versioned docs issue, an alternative is to
>>>>> host
>>>>> > only
>>>>> > > > the
>>>>> > > > > > > user docs along with each version, like Flink does[9]. But
>>>>> it
>>>>> > both
>>>>> > > > > detaches
>>>>> > > > > > > from the Docu framework and requires significant
>>>>> development
>>>>> > > efforts.
>>>>> > > > > > >
>>>>> > > > > > > Since it can explicitly change the development flow (that
>>>>> is, you
>>>>> > > > > should
>>>>> > > > > > > now update docs separately), I am starting this discussion
>>>>> here
>>>>> > to
>>>>> > > > > reach
>>>>> > > > > > > for your feedback.
>>>>> > > > > > >
>>>>> > > > > > > Welcome to leave your comments!
>>>>> > > > > > >
>>>>> > > > > > > Best,
>>>>> > > > > > > tison.
>>>>> > > > > > >
>>>>> > > > > > > [1] https://pulsar.apache.org/
>>>>> > > > > > > [2] https://github.com/apache/pulsar-site
>>>>> > > > > > > [3] https://github.com/apache/pulsar/issues/18014
>>>>> > > > > > > [4] https://github.com/apache/pulsar/issues/17599
>>>>> > > > > > > [5]
>>>>> > > >
>>>>> https://github.com/apache/pulsar/pull/17863#discussion_r990174850
>>>>> > > > > > > [6]
>>>>> > > >
>>>>> https://github.com/apache/pulsar/pull/17853#discussion_r991803704
>>>>> > > > > > > [7]
>>>>> > > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > >
>>>>> >
>>>>> https://github.com/apache/pulsar/blob/b1f9e351fa4d5aba197d33cfc0c536516b55b61f/site2/website/start.sh
>>>>> > > > > > > [8]
>>>>> > > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > >
>>>>> >
>>>>> https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
>>>>> > > > > > > [9] https://github.com/apache/flink/tree/master/docs
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > >
>>>>> >
>>>>>
>>>>

Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

Posted by tison <wa...@gmail.com>.
Landed.

Best,
tison.


tison <wa...@gmail.com> 于2022年12月29日周四 17:51写道:

> Here are the related PRs:
>
> * https://github.com/apache/pulsar/pull/19100
> * https://github.com/apache/pulsar-site/pull/348
>
> Best,
> tison.
>
>
> tison <wa...@gmail.com> 于2022年12月26日周一 21:45写道:
>
>> FYI tracking issue has been created:
>> https://github.com/apache/pulsar/issues/19064
>>
>> I plan to finish it by the end of next month.
>>
>> Best,
>> tison.
>>
>>
>> tison <wa...@gmail.com> 于2022年12月21日周三 11:33写道:
>>
>>> Thanks for your feedback!
>>>
>>> @Yu
>>>
>>> Thanks for sharing the previous thread. I looped in @michaeljmarshall
>>> here.
>>>
>>> @Jun
>>>
>>> It's possible but causes a new shortcoming: Now you should tell the
>>> contributor that the versioned docs are different from the NEXT version
>>> docs, lol.
>>>
>>> If our developers don't complain about these separated sources. Like @Asaf
>>> comment:
>>>
>>> > We can take, let's say, five features and see if they were actually
>>> done in
>>> > the same PR or separate PR. I guess that most documentation is actually
>>> > updated separately. Thus, from that perspective, maybe it’s not a con.
>>>
>>> Then we can do this refactor thoroughgoing.
>>>
>>> Also, if we keep, somehow several sources in the main repo. We still
>>> have shortcomings:
>>>
>>> 1. Duplicated CI workflows.
>>> 2. Cumbersome preview scaffolding in the main repo.
>>>
>>> ... which is the original purpose I'd like to overcome.
>>>
>>> Best,
>>> tison.
>>>
>>>
>>> Jun Ma <mo...@hotmail.com> 于2022年12月21日周三 11:19写道:
>>>
>>>> Is it possible to come up with a compromised solution that has the pros
>>>> of both sides but minimizes the side effect? I'm thinking maybe it's not
>>>> necessary to sacrifice the current contribution process, as long as it can
>>>> greatly reduce the load of back-end actions and source size. For example,
>>>> if we only move out the versioned docs to the site repo but keep the source
>>>> of the NEXT docs in the pulsar repo, does this help to win a large
>>>> proportion of those pros when people can still contribute as usual?
>>>>
>>>> ________________________________
>>>> From: Jiaqi Shen <gl...@gmail.com>
>>>> Sent: Tuesday, December 20, 2022 17:15
>>>> To: dev@pulsar.apache.org <de...@pulsar.apache.org>
>>>> Subject: Re: [PROPOSAL] Website precommit and move the source of docs
>>>> to the site repo
>>>>
>>>> +1, it makes sense to me.
>>>>
>>>> Thanks,
>>>> Jiaqi Shen
>>>>
>>>>
>>>> Yu <li...@apache.org> 于2022年12月19日周一 20:57写道:
>>>>
>>>> > Hi tison,
>>>> >
>>>> > Thanks for raising this up!
>>>> >
>>>> > Our community had a similar discussion previously and chose to "keep"
>>>> the
>>>> > doc repo stay in the Pulsar main repo at that time.
>>>> >
>>>> > [1] lists the pros and cons of "keep" and "not keep" solutions.
>>>> >
>>>> > I'm +0 on this proposal because I think the total scores of these two
>>>> > solutions are almost equal after weighing the pros and cons.
>>>> >
>>>> > ~~~~~~~~~~~~~~~~~~~~
>>>> >
>>>> > [1] https://lists.apache.org/thread/mf2xwntfgn84dq78ksqv22jk3drq6xb3
>>>> >
>>>> >
>>>> > On Mon, Dec 19, 2022 at 5:40 PM tison <wa...@gmail.com> wrote:
>>>> >
>>>> > > Thanks for your feedback!
>>>> > >
>>>> > > @Asaf
>>>> > >
>>>> > > > pre-commit
>>>> > >
>>>> > > I mean CI checks before merging a patch. Currently, we don't run
>>>> checks
>>>> > for
>>>> > > the content before merging them. This causes a series of syntax
>>>> errors
>>>> > and
>>>> > > broken links issues. If we hold docs under site2 folder in the main
>>>> repo
>>>> > > and then copied to the site repo, we have two places to build such
>>>> CI
>>>> > > checks. What's worse, the checks for the main repo will be quite
>>>> > > cumbersome (that you do some if-else logic in the whole Pulsar CI
>>>> > > workflows, and do the sync sequentially in that workflow).
>>>> > >
>>>> > > If we hold the source of docs only in the site repo, we can extend
>>>> the
>>>> > > "precommit" workflow[1] I added recently to check for syntax errors
>>>> and
>>>> > > broken links also.
>>>> > >
>>>> > > > What does the apache/pulsar-site repo contain today?
>>>> > >
>>>> > > It should be covered by the documentation guide page[2]. It holds
>>>> the
>>>> > > source of the official website and the user docs are synced from
>>>> the main
>>>> > > repo.
>>>> > >
>>>> > > > What content do we have today in the pulsar repo related to the
>>>> site?
>>>> > >
>>>> > > After issue-18014[3] is done, we host only user docs and some JSON
>>>> > metadata
>>>> > > in the main repo, which is synced by site_syncer.py[4].
>>>> > >
>>>> > > > Can you explain that better? Are you saying pulsar source JARs
>>>> contain
>>>> > > the documentation?
>>>> > >
>>>> > > No. Source JARs contain only the Java files and necessary copyrights
>>>> > info.
>>>> > > The source release is, for example,
>>>> > >
>>>> > >
>>>> >
>>>> https://archive.apache.org/dist/pulsar/pulsar-2.10.2/apache-pulsar-2.10.2-src.tar.gz
>>>> > > ,
>>>> > > which is extracted to 173M where 129M is occupied by the site2
>>>> folder.
>>>> > >
>>>> > > This also affects when developers do git clone to clone the repo.
>>>> > >
>>>> > > > I mean, if you wish to document a bug fix in 2.9.x, for example,
>>>> would
>>>> > > you do it in the 2.9.x branch under site2/docs or
>>>> > > site2/website/versioned_docs/2.9.5?
>>>> > >
>>>> > > This is another question. Ideally, we should have hosted versioned
>>>> docs
>>>> > > associated with the specific version to that branch, like Apache
>>>> Flink
>>>> > does
>>>> > > as I mentioned[5]. But we do not, and actually the situation is we
>>>> update
>>>> > > the versioned docs under the master branch and thus, the docs can be
>>>> > synced
>>>> > > properly.
>>>> > >
>>>> > > See also the "Alternatives" section in the original email.
>>>> > >
>>>> > > @All
>>>> > >
>>>> > > Since we don't have objections to the possible cons listed above or
>>>> any
>>>> > new
>>>> > > ones, I'm going to create a tracking issue later this week and show
>>>> what
>>>> > > will be changed in PRs for further review.
>>>> > >
>>>> > > Best,
>>>> > > tison.
>>>> > >
>>>> > > [1]
>>>> > >
>>>> > >
>>>> >
>>>> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/.github/workflows/ci-precommit.yml
>>>> > > [2]
>>>> > >
>>>> > >
>>>> >
>>>> https://pulsar.apache.org/contribute/document-contribution/#source-repositories
>>>> > > [3] https://github.com/apache/pulsar/issues/18014
>>>> > > [4]
>>>> > >
>>>> > >
>>>> >
>>>> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/tools/pytools/lib/execute/site_syncer.py
>>>> > > [5] https://github.com/apache/flink/tree/master/docs
>>>> > >
>>>> > >
>>>> > > PengHui Li <pe...@apache.org> 于2022年12月19日周一 16:26写道:
>>>> > >
>>>> > > > +1
>>>> > > >
>>>> > > > I support moving them to the website repo.
>>>> > > >
>>>> > > > Thanks,
>>>> > > > Penghui
>>>> > > >
>>>> > > > On Mon, Dec 19, 2022 at 12:04 PM Yunze Xu
>>>> <yzxu@streamnative.io.invalid
>>>> > >
>>>> > > > wrote:
>>>> > > >
>>>> > > > > +1. The most significant point to me is that we can preview all
>>>> the
>>>> > > > > content of the website without synchronizing contents from the
>>>> > > > > apache/pulsar repo.
>>>> > > > >
>>>> > > > > Thanks,
>>>> > > > > Yunze
>>>> > > > >
>>>> > > > > On Mon, Dec 19, 2022 at 9:53 AM Li Li <ur...@apache.org>
>>>> wrote:
>>>> > > > > >
>>>> > > > > > +1, That’s a good idea.
>>>> > > > > >
>>>> > > > > > > On Dec 16, 2022, at 07:07, tison <wa...@gmail.com>
>>>> wrote:
>>>> > > > > > >
>>>> > > > > > > Hi,
>>>> > > > > > >
>>>> > > > > > > After several works around the build flow of our official
>>>> > > > > website[1][2][3],
>>>> > > > > > > the content sync and site build flow is debuggable and
>>>> > reproducible
>>>> > > > > now.
>>>> > > > > > >
>>>> > > > > > > However, compared to other Apache projects' websites'
>>>> project
>>>> > > layouts
>>>> > > > > and
>>>> > > > > > > workflow, we still meet two challenges on the Pulsar site:
>>>> > > > > > >
>>>> > > > > > > 1. We don't have a pre-commit workflow for any
>>>> website-related
>>>> > > > changes.
>>>> > > > > > > Thus, we don't detect broken links or syntax errors when
>>>> > reviewing
>>>> > > > new
>>>> > > > > > > patches[4][5][6].
>>>> > > > > > > 2. The website's content is two-level down in
>>>> > `site2/website-next`
>>>> > > > for
>>>> > > > > > > historical reasons, which is confusing for contributors.
>>>> > > > > > >
>>>> > > > > > > To overcome these two shortcomings, I propose the following:
>>>> > > > > > >
>>>> > > > > > > 1. Move the website's content to the root level, then we
>>>> have a
>>>> > > > > first-class
>>>> > > > > > > Docu&yarn-based JS project layout. It's more convenient and
>>>> > > familiar
>>>> > > > to
>>>> > > > > > > related developers.
>>>> > > > > > > 2. Host the source of docs in the site repo
>>>> (apache/pulsar-site)
>>>> > > > > instead of
>>>> > > > > > > under `site2` folder in the main repo and do content sync.
>>>> > > > > > >
>>>> > > > > > > Below are the pros and cons:
>>>> > > > > > >
>>>> > > > > > > Pros
>>>> > > > > > >
>>>> > > > > > > 1. Obviously, we have the pre-commit workflow now. And
>>>> since we
>>>> > > host
>>>> > > > > the
>>>> > > > > > > source of docs in one repo, we don't have to run the
>>>> pre-commit
>>>> > > > > workflow in
>>>> > > > > > > two places, which can be quite cumbersome to implement.
>>>> > > > > > > 2. The size of the source release of the main repo can be
>>>> > reduced a
>>>> > > > > lot.
>>>> > > > > > > Currently, 63MB out of 140MB of the sources are taken by the
>>>> > site2
>>>> > > > > folder,
>>>> > > > > > > which we can remove totally. In addition, we carry out
>>>> > > full-versioned
>>>> > > > > docs
>>>> > > > > > > every release.
>>>> > > > > > > 3. We can clean up a large portion of "integration" to
>>>> debug the
>>>> > > site
>>>> > > > > > > brittlely on the main repo[7]  (etc.) and redundant
>>>> contribution
>>>> > > > > guide[8].
>>>> > > > > > > This way, when updating docs, we can preview the result in
>>>> one
>>>> > repo
>>>> > > > > instead
>>>> > > > > > > of actually doing the sync on the fly. In addition, this
>>>> > > integration
>>>> > > > > blocks
>>>> > > > > > > we move the website content to the top level since it makes
>>>> > strong
>>>> > > > > > > assumptions about the relative layout.
>>>> > > > > > >
>>>> > > > > > > Cons
>>>> > > > > > >
>>>> > > > > > > The most significant con is that we cannot update the code
>>>> and
>>>> > docs
>>>> > > > in
>>>> > > > > one
>>>> > > > > > > patch against apache/pulsar now. You must open a new pull
>>>> request
>>>> > > to
>>>> > > > > > > apache/pulsar-site, cross-reference each other and manage
>>>> the
>>>> > merge
>>>> > > > > order
>>>> > > > > > > (synchronization).
>>>> > > > > > >
>>>> > > > > > > Alternatives:
>>>> > > > > > >
>>>> > > > > > > To resolve the versioned docs issue, an alternative is to
>>>> host
>>>> > only
>>>> > > > the
>>>> > > > > > > user docs along with each version, like Flink does[9]. But
>>>> it
>>>> > both
>>>> > > > > detaches
>>>> > > > > > > from the Docu framework and requires significant development
>>>> > > efforts.
>>>> > > > > > >
>>>> > > > > > > Since it can explicitly change the development flow (that
>>>> is, you
>>>> > > > > should
>>>> > > > > > > now update docs separately), I am starting this discussion
>>>> here
>>>> > to
>>>> > > > > reach
>>>> > > > > > > for your feedback.
>>>> > > > > > >
>>>> > > > > > > Welcome to leave your comments!
>>>> > > > > > >
>>>> > > > > > > Best,
>>>> > > > > > > tison.
>>>> > > > > > >
>>>> > > > > > > [1] https://pulsar.apache.org/
>>>> > > > > > > [2] https://github.com/apache/pulsar-site
>>>> > > > > > > [3] https://github.com/apache/pulsar/issues/18014
>>>> > > > > > > [4] https://github.com/apache/pulsar/issues/17599
>>>> > > > > > > [5]
>>>> > > > https://github.com/apache/pulsar/pull/17863#discussion_r990174850
>>>> > > > > > > [6]
>>>> > > > https://github.com/apache/pulsar/pull/17853#discussion_r991803704
>>>> > > > > > > [7]
>>>> > > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>> https://github.com/apache/pulsar/blob/b1f9e351fa4d5aba197d33cfc0c536516b55b61f/site2/website/start.sh
>>>> > > > > > > [8]
>>>> > > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>> https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
>>>> > > > > > > [9] https://github.com/apache/flink/tree/master/docs
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>>
>>>

Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

Posted by tison <wa...@gmail.com>.
Here are the related PRs:

* https://github.com/apache/pulsar/pull/19100
* https://github.com/apache/pulsar-site/pull/348

Best,
tison.


tison <wa...@gmail.com> 于2022年12月26日周一 21:45写道:

> FYI tracking issue has been created:
> https://github.com/apache/pulsar/issues/19064
>
> I plan to finish it by the end of next month.
>
> Best,
> tison.
>
>
> tison <wa...@gmail.com> 于2022年12月21日周三 11:33写道:
>
>> Thanks for your feedback!
>>
>> @Yu
>>
>> Thanks for sharing the previous thread. I looped in @michaeljmarshall
>> here.
>>
>> @Jun
>>
>> It's possible but causes a new shortcoming: Now you should tell the
>> contributor that the versioned docs are different from the NEXT version
>> docs, lol.
>>
>> If our developers don't complain about these separated sources. Like @Asaf
>> comment:
>>
>> > We can take, let's say, five features and see if they were actually
>> done in
>> > the same PR or separate PR. I guess that most documentation is actually
>> > updated separately. Thus, from that perspective, maybe it’s not a con.
>>
>> Then we can do this refactor thoroughgoing.
>>
>> Also, if we keep, somehow several sources in the main repo. We still have
>> shortcomings:
>>
>> 1. Duplicated CI workflows.
>> 2. Cumbersome preview scaffolding in the main repo.
>>
>> ... which is the original purpose I'd like to overcome.
>>
>> Best,
>> tison.
>>
>>
>> Jun Ma <mo...@hotmail.com> 于2022年12月21日周三 11:19写道:
>>
>>> Is it possible to come up with a compromised solution that has the pros
>>> of both sides but minimizes the side effect? I'm thinking maybe it's not
>>> necessary to sacrifice the current contribution process, as long as it can
>>> greatly reduce the load of back-end actions and source size. For example,
>>> if we only move out the versioned docs to the site repo but keep the source
>>> of the NEXT docs in the pulsar repo, does this help to win a large
>>> proportion of those pros when people can still contribute as usual?
>>>
>>> ________________________________
>>> From: Jiaqi Shen <gl...@gmail.com>
>>> Sent: Tuesday, December 20, 2022 17:15
>>> To: dev@pulsar.apache.org <de...@pulsar.apache.org>
>>> Subject: Re: [PROPOSAL] Website precommit and move the source of docs to
>>> the site repo
>>>
>>> +1, it makes sense to me.
>>>
>>> Thanks,
>>> Jiaqi Shen
>>>
>>>
>>> Yu <li...@apache.org> 于2022年12月19日周一 20:57写道:
>>>
>>> > Hi tison,
>>> >
>>> > Thanks for raising this up!
>>> >
>>> > Our community had a similar discussion previously and chose to "keep"
>>> the
>>> > doc repo stay in the Pulsar main repo at that time.
>>> >
>>> > [1] lists the pros and cons of "keep" and "not keep" solutions.
>>> >
>>> > I'm +0 on this proposal because I think the total scores of these two
>>> > solutions are almost equal after weighing the pros and cons.
>>> >
>>> > ~~~~~~~~~~~~~~~~~~~~
>>> >
>>> > [1] https://lists.apache.org/thread/mf2xwntfgn84dq78ksqv22jk3drq6xb3
>>> >
>>> >
>>> > On Mon, Dec 19, 2022 at 5:40 PM tison <wa...@gmail.com> wrote:
>>> >
>>> > > Thanks for your feedback!
>>> > >
>>> > > @Asaf
>>> > >
>>> > > > pre-commit
>>> > >
>>> > > I mean CI checks before merging a patch. Currently, we don't run
>>> checks
>>> > for
>>> > > the content before merging them. This causes a series of syntax
>>> errors
>>> > and
>>> > > broken links issues. If we hold docs under site2 folder in the main
>>> repo
>>> > > and then copied to the site repo, we have two places to build such CI
>>> > > checks. What's worse, the checks for the main repo will be quite
>>> > > cumbersome (that you do some if-else logic in the whole Pulsar CI
>>> > > workflows, and do the sync sequentially in that workflow).
>>> > >
>>> > > If we hold the source of docs only in the site repo, we can extend
>>> the
>>> > > "precommit" workflow[1] I added recently to check for syntax errors
>>> and
>>> > > broken links also.
>>> > >
>>> > > > What does the apache/pulsar-site repo contain today?
>>> > >
>>> > > It should be covered by the documentation guide page[2]. It holds the
>>> > > source of the official website and the user docs are synced from the
>>> main
>>> > > repo.
>>> > >
>>> > > > What content do we have today in the pulsar repo related to the
>>> site?
>>> > >
>>> > > After issue-18014[3] is done, we host only user docs and some JSON
>>> > metadata
>>> > > in the main repo, which is synced by site_syncer.py[4].
>>> > >
>>> > > > Can you explain that better? Are you saying pulsar source JARs
>>> contain
>>> > > the documentation?
>>> > >
>>> > > No. Source JARs contain only the Java files and necessary copyrights
>>> > info.
>>> > > The source release is, for example,
>>> > >
>>> > >
>>> >
>>> https://archive.apache.org/dist/pulsar/pulsar-2.10.2/apache-pulsar-2.10.2-src.tar.gz
>>> > > ,
>>> > > which is extracted to 173M where 129M is occupied by the site2
>>> folder.
>>> > >
>>> > > This also affects when developers do git clone to clone the repo.
>>> > >
>>> > > > I mean, if you wish to document a bug fix in 2.9.x, for example,
>>> would
>>> > > you do it in the 2.9.x branch under site2/docs or
>>> > > site2/website/versioned_docs/2.9.5?
>>> > >
>>> > > This is another question. Ideally, we should have hosted versioned
>>> docs
>>> > > associated with the specific version to that branch, like Apache
>>> Flink
>>> > does
>>> > > as I mentioned[5]. But we do not, and actually the situation is we
>>> update
>>> > > the versioned docs under the master branch and thus, the docs can be
>>> > synced
>>> > > properly.
>>> > >
>>> > > See also the "Alternatives" section in the original email.
>>> > >
>>> > > @All
>>> > >
>>> > > Since we don't have objections to the possible cons listed above or
>>> any
>>> > new
>>> > > ones, I'm going to create a tracking issue later this week and show
>>> what
>>> > > will be changed in PRs for further review.
>>> > >
>>> > > Best,
>>> > > tison.
>>> > >
>>> > > [1]
>>> > >
>>> > >
>>> >
>>> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/.github/workflows/ci-precommit.yml
>>> > > [2]
>>> > >
>>> > >
>>> >
>>> https://pulsar.apache.org/contribute/document-contribution/#source-repositories
>>> > > [3] https://github.com/apache/pulsar/issues/18014
>>> > > [4]
>>> > >
>>> > >
>>> >
>>> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/tools/pytools/lib/execute/site_syncer.py
>>> > > [5] https://github.com/apache/flink/tree/master/docs
>>> > >
>>> > >
>>> > > PengHui Li <pe...@apache.org> 于2022年12月19日周一 16:26写道:
>>> > >
>>> > > > +1
>>> > > >
>>> > > > I support moving them to the website repo.
>>> > > >
>>> > > > Thanks,
>>> > > > Penghui
>>> > > >
>>> > > > On Mon, Dec 19, 2022 at 12:04 PM Yunze Xu
>>> <yzxu@streamnative.io.invalid
>>> > >
>>> > > > wrote:
>>> > > >
>>> > > > > +1. The most significant point to me is that we can preview all
>>> the
>>> > > > > content of the website without synchronizing contents from the
>>> > > > > apache/pulsar repo.
>>> > > > >
>>> > > > > Thanks,
>>> > > > > Yunze
>>> > > > >
>>> > > > > On Mon, Dec 19, 2022 at 9:53 AM Li Li <ur...@apache.org> wrote:
>>> > > > > >
>>> > > > > > +1, That’s a good idea.
>>> > > > > >
>>> > > > > > > On Dec 16, 2022, at 07:07, tison <wa...@gmail.com>
>>> wrote:
>>> > > > > > >
>>> > > > > > > Hi,
>>> > > > > > >
>>> > > > > > > After several works around the build flow of our official
>>> > > > > website[1][2][3],
>>> > > > > > > the content sync and site build flow is debuggable and
>>> > reproducible
>>> > > > > now.
>>> > > > > > >
>>> > > > > > > However, compared to other Apache projects' websites' project
>>> > > layouts
>>> > > > > and
>>> > > > > > > workflow, we still meet two challenges on the Pulsar site:
>>> > > > > > >
>>> > > > > > > 1. We don't have a pre-commit workflow for any
>>> website-related
>>> > > > changes.
>>> > > > > > > Thus, we don't detect broken links or syntax errors when
>>> > reviewing
>>> > > > new
>>> > > > > > > patches[4][5][6].
>>> > > > > > > 2. The website's content is two-level down in
>>> > `site2/website-next`
>>> > > > for
>>> > > > > > > historical reasons, which is confusing for contributors.
>>> > > > > > >
>>> > > > > > > To overcome these two shortcomings, I propose the following:
>>> > > > > > >
>>> > > > > > > 1. Move the website's content to the root level, then we
>>> have a
>>> > > > > first-class
>>> > > > > > > Docu&yarn-based JS project layout. It's more convenient and
>>> > > familiar
>>> > > > to
>>> > > > > > > related developers.
>>> > > > > > > 2. Host the source of docs in the site repo
>>> (apache/pulsar-site)
>>> > > > > instead of
>>> > > > > > > under `site2` folder in the main repo and do content sync.
>>> > > > > > >
>>> > > > > > > Below are the pros and cons:
>>> > > > > > >
>>> > > > > > > Pros
>>> > > > > > >
>>> > > > > > > 1. Obviously, we have the pre-commit workflow now. And since
>>> we
>>> > > host
>>> > > > > the
>>> > > > > > > source of docs in one repo, we don't have to run the
>>> pre-commit
>>> > > > > workflow in
>>> > > > > > > two places, which can be quite cumbersome to implement.
>>> > > > > > > 2. The size of the source release of the main repo can be
>>> > reduced a
>>> > > > > lot.
>>> > > > > > > Currently, 63MB out of 140MB of the sources are taken by the
>>> > site2
>>> > > > > folder,
>>> > > > > > > which we can remove totally. In addition, we carry out
>>> > > full-versioned
>>> > > > > docs
>>> > > > > > > every release.
>>> > > > > > > 3. We can clean up a large portion of "integration" to debug
>>> the
>>> > > site
>>> > > > > > > brittlely on the main repo[7]  (etc.) and redundant
>>> contribution
>>> > > > > guide[8].
>>> > > > > > > This way, when updating docs, we can preview the result in
>>> one
>>> > repo
>>> > > > > instead
>>> > > > > > > of actually doing the sync on the fly. In addition, this
>>> > > integration
>>> > > > > blocks
>>> > > > > > > we move the website content to the top level since it makes
>>> > strong
>>> > > > > > > assumptions about the relative layout.
>>> > > > > > >
>>> > > > > > > Cons
>>> > > > > > >
>>> > > > > > > The most significant con is that we cannot update the code
>>> and
>>> > docs
>>> > > > in
>>> > > > > one
>>> > > > > > > patch against apache/pulsar now. You must open a new pull
>>> request
>>> > > to
>>> > > > > > > apache/pulsar-site, cross-reference each other and manage the
>>> > merge
>>> > > > > order
>>> > > > > > > (synchronization).
>>> > > > > > >
>>> > > > > > > Alternatives:
>>> > > > > > >
>>> > > > > > > To resolve the versioned docs issue, an alternative is to
>>> host
>>> > only
>>> > > > the
>>> > > > > > > user docs along with each version, like Flink does[9]. But it
>>> > both
>>> > > > > detaches
>>> > > > > > > from the Docu framework and requires significant development
>>> > > efforts.
>>> > > > > > >
>>> > > > > > > Since it can explicitly change the development flow (that
>>> is, you
>>> > > > > should
>>> > > > > > > now update docs separately), I am starting this discussion
>>> here
>>> > to
>>> > > > > reach
>>> > > > > > > for your feedback.
>>> > > > > > >
>>> > > > > > > Welcome to leave your comments!
>>> > > > > > >
>>> > > > > > > Best,
>>> > > > > > > tison.
>>> > > > > > >
>>> > > > > > > [1] https://pulsar.apache.org/
>>> > > > > > > [2] https://github.com/apache/pulsar-site
>>> > > > > > > [3] https://github.com/apache/pulsar/issues/18014
>>> > > > > > > [4] https://github.com/apache/pulsar/issues/17599
>>> > > > > > > [5]
>>> > > > https://github.com/apache/pulsar/pull/17863#discussion_r990174850
>>> > > > > > > [6]
>>> > > > https://github.com/apache/pulsar/pull/17853#discussion_r991803704
>>> > > > > > > [7]
>>> > > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://github.com/apache/pulsar/blob/b1f9e351fa4d5aba197d33cfc0c536516b55b61f/site2/website/start.sh
>>> > > > > > > [8]
>>> > > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
>>> > > > > > > [9] https://github.com/apache/flink/tree/master/docs
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>

Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

Posted by tison <wa...@gmail.com>.
FYI tracking issue has been created:
https://github.com/apache/pulsar/issues/19064

I plan to finish it by the end of next month.

Best,
tison.


tison <wa...@gmail.com> 于2022年12月21日周三 11:33写道:

> Thanks for your feedback!
>
> @Yu
>
> Thanks for sharing the previous thread. I looped in @michaeljmarshall
> here.
>
> @Jun
>
> It's possible but causes a new shortcoming: Now you should tell the
> contributor that the versioned docs are different from the NEXT version
> docs, lol.
>
> If our developers don't complain about these separated sources. Like @Asaf
> comment:
>
> > We can take, let's say, five features and see if they were actually done
> in
> > the same PR or separate PR. I guess that most documentation is actually
> > updated separately. Thus, from that perspective, maybe it’s not a con.
>
> Then we can do this refactor thoroughgoing.
>
> Also, if we keep, somehow several sources in the main repo. We still have
> shortcomings:
>
> 1. Duplicated CI workflows.
> 2. Cumbersome preview scaffolding in the main repo.
>
> ... which is the original purpose I'd like to overcome.
>
> Best,
> tison.
>
>
> Jun Ma <mo...@hotmail.com> 于2022年12月21日周三 11:19写道:
>
>> Is it possible to come up with a compromised solution that has the pros
>> of both sides but minimizes the side effect? I'm thinking maybe it's not
>> necessary to sacrifice the current contribution process, as long as it can
>> greatly reduce the load of back-end actions and source size. For example,
>> if we only move out the versioned docs to the site repo but keep the source
>> of the NEXT docs in the pulsar repo, does this help to win a large
>> proportion of those pros when people can still contribute as usual?
>>
>> ________________________________
>> From: Jiaqi Shen <gl...@gmail.com>
>> Sent: Tuesday, December 20, 2022 17:15
>> To: dev@pulsar.apache.org <de...@pulsar.apache.org>
>> Subject: Re: [PROPOSAL] Website precommit and move the source of docs to
>> the site repo
>>
>> +1, it makes sense to me.
>>
>> Thanks,
>> Jiaqi Shen
>>
>>
>> Yu <li...@apache.org> 于2022年12月19日周一 20:57写道:
>>
>> > Hi tison,
>> >
>> > Thanks for raising this up!
>> >
>> > Our community had a similar discussion previously and chose to "keep"
>> the
>> > doc repo stay in the Pulsar main repo at that time.
>> >
>> > [1] lists the pros and cons of "keep" and "not keep" solutions.
>> >
>> > I'm +0 on this proposal because I think the total scores of these two
>> > solutions are almost equal after weighing the pros and cons.
>> >
>> > ~~~~~~~~~~~~~~~~~~~~
>> >
>> > [1] https://lists.apache.org/thread/mf2xwntfgn84dq78ksqv22jk3drq6xb3
>> >
>> >
>> > On Mon, Dec 19, 2022 at 5:40 PM tison <wa...@gmail.com> wrote:
>> >
>> > > Thanks for your feedback!
>> > >
>> > > @Asaf
>> > >
>> > > > pre-commit
>> > >
>> > > I mean CI checks before merging a patch. Currently, we don't run
>> checks
>> > for
>> > > the content before merging them. This causes a series of syntax errors
>> > and
>> > > broken links issues. If we hold docs under site2 folder in the main
>> repo
>> > > and then copied to the site repo, we have two places to build such CI
>> > > checks. What's worse, the checks for the main repo will be quite
>> > > cumbersome (that you do some if-else logic in the whole Pulsar CI
>> > > workflows, and do the sync sequentially in that workflow).
>> > >
>> > > If we hold the source of docs only in the site repo, we can extend the
>> > > "precommit" workflow[1] I added recently to check for syntax errors
>> and
>> > > broken links also.
>> > >
>> > > > What does the apache/pulsar-site repo contain today?
>> > >
>> > > It should be covered by the documentation guide page[2]. It holds the
>> > > source of the official website and the user docs are synced from the
>> main
>> > > repo.
>> > >
>> > > > What content do we have today in the pulsar repo related to the
>> site?
>> > >
>> > > After issue-18014[3] is done, we host only user docs and some JSON
>> > metadata
>> > > in the main repo, which is synced by site_syncer.py[4].
>> > >
>> > > > Can you explain that better? Are you saying pulsar source JARs
>> contain
>> > > the documentation?
>> > >
>> > > No. Source JARs contain only the Java files and necessary copyrights
>> > info.
>> > > The source release is, for example,
>> > >
>> > >
>> >
>> https://archive.apache.org/dist/pulsar/pulsar-2.10.2/apache-pulsar-2.10.2-src.tar.gz
>> > > ,
>> > > which is extracted to 173M where 129M is occupied by the site2 folder.
>> > >
>> > > This also affects when developers do git clone to clone the repo.
>> > >
>> > > > I mean, if you wish to document a bug fix in 2.9.x, for example,
>> would
>> > > you do it in the 2.9.x branch under site2/docs or
>> > > site2/website/versioned_docs/2.9.5?
>> > >
>> > > This is another question. Ideally, we should have hosted versioned
>> docs
>> > > associated with the specific version to that branch, like Apache Flink
>> > does
>> > > as I mentioned[5]. But we do not, and actually the situation is we
>> update
>> > > the versioned docs under the master branch and thus, the docs can be
>> > synced
>> > > properly.
>> > >
>> > > See also the "Alternatives" section in the original email.
>> > >
>> > > @All
>> > >
>> > > Since we don't have objections to the possible cons listed above or
>> any
>> > new
>> > > ones, I'm going to create a tracking issue later this week and show
>> what
>> > > will be changed in PRs for further review.
>> > >
>> > > Best,
>> > > tison.
>> > >
>> > > [1]
>> > >
>> > >
>> >
>> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/.github/workflows/ci-precommit.yml
>> > > [2]
>> > >
>> > >
>> >
>> https://pulsar.apache.org/contribute/document-contribution/#source-repositories
>> > > [3] https://github.com/apache/pulsar/issues/18014
>> > > [4]
>> > >
>> > >
>> >
>> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/tools/pytools/lib/execute/site_syncer.py
>> > > [5] https://github.com/apache/flink/tree/master/docs
>> > >
>> > >
>> > > PengHui Li <pe...@apache.org> 于2022年12月19日周一 16:26写道:
>> > >
>> > > > +1
>> > > >
>> > > > I support moving them to the website repo.
>> > > >
>> > > > Thanks,
>> > > > Penghui
>> > > >
>> > > > On Mon, Dec 19, 2022 at 12:04 PM Yunze Xu
>> <yzxu@streamnative.io.invalid
>> > >
>> > > > wrote:
>> > > >
>> > > > > +1. The most significant point to me is that we can preview all
>> the
>> > > > > content of the website without synchronizing contents from the
>> > > > > apache/pulsar repo.
>> > > > >
>> > > > > Thanks,
>> > > > > Yunze
>> > > > >
>> > > > > On Mon, Dec 19, 2022 at 9:53 AM Li Li <ur...@apache.org> wrote:
>> > > > > >
>> > > > > > +1, That’s a good idea.
>> > > > > >
>> > > > > > > On Dec 16, 2022, at 07:07, tison <wa...@gmail.com>
>> wrote:
>> > > > > > >
>> > > > > > > Hi,
>> > > > > > >
>> > > > > > > After several works around the build flow of our official
>> > > > > website[1][2][3],
>> > > > > > > the content sync and site build flow is debuggable and
>> > reproducible
>> > > > > now.
>> > > > > > >
>> > > > > > > However, compared to other Apache projects' websites' project
>> > > layouts
>> > > > > and
>> > > > > > > workflow, we still meet two challenges on the Pulsar site:
>> > > > > > >
>> > > > > > > 1. We don't have a pre-commit workflow for any website-related
>> > > > changes.
>> > > > > > > Thus, we don't detect broken links or syntax errors when
>> > reviewing
>> > > > new
>> > > > > > > patches[4][5][6].
>> > > > > > > 2. The website's content is two-level down in
>> > `site2/website-next`
>> > > > for
>> > > > > > > historical reasons, which is confusing for contributors.
>> > > > > > >
>> > > > > > > To overcome these two shortcomings, I propose the following:
>> > > > > > >
>> > > > > > > 1. Move the website's content to the root level, then we have
>> a
>> > > > > first-class
>> > > > > > > Docu&yarn-based JS project layout. It's more convenient and
>> > > familiar
>> > > > to
>> > > > > > > related developers.
>> > > > > > > 2. Host the source of docs in the site repo
>> (apache/pulsar-site)
>> > > > > instead of
>> > > > > > > under `site2` folder in the main repo and do content sync.
>> > > > > > >
>> > > > > > > Below are the pros and cons:
>> > > > > > >
>> > > > > > > Pros
>> > > > > > >
>> > > > > > > 1. Obviously, we have the pre-commit workflow now. And since
>> we
>> > > host
>> > > > > the
>> > > > > > > source of docs in one repo, we don't have to run the
>> pre-commit
>> > > > > workflow in
>> > > > > > > two places, which can be quite cumbersome to implement.
>> > > > > > > 2. The size of the source release of the main repo can be
>> > reduced a
>> > > > > lot.
>> > > > > > > Currently, 63MB out of 140MB of the sources are taken by the
>> > site2
>> > > > > folder,
>> > > > > > > which we can remove totally. In addition, we carry out
>> > > full-versioned
>> > > > > docs
>> > > > > > > every release.
>> > > > > > > 3. We can clean up a large portion of "integration" to debug
>> the
>> > > site
>> > > > > > > brittlely on the main repo[7]  (etc.) and redundant
>> contribution
>> > > > > guide[8].
>> > > > > > > This way, when updating docs, we can preview the result in one
>> > repo
>> > > > > instead
>> > > > > > > of actually doing the sync on the fly. In addition, this
>> > > integration
>> > > > > blocks
>> > > > > > > we move the website content to the top level since it makes
>> > strong
>> > > > > > > assumptions about the relative layout.
>> > > > > > >
>> > > > > > > Cons
>> > > > > > >
>> > > > > > > The most significant con is that we cannot update the code and
>> > docs
>> > > > in
>> > > > > one
>> > > > > > > patch against apache/pulsar now. You must open a new pull
>> request
>> > > to
>> > > > > > > apache/pulsar-site, cross-reference each other and manage the
>> > merge
>> > > > > order
>> > > > > > > (synchronization).
>> > > > > > >
>> > > > > > > Alternatives:
>> > > > > > >
>> > > > > > > To resolve the versioned docs issue, an alternative is to host
>> > only
>> > > > the
>> > > > > > > user docs along with each version, like Flink does[9]. But it
>> > both
>> > > > > detaches
>> > > > > > > from the Docu framework and requires significant development
>> > > efforts.
>> > > > > > >
>> > > > > > > Since it can explicitly change the development flow (that is,
>> you
>> > > > > should
>> > > > > > > now update docs separately), I am starting this discussion
>> here
>> > to
>> > > > > reach
>> > > > > > > for your feedback.
>> > > > > > >
>> > > > > > > Welcome to leave your comments!
>> > > > > > >
>> > > > > > > Best,
>> > > > > > > tison.
>> > > > > > >
>> > > > > > > [1] https://pulsar.apache.org/
>> > > > > > > [2] https://github.com/apache/pulsar-site
>> > > > > > > [3] https://github.com/apache/pulsar/issues/18014
>> > > > > > > [4] https://github.com/apache/pulsar/issues/17599
>> > > > > > > [5]
>> > > > https://github.com/apache/pulsar/pull/17863#discussion_r990174850
>> > > > > > > [6]
>> > > > https://github.com/apache/pulsar/pull/17853#discussion_r991803704
>> > > > > > > [7]
>> > > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/pulsar/blob/b1f9e351fa4d5aba197d33cfc0c536516b55b61f/site2/website/start.sh
>> > > > > > > [8]
>> > > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
>> > > > > > > [9] https://github.com/apache/flink/tree/master/docs
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

Posted by tison <wa...@gmail.com>.
Thanks for your feedback!

@Yu

Thanks for sharing the previous thread. I looped in @michaeljmarshall here.

@Jun

It's possible but causes a new shortcoming: Now you should tell the
contributor that the versioned docs are different from the NEXT version
docs, lol.

If our developers don't complain about these separated sources. Like @Asaf
comment:

> We can take, let's say, five features and see if they were actually done
in
> the same PR or separate PR. I guess that most documentation is actually
> updated separately. Thus, from that perspective, maybe it’s not a con.

Then we can do this refactor thoroughgoing.

Also, if we keep, somehow several sources in the main repo. We still have
shortcomings:

1. Duplicated CI workflows.
2. Cumbersome preview scaffolding in the main repo.

... which is the original purpose I'd like to overcome.

Best,
tison.


Jun Ma <mo...@hotmail.com> 于2022年12月21日周三 11:19写道:

> Is it possible to come up with a compromised solution that has the pros of
> both sides but minimizes the side effect? I'm thinking maybe it's not
> necessary to sacrifice the current contribution process, as long as it can
> greatly reduce the load of back-end actions and source size. For example,
> if we only move out the versioned docs to the site repo but keep the source
> of the NEXT docs in the pulsar repo, does this help to win a large
> proportion of those pros when people can still contribute as usual?
>
> ________________________________
> From: Jiaqi Shen <gl...@gmail.com>
> Sent: Tuesday, December 20, 2022 17:15
> To: dev@pulsar.apache.org <de...@pulsar.apache.org>
> Subject: Re: [PROPOSAL] Website precommit and move the source of docs to
> the site repo
>
> +1, it makes sense to me.
>
> Thanks,
> Jiaqi Shen
>
>
> Yu <li...@apache.org> 于2022年12月19日周一 20:57写道:
>
> > Hi tison,
> >
> > Thanks for raising this up!
> >
> > Our community had a similar discussion previously and chose to "keep" the
> > doc repo stay in the Pulsar main repo at that time.
> >
> > [1] lists the pros and cons of "keep" and "not keep" solutions.
> >
> > I'm +0 on this proposal because I think the total scores of these two
> > solutions are almost equal after weighing the pros and cons.
> >
> > ~~~~~~~~~~~~~~~~~~~~
> >
> > [1] https://lists.apache.org/thread/mf2xwntfgn84dq78ksqv22jk3drq6xb3
> >
> >
> > On Mon, Dec 19, 2022 at 5:40 PM tison <wa...@gmail.com> wrote:
> >
> > > Thanks for your feedback!
> > >
> > > @Asaf
> > >
> > > > pre-commit
> > >
> > > I mean CI checks before merging a patch. Currently, we don't run checks
> > for
> > > the content before merging them. This causes a series of syntax errors
> > and
> > > broken links issues. If we hold docs under site2 folder in the main
> repo
> > > and then copied to the site repo, we have two places to build such CI
> > > checks. What's worse, the checks for the main repo will be quite
> > > cumbersome (that you do some if-else logic in the whole Pulsar CI
> > > workflows, and do the sync sequentially in that workflow).
> > >
> > > If we hold the source of docs only in the site repo, we can extend the
> > > "precommit" workflow[1] I added recently to check for syntax errors and
> > > broken links also.
> > >
> > > > What does the apache/pulsar-site repo contain today?
> > >
> > > It should be covered by the documentation guide page[2]. It holds the
> > > source of the official website and the user docs are synced from the
> main
> > > repo.
> > >
> > > > What content do we have today in the pulsar repo related to the site?
> > >
> > > After issue-18014[3] is done, we host only user docs and some JSON
> > metadata
> > > in the main repo, which is synced by site_syncer.py[4].
> > >
> > > > Can you explain that better? Are you saying pulsar source JARs
> contain
> > > the documentation?
> > >
> > > No. Source JARs contain only the Java files and necessary copyrights
> > info.
> > > The source release is, for example,
> > >
> > >
> >
> https://archive.apache.org/dist/pulsar/pulsar-2.10.2/apache-pulsar-2.10.2-src.tar.gz
> > > ,
> > > which is extracted to 173M where 129M is occupied by the site2 folder.
> > >
> > > This also affects when developers do git clone to clone the repo.
> > >
> > > > I mean, if you wish to document a bug fix in 2.9.x, for example,
> would
> > > you do it in the 2.9.x branch under site2/docs or
> > > site2/website/versioned_docs/2.9.5?
> > >
> > > This is another question. Ideally, we should have hosted versioned docs
> > > associated with the specific version to that branch, like Apache Flink
> > does
> > > as I mentioned[5]. But we do not, and actually the situation is we
> update
> > > the versioned docs under the master branch and thus, the docs can be
> > synced
> > > properly.
> > >
> > > See also the "Alternatives" section in the original email.
> > >
> > > @All
> > >
> > > Since we don't have objections to the possible cons listed above or any
> > new
> > > ones, I'm going to create a tracking issue later this week and show
> what
> > > will be changed in PRs for further review.
> > >
> > > Best,
> > > tison.
> > >
> > > [1]
> > >
> > >
> >
> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/.github/workflows/ci-precommit.yml
> > > [2]
> > >
> > >
> >
> https://pulsar.apache.org/contribute/document-contribution/#source-repositories
> > > [3] https://github.com/apache/pulsar/issues/18014
> > > [4]
> > >
> > >
> >
> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/tools/pytools/lib/execute/site_syncer.py
> > > [5] https://github.com/apache/flink/tree/master/docs
> > >
> > >
> > > PengHui Li <pe...@apache.org> 于2022年12月19日周一 16:26写道:
> > >
> > > > +1
> > > >
> > > > I support moving them to the website repo.
> > > >
> > > > Thanks,
> > > > Penghui
> > > >
> > > > On Mon, Dec 19, 2022 at 12:04 PM Yunze Xu
> <yzxu@streamnative.io.invalid
> > >
> > > > wrote:
> > > >
> > > > > +1. The most significant point to me is that we can preview all the
> > > > > content of the website without synchronizing contents from the
> > > > > apache/pulsar repo.
> > > > >
> > > > > Thanks,
> > > > > Yunze
> > > > >
> > > > > On Mon, Dec 19, 2022 at 9:53 AM Li Li <ur...@apache.org> wrote:
> > > > > >
> > > > > > +1, That’s a good idea.
> > > > > >
> > > > > > > On Dec 16, 2022, at 07:07, tison <wa...@gmail.com> wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > After several works around the build flow of our official
> > > > > website[1][2][3],
> > > > > > > the content sync and site build flow is debuggable and
> > reproducible
> > > > > now.
> > > > > > >
> > > > > > > However, compared to other Apache projects' websites' project
> > > layouts
> > > > > and
> > > > > > > workflow, we still meet two challenges on the Pulsar site:
> > > > > > >
> > > > > > > 1. We don't have a pre-commit workflow for any website-related
> > > > changes.
> > > > > > > Thus, we don't detect broken links or syntax errors when
> > reviewing
> > > > new
> > > > > > > patches[4][5][6].
> > > > > > > 2. The website's content is two-level down in
> > `site2/website-next`
> > > > for
> > > > > > > historical reasons, which is confusing for contributors.
> > > > > > >
> > > > > > > To overcome these two shortcomings, I propose the following:
> > > > > > >
> > > > > > > 1. Move the website's content to the root level, then we have a
> > > > > first-class
> > > > > > > Docu&yarn-based JS project layout. It's more convenient and
> > > familiar
> > > > to
> > > > > > > related developers.
> > > > > > > 2. Host the source of docs in the site repo
> (apache/pulsar-site)
> > > > > instead of
> > > > > > > under `site2` folder in the main repo and do content sync.
> > > > > > >
> > > > > > > Below are the pros and cons:
> > > > > > >
> > > > > > > Pros
> > > > > > >
> > > > > > > 1. Obviously, we have the pre-commit workflow now. And since we
> > > host
> > > > > the
> > > > > > > source of docs in one repo, we don't have to run the pre-commit
> > > > > workflow in
> > > > > > > two places, which can be quite cumbersome to implement.
> > > > > > > 2. The size of the source release of the main repo can be
> > reduced a
> > > > > lot.
> > > > > > > Currently, 63MB out of 140MB of the sources are taken by the
> > site2
> > > > > folder,
> > > > > > > which we can remove totally. In addition, we carry out
> > > full-versioned
> > > > > docs
> > > > > > > every release.
> > > > > > > 3. We can clean up a large portion of "integration" to debug
> the
> > > site
> > > > > > > brittlely on the main repo[7]  (etc.) and redundant
> contribution
> > > > > guide[8].
> > > > > > > This way, when updating docs, we can preview the result in one
> > repo
> > > > > instead
> > > > > > > of actually doing the sync on the fly. In addition, this
> > > integration
> > > > > blocks
> > > > > > > we move the website content to the top level since it makes
> > strong
> > > > > > > assumptions about the relative layout.
> > > > > > >
> > > > > > > Cons
> > > > > > >
> > > > > > > The most significant con is that we cannot update the code and
> > docs
> > > > in
> > > > > one
> > > > > > > patch against apache/pulsar now. You must open a new pull
> request
> > > to
> > > > > > > apache/pulsar-site, cross-reference each other and manage the
> > merge
> > > > > order
> > > > > > > (synchronization).
> > > > > > >
> > > > > > > Alternatives:
> > > > > > >
> > > > > > > To resolve the versioned docs issue, an alternative is to host
> > only
> > > > the
> > > > > > > user docs along with each version, like Flink does[9]. But it
> > both
> > > > > detaches
> > > > > > > from the Docu framework and requires significant development
> > > efforts.
> > > > > > >
> > > > > > > Since it can explicitly change the development flow (that is,
> you
> > > > > should
> > > > > > > now update docs separately), I am starting this discussion here
> > to
> > > > > reach
> > > > > > > for your feedback.
> > > > > > >
> > > > > > > Welcome to leave your comments!
> > > > > > >
> > > > > > > Best,
> > > > > > > tison.
> > > > > > >
> > > > > > > [1] https://pulsar.apache.org/
> > > > > > > [2] https://github.com/apache/pulsar-site
> > > > > > > [3] https://github.com/apache/pulsar/issues/18014
> > > > > > > [4] https://github.com/apache/pulsar/issues/17599
> > > > > > > [5]
> > > > https://github.com/apache/pulsar/pull/17863#discussion_r990174850
> > > > > > > [6]
> > > > https://github.com/apache/pulsar/pull/17853#discussion_r991803704
> > > > > > > [7]
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/pulsar/blob/b1f9e351fa4d5aba197d33cfc0c536516b55b61f/site2/website/start.sh
> > > > > > > [8]
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
> > > > > > > [9] https://github.com/apache/flink/tree/master/docs
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

Posted by Jun Ma <mo...@hotmail.com>.
Is it possible to come up with a compromised solution that has the pros of both sides but minimizes the side effect? I'm thinking maybe it's not necessary to sacrifice the current contribution process, as long as it can greatly reduce the load of back-end actions and source size. For example, if we only move out the versioned docs to the site repo but keep the source of the NEXT docs in the pulsar repo, does this help to win a large proportion of those pros when people can still contribute as usual?

________________________________
From: Jiaqi Shen <gl...@gmail.com>
Sent: Tuesday, December 20, 2022 17:15
To: dev@pulsar.apache.org <de...@pulsar.apache.org>
Subject: Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

+1, it makes sense to me.

Thanks,
Jiaqi Shen


Yu <li...@apache.org> 于2022年12月19日周一 20:57写道:

> Hi tison,
>
> Thanks for raising this up!
>
> Our community had a similar discussion previously and chose to "keep" the
> doc repo stay in the Pulsar main repo at that time.
>
> [1] lists the pros and cons of "keep" and "not keep" solutions.
>
> I'm +0 on this proposal because I think the total scores of these two
> solutions are almost equal after weighing the pros and cons.
>
> ~~~~~~~~~~~~~~~~~~~~
>
> [1] https://lists.apache.org/thread/mf2xwntfgn84dq78ksqv22jk3drq6xb3
>
>
> On Mon, Dec 19, 2022 at 5:40 PM tison <wa...@gmail.com> wrote:
>
> > Thanks for your feedback!
> >
> > @Asaf
> >
> > > pre-commit
> >
> > I mean CI checks before merging a patch. Currently, we don't run checks
> for
> > the content before merging them. This causes a series of syntax errors
> and
> > broken links issues. If we hold docs under site2 folder in the main repo
> > and then copied to the site repo, we have two places to build such CI
> > checks. What's worse, the checks for the main repo will be quite
> > cumbersome (that you do some if-else logic in the whole Pulsar CI
> > workflows, and do the sync sequentially in that workflow).
> >
> > If we hold the source of docs only in the site repo, we can extend the
> > "precommit" workflow[1] I added recently to check for syntax errors and
> > broken links also.
> >
> > > What does the apache/pulsar-site repo contain today?
> >
> > It should be covered by the documentation guide page[2]. It holds the
> > source of the official website and the user docs are synced from the main
> > repo.
> >
> > > What content do we have today in the pulsar repo related to the site?
> >
> > After issue-18014[3] is done, we host only user docs and some JSON
> metadata
> > in the main repo, which is synced by site_syncer.py[4].
> >
> > > Can you explain that better? Are you saying pulsar source JARs contain
> > the documentation?
> >
> > No. Source JARs contain only the Java files and necessary copyrights
> info.
> > The source release is, for example,
> >
> >
> https://archive.apache.org/dist/pulsar/pulsar-2.10.2/apache-pulsar-2.10.2-src.tar.gz
> > ,
> > which is extracted to 173M where 129M is occupied by the site2 folder.
> >
> > This also affects when developers do git clone to clone the repo.
> >
> > > I mean, if you wish to document a bug fix in 2.9.x, for example, would
> > you do it in the 2.9.x branch under site2/docs or
> > site2/website/versioned_docs/2.9.5?
> >
> > This is another question. Ideally, we should have hosted versioned docs
> > associated with the specific version to that branch, like Apache Flink
> does
> > as I mentioned[5]. But we do not, and actually the situation is we update
> > the versioned docs under the master branch and thus, the docs can be
> synced
> > properly.
> >
> > See also the "Alternatives" section in the original email.
> >
> > @All
> >
> > Since we don't have objections to the possible cons listed above or any
> new
> > ones, I'm going to create a tracking issue later this week and show what
> > will be changed in PRs for further review.
> >
> > Best,
> > tison.
> >
> > [1]
> >
> >
> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/.github/workflows/ci-precommit.yml
> > [2]
> >
> >
> https://pulsar.apache.org/contribute/document-contribution/#source-repositories
> > [3] https://github.com/apache/pulsar/issues/18014
> > [4]
> >
> >
> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/tools/pytools/lib/execute/site_syncer.py
> > [5] https://github.com/apache/flink/tree/master/docs
> >
> >
> > PengHui Li <pe...@apache.org> 于2022年12月19日周一 16:26写道:
> >
> > > +1
> > >
> > > I support moving them to the website repo.
> > >
> > > Thanks,
> > > Penghui
> > >
> > > On Mon, Dec 19, 2022 at 12:04 PM Yunze Xu <yzxu@streamnative.io.invalid
> >
> > > wrote:
> > >
> > > > +1. The most significant point to me is that we can preview all the
> > > > content of the website without synchronizing contents from the
> > > > apache/pulsar repo.
> > > >
> > > > Thanks,
> > > > Yunze
> > > >
> > > > On Mon, Dec 19, 2022 at 9:53 AM Li Li <ur...@apache.org> wrote:
> > > > >
> > > > > +1, That’s a good idea.
> > > > >
> > > > > > On Dec 16, 2022, at 07:07, tison <wa...@gmail.com> wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > After several works around the build flow of our official
> > > > website[1][2][3],
> > > > > > the content sync and site build flow is debuggable and
> reproducible
> > > > now.
> > > > > >
> > > > > > However, compared to other Apache projects' websites' project
> > layouts
> > > > and
> > > > > > workflow, we still meet two challenges on the Pulsar site:
> > > > > >
> > > > > > 1. We don't have a pre-commit workflow for any website-related
> > > changes.
> > > > > > Thus, we don't detect broken links or syntax errors when
> reviewing
> > > new
> > > > > > patches[4][5][6].
> > > > > > 2. The website's content is two-level down in
> `site2/website-next`
> > > for
> > > > > > historical reasons, which is confusing for contributors.
> > > > > >
> > > > > > To overcome these two shortcomings, I propose the following:
> > > > > >
> > > > > > 1. Move the website's content to the root level, then we have a
> > > > first-class
> > > > > > Docu&yarn-based JS project layout. It's more convenient and
> > familiar
> > > to
> > > > > > related developers.
> > > > > > 2. Host the source of docs in the site repo (apache/pulsar-site)
> > > > instead of
> > > > > > under `site2` folder in the main repo and do content sync.
> > > > > >
> > > > > > Below are the pros and cons:
> > > > > >
> > > > > > Pros
> > > > > >
> > > > > > 1. Obviously, we have the pre-commit workflow now. And since we
> > host
> > > > the
> > > > > > source of docs in one repo, we don't have to run the pre-commit
> > > > workflow in
> > > > > > two places, which can be quite cumbersome to implement.
> > > > > > 2. The size of the source release of the main repo can be
> reduced a
> > > > lot.
> > > > > > Currently, 63MB out of 140MB of the sources are taken by the
> site2
> > > > folder,
> > > > > > which we can remove totally. In addition, we carry out
> > full-versioned
> > > > docs
> > > > > > every release.
> > > > > > 3. We can clean up a large portion of "integration" to debug the
> > site
> > > > > > brittlely on the main repo[7]  (etc.) and redundant contribution
> > > > guide[8].
> > > > > > This way, when updating docs, we can preview the result in one
> repo
> > > > instead
> > > > > > of actually doing the sync on the fly. In addition, this
> > integration
> > > > blocks
> > > > > > we move the website content to the top level since it makes
> strong
> > > > > > assumptions about the relative layout.
> > > > > >
> > > > > > Cons
> > > > > >
> > > > > > The most significant con is that we cannot update the code and
> docs
> > > in
> > > > one
> > > > > > patch against apache/pulsar now. You must open a new pull request
> > to
> > > > > > apache/pulsar-site, cross-reference each other and manage the
> merge
> > > > order
> > > > > > (synchronization).
> > > > > >
> > > > > > Alternatives:
> > > > > >
> > > > > > To resolve the versioned docs issue, an alternative is to host
> only
> > > the
> > > > > > user docs along with each version, like Flink does[9]. But it
> both
> > > > detaches
> > > > > > from the Docu framework and requires significant development
> > efforts.
> > > > > >
> > > > > > Since it can explicitly change the development flow (that is, you
> > > > should
> > > > > > now update docs separately), I am starting this discussion here
> to
> > > > reach
> > > > > > for your feedback.
> > > > > >
> > > > > > Welcome to leave your comments!
> > > > > >
> > > > > > Best,
> > > > > > tison.
> > > > > >
> > > > > > [1] https://pulsar.apache.org/
> > > > > > [2] https://github.com/apache/pulsar-site
> > > > > > [3] https://github.com/apache/pulsar/issues/18014
> > > > > > [4] https://github.com/apache/pulsar/issues/17599
> > > > > > [5]
> > > https://github.com/apache/pulsar/pull/17863#discussion_r990174850
> > > > > > [6]
> > > https://github.com/apache/pulsar/pull/17853#discussion_r991803704
> > > > > > [7]
> > > > > >
> > > >
> > >
> >
> https://github.com/apache/pulsar/blob/b1f9e351fa4d5aba197d33cfc0c536516b55b61f/site2/website/start.sh
> > > > > > [8]
> > > > > >
> > > >
> > >
> >
> https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
> > > > > > [9] https://github.com/apache/flink/tree/master/docs
> > > > >
> > > >
> > >
> >
>

Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

Posted by Jiaqi Shen <gl...@gmail.com>.
+1, it makes sense to me.

Thanks,
Jiaqi Shen


Yu <li...@apache.org> 于2022年12月19日周一 20:57写道:

> Hi tison,
>
> Thanks for raising this up!
>
> Our community had a similar discussion previously and chose to "keep" the
> doc repo stay in the Pulsar main repo at that time.
>
> [1] lists the pros and cons of "keep" and "not keep" solutions.
>
> I'm +0 on this proposal because I think the total scores of these two
> solutions are almost equal after weighing the pros and cons.
>
> ~~~~~~~~~~~~~~~~~~~~
>
> [1] https://lists.apache.org/thread/mf2xwntfgn84dq78ksqv22jk3drq6xb3
>
>
> On Mon, Dec 19, 2022 at 5:40 PM tison <wa...@gmail.com> wrote:
>
> > Thanks for your feedback!
> >
> > @Asaf
> >
> > > pre-commit
> >
> > I mean CI checks before merging a patch. Currently, we don't run checks
> for
> > the content before merging them. This causes a series of syntax errors
> and
> > broken links issues. If we hold docs under site2 folder in the main repo
> > and then copied to the site repo, we have two places to build such CI
> > checks. What's worse, the checks for the main repo will be quite
> > cumbersome (that you do some if-else logic in the whole Pulsar CI
> > workflows, and do the sync sequentially in that workflow).
> >
> > If we hold the source of docs only in the site repo, we can extend the
> > "precommit" workflow[1] I added recently to check for syntax errors and
> > broken links also.
> >
> > > What does the apache/pulsar-site repo contain today?
> >
> > It should be covered by the documentation guide page[2]. It holds the
> > source of the official website and the user docs are synced from the main
> > repo.
> >
> > > What content do we have today in the pulsar repo related to the site?
> >
> > After issue-18014[3] is done, we host only user docs and some JSON
> metadata
> > in the main repo, which is synced by site_syncer.py[4].
> >
> > > Can you explain that better? Are you saying pulsar source JARs contain
> > the documentation?
> >
> > No. Source JARs contain only the Java files and necessary copyrights
> info.
> > The source release is, for example,
> >
> >
> https://archive.apache.org/dist/pulsar/pulsar-2.10.2/apache-pulsar-2.10.2-src.tar.gz
> > ,
> > which is extracted to 173M where 129M is occupied by the site2 folder.
> >
> > This also affects when developers do git clone to clone the repo.
> >
> > > I mean, if you wish to document a bug fix in 2.9.x, for example, would
> > you do it in the 2.9.x branch under site2/docs or
> > site2/website/versioned_docs/2.9.5?
> >
> > This is another question. Ideally, we should have hosted versioned docs
> > associated with the specific version to that branch, like Apache Flink
> does
> > as I mentioned[5]. But we do not, and actually the situation is we update
> > the versioned docs under the master branch and thus, the docs can be
> synced
> > properly.
> >
> > See also the "Alternatives" section in the original email.
> >
> > @All
> >
> > Since we don't have objections to the possible cons listed above or any
> new
> > ones, I'm going to create a tracking issue later this week and show what
> > will be changed in PRs for further review.
> >
> > Best,
> > tison.
> >
> > [1]
> >
> >
> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/.github/workflows/ci-precommit.yml
> > [2]
> >
> >
> https://pulsar.apache.org/contribute/document-contribution/#source-repositories
> > [3] https://github.com/apache/pulsar/issues/18014
> > [4]
> >
> >
> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/tools/pytools/lib/execute/site_syncer.py
> > [5] https://github.com/apache/flink/tree/master/docs
> >
> >
> > PengHui Li <pe...@apache.org> 于2022年12月19日周一 16:26写道:
> >
> > > +1
> > >
> > > I support moving them to the website repo.
> > >
> > > Thanks,
> > > Penghui
> > >
> > > On Mon, Dec 19, 2022 at 12:04 PM Yunze Xu <yzxu@streamnative.io.invalid
> >
> > > wrote:
> > >
> > > > +1. The most significant point to me is that we can preview all the
> > > > content of the website without synchronizing contents from the
> > > > apache/pulsar repo.
> > > >
> > > > Thanks,
> > > > Yunze
> > > >
> > > > On Mon, Dec 19, 2022 at 9:53 AM Li Li <ur...@apache.org> wrote:
> > > > >
> > > > > +1, That’s a good idea.
> > > > >
> > > > > > On Dec 16, 2022, at 07:07, tison <wa...@gmail.com> wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > After several works around the build flow of our official
> > > > website[1][2][3],
> > > > > > the content sync and site build flow is debuggable and
> reproducible
> > > > now.
> > > > > >
> > > > > > However, compared to other Apache projects' websites' project
> > layouts
> > > > and
> > > > > > workflow, we still meet two challenges on the Pulsar site:
> > > > > >
> > > > > > 1. We don't have a pre-commit workflow for any website-related
> > > changes.
> > > > > > Thus, we don't detect broken links or syntax errors when
> reviewing
> > > new
> > > > > > patches[4][5][6].
> > > > > > 2. The website's content is two-level down in
> `site2/website-next`
> > > for
> > > > > > historical reasons, which is confusing for contributors.
> > > > > >
> > > > > > To overcome these two shortcomings, I propose the following:
> > > > > >
> > > > > > 1. Move the website's content to the root level, then we have a
> > > > first-class
> > > > > > Docu&yarn-based JS project layout. It's more convenient and
> > familiar
> > > to
> > > > > > related developers.
> > > > > > 2. Host the source of docs in the site repo (apache/pulsar-site)
> > > > instead of
> > > > > > under `site2` folder in the main repo and do content sync.
> > > > > >
> > > > > > Below are the pros and cons:
> > > > > >
> > > > > > Pros
> > > > > >
> > > > > > 1. Obviously, we have the pre-commit workflow now. And since we
> > host
> > > > the
> > > > > > source of docs in one repo, we don't have to run the pre-commit
> > > > workflow in
> > > > > > two places, which can be quite cumbersome to implement.
> > > > > > 2. The size of the source release of the main repo can be
> reduced a
> > > > lot.
> > > > > > Currently, 63MB out of 140MB of the sources are taken by the
> site2
> > > > folder,
> > > > > > which we can remove totally. In addition, we carry out
> > full-versioned
> > > > docs
> > > > > > every release.
> > > > > > 3. We can clean up a large portion of "integration" to debug the
> > site
> > > > > > brittlely on the main repo[7]  (etc.) and redundant contribution
> > > > guide[8].
> > > > > > This way, when updating docs, we can preview the result in one
> repo
> > > > instead
> > > > > > of actually doing the sync on the fly. In addition, this
> > integration
> > > > blocks
> > > > > > we move the website content to the top level since it makes
> strong
> > > > > > assumptions about the relative layout.
> > > > > >
> > > > > > Cons
> > > > > >
> > > > > > The most significant con is that we cannot update the code and
> docs
> > > in
> > > > one
> > > > > > patch against apache/pulsar now. You must open a new pull request
> > to
> > > > > > apache/pulsar-site, cross-reference each other and manage the
> merge
> > > > order
> > > > > > (synchronization).
> > > > > >
> > > > > > Alternatives:
> > > > > >
> > > > > > To resolve the versioned docs issue, an alternative is to host
> only
> > > the
> > > > > > user docs along with each version, like Flink does[9]. But it
> both
> > > > detaches
> > > > > > from the Docu framework and requires significant development
> > efforts.
> > > > > >
> > > > > > Since it can explicitly change the development flow (that is, you
> > > > should
> > > > > > now update docs separately), I am starting this discussion here
> to
> > > > reach
> > > > > > for your feedback.
> > > > > >
> > > > > > Welcome to leave your comments!
> > > > > >
> > > > > > Best,
> > > > > > tison.
> > > > > >
> > > > > > [1] https://pulsar.apache.org/
> > > > > > [2] https://github.com/apache/pulsar-site
> > > > > > [3] https://github.com/apache/pulsar/issues/18014
> > > > > > [4] https://github.com/apache/pulsar/issues/17599
> > > > > > [5]
> > > https://github.com/apache/pulsar/pull/17863#discussion_r990174850
> > > > > > [6]
> > > https://github.com/apache/pulsar/pull/17853#discussion_r991803704
> > > > > > [7]
> > > > > >
> > > >
> > >
> >
> https://github.com/apache/pulsar/blob/b1f9e351fa4d5aba197d33cfc0c536516b55b61f/site2/website/start.sh
> > > > > > [8]
> > > > > >
> > > >
> > >
> >
> https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
> > > > > > [9] https://github.com/apache/flink/tree/master/docs
> > > > >
> > > >
> > >
> >
>

Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

Posted by Yu <li...@apache.org>.
Hi tison,

Thanks for raising this up!

Our community had a similar discussion previously and chose to "keep" the
doc repo stay in the Pulsar main repo at that time.

[1] lists the pros and cons of "keep" and "not keep" solutions.

I'm +0 on this proposal because I think the total scores of these two
solutions are almost equal after weighing the pros and cons.

~~~~~~~~~~~~~~~~~~~~

[1] https://lists.apache.org/thread/mf2xwntfgn84dq78ksqv22jk3drq6xb3


On Mon, Dec 19, 2022 at 5:40 PM tison <wa...@gmail.com> wrote:

> Thanks for your feedback!
>
> @Asaf
>
> > pre-commit
>
> I mean CI checks before merging a patch. Currently, we don't run checks for
> the content before merging them. This causes a series of syntax errors and
> broken links issues. If we hold docs under site2 folder in the main repo
> and then copied to the site repo, we have two places to build such CI
> checks. What's worse, the checks for the main repo will be quite
> cumbersome (that you do some if-else logic in the whole Pulsar CI
> workflows, and do the sync sequentially in that workflow).
>
> If we hold the source of docs only in the site repo, we can extend the
> "precommit" workflow[1] I added recently to check for syntax errors and
> broken links also.
>
> > What does the apache/pulsar-site repo contain today?
>
> It should be covered by the documentation guide page[2]. It holds the
> source of the official website and the user docs are synced from the main
> repo.
>
> > What content do we have today in the pulsar repo related to the site?
>
> After issue-18014[3] is done, we host only user docs and some JSON metadata
> in the main repo, which is synced by site_syncer.py[4].
>
> > Can you explain that better? Are you saying pulsar source JARs contain
> the documentation?
>
> No. Source JARs contain only the Java files and necessary copyrights info.
> The source release is, for example,
>
> https://archive.apache.org/dist/pulsar/pulsar-2.10.2/apache-pulsar-2.10.2-src.tar.gz
> ,
> which is extracted to 173M where 129M is occupied by the site2 folder.
>
> This also affects when developers do git clone to clone the repo.
>
> > I mean, if you wish to document a bug fix in 2.9.x, for example, would
> you do it in the 2.9.x branch under site2/docs or
> site2/website/versioned_docs/2.9.5?
>
> This is another question. Ideally, we should have hosted versioned docs
> associated with the specific version to that branch, like Apache Flink does
> as I mentioned[5]. But we do not, and actually the situation is we update
> the versioned docs under the master branch and thus, the docs can be synced
> properly.
>
> See also the "Alternatives" section in the original email.
>
> @All
>
> Since we don't have objections to the possible cons listed above or any new
> ones, I'm going to create a tracking issue later this week and show what
> will be changed in PRs for further review.
>
> Best,
> tison.
>
> [1]
>
> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/.github/workflows/ci-precommit.yml
> [2]
>
> https://pulsar.apache.org/contribute/document-contribution/#source-repositories
> [3] https://github.com/apache/pulsar/issues/18014
> [4]
>
> https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/tools/pytools/lib/execute/site_syncer.py
> [5] https://github.com/apache/flink/tree/master/docs
>
>
> PengHui Li <pe...@apache.org> 于2022年12月19日周一 16:26写道:
>
> > +1
> >
> > I support moving them to the website repo.
> >
> > Thanks,
> > Penghui
> >
> > On Mon, Dec 19, 2022 at 12:04 PM Yunze Xu <yz...@streamnative.io.invalid>
> > wrote:
> >
> > > +1. The most significant point to me is that we can preview all the
> > > content of the website without synchronizing contents from the
> > > apache/pulsar repo.
> > >
> > > Thanks,
> > > Yunze
> > >
> > > On Mon, Dec 19, 2022 at 9:53 AM Li Li <ur...@apache.org> wrote:
> > > >
> > > > +1, That’s a good idea.
> > > >
> > > > > On Dec 16, 2022, at 07:07, tison <wa...@gmail.com> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > After several works around the build flow of our official
> > > website[1][2][3],
> > > > > the content sync and site build flow is debuggable and reproducible
> > > now.
> > > > >
> > > > > However, compared to other Apache projects' websites' project
> layouts
> > > and
> > > > > workflow, we still meet two challenges on the Pulsar site:
> > > > >
> > > > > 1. We don't have a pre-commit workflow for any website-related
> > changes.
> > > > > Thus, we don't detect broken links or syntax errors when reviewing
> > new
> > > > > patches[4][5][6].
> > > > > 2. The website's content is two-level down in `site2/website-next`
> > for
> > > > > historical reasons, which is confusing for contributors.
> > > > >
> > > > > To overcome these two shortcomings, I propose the following:
> > > > >
> > > > > 1. Move the website's content to the root level, then we have a
> > > first-class
> > > > > Docu&yarn-based JS project layout. It's more convenient and
> familiar
> > to
> > > > > related developers.
> > > > > 2. Host the source of docs in the site repo (apache/pulsar-site)
> > > instead of
> > > > > under `site2` folder in the main repo and do content sync.
> > > > >
> > > > > Below are the pros and cons:
> > > > >
> > > > > Pros
> > > > >
> > > > > 1. Obviously, we have the pre-commit workflow now. And since we
> host
> > > the
> > > > > source of docs in one repo, we don't have to run the pre-commit
> > > workflow in
> > > > > two places, which can be quite cumbersome to implement.
> > > > > 2. The size of the source release of the main repo can be reduced a
> > > lot.
> > > > > Currently, 63MB out of 140MB of the sources are taken by the site2
> > > folder,
> > > > > which we can remove totally. In addition, we carry out
> full-versioned
> > > docs
> > > > > every release.
> > > > > 3. We can clean up a large portion of "integration" to debug the
> site
> > > > > brittlely on the main repo[7]  (etc.) and redundant contribution
> > > guide[8].
> > > > > This way, when updating docs, we can preview the result in one repo
> > > instead
> > > > > of actually doing the sync on the fly. In addition, this
> integration
> > > blocks
> > > > > we move the website content to the top level since it makes strong
> > > > > assumptions about the relative layout.
> > > > >
> > > > > Cons
> > > > >
> > > > > The most significant con is that we cannot update the code and docs
> > in
> > > one
> > > > > patch against apache/pulsar now. You must open a new pull request
> to
> > > > > apache/pulsar-site, cross-reference each other and manage the merge
> > > order
> > > > > (synchronization).
> > > > >
> > > > > Alternatives:
> > > > >
> > > > > To resolve the versioned docs issue, an alternative is to host only
> > the
> > > > > user docs along with each version, like Flink does[9]. But it both
> > > detaches
> > > > > from the Docu framework and requires significant development
> efforts.
> > > > >
> > > > > Since it can explicitly change the development flow (that is, you
> > > should
> > > > > now update docs separately), I am starting this discussion here to
> > > reach
> > > > > for your feedback.
> > > > >
> > > > > Welcome to leave your comments!
> > > > >
> > > > > Best,
> > > > > tison.
> > > > >
> > > > > [1] https://pulsar.apache.org/
> > > > > [2] https://github.com/apache/pulsar-site
> > > > > [3] https://github.com/apache/pulsar/issues/18014
> > > > > [4] https://github.com/apache/pulsar/issues/17599
> > > > > [5]
> > https://github.com/apache/pulsar/pull/17863#discussion_r990174850
> > > > > [6]
> > https://github.com/apache/pulsar/pull/17853#discussion_r991803704
> > > > > [7]
> > > > >
> > >
> >
> https://github.com/apache/pulsar/blob/b1f9e351fa4d5aba197d33cfc0c536516b55b61f/site2/website/start.sh
> > > > > [8]
> > > > >
> > >
> >
> https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
> > > > > [9] https://github.com/apache/flink/tree/master/docs
> > > >
> > >
> >
>

Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

Posted by tison <wa...@gmail.com>.
Thanks for your feedback!

@Asaf

> pre-commit

I mean CI checks before merging a patch. Currently, we don't run checks for
the content before merging them. This causes a series of syntax errors and
broken links issues. If we hold docs under site2 folder in the main repo
and then copied to the site repo, we have two places to build such CI
checks. What's worse, the checks for the main repo will be quite
cumbersome (that you do some if-else logic in the whole Pulsar CI
workflows, and do the sync sequentially in that workflow).

If we hold the source of docs only in the site repo, we can extend the
"precommit" workflow[1] I added recently to check for syntax errors and
broken links also.

> What does the apache/pulsar-site repo contain today?

It should be covered by the documentation guide page[2]. It holds the
source of the official website and the user docs are synced from the main
repo.

> What content do we have today in the pulsar repo related to the site?

After issue-18014[3] is done, we host only user docs and some JSON metadata
in the main repo, which is synced by site_syncer.py[4].

> Can you explain that better? Are you saying pulsar source JARs contain
the documentation?

No. Source JARs contain only the Java files and necessary copyrights info.
The source release is, for example,
https://archive.apache.org/dist/pulsar/pulsar-2.10.2/apache-pulsar-2.10.2-src.tar.gz,
which is extracted to 173M where 129M is occupied by the site2 folder.

This also affects when developers do git clone to clone the repo.

> I mean, if you wish to document a bug fix in 2.9.x, for example, would
you do it in the 2.9.x branch under site2/docs or
site2/website/versioned_docs/2.9.5?

This is another question. Ideally, we should have hosted versioned docs
associated with the specific version to that branch, like Apache Flink does
as I mentioned[5]. But we do not, and actually the situation is we update
the versioned docs under the master branch and thus, the docs can be synced
properly.

See also the "Alternatives" section in the original email.

@All

Since we don't have objections to the possible cons listed above or any new
ones, I'm going to create a tracking issue later this week and show what
will be changed in PRs for further review.

Best,
tison.

[1]
https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/.github/workflows/ci-precommit.yml
[2]
https://pulsar.apache.org/contribute/document-contribution/#source-repositories
[3] https://github.com/apache/pulsar/issues/18014
[4]
https://github.com/apache/pulsar-site/blob/f7abc615d57d9846ed093922d24bff952dc0e838/tools/pytools/lib/execute/site_syncer.py
[5] https://github.com/apache/flink/tree/master/docs


PengHui Li <pe...@apache.org> 于2022年12月19日周一 16:26写道:

> +1
>
> I support moving them to the website repo.
>
> Thanks,
> Penghui
>
> On Mon, Dec 19, 2022 at 12:04 PM Yunze Xu <yz...@streamnative.io.invalid>
> wrote:
>
> > +1. The most significant point to me is that we can preview all the
> > content of the website without synchronizing contents from the
> > apache/pulsar repo.
> >
> > Thanks,
> > Yunze
> >
> > On Mon, Dec 19, 2022 at 9:53 AM Li Li <ur...@apache.org> wrote:
> > >
> > > +1, That’s a good idea.
> > >
> > > > On Dec 16, 2022, at 07:07, tison <wa...@gmail.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > > After several works around the build flow of our official
> > website[1][2][3],
> > > > the content sync and site build flow is debuggable and reproducible
> > now.
> > > >
> > > > However, compared to other Apache projects' websites' project layouts
> > and
> > > > workflow, we still meet two challenges on the Pulsar site:
> > > >
> > > > 1. We don't have a pre-commit workflow for any website-related
> changes.
> > > > Thus, we don't detect broken links or syntax errors when reviewing
> new
> > > > patches[4][5][6].
> > > > 2. The website's content is two-level down in `site2/website-next`
> for
> > > > historical reasons, which is confusing for contributors.
> > > >
> > > > To overcome these two shortcomings, I propose the following:
> > > >
> > > > 1. Move the website's content to the root level, then we have a
> > first-class
> > > > Docu&yarn-based JS project layout. It's more convenient and familiar
> to
> > > > related developers.
> > > > 2. Host the source of docs in the site repo (apache/pulsar-site)
> > instead of
> > > > under `site2` folder in the main repo and do content sync.
> > > >
> > > > Below are the pros and cons:
> > > >
> > > > Pros
> > > >
> > > > 1. Obviously, we have the pre-commit workflow now. And since we host
> > the
> > > > source of docs in one repo, we don't have to run the pre-commit
> > workflow in
> > > > two places, which can be quite cumbersome to implement.
> > > > 2. The size of the source release of the main repo can be reduced a
> > lot.
> > > > Currently, 63MB out of 140MB of the sources are taken by the site2
> > folder,
> > > > which we can remove totally. In addition, we carry out full-versioned
> > docs
> > > > every release.
> > > > 3. We can clean up a large portion of "integration" to debug the site
> > > > brittlely on the main repo[7]  (etc.) and redundant contribution
> > guide[8].
> > > > This way, when updating docs, we can preview the result in one repo
> > instead
> > > > of actually doing the sync on the fly. In addition, this integration
> > blocks
> > > > we move the website content to the top level since it makes strong
> > > > assumptions about the relative layout.
> > > >
> > > > Cons
> > > >
> > > > The most significant con is that we cannot update the code and docs
> in
> > one
> > > > patch against apache/pulsar now. You must open a new pull request to
> > > > apache/pulsar-site, cross-reference each other and manage the merge
> > order
> > > > (synchronization).
> > > >
> > > > Alternatives:
> > > >
> > > > To resolve the versioned docs issue, an alternative is to host only
> the
> > > > user docs along with each version, like Flink does[9]. But it both
> > detaches
> > > > from the Docu framework and requires significant development efforts.
> > > >
> > > > Since it can explicitly change the development flow (that is, you
> > should
> > > > now update docs separately), I am starting this discussion here to
> > reach
> > > > for your feedback.
> > > >
> > > > Welcome to leave your comments!
> > > >
> > > > Best,
> > > > tison.
> > > >
> > > > [1] https://pulsar.apache.org/
> > > > [2] https://github.com/apache/pulsar-site
> > > > [3] https://github.com/apache/pulsar/issues/18014
> > > > [4] https://github.com/apache/pulsar/issues/17599
> > > > [5]
> https://github.com/apache/pulsar/pull/17863#discussion_r990174850
> > > > [6]
> https://github.com/apache/pulsar/pull/17853#discussion_r991803704
> > > > [7]
> > > >
> >
> https://github.com/apache/pulsar/blob/b1f9e351fa4d5aba197d33cfc0c536516b55b61f/site2/website/start.sh
> > > > [8]
> > > >
> >
> https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
> > > > [9] https://github.com/apache/flink/tree/master/docs
> > >
> >
>

Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

Posted by PengHui Li <pe...@apache.org>.
+1

I support moving them to the website repo.

Thanks,
Penghui

On Mon, Dec 19, 2022 at 12:04 PM Yunze Xu <yz...@streamnative.io.invalid>
wrote:

> +1. The most significant point to me is that we can preview all the
> content of the website without synchronizing contents from the
> apache/pulsar repo.
>
> Thanks,
> Yunze
>
> On Mon, Dec 19, 2022 at 9:53 AM Li Li <ur...@apache.org> wrote:
> >
> > +1, That’s a good idea.
> >
> > > On Dec 16, 2022, at 07:07, tison <wa...@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > After several works around the build flow of our official
> website[1][2][3],
> > > the content sync and site build flow is debuggable and reproducible
> now.
> > >
> > > However, compared to other Apache projects' websites' project layouts
> and
> > > workflow, we still meet two challenges on the Pulsar site:
> > >
> > > 1. We don't have a pre-commit workflow for any website-related changes.
> > > Thus, we don't detect broken links or syntax errors when reviewing new
> > > patches[4][5][6].
> > > 2. The website's content is two-level down in `site2/website-next` for
> > > historical reasons, which is confusing for contributors.
> > >
> > > To overcome these two shortcomings, I propose the following:
> > >
> > > 1. Move the website's content to the root level, then we have a
> first-class
> > > Docu&yarn-based JS project layout. It's more convenient and familiar to
> > > related developers.
> > > 2. Host the source of docs in the site repo (apache/pulsar-site)
> instead of
> > > under `site2` folder in the main repo and do content sync.
> > >
> > > Below are the pros and cons:
> > >
> > > Pros
> > >
> > > 1. Obviously, we have the pre-commit workflow now. And since we host
> the
> > > source of docs in one repo, we don't have to run the pre-commit
> workflow in
> > > two places, which can be quite cumbersome to implement.
> > > 2. The size of the source release of the main repo can be reduced a
> lot.
> > > Currently, 63MB out of 140MB of the sources are taken by the site2
> folder,
> > > which we can remove totally. In addition, we carry out full-versioned
> docs
> > > every release.
> > > 3. We can clean up a large portion of "integration" to debug the site
> > > brittlely on the main repo[7]  (etc.) and redundant contribution
> guide[8].
> > > This way, when updating docs, we can preview the result in one repo
> instead
> > > of actually doing the sync on the fly. In addition, this integration
> blocks
> > > we move the website content to the top level since it makes strong
> > > assumptions about the relative layout.
> > >
> > > Cons
> > >
> > > The most significant con is that we cannot update the code and docs in
> one
> > > patch against apache/pulsar now. You must open a new pull request to
> > > apache/pulsar-site, cross-reference each other and manage the merge
> order
> > > (synchronization).
> > >
> > > Alternatives:
> > >
> > > To resolve the versioned docs issue, an alternative is to host only the
> > > user docs along with each version, like Flink does[9]. But it both
> detaches
> > > from the Docu framework and requires significant development efforts.
> > >
> > > Since it can explicitly change the development flow (that is, you
> should
> > > now update docs separately), I am starting this discussion here to
> reach
> > > for your feedback.
> > >
> > > Welcome to leave your comments!
> > >
> > > Best,
> > > tison.
> > >
> > > [1] https://pulsar.apache.org/
> > > [2] https://github.com/apache/pulsar-site
> > > [3] https://github.com/apache/pulsar/issues/18014
> > > [4] https://github.com/apache/pulsar/issues/17599
> > > [5] https://github.com/apache/pulsar/pull/17863#discussion_r990174850
> > > [6] https://github.com/apache/pulsar/pull/17853#discussion_r991803704
> > > [7]
> > >
> https://github.com/apache/pulsar/blob/b1f9e351fa4d5aba197d33cfc0c536516b55b61f/site2/website/start.sh
> > > [8]
> > >
> https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
> > > [9] https://github.com/apache/flink/tree/master/docs
> >
>

Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

Posted by Yunze Xu <yz...@streamnative.io.INVALID>.
+1. The most significant point to me is that we can preview all the
content of the website without synchronizing contents from the
apache/pulsar repo.

Thanks,
Yunze

On Mon, Dec 19, 2022 at 9:53 AM Li Li <ur...@apache.org> wrote:
>
> +1, That’s a good idea.
>
> > On Dec 16, 2022, at 07:07, tison <wa...@gmail.com> wrote:
> >
> > Hi,
> >
> > After several works around the build flow of our official website[1][2][3],
> > the content sync and site build flow is debuggable and reproducible now.
> >
> > However, compared to other Apache projects' websites' project layouts and
> > workflow, we still meet two challenges on the Pulsar site:
> >
> > 1. We don't have a pre-commit workflow for any website-related changes.
> > Thus, we don't detect broken links or syntax errors when reviewing new
> > patches[4][5][6].
> > 2. The website's content is two-level down in `site2/website-next` for
> > historical reasons, which is confusing for contributors.
> >
> > To overcome these two shortcomings, I propose the following:
> >
> > 1. Move the website's content to the root level, then we have a first-class
> > Docu&yarn-based JS project layout. It's more convenient and familiar to
> > related developers.
> > 2. Host the source of docs in the site repo (apache/pulsar-site) instead of
> > under `site2` folder in the main repo and do content sync.
> >
> > Below are the pros and cons:
> >
> > Pros
> >
> > 1. Obviously, we have the pre-commit workflow now. And since we host the
> > source of docs in one repo, we don't have to run the pre-commit workflow in
> > two places, which can be quite cumbersome to implement.
> > 2. The size of the source release of the main repo can be reduced a lot.
> > Currently, 63MB out of 140MB of the sources are taken by the site2 folder,
> > which we can remove totally. In addition, we carry out full-versioned docs
> > every release.
> > 3. We can clean up a large portion of "integration" to debug the site
> > brittlely on the main repo[7]  (etc.) and redundant contribution guide[8].
> > This way, when updating docs, we can preview the result in one repo instead
> > of actually doing the sync on the fly. In addition, this integration blocks
> > we move the website content to the top level since it makes strong
> > assumptions about the relative layout.
> >
> > Cons
> >
> > The most significant con is that we cannot update the code and docs in one
> > patch against apache/pulsar now. You must open a new pull request to
> > apache/pulsar-site, cross-reference each other and manage the merge order
> > (synchronization).
> >
> > Alternatives:
> >
> > To resolve the versioned docs issue, an alternative is to host only the
> > user docs along with each version, like Flink does[9]. But it both detaches
> > from the Docu framework and requires significant development efforts.
> >
> > Since it can explicitly change the development flow (that is, you should
> > now update docs separately), I am starting this discussion here to reach
> > for your feedback.
> >
> > Welcome to leave your comments!
> >
> > Best,
> > tison.
> >
> > [1] https://pulsar.apache.org/
> > [2] https://github.com/apache/pulsar-site
> > [3] https://github.com/apache/pulsar/issues/18014
> > [4] https://github.com/apache/pulsar/issues/17599
> > [5] https://github.com/apache/pulsar/pull/17863#discussion_r990174850
> > [6] https://github.com/apache/pulsar/pull/17853#discussion_r991803704
> > [7]
> > https://github.com/apache/pulsar/blob/b1f9e351fa4d5aba197d33cfc0c536516b55b61f/site2/website/start.sh
> > [8]
> > https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
> > [9] https://github.com/apache/flink/tree/master/docs
>

Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

Posted by Li Li <ur...@apache.org>.
+1, That’s a good idea.

> On Dec 16, 2022, at 07:07, tison <wa...@gmail.com> wrote:
> 
> Hi,
> 
> After several works around the build flow of our official website[1][2][3],
> the content sync and site build flow is debuggable and reproducible now.
> 
> However, compared to other Apache projects' websites' project layouts and
> workflow, we still meet two challenges on the Pulsar site:
> 
> 1. We don't have a pre-commit workflow for any website-related changes.
> Thus, we don't detect broken links or syntax errors when reviewing new
> patches[4][5][6].
> 2. The website's content is two-level down in `site2/website-next` for
> historical reasons, which is confusing for contributors.
> 
> To overcome these two shortcomings, I propose the following:
> 
> 1. Move the website's content to the root level, then we have a first-class
> Docu&yarn-based JS project layout. It's more convenient and familiar to
> related developers.
> 2. Host the source of docs in the site repo (apache/pulsar-site) instead of
> under `site2` folder in the main repo and do content sync.
> 
> Below are the pros and cons:
> 
> Pros
> 
> 1. Obviously, we have the pre-commit workflow now. And since we host the
> source of docs in one repo, we don't have to run the pre-commit workflow in
> two places, which can be quite cumbersome to implement.
> 2. The size of the source release of the main repo can be reduced a lot.
> Currently, 63MB out of 140MB of the sources are taken by the site2 folder,
> which we can remove totally. In addition, we carry out full-versioned docs
> every release.
> 3. We can clean up a large portion of "integration" to debug the site
> brittlely on the main repo[7]  (etc.) and redundant contribution guide[8].
> This way, when updating docs, we can preview the result in one repo instead
> of actually doing the sync on the fly. In addition, this integration blocks
> we move the website content to the top level since it makes strong
> assumptions about the relative layout.
> 
> Cons
> 
> The most significant con is that we cannot update the code and docs in one
> patch against apache/pulsar now. You must open a new pull request to
> apache/pulsar-site, cross-reference each other and manage the merge order
> (synchronization).
> 
> Alternatives:
> 
> To resolve the versioned docs issue, an alternative is to host only the
> user docs along with each version, like Flink does[9]. But it both detaches
> from the Docu framework and requires significant development efforts.
> 
> Since it can explicitly change the development flow (that is, you should
> now update docs separately), I am starting this discussion here to reach
> for your feedback.
> 
> Welcome to leave your comments!
> 
> Best,
> tison.
> 
> [1] https://pulsar.apache.org/
> [2] https://github.com/apache/pulsar-site
> [3] https://github.com/apache/pulsar/issues/18014
> [4] https://github.com/apache/pulsar/issues/17599
> [5] https://github.com/apache/pulsar/pull/17863#discussion_r990174850
> [6] https://github.com/apache/pulsar/pull/17853#discussion_r991803704
> [7]
> https://github.com/apache/pulsar/blob/b1f9e351fa4d5aba197d33cfc0c536516b55b61f/site2/website/start.sh
> [8]
> https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
> [9] https://github.com/apache/flink/tree/master/docs


Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

Posted by Asaf Mesika <as...@gmail.com>.
>
> We don't have a pre-commit workflow for any website-related changes.
> Thus, we don't detect broken links or syntax errors when reviewing new
> patches[4][5][6].

When you say "pre-commit work-flow,” do you mean what exactly? Usually, a
pre-commit is something you run locally upon committing. Do you mean this?
When you say “new patches,” - do you mean content? Code? Or documentation?

1. Move the website's content to the root level, then we have a first-class
> Docu&yarn-based JS project layout. It's more convenient and familiar to
> related developers.

2. Host the source of docs in the site repo (apache/pulsar-site) instead of
> under `site2` folder in the main repo and do content sync.


Maybe you can give some context?
What does the apache/pulsar-site repo contain today?
What content do we have today in the pulsar repo related to the site? I see
we have the docs and also the Javascript code. I would love some context
here.

2. The size of the source release of the main repo can be reduced a lot.
> Currently, 63MB out of 140MB of the sources are taken by the site2 folder,
> which we can remove totally. In addition, we carry out full-versioned docs
> every release.

Can you explain that better? Are you saying pulsar source JARs contain the
documentation?

Cons
>
> The most significant con is that we cannot update the code and docs in one
> patch against apache/pulsar now. You must open a new pull request to
> apache/pulsar-site, cross-reference each other and manage the merge order
> (synchronization).

We can take, let's say, five features and see if they were actually done in
the same PR or separate PR. I guess that most documentation is actually
updated separately. Thus, from that perspective, maybe it’s not a con.

Since we actually maintain separate directories per version, at least for
the documentation, we lost the actual benefit of saying the code is in sync
with the docs. I mean, if you wish to document a bug fix in 2.9.x, for
example, would you do it in the 2.9.x branch under site2/docs or
site2/website/versioned_docs/2.9.5?

The site2/docs and site2/website/versioned_docs are super confusing, IMO.
Why are the first called docs and the other actually under the website? I
would prefer everything related to docs to have sub-folders like
“previous_versions" and “next,” thus not separated as it is today.


Thanks,

Asaf


On 16 Dec 2022 at 1:07:47, tison <wa...@gmail.com> wrote:

> Hi,
>
> After several works around the build flow of our official website[1][2][3],
> the content sync and site build flow is debuggable and reproducible now.
>
> However, compared to other Apache projects' websites' project layouts and
> workflow, we still meet two challenges on the Pulsar site:
>
> 1. We don't have a pre-commit workflow for any website-related changes.
> Thus, we don't detect broken links or syntax errors when reviewing new
> patches[4][5][6].
> 2. The website's content is two-level down in `site2/website-next` for
> historical reasons, which is confusing for contributors.
>
> To overcome these two shortcomings, I propose the following:
>
> 1. Move the website's content to the root level, then we have a first-class
> Docu&yarn-based JS project layout. It's more convenient and familiar to
> related developers.
> 2. Host the source of docs in the site repo (apache/pulsar-site) instead of
> under `site2` folder in the main repo and do content sync.
>
> Below are the pros and cons:
>
> Pros
>
> 1. Obviously, we have the pre-commit workflow now. And since we host the
> source of docs in one repo, we don't have to run the pre-commit workflow in
> two places, which can be quite cumbersome to implement.
> 2. The size of the source release of the main repo can be reduced a lot.
> Currently, 63MB out of 140MB of the sources are taken by the site2 folder,
> which we can remove totally. In addition, we carry out full-versioned docs
> every release.
> 3. We can clean up a large portion of "integration" to debug the site
> brittlely on the main repo[7]  (etc.) and redundant contribution guide[8].
> This way, when updating docs, we can preview the result in one repo instead
> of actually doing the sync on the fly. In addition, this integration blocks
> we move the website content to the top level since it makes strong
> assumptions about the relative layout.
>
> Cons
>
> The most significant con is that we cannot update the code and docs in one
> patch against apache/pulsar now. You must open a new pull request to
> apache/pulsar-site, cross-reference each other and manage the merge order
> (synchronization).
>
> Alternatives:
>
> To resolve the versioned docs issue, an alternative is to host only the
> user docs along with each version, like Flink does[9]. But it both detaches
> from the Docu framework and requires significant development efforts.
>
> Since it can explicitly change the development flow (that is, you should
> now update docs separately), I am starting this discussion here to reach
> for your feedback.
>
> Welcome to leave your comments!
>
> Best,
> tison.
>
> [1] https://pulsar.apache.org/
> [2] https://github.com/apache/pulsar-site
> [3] https://github.com/apache/pulsar/issues/18014
> [4] https://github.com/apache/pulsar/issues/17599
> [5] https://github.com/apache/pulsar/pull/17863#discussion_r990174850
> [6] https://github.com/apache/pulsar/pull/17853#discussion_r991803704
> [7]
>
> https://github.com/apache/pulsar/blob/b1f9e351fa4d5aba197d33cfc0c536516b55b61f/site2/website/start.sh
> [8]
>
> https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
> [9] https://github.com/apache/flink/tree/master/docs
>

Re: [PROPOSAL] Website precommit and move the source of docs to the site repo

Posted by tison <wa...@gmail.com>.
About the final status, you can take the inlong-website repo as a
reference: https://github.com/apache/inlong-website

Best,
tison.


tison <wa...@gmail.com> 于2022年12月16日周五 07:07写道:

> Hi,
>
> After several works around the build flow of our official
> website[1][2][3], the content sync and site build flow is debuggable and
> reproducible now.
>
> However, compared to other Apache projects' websites' project layouts and
> workflow, we still meet two challenges on the Pulsar site:
>
> 1. We don't have a pre-commit workflow for any website-related changes.
> Thus, we don't detect broken links or syntax errors when reviewing new
> patches[4][5][6].
> 2. The website's content is two-level down in `site2/website-next` for
> historical reasons, which is confusing for contributors.
>
> To overcome these two shortcomings, I propose the following:
>
> 1. Move the website's content to the root level, then we have a
> first-class Docu&yarn-based JS project layout. It's more convenient and
> familiar to related developers.
> 2. Host the source of docs in the site repo (apache/pulsar-site) instead
> of under `site2` folder in the main repo and do content sync.
>
> Below are the pros and cons:
>
> Pros
>
> 1. Obviously, we have the pre-commit workflow now. And since we host the
> source of docs in one repo, we don't have to run the pre-commit workflow in
> two places, which can be quite cumbersome to implement.
> 2. The size of the source release of the main repo can be reduced a lot.
> Currently, 63MB out of 140MB of the sources are taken by the site2 folder,
> which we can remove totally. In addition, we carry out full-versioned docs
> every release.
> 3. We can clean up a large portion of "integration" to debug the site
> brittlely on the main repo[7]  (etc.) and redundant contribution guide[8].
> This way, when updating docs, we can preview the result in one repo instead
> of actually doing the sync on the fly. In addition, this integration blocks
> we move the website content to the top level since it makes strong
> assumptions about the relative layout.
>
> Cons
>
> The most significant con is that we cannot update the code and docs in one
> patch against apache/pulsar now. You must open a new pull request to
> apache/pulsar-site, cross-reference each other and manage the merge order
> (synchronization).
>
> Alternatives:
>
> To resolve the versioned docs issue, an alternative is to host only the
> user docs along with each version, like Flink does[9]. But it both detaches
> from the Docu framework and requires significant development efforts.
>
> Since it can explicitly change the development flow (that is, you should
> now update docs separately), I am starting this discussion here to reach
> for your feedback.
>
> Welcome to leave your comments!
>
> Best,
> tison.
>
> [1] https://pulsar.apache.org/
> [2] https://github.com/apache/pulsar-site
> [3] https://github.com/apache/pulsar/issues/18014
> [4] https://github.com/apache/pulsar/issues/17599
> [5] https://github.com/apache/pulsar/pull/17863#discussion_r990174850
> [6] https://github.com/apache/pulsar/pull/17853#discussion_r991803704
> [7]
> https://github.com/apache/pulsar/blob/b1f9e351fa4d5aba197d33cfc0c536516b55b61f/site2/website/start.sh
> [8]
> https://pulsar.apache.org/contribute/document-preview/#preview-documentation-changes
> [9] https://github.com/apache/flink/tree/master/docs
>