You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Francis Chuang <fr...@apache.org> on 2019/07/04 11:21:46 UTC
[DISCUSS] Automated builds for Calcite website
Hey everyone,
Last month, I mentioned that I was going to start working on automated
website builds for Calcite and its sub-projects. I have completed most
of the work but am current stuck on how to trigger them.
The commits are currently in this branch:
https://github.com/apache/calcite/tree/test-site
The jenkins job is here: https://builds.apache.org/job/calcite-site/
First off, here are my proposed changes to the way websites are built:
1. Force the Jekyll build environment/docker image to use UTC as its
default timezone:
https://github.com/apache/calcite/blob/test-site/site/_config.yml#L58
and
https://github.com/apache/calcite/blob/test-site/site/docker-compose.yml#L33
This allows us to have a definite reference point to calculate the dates
for news items. I think UTC is a pretty "neutral" timezone compared to
other ones and it's often much easier for people to calculate the date
mentally as we tend to know how many hours ahead or behind UTC we are in.
2. Allow publication of future
posts:https://github.com/apache/calcite/blob/test-site/site/_config.yml#L56
When releasing avatica and avatica-go, I often find it annoying to have
to wait for a certain time before building and publishing the release
item (for it to show up) due to the release date. I think if the news
item is published a couple of hours earlier, not much harm is done,
considering that if we have automated website builds in place, it would
be quite annoying to have to push another commit and have it build to
show up.
3. Add a site branch for calcite-avatica and calcite-avatica-go. The
avatica and avatica-go sites don't tend to get updated all that much. We
usually just push it, along with the news item after a release. However,
there may be some commits that make changes to the documentation on the
site as part of a PR and we do not want to build the site from master.
If we build when the repository is tagged, the news item won't be
published as it's committed after the tag's commit. Therefore, we should
have a site branch (like calcite) for those 2 repositories where
everything on site is build immediately, and after a release, the RM
should even the master and site branches.
4. Build triggers:
- For Javadocs (Calcite and Avatica) trigger and build when a final
version is tagged.
- Build the site branch (Calcite, Avatica and Avatica Go) every time a
new commit is added.
5. For the calcite-site repository, we need to push the changes to the
`asf-site` branch and ask infra to set gitpubsub to that branch. The
reason for this is that the git credentials on the gitwebsites build
nodes can only push to a branch called `asf-site`.
Please let me know what you guys think of these proposals.
Problem:
So, the build works correctly and I have verified this using the Jenkins
job. However, when it comes to triggering the jobs, it seems to not be
working correctly. My memory is a bit hazy as I last worked on this a
few weeks ago, but if I am just triggering the build on a push to the
`test-site` branch, everything works correctly. However, once I try to
get it to trigger on a tag, sometimes it doesn't trigger and sometimes,
it just skips all the builds in the Jenkins pipeline.
I think Jenkins is a much more complex beast than I've anticipated and I
currently don't have enough bandwidth to dig into its internals. Could
someone who knows more about Jenkins find some time to look into it?
Another solution might be to use an external CI such as Travis (which is
easier to configure), but I am not sure how happy INFRA would be with
putting the git credential as a secret on those services. There is
currently some general discussion for external CI services on the builds
list here:
https://lists.apache.org/thread.html/af52e2a3e865c01596d46374e8b294f2740587dbd59d85e132429b6c@%3Cbuilds.apache.org%3E
Francis
Re: [DISCUSS] Automated builds for Calcite website
Posted by Michael Mior <mm...@apache.org>.
All sounds great to me. Thanks Francis! Unfortunately I don't know
much about Jenkins so I'm not sure how much help I can be here.
--
Michael Mior
mmior@apache.org
Le jeu. 4 juil. 2019 à 13:21, Francis Chuang
<fr...@apache.org> a écrit :
>
> Hey everyone,
>
> Last month, I mentioned that I was going to start working on automated
> website builds for Calcite and its sub-projects. I have completed most
> of the work but am current stuck on how to trigger them.
>
> The commits are currently in this branch:
> https://github.com/apache/calcite/tree/test-site
> The jenkins job is here: https://builds.apache.org/job/calcite-site/
>
> First off, here are my proposed changes to the way websites are built:
> 1. Force the Jekyll build environment/docker image to use UTC as its
> default timezone:
> https://github.com/apache/calcite/blob/test-site/site/_config.yml#L58
> and
> https://github.com/apache/calcite/blob/test-site/site/docker-compose.yml#L33
>
> This allows us to have a definite reference point to calculate the dates
> for news items. I think UTC is a pretty "neutral" timezone compared to
> other ones and it's often much easier for people to calculate the date
> mentally as we tend to know how many hours ahead or behind UTC we are in.
>
> 2. Allow publication of future
> posts:https://github.com/apache/calcite/blob/test-site/site/_config.yml#L56
>
> When releasing avatica and avatica-go, I often find it annoying to have
> to wait for a certain time before building and publishing the release
> item (for it to show up) due to the release date. I think if the news
> item is published a couple of hours earlier, not much harm is done,
> considering that if we have automated website builds in place, it would
> be quite annoying to have to push another commit and have it build to
> show up.
>
> 3. Add a site branch for calcite-avatica and calcite-avatica-go. The
> avatica and avatica-go sites don't tend to get updated all that much. We
> usually just push it, along with the news item after a release. However,
> there may be some commits that make changes to the documentation on the
> site as part of a PR and we do not want to build the site from master.
> If we build when the repository is tagged, the news item won't be
> published as it's committed after the tag's commit. Therefore, we should
> have a site branch (like calcite) for those 2 repositories where
> everything on site is build immediately, and after a release, the RM
> should even the master and site branches.
>
> 4. Build triggers:
> - For Javadocs (Calcite and Avatica) trigger and build when a final
> version is tagged.
> - Build the site branch (Calcite, Avatica and Avatica Go) every time a
> new commit is added.
>
> 5. For the calcite-site repository, we need to push the changes to the
> `asf-site` branch and ask infra to set gitpubsub to that branch. The
> reason for this is that the git credentials on the gitwebsites build
> nodes can only push to a branch called `asf-site`.
>
> Please let me know what you guys think of these proposals.
>
> Problem:
> So, the build works correctly and I have verified this using the Jenkins
> job. However, when it comes to triggering the jobs, it seems to not be
> working correctly. My memory is a bit hazy as I last worked on this a
> few weeks ago, but if I am just triggering the build on a push to the
> `test-site` branch, everything works correctly. However, once I try to
> get it to trigger on a tag, sometimes it doesn't trigger and sometimes,
> it just skips all the builds in the Jenkins pipeline.
>
> I think Jenkins is a much more complex beast than I've anticipated and I
> currently don't have enough bandwidth to dig into its internals. Could
> someone who knows more about Jenkins find some time to look into it?
>
> Another solution might be to use an external CI such as Travis (which is
> easier to configure), but I am not sure how happy INFRA would be with
> putting the git credential as a secret on those services. There is
> currently some general discussion for external CI services on the builds
> list here:
> https://lists.apache.org/thread.html/af52e2a3e865c01596d46374e8b294f2740587dbd59d85e132429b6c@%3Cbuilds.apache.org%3E
>
> Francis
Re: [DISCUSS] Automated builds for Calcite website
Posted by Stamatis Zampetakis <za...@gmail.com>.
Many projects and also JDK adopts the approach of version selector in the
Javadoc so I think people are accustomed to searching for the right
documentation.
I don't really mind if we follow this approach or not but if it simplifies
the automation process maybe this would be another advantage.
Best,
Stamatis
On Tue, Jul 16, 2019, 7:53 AM Vladimir Sitnikov <si...@gmail.com>
wrote:
> Francis> I've been burnt in the past where generated documentation is for
> master
> Francis> and not the latest release, causing some confusion.
>
> On contrary, it is better when the documentation has version selector, so
> one could pick an appropriate version.
> Then "snapshot" documentation is good for the review purposes.
>
> The documentation for "past" releases is helpful as well.
>
> Vladimir
>
Re: [DISCUSS] Automated builds for Calcite website
Posted by Julian Hyde <jh...@apache.org>.
I find my documentation - even for Calcite - using google. The more versions we keep on the site, the greater chance that I am looking at the wrong version.
So I vote for keeping only one version, the latest release.
Julian
> On Jul 15, 2019, at 10:53 PM, Vladimir Sitnikov <si...@gmail.com> wrote:
>
> Francis> I've been burnt in the past where generated documentation is for
> master
> Francis> and not the latest release, causing some confusion.
>
> On contrary, it is better when the documentation has version selector, so
> one could pick an appropriate version.
> Then "snapshot" documentation is good for the review purposes.
>
> The documentation for "past" releases is helpful as well.
>
> Vladimir
Re: [DISCUSS] Automated builds for Calcite website
Posted by Vladimir Sitnikov <si...@gmail.com>.
Francis> I've been burnt in the past where generated documentation is for
master
Francis> and not the latest release, causing some confusion.
On contrary, it is better when the documentation has version selector, so
one could pick an appropriate version.
Then "snapshot" documentation is good for the review purposes.
The documentation for "past" releases is helpful as well.
Vladimir
Re: [DISCUSS] Automated builds for Calcite website
Posted by Francis Chuang <fr...@apache.org>.
In general, I prefer the Javadocs to reflect the latest release. I find
that when pulling dependencies into my projects (in languages such as
Go, Javascript, PHP, etc), I tend to pin to the latest release, unless
there's a critical and unreleased change on master that I really need.
I've been burnt in the past where generated documentation is for master
and not the latest release, causing some confusion.
On 16/07/2019 4:35 am, Julian Hyde wrote:
>
>
>> On Jul 14, 2019, at 3:27 PM, Francis Chuang <fr...@apache.org> wrote:
>>
>> In the past, the Javadocs were only ever built right after finalizing a release. So, in general, the javadocs do not reflect snapshot versions.
>>
>> Can Julian confirm if the javadocs must be for the latest final release?
>
>
> That has been the policy. I don’t think we ever debated it. We can of course we can debate it.
>
> Remember that “snapshot versions” are not releases, and we should not encourage people to believe that the code in Github’s master branch is approved.
>
> The policy has been that the web site contains documentation and API docs for the latest release. So, the release manager would roll out the site immediately after a release is final. Other aspects of the web site (e.g. committer list, upcoming talks, and process documents) are changed as and when needed. I believe that is the most useful thing for our users.
>
> Julian
>
Re: [DISCUSS] Automated builds for Calcite website
Posted by Julian Hyde <jh...@apache.org>.
> On Jul 14, 2019, at 3:27 PM, Francis Chuang <fr...@apache.org> wrote:
>
> In the past, the Javadocs were only ever built right after finalizing a release. So, in general, the javadocs do not reflect snapshot versions.
>
> Can Julian confirm if the javadocs must be for the latest final release?
That has been the policy. I don’t think we ever debated it. We can of course we can debate it.
Remember that “snapshot versions” are not releases, and we should not encourage people to believe that the code in Github’s master branch is approved.
The policy has been that the web site contains documentation and API docs for the latest release. So, the release manager would roll out the site immediately after a release is final. Other aspects of the web site (e.g. committer list, upcoming talks, and process documents) are changed as and when needed. I believe that is the most useful thing for our users.
Julian
Re: [DISCUSS] Automated builds for Calcite website
Posted by Francis Chuang <fr...@apache.org>.
In the past, the Javadocs were only ever built right after finalizing a
release. So, in general, the javadocs do not reflect snapshot versions.
Can Julian confirm if the javadocs must be for the latest final release?
I will ask the Jenkins list to see if there are any tips for getting the
javadocs to built on a new tag for a final release. I did ask a question
regarding docker hanging (which is now fixed due to upgrades to docker
on all jenkins nodes last week), but received no response. Hopefully,
better luck this time.
On 14/07/2019 10:49 pm, Stamatis Zampetakis wrote:
> From a quick look it seems that the build fails because of the following:
>
> Could not find artifact
> org.apache.calcite:calcite-core:jar:tests:1.21.0-SNAPSHOT in
> apache.snapshots (https://repository.apache.org/snapshots) -> [Help 1]
>
> So apparently the Apache Nexus repository (
> https://repository.apache.org/snapshots) does not have the 1.21.0-SNAPSHOT
> so perhaps we just have to push SNAPSHOT artifacts at a more regular basis.
>
>
> On Sun, Jul 14, 2019 at 8:56 AM Francis Chuang <fr...@apache.org>
> wrote:
>
>> Just a follow up to this. I don't think it will be possible to build the
>> javadoc and site at every time a change is pushed to the site branch.
>>
>> When the site branch is reset to master after a release, the
>> `[maven-release-plugin] prepare for next development iteration` commit
>> will be on the site branch as well. In the last release, this is
>>
>> https://github.com/apache/calcite/commit/8b5fae5deccfb69b9a9a571ddcdc9bef5819948b
>>
>> Since the artifacts for the 1.21.0-snapshot release does not exist on
>> maven central, the javadoc build will fail. See
>> https://builds.apache.org/job/Calcite-Site/84/console
>>
>> On 12/07/2019 3:01 am, Michael Mior wrote:
>>> Francis,
>>>
>>> I just confirmed that there were no changes in site that were not on
>>> master and then did
>>>
>>> git checkout site
>>> git reset --hard origin/master
>>> git push -f origin site
>>>
>>> Essentially just making sure that site and master are exactly the same
>>> after the release.
>>>
>>> --
>>> Michael Mior
>>> mmior@apache.org
>>>
>>> Le jeu. 11 juil. 2019 à 04:50, Francis Chuang
>>> <fr...@apache.org> a écrit :
>>>>
>>>> I meant to ask this in my previous email, but forgot.
>>>>
>>>> Michael, when you were RM for the last Calcite release, what was the git
>>>> command used to even the master and site branches when the release was
>>>> finalized?
>>>>
>>>> I'd like to have that documented as part of this change as well.
>>>>
>>>> On 11/07/2019 9:33 am, Stamatis Zampetakis wrote:
>>>>> Thanks for working on this Francis, great progress!
>>>>>
>>>>> As far as I can tell there is nothing really blocking to start using
>> the
>>>>> automated builds.
>>>>> Since at the moment we don't really have a good solution for
>> triggering the
>>>>> javadoc build on tag creation I would suggest to go on with the naive
>>>>> solution (i.e., build on every push).
>>>>> The site is not updated too often so I guess it is acceptable to have a
>>>>> long build pipeline once in a while.
>>>>> We can create a JIRA for improving the time and leave it open till we
>> find
>>>>> a better solution on this (Gradle, Jenkins, or other trick).
>>>>>
>>>>> Best,
>>>>> Stamatis
>>>>>
>>>>> On Fri, Jul 5, 2019 at 9:16 AM Vladimir Sitnikov <
>>>>> sitnikov.vladimir@gmail.com> wrote:
>>>>>
>>>>>> Francis>javadocs takes around 20 minutes to build
>>>>>>
>>>>>> I did not thought it takes so much time.
>>>>>> "tag" trigger for javadocs is clever, and I just thought we might
>> want to
>>>>>> be able to update the wording on the site javadoc without releasing
>>>>>> Calcite.
>>>>>> That is why I suggested to build javadoc for all site pushes.
>>>>>>
>>>>>> I wonder if the job can reuse the workspace.
>>>>>> I guess it can see the results of the previous builds, so it could
>> just
>>>>>> reuse the javadocs if they are the same.
>>>>>>
>>>>>> Here's what I have for Avatica:
>>>>>>
>>>>>> $ time ./gradlew javadoc
>>>>>> real 0m33.714s
>>>>>> user 0m5.499s
>>>>>> sys 0m0.399s
>>>>>>
>>>>>> $ time ./gradlew javadoc
>>>>>> real 0m2.916s
>>>>>> user 0m2.646s
>>>>>> sys 0m0.208s
>>>>>>
>>>>>> It skips the processing provided no modifications to the javadocs was
>> made.
>>>>>>
>>>>>> Vladimir
>>>>>>
>>>>>
>>
>
Re: [DISCUSS] Automated builds for Calcite website
Posted by Stamatis Zampetakis <za...@gmail.com>.
From a quick look it seems that the build fails because of the following:
Could not find artifact
org.apache.calcite:calcite-core:jar:tests:1.21.0-SNAPSHOT in
apache.snapshots (https://repository.apache.org/snapshots) -> [Help 1]
So apparently the Apache Nexus repository (
https://repository.apache.org/snapshots) does not have the 1.21.0-SNAPSHOT
so perhaps we just have to push SNAPSHOT artifacts at a more regular basis.
On Sun, Jul 14, 2019 at 8:56 AM Francis Chuang <fr...@apache.org>
wrote:
> Just a follow up to this. I don't think it will be possible to build the
> javadoc and site at every time a change is pushed to the site branch.
>
> When the site branch is reset to master after a release, the
> `[maven-release-plugin] prepare for next development iteration` commit
> will be on the site branch as well. In the last release, this is
>
> https://github.com/apache/calcite/commit/8b5fae5deccfb69b9a9a571ddcdc9bef5819948b
>
> Since the artifacts for the 1.21.0-snapshot release does not exist on
> maven central, the javadoc build will fail. See
> https://builds.apache.org/job/Calcite-Site/84/console
>
> On 12/07/2019 3:01 am, Michael Mior wrote:
> > Francis,
> >
> > I just confirmed that there were no changes in site that were not on
> > master and then did
> >
> > git checkout site
> > git reset --hard origin/master
> > git push -f origin site
> >
> > Essentially just making sure that site and master are exactly the same
> > after the release.
> >
> > --
> > Michael Mior
> > mmior@apache.org
> >
> > Le jeu. 11 juil. 2019 à 04:50, Francis Chuang
> > <fr...@apache.org> a écrit :
> >>
> >> I meant to ask this in my previous email, but forgot.
> >>
> >> Michael, when you were RM for the last Calcite release, what was the git
> >> command used to even the master and site branches when the release was
> >> finalized?
> >>
> >> I'd like to have that documented as part of this change as well.
> >>
> >> On 11/07/2019 9:33 am, Stamatis Zampetakis wrote:
> >>> Thanks for working on this Francis, great progress!
> >>>
> >>> As far as I can tell there is nothing really blocking to start using
> the
> >>> automated builds.
> >>> Since at the moment we don't really have a good solution for
> triggering the
> >>> javadoc build on tag creation I would suggest to go on with the naive
> >>> solution (i.e., build on every push).
> >>> The site is not updated too often so I guess it is acceptable to have a
> >>> long build pipeline once in a while.
> >>> We can create a JIRA for improving the time and leave it open till we
> find
> >>> a better solution on this (Gradle, Jenkins, or other trick).
> >>>
> >>> Best,
> >>> Stamatis
> >>>
> >>> On Fri, Jul 5, 2019 at 9:16 AM Vladimir Sitnikov <
> >>> sitnikov.vladimir@gmail.com> wrote:
> >>>
> >>>> Francis>javadocs takes around 20 minutes to build
> >>>>
> >>>> I did not thought it takes so much time.
> >>>> "tag" trigger for javadocs is clever, and I just thought we might
> want to
> >>>> be able to update the wording on the site javadoc without releasing
> >>>> Calcite.
> >>>> That is why I suggested to build javadoc for all site pushes.
> >>>>
> >>>> I wonder if the job can reuse the workspace.
> >>>> I guess it can see the results of the previous builds, so it could
> just
> >>>> reuse the javadocs if they are the same.
> >>>>
> >>>> Here's what I have for Avatica:
> >>>>
> >>>> $ time ./gradlew javadoc
> >>>> real 0m33.714s
> >>>> user 0m5.499s
> >>>> sys 0m0.399s
> >>>>
> >>>> $ time ./gradlew javadoc
> >>>> real 0m2.916s
> >>>> user 0m2.646s
> >>>> sys 0m0.208s
> >>>>
> >>>> It skips the processing provided no modifications to the javadocs was
> made.
> >>>>
> >>>> Vladimir
> >>>>
> >>>
>
Re: [DISCUSS] Automated builds for Calcite website
Posted by Francis Chuang <fr...@apache.org>.
Just a follow up to this. I don't think it will be possible to build the
javadoc and site at every time a change is pushed to the site branch.
When the site branch is reset to master after a release, the
`[maven-release-plugin] prepare for next development iteration` commit
will be on the site branch as well. In the last release, this is
https://github.com/apache/calcite/commit/8b5fae5deccfb69b9a9a571ddcdc9bef5819948b
Since the artifacts for the 1.21.0-snapshot release does not exist on
maven central, the javadoc build will fail. See
https://builds.apache.org/job/Calcite-Site/84/console
On 12/07/2019 3:01 am, Michael Mior wrote:
> Francis,
>
> I just confirmed that there were no changes in site that were not on
> master and then did
>
> git checkout site
> git reset --hard origin/master
> git push -f origin site
>
> Essentially just making sure that site and master are exactly the same
> after the release.
>
> --
> Michael Mior
> mmior@apache.org
>
> Le jeu. 11 juil. 2019 à 04:50, Francis Chuang
> <fr...@apache.org> a écrit :
>>
>> I meant to ask this in my previous email, but forgot.
>>
>> Michael, when you were RM for the last Calcite release, what was the git
>> command used to even the master and site branches when the release was
>> finalized?
>>
>> I'd like to have that documented as part of this change as well.
>>
>> On 11/07/2019 9:33 am, Stamatis Zampetakis wrote:
>>> Thanks for working on this Francis, great progress!
>>>
>>> As far as I can tell there is nothing really blocking to start using the
>>> automated builds.
>>> Since at the moment we don't really have a good solution for triggering the
>>> javadoc build on tag creation I would suggest to go on with the naive
>>> solution (i.e., build on every push).
>>> The site is not updated too often so I guess it is acceptable to have a
>>> long build pipeline once in a while.
>>> We can create a JIRA for improving the time and leave it open till we find
>>> a better solution on this (Gradle, Jenkins, or other trick).
>>>
>>> Best,
>>> Stamatis
>>>
>>> On Fri, Jul 5, 2019 at 9:16 AM Vladimir Sitnikov <
>>> sitnikov.vladimir@gmail.com> wrote:
>>>
>>>> Francis>javadocs takes around 20 minutes to build
>>>>
>>>> I did not thought it takes so much time.
>>>> "tag" trigger for javadocs is clever, and I just thought we might want to
>>>> be able to update the wording on the site javadoc without releasing
>>>> Calcite.
>>>> That is why I suggested to build javadoc for all site pushes.
>>>>
>>>> I wonder if the job can reuse the workspace.
>>>> I guess it can see the results of the previous builds, so it could just
>>>> reuse the javadocs if they are the same.
>>>>
>>>> Here's what I have for Avatica:
>>>>
>>>> $ time ./gradlew javadoc
>>>> real 0m33.714s
>>>> user 0m5.499s
>>>> sys 0m0.399s
>>>>
>>>> $ time ./gradlew javadoc
>>>> real 0m2.916s
>>>> user 0m2.646s
>>>> sys 0m0.208s
>>>>
>>>> It skips the processing provided no modifications to the javadocs was made.
>>>>
>>>> Vladimir
>>>>
>>>
Re: [DISCUSS] Automated builds for Calcite website
Posted by Michael Mior <mm...@apache.org>.
Francis,
I just confirmed that there were no changes in site that were not on
master and then did
git checkout site
git reset --hard origin/master
git push -f origin site
Essentially just making sure that site and master are exactly the same
after the release.
--
Michael Mior
mmior@apache.org
Le jeu. 11 juil. 2019 à 04:50, Francis Chuang
<fr...@apache.org> a écrit :
>
> I meant to ask this in my previous email, but forgot.
>
> Michael, when you were RM for the last Calcite release, what was the git
> command used to even the master and site branches when the release was
> finalized?
>
> I'd like to have that documented as part of this change as well.
>
> On 11/07/2019 9:33 am, Stamatis Zampetakis wrote:
> > Thanks for working on this Francis, great progress!
> >
> > As far as I can tell there is nothing really blocking to start using the
> > automated builds.
> > Since at the moment we don't really have a good solution for triggering the
> > javadoc build on tag creation I would suggest to go on with the naive
> > solution (i.e., build on every push).
> > The site is not updated too often so I guess it is acceptable to have a
> > long build pipeline once in a while.
> > We can create a JIRA for improving the time and leave it open till we find
> > a better solution on this (Gradle, Jenkins, or other trick).
> >
> > Best,
> > Stamatis
> >
> > On Fri, Jul 5, 2019 at 9:16 AM Vladimir Sitnikov <
> > sitnikov.vladimir@gmail.com> wrote:
> >
> >> Francis>javadocs takes around 20 minutes to build
> >>
> >> I did not thought it takes so much time.
> >> "tag" trigger for javadocs is clever, and I just thought we might want to
> >> be able to update the wording on the site javadoc without releasing
> >> Calcite.
> >> That is why I suggested to build javadoc for all site pushes.
> >>
> >> I wonder if the job can reuse the workspace.
> >> I guess it can see the results of the previous builds, so it could just
> >> reuse the javadocs if they are the same.
> >>
> >> Here's what I have for Avatica:
> >>
> >> $ time ./gradlew javadoc
> >> real 0m33.714s
> >> user 0m5.499s
> >> sys 0m0.399s
> >>
> >> $ time ./gradlew javadoc
> >> real 0m2.916s
> >> user 0m2.646s
> >> sys 0m0.208s
> >>
> >> It skips the processing provided no modifications to the javadocs was made.
> >>
> >> Vladimir
> >>
> >
Re: [DISCUSS] Automated builds for Calcite website
Posted by Francis Chuang <fr...@apache.org>.
I meant to ask this in my previous email, but forgot.
Michael, when you were RM for the last Calcite release, what was the git
command used to even the master and site branches when the release was
finalized?
I'd like to have that documented as part of this change as well.
On 11/07/2019 9:33 am, Stamatis Zampetakis wrote:
> Thanks for working on this Francis, great progress!
>
> As far as I can tell there is nothing really blocking to start using the
> automated builds.
> Since at the moment we don't really have a good solution for triggering the
> javadoc build on tag creation I would suggest to go on with the naive
> solution (i.e., build on every push).
> The site is not updated too often so I guess it is acceptable to have a
> long build pipeline once in a while.
> We can create a JIRA for improving the time and leave it open till we find
> a better solution on this (Gradle, Jenkins, or other trick).
>
> Best,
> Stamatis
>
> On Fri, Jul 5, 2019 at 9:16 AM Vladimir Sitnikov <
> sitnikov.vladimir@gmail.com> wrote:
>
>> Francis>javadocs takes around 20 minutes to build
>>
>> I did not thought it takes so much time.
>> "tag" trigger for javadocs is clever, and I just thought we might want to
>> be able to update the wording on the site javadoc without releasing
>> Calcite.
>> That is why I suggested to build javadoc for all site pushes.
>>
>> I wonder if the job can reuse the workspace.
>> I guess it can see the results of the previous builds, so it could just
>> reuse the javadocs if they are the same.
>>
>> Here's what I have for Avatica:
>>
>> $ time ./gradlew javadoc
>> real 0m33.714s
>> user 0m5.499s
>> sys 0m0.399s
>>
>> $ time ./gradlew javadoc
>> real 0m2.916s
>> user 0m2.646s
>> sys 0m0.208s
>>
>> It skips the processing provided no modifications to the javadocs was made.
>>
>> Vladimir
>>
>
Re: [DISCUSS] Automated builds for Calcite website
Posted by Francis Chuang <fr...@apache.org>.
I plan to finalize this over the next few days. I think the naive
solution of building the javadocs on each push is the best solution so far.
I've considered Vladimir's solution of using the Jenkins workspace to
store previous builds, but I've had issues where builds fail if the
previous build failed and the workspace was dirty while building
releases. Due to that, I am more inclined to start with a clean
workspace for every build until we get a more performant solution in place.
Francis
On 11/07/2019 9:33 am, Stamatis Zampetakis wrote:
> Thanks for working on this Francis, great progress!
>
> As far as I can tell there is nothing really blocking to start using the
> automated builds.
> Since at the moment we don't really have a good solution for triggering the
> javadoc build on tag creation I would suggest to go on with the naive
> solution (i.e., build on every push).
> The site is not updated too often so I guess it is acceptable to have a
> long build pipeline once in a while.
> We can create a JIRA for improving the time and leave it open till we find
> a better solution on this (Gradle, Jenkins, or other trick).
>
> Best,
> Stamatis
>
> On Fri, Jul 5, 2019 at 9:16 AM Vladimir Sitnikov <
> sitnikov.vladimir@gmail.com> wrote:
>
>> Francis>javadocs takes around 20 minutes to build
>>
>> I did not thought it takes so much time.
>> "tag" trigger for javadocs is clever, and I just thought we might want to
>> be able to update the wording on the site javadoc without releasing
>> Calcite.
>> That is why I suggested to build javadoc for all site pushes.
>>
>> I wonder if the job can reuse the workspace.
>> I guess it can see the results of the previous builds, so it could just
>> reuse the javadocs if they are the same.
>>
>> Here's what I have for Avatica:
>>
>> $ time ./gradlew javadoc
>> real 0m33.714s
>> user 0m5.499s
>> sys 0m0.399s
>>
>> $ time ./gradlew javadoc
>> real 0m2.916s
>> user 0m2.646s
>> sys 0m0.208s
>>
>> It skips the processing provided no modifications to the javadocs was made.
>>
>> Vladimir
>>
>
Re: [DISCUSS] Automated builds for Calcite website
Posted by Stamatis Zampetakis <za...@gmail.com>.
Thanks for working on this Francis, great progress!
As far as I can tell there is nothing really blocking to start using the
automated builds.
Since at the moment we don't really have a good solution for triggering the
javadoc build on tag creation I would suggest to go on with the naive
solution (i.e., build on every push).
The site is not updated too often so I guess it is acceptable to have a
long build pipeline once in a while.
We can create a JIRA for improving the time and leave it open till we find
a better solution on this (Gradle, Jenkins, or other trick).
Best,
Stamatis
On Fri, Jul 5, 2019 at 9:16 AM Vladimir Sitnikov <
sitnikov.vladimir@gmail.com> wrote:
> Francis>javadocs takes around 20 minutes to build
>
> I did not thought it takes so much time.
> "tag" trigger for javadocs is clever, and I just thought we might want to
> be able to update the wording on the site javadoc without releasing
> Calcite.
> That is why I suggested to build javadoc for all site pushes.
>
> I wonder if the job can reuse the workspace.
> I guess it can see the results of the previous builds, so it could just
> reuse the javadocs if they are the same.
>
> Here's what I have for Avatica:
>
> $ time ./gradlew javadoc
> real 0m33.714s
> user 0m5.499s
> sys 0m0.399s
>
> $ time ./gradlew javadoc
> real 0m2.916s
> user 0m2.646s
> sys 0m0.208s
>
> It skips the processing provided no modifications to the javadocs was made.
>
> Vladimir
>
Re: [DISCUSS] Automated builds for Calcite website
Posted by Vladimir Sitnikov <si...@gmail.com>.
Francis>javadocs takes around 20 minutes to build
I did not thought it takes so much time.
"tag" trigger for javadocs is clever, and I just thought we might want to
be able to update the wording on the site javadoc without releasing Calcite.
That is why I suggested to build javadoc for all site pushes.
I wonder if the job can reuse the workspace.
I guess it can see the results of the previous builds, so it could just
reuse the javadocs if they are the same.
Here's what I have for Avatica:
$ time ./gradlew javadoc
real 0m33.714s
user 0m5.499s
sys 0m0.399s
$ time ./gradlew javadoc
real 0m2.916s
user 0m2.646s
sys 0m0.208s
It skips the processing provided no modifications to the javadocs was made.
Vladimir
Re: [DISCUSS] Automated builds for Calcite website
Posted by Francis Chuang <fr...@apache.org>.
This will be my fallback solution if I can't get the job to trigger on a
tag push or a push to the site branch correctly.
The reason I wanted to build the javadocs only when a final release's
tag is pushed is because the javadocs takes around 20 minutes to build
on the node, whereas the site takes between 5 to 6 minutes. I think
needlessly building the javadocs every time the site is pushed is not as
efficient as I'd like the job to be.
On 4/07/2019 9:57 pm, Vladimir Sitnikov wrote:
> Francis>- For Javadocs (Calcite and Avatica) trigger and build when a final
> version is tagged.
>
> I think javadocs could be built on each push to site branch.
>
> Vladimir
>
Re: [DISCUSS] Automated builds for Calcite website
Posted by Vladimir Sitnikov <si...@gmail.com>.
Francis>- For Javadocs (Calcite and Avatica) trigger and build when a final
version is tagged.
I think javadocs could be built on each push to site branch.
Vladimir