You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by ab...@apache.org on 2020/08/27 10:38:27 UTC
[kudu] branch master updated: Upgrade docs/site building tools
This is an automated email from the ASF dual-hosted git repository.
abukor pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new de31237 Upgrade docs/site building tools
de31237 is described below
commit de312377d2fa71a4a39f7f9b8cf78615e8d47715
Author: Attila Bukor <ab...@apache.org>
AuthorDate: Wed Aug 26 13:57:28 2020 +0200
Upgrade docs/site building tools
The tooling to build the documentation for the Kudu website are very
old making it difficult to run it on newer OSes.
This patch upgrades the Ruby dependencies along with the required
Doxygen version to make it easier. As the new dependencies are fairly
new, they require Ruby and Doxygen versions newer than what's included
by default on CentOS 7 and likely on other OSes that we might want to
use to build the site. It's fairly simple to upgrade them on the other
hand, this patch also adds instructions to README.adoc.
Change-Id: I5eddf689da90123ca52075d09de07981d11f8ffe
Reviewed-on: http://gerrit.cloudera.org:8080/16371
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <as...@cloudera.com>
---
CMakeLists.txt | 8 +--
README.adoc | 22 +++++++
RELEASING.adoc | 9 ++-
docs/support/doxygen/client_api.doxy.in | 4 ++
docs/support/scripts/Gemfile | 5 --
docs/support/scripts/Gemfile.lock | 101 +++++++++++++++++---------------
docs/support/scripts/make_docs.sh | 30 +++++++---
7 files changed, 112 insertions(+), 67 deletions(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index b34c8b2..4b5dcbd 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -1369,13 +1369,13 @@ endif (UNIX)
############################################################
# "make doxygen" target
-# Requires doxygen of version >= 1.8.11 to leverage the
-# WARN_AS_ERROR configuration parameter (introduced in
-# version 1.8.11) for spotting various issues in the Kudu
+# Requires doxygen of version >= 1.8.19 that includes the
+# fix of a bug that would otherwise fail the build as we
+# treat warnings as errors to spot various issues in the
# C++ client API documentation.
############################################################
if (UNIX)
- find_package(Doxygen 1.8.11)
+ find_package(Doxygen 1.8.19)
if (NOT (DOXYGEN_FOUND AND DOXYGEN_DOT_FOUND))
message(WARNING "Doxygen with Dot support (graphviz) not found: 'doxygen' target is not available")
else ()
diff --git a/README.adoc b/README.adoc
index 6a546da..4603a9f 100644
--- a/README.adoc
+++ b/README.adoc
@@ -279,6 +279,7 @@ $ ./build-support/iwyu.py
See the output of `iwyu.py --help` for details on various modes of operation.
+[[building-docs]]
=== Building Kudu documentation
Kudu's documentation is written in asciidoc and lives in the _docs_ subdirectory.
@@ -297,6 +298,17 @@ emitted to _build/docs_. This script requires `ruby` and `gem` to be installed
on the system path, and will attempt to install `asciidoctor` and other related
dependencies into `$HOME/.gems` using https://bundler.io/[bundler].
+Some of the dependencies require a recent version of Ruby. To build the
+documentation on a system that comes with an older Ruby version (such as Ruby
+2.0 on CentOS 7), it is easiest to use https://github.com/rbenv/rbenv[rbenv] to
+install Ruby 2.7.
+
+WARNING: As the default values for some configuration options differ between Mac
+and Linux (e.g. file vs log block manager) and the configuration reference is
+generated by running the binaries with `-help`, the documentation *MUST NOT* be
+generated on Mac for publishing purposes, only for verification.
+
+[[updating-the-site]]
=== Updating the Kudu web site documentation
To update the documentation that is integrated into the Kudu web site,
@@ -315,6 +327,13 @@ These will be placed inside the checked-out web site, along with a tarball
containing only the generated documentation (the _docs/_ and _apidocs/_ paths
on the web site). Everything can be found in the _build/site_ subdirectory.
+To build the C++ Client API you need to have Doxygen 1.8.19 or later which is
+fairly new so you might need to
+https://www.doxygen.nl/manual/install.html#install_src_unix[build it from
+source]. To build it on RHEL/CentOS you'll also need
+https://www.softwarecollections.org/en/scls/rhscl/devtoolset-8/[devtoolset] as
+Doxygen uses {cpp}14 since 1.8.17.
+
You can proceed to commit the changes in the pages repository and send a code
review for your changes. In the future, this step may be automated whenever
changes are checked into the main Kudu repository.
@@ -322,6 +341,9 @@ changes are checked into the main Kudu repository.
After making changes to the `gh-pages` branch, follow the instructions below
when you want to deploy those changes to the live web site.
+WARNING: The site *MUST NOT* be built on Mac for publishing purposes, only for
+verification. See the warning in <<building-docs>> for details.
+
=== Deploying changes to the Apache Kudu web site
When the documentation is updated on the `gh-pages` branch, or when other web
diff --git a/RELEASING.adoc b/RELEASING.adoc
index e0e3f1f..9bd82ad 100644
--- a/RELEASING.adoc
+++ b/RELEASING.adoc
@@ -332,8 +332,13 @@ Note: You can upload to the test PyPi by adding
. Add the previous release notes to the `prior_release_notes.adoc` and submit the changes.
. Generate the version-specific documentation from that branch following these
- link:https://github.com/apache/kudu/#updating-the-kudu-web-site-documentation[instructions].
+ <<README.adoc#updating-the-site,instructions>>.
++
+
+WARNING: The site *MUST NOT* be built on Mac. See the
+<<README.adoc#building-docs,documentation building instructions>> for details.
++
. Update the `index.md` file in the `releases` directory. Add a line about
the newly released version into the `Latest release` section, move line
about the previous one into the `Previous releases`. Overall, update
@@ -368,7 +373,7 @@ Note: You can upload to the test PyPi by adding
. Submit these changes to the `gh-pages` Gerrit branch and get them reviewed.
. Once the review is finished and the commit is pushed, update the website following
- these link:https://github.com/apache/kudu/#deploying-changes-to-the-apache-kudu-web-site[instructions].
+ these <<README.adoc#deploying-changes-to-the-apache-kudu-web-site,instructions>>.
. Add a release blog post about the new release, similar to
link:https://kudu.apache.org/2019/07/09/apache-kudu-1-10-0-release.html[this].
diff --git a/docs/support/doxygen/client_api.doxy.in b/docs/support/doxygen/client_api.doxy.in
index f7e862c..c438bf5 100644
--- a/docs/support/doxygen/client_api.doxy.in
+++ b/docs/support/doxygen/client_api.doxy.in
@@ -67,3 +67,7 @@ GENERATE_LATEX = NO
# Do not output warnings on undocumented friend classes/structs/unions.
HIDE_FRIEND_COMPOUNDS = YES
+
+# Copy the first member's documentation to the rest in the group if they aren't
+# documented.
+DISTRIBUTE_GROUP_DOC = YES
diff --git a/docs/support/scripts/Gemfile b/docs/support/scripts/Gemfile
index 0ac227b..e1ffd74 100644
--- a/docs/support/scripts/Gemfile
+++ b/docs/support/scripts/Gemfile
@@ -15,20 +15,15 @@
# specific language governing permissions and limitations
# under the License.
-# We need to use versions that can run on CentOS 7, which ships Ruby 2.0.0.
-ruby '>= 2.0.0'
-
source 'https://rubygems.org'
gem 'thread_safe'
gem 'redcarpet'
gem 'jekyll'
-gem 'therubyracer' # V8 runtime installer for Jekyll.
gem 'ref'
gem 'json'
gem 'rdoc'
gem 'yajl-ruby', '>= 1.3.1' # Due to CVE-2017-16516.
-gem 'libv8', '>= 3.16.14.14' # Needed for macOS.
gem 'asciidoctor'
gem 'tilt'
diff --git a/docs/support/scripts/Gemfile.lock b/docs/support/scripts/Gemfile.lock
index f0587e3..2e1e038 100644
--- a/docs/support/scripts/Gemfile.lock
+++ b/docs/support/scripts/Gemfile.lock
@@ -1,57 +1,69 @@
GEM
remote: https://rubygems.org/
specs:
- addressable (2.5.2)
- public_suffix (>= 2.0.2, < 4.0)
- asciidoctor (1.5.6.1)
+ addressable (2.7.0)
+ public_suffix (>= 2.0.2, < 5.0)
+ asciidoctor (2.0.10)
colorator (1.1.0)
- ffi (1.9.24)
+ concurrent-ruby (1.1.7)
+ em-websocket (0.5.1)
+ eventmachine (>= 0.12.9)
+ http_parser.rb (~> 0.6.0)
+ eventmachine (1.2.7)
+ ffi (1.13.1)
forwardable-extended (2.6.0)
- jekyll (3.6.3)
+ http_parser.rb (0.6.0)
+ i18n (1.8.5)
+ concurrent-ruby (~> 1.0)
+ jekyll (4.1.1)
addressable (~> 2.4)
colorator (~> 1.0)
- jekyll-sass-converter (~> 1.0)
- jekyll-watch (~> 1.1)
- kramdown (~> 1.14)
+ em-websocket (~> 0.5)
+ i18n (~> 1.0)
+ jekyll-sass-converter (~> 2.0)
+ jekyll-watch (~> 2.0)
+ kramdown (~> 2.1)
+ kramdown-parser-gfm (~> 1.0)
liquid (~> 4.0)
- mercenary (~> 0.3.3)
+ mercenary (~> 0.4.0)
pathutil (~> 0.9)
- rouge (>= 1.7, < 3)
+ rouge (~> 3.0)
safe_yaml (~> 1.0)
- jekyll-sass-converter (1.5.0)
- sass (~> 3.4)
- jekyll-watch (1.5.0)
- listen (~> 3.0, < 3.1)
- json (2.1.0)
- kramdown (1.16.1)
- libv8 (3.16.14.19)
- liquid (4.0.0)
- listen (3.0.8)
- rb-fsevent (~> 0.9, >= 0.9.4)
- rb-inotify (~> 0.9, >= 0.9.7)
- mercenary (0.3.6)
- pathutil (0.16.0)
+ terminal-table (~> 1.8)
+ jekyll-sass-converter (2.1.0)
+ sassc (> 2.0.1, < 3.0)
+ jekyll-watch (2.2.1)
+ listen (~> 3.0)
+ json (2.3.1)
+ kramdown (2.3.0)
+ rexml
+ kramdown-parser-gfm (1.1.0)
+ kramdown (~> 2.0)
+ liquid (4.0.3)
+ listen (3.2.1)
+ rb-fsevent (~> 0.10, >= 0.10.3)
+ rb-inotify (~> 0.9, >= 0.9.10)
+ mercenary (0.4.0)
+ pathutil (0.16.2)
forwardable-extended (~> 2.6)
- public_suffix (2.0.5)
- rb-fsevent (0.10.2)
- rb-inotify (0.9.10)
- ffi (>= 0.5.0, < 2)
- rdoc (4.3.0)
- redcarpet (3.4.0)
+ public_suffix (4.0.5)
+ rb-fsevent (0.10.4)
+ rb-inotify (0.10.1)
+ ffi (~> 1.0)
+ rdoc (6.2.1)
+ redcarpet (3.5.0)
ref (2.0.0)
- rouge (1.11.1)
- safe_yaml (1.0.4)
- sass (3.5.3)
- sass-listen (~> 4.0.0)
- sass-listen (4.0.0)
- rb-fsevent (~> 0.9, >= 0.9.4)
- rb-inotify (~> 0.9, >= 0.9.7)
- therubyracer (0.12.3)
- libv8 (~> 3.16.14.15)
- ref
+ rexml (3.2.4)
+ rouge (3.22.0)
+ safe_yaml (1.0.5)
+ sassc (2.4.0)
+ ffi (~> 1.9)
+ terminal-table (1.8.0)
+ unicode-display_width (~> 1.1, >= 1.1.1)
thread_safe (0.3.6)
- tilt (2.0.8)
- yajl-ruby (1.3.1)
+ tilt (2.0.10)
+ unicode-display_width (1.7.0)
+ yajl-ruby (1.4.1)
PLATFORMS
ruby
@@ -60,17 +72,12 @@ DEPENDENCIES
asciidoctor
jekyll
json
- libv8 (>= 3.16.14.14)
rdoc
redcarpet
ref
- therubyracer
thread_safe
tilt
yajl-ruby (>= 1.3.1)
-RUBY VERSION
- ruby 2.0.0p648
-
BUNDLED WITH
- 1.16.0
+ 2.1.4
diff --git a/docs/support/scripts/make_docs.sh b/docs/support/scripts/make_docs.sh
index 1b0b742..7738374 100755
--- a/docs/support/scripts/make_docs.sh
+++ b/docs/support/scripts/make_docs.sh
@@ -34,6 +34,21 @@ usage() {
echo usage: "$0 --build_root <path> --output_subdir <relative path> [--site <path to gh-pages checkout>]"
}
+if [[ "$OSTYPE" == "linux-gnu"* ]]; then
+ SED="sed -i -e"
+elif [[ "$OSTYPE" == "darwin"* ]]; then
+ echo "WARNING: The docs MUST NOT be built on Mac for publishing purposes.
+ Press 'y' to continue"
+ read -n 1
+ if [[ "$REPLY" != "y" ]]; then
+ exit 1;
+ fi
+ SED="sed -i~ -e"
+else
+ echo "Unsupported OS"
+ exit 1
+fi
+
while [[ $# > 0 ]] ; do
arg=$1
case $arg in
@@ -113,10 +128,7 @@ BUNDLE="$GEM_PATH/bin/bundle"
echo "Locally installing ruby gems needed to build docs."
if [ ! -x "$BUNDLE" ]; then
set -x
- # The bundler install is pinned to an explicit version because versions after
- # 2.0.0 require ruby >= 2.3.0 which is not available on many supported
- # operating systems.
- gem install --no-ri --no-rdoc -q --install-dir "$GEM_PATH" bundler -v '1.17.3'
+ gem install --no-document -q --install-dir "$GEM_PATH" bundler
set +x
fi
@@ -163,7 +175,7 @@ for binary in ${binaries[@]}; do
-o $GEN_DOC_DIR/${binary}_configuration_reference.adoc \
$SOURCE_ROOT/docs/support/xsl/gflags_to_asciidoc.xsl \
${GEN_DOC_DIR}/$binary.xml
- INCLUSIONS_SUPPORTED+="include::${binary}_configuration_reference.adoc[leveloffset=+1]\n"
+ INCLUSIONS_SUPPORTED+="include::${binary}_configuration_reference.adoc[leveloffset=+1]\\"$'\n'
# Create the unsupported config reference
xsltproc \
@@ -172,13 +184,13 @@ for binary in ${binaries[@]}; do
-o $GEN_DOC_DIR/${binary}_configuration_reference_unsupported.adoc \
$SOURCE_ROOT/docs/support/xsl/gflags_to_asciidoc.xsl \
${GEN_DOC_DIR}/$binary.xml
- INCLUSIONS_UNSUPPORTED+="include::${binary}_configuration_reference_unsupported.adoc[leveloffset=+1]\n"
+ INCLUSIONS_UNSUPPORTED+="include::${binary}_configuration_reference_unsupported.adoc[leveloffset=+1]\\"$'\n'
done
# Add the includes to the configuration reference files, replacing the template lines
cp $SOURCE_ROOT/docs/configuration_reference* $GEN_DOC_DIR/
-sed -i "s#@@CONFIGURATION_REFERENCE@@#${INCLUSIONS_SUPPORTED}#" ${GEN_DOC_DIR}/configuration_reference.adoc
-sed -i "s#@@CONFIGURATION_REFERENCE@@#${INCLUSIONS_UNSUPPORTED}#" ${GEN_DOC_DIR}/configuration_reference_unsupported.adoc
+$SED "s#@@CONFIGURATION_REFERENCE@@#${INCLUSIONS_SUPPORTED}#" ${GEN_DOC_DIR}/configuration_reference.adoc
+$SED "s#@@CONFIGURATION_REFERENCE@@#${INCLUSIONS_UNSUPPORTED}#" ${GEN_DOC_DIR}/configuration_reference_unsupported.adoc
# Create tool references
echo "Running kudu --helpxml"
@@ -202,7 +214,7 @@ ${GEN_DOC_DIR}/kudu.xml
# Add the includes to the cli tools reference files, replacing the template lines
cp $SOURCE_ROOT/docs/command_line_tools_reference.adoc $GEN_DOC_DIR/
-sed -i "s#@@TOOLS_REFERENCE@@#include::command_line_tools.adoc[leveloffset=+1]\n#" ${GEN_DOC_DIR}/command_line_tools_reference.adoc
+$SED "s#@@TOOLS_REFERENCE@@#include::command_line_tools.adoc[leveloffset=+1]\\"$'\n#' ${GEN_DOC_DIR}/command_line_tools_reference.adoc
# If we're generating the web site, pass the template which causes us
# to generate Jekyll templates instead of full HTML.