You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by mc...@apache.org on 2021/09/02 11:43:42 UTC
[cassandra-website] 03/07: CASSANDRA-16066: Add website generation
tooling
This is an automated email from the ASF dual-hosted git repository.
mck pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra-website.git
commit 56a532993af73f1f5ddf548cfd3da48831473d50
Author: Anthony Grasso <an...@thelastpickle.com>
AuthorDate: Fri Oct 16 23:43:17 2020 +1100
CASSANDRA-16066: Add website generation tooling
This commit replaces the existing Docker website generation tool. The original
website generation tool was Jekyll. It has been replaced with Antora which
is purpose built for handling different document versions in a repository.
A Docker container has been added which renders the site using Antora. Unlike
other Antora configurations which have a site.yaml file committed, this
implementaiton uses a site.yaml template and python script to generate the
final site.yaml. The Docker container is responsible for calling the script
to generate the site.yaml file using the template. The generated file is then
passed to Antora to render the final site. This has been done so that different
document sources can be passed to Antora and newly released Cassandra versions
will automatically appear in the Downloads page.
The source content and styling which Antora uses to generate the site can be
controlled via the container environment variables.
The container includes a preview mode which monitors the content directories
and runs Antora when any file changes in the content directories.
By default, only the cassandra-website will be used as the source for Antora.
That is unless the Cassandra document generation is specified via the
generate-docs command, the container will only generate the website HTML.
patch by Anthony Grasso; reviewed by Mick Semb Wever, Lorina Poland, Melissa Logan, Paul Au for CASSANDRA-16066
---
.gitignore | 3 +
Dockerfile | 58 -------
README.md | 116 ++++++++++---
docker-compose.yaml | 15 ++
docker-entrypoint.sh | 51 ------
site-content/Dockerfile | 124 +++++++++++++
site-content/bin/site_yaml_generator.py | 158 +++++++++++++++++
site-content/docker-entrypoint.sh | 299 ++++++++++++++++++++++++++++++++
site-content/site.template.yaml | 80 +++++++++
9 files changed, 773 insertions(+), 131 deletions(-)
diff --git a/.gitignore b/.gitignore
index 7ca2a18..9dd1f7b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,6 +1,9 @@
.DS_Store
/.idea/
+.env*
+
+/site-content/site.yaml
/site-content/build/
/site-ui/build/
diff --git a/Dockerfile b/Dockerfile
deleted file mode 100644
index e8ced6f..0000000
--- a/Dockerfile
+++ /dev/null
@@ -1,58 +0,0 @@
-FROM debian:stretch
-
-# Set up non-root user, 'build', with default uid:gid
-# This allows passing --build-arg to use local host user's uid:gid:
-# $ docker-compose build \
-# --build-arg UID=$(id -u) \
-# --build-arg GID=$(id -g) \
-# cassandra-website
-ARG UID=1000
-ARG GID=1000
-RUN echo "Setting up user 'build' with UID=${UID} GID=${GID}"
-RUN groupadd --gid $GID --non-unique build
-RUN useradd --create-home --shell /bin/bash \
- --uid $UID --gid $GID --non-unique build
-
-# Install tools
-RUN apt-get update && \
- apt-get install -y \
- openjdk-8-jdk \
- procps \
- git \
- python2.7 \
- python-pip \
- ruby-full \
- make \
- ant \
- ant-optional \
- maven \
- wget
-
-# Install Sphinx for generating Cassandra docs
-RUN pip install --no-cache-dir \
- sphinx \
- sphinx_rtd_theme
-
-COPY ./src/Gemfile /
-COPY ./src/Gemfile.lock /
-
-RUN gem install bundler && \
- bundle install && \
- rm /Gemfile /Gemfile.lock
-
-ENV BUILD_DIR="/home/build"
-
-# Setup directories for building the docs
-# Give the build user rw access to everything in the build directory,
-# neccessary for the ASF 'websites' jenkins agent (which can't chown)
-RUN mkdir -p ${BUILD_DIR}/cassandra-site && \
- git clone https://gitbox.apache.org/repos/asf/cassandra.git ${BUILD_DIR}/cassandra && \
- chmod -R a+rw ${BUILD_DIR}
-
-EXPOSE 4000/tcp
-
-# Run as build user from here
-USER build
-COPY docker-entrypoint.sh /home/build/
-ENTRYPOINT ["/home/build/docker-entrypoint.sh"]
-CMD [""]
diff --git a/README.md b/README.md
index 5c87a9a..8103c1a 100644
--- a/README.md
+++ b/README.md
@@ -1,8 +1,8 @@
-Apache Cassandra website
-========================
+# Apache Cassandra website
-Development Cycle
------------------
+TODO: Add notes about the repository structure
+
+## Development Cycle
Making changes to the website content can be done using the following steps.
@@ -14,34 +14,29 @@ Making changes to the website content can be done using the following steps.
5. Merge `asf-staging` to `asf-site`.
6. View the rendered site on https://cassandra.apache.org/.
+# Developer Quickstart
-To test changes before committing, it is a requirement that you build the website locally. Building the Apache Cassandra website takes a number of steps. To make things easier we have provided a Docker container which can build the full website in two simple commands and have it ready to commit via git.
+To test changes before committing, it is a requirement that you build the website locally. Building the Apache Cassandra website takes a number of steps. To make things easier we have provided a Docker container which can build the full website in a few simple commands and have it ready to commit via git.
-Building Prerequisites
-----------------------
+## Building Prerequisites
To build and run the Docker container you will need `Docker` version 2.0.0.0 or greater. If you need a copy of the site code you will need `git` as well.
-
-# WARNING: The content below is incorrect and needs to be updated.
-
-
-Building the site
------------------
+## Building the Website
If you need a copy of the site code run this command:
```bash
$ git clone https://github.com/apache/cassandra-website.git
$ cd ./cassandra-website
-
```
To build the website run the following commands from within the `./cassandra-website` directory (assuming you used the above clone command):
```bash
-$ docker-compose build cassandra-website
-$ docker-compose run cassandra-website
+$ export BUILD_USER=build
+$ docker-compose build website
+$ docker-compose run website
```
:warning: *Tip:* In order to prevent root-owned modified files in your repository, the container user, `build`, is set up with a default UID=1000:GID=1000, which is usually the first user configured on a linux machine. If your local user is different you should set up the container user with your local host user's UID:GID, replace the above with:
@@ -49,17 +44,57 @@ $ docker-compose run cassandra-website
:warning: *Tip:* Building cassandra-website may fail if the `CLOUDSDK_PYTHON` environment variable is not set on your machine. For example, set the environment variable to export `CLOUDSDK_PYTHON=/usr/bin/python2`.
```bash
-$ docker-compose build --build-arg UID=$(id -u) --build-arg GID=$(id -g) cassandra-website
-$ docker-compose run cassandra-website
+$ export BUILD_USER=build
+$ docker-compose build --build-arg UID_ARG=$(id -u) --build-arg GID_ARG=$(id -g) website
+$ docker-compose run website
```
-Go make yourself a cup of coffee, this will take a while...
-
Once building has completed, the site content will be in the `./cassandra-website/site-content/build` directory ready to be tested.
+### Customising the Website Build for Development
+
+#### You can use your own Cassandra fork
+
+The container contains a local copy of the Apache Cassandra repository. The document generation process commits the generated AsciiDoc to the local repository. This is done to allow Antora to render to HTML the versioned documentation and website using the committed AsciiDoc in each branch.
+
+If you have a Cassandra fork you would like to use instead, you can override the repository used when building the container:
+
+```
+$ export BUILD_USER=build
+$ docker-compose build --build-arg CASSANDRA_REPOSITORY_URL_ARG=<git_url_to_your_fork> website
+$ docker-compose run website
+```
+
+Note that you can specify multiple `--build-arg` options when calling `docker-compose build`. So if you wanted to use a different user ID, group ID and repository you could run the following command:
+
+```
+$ export BUILD_USER=build
+$ docker-compose build \
+ --build-arg CASSANDRA_REPOSITORY_URL_ARG=<git_url_to_your_fork> \
+ --build-arg UID_ARG=$(id -u) \
+ --build-arg GID_ARG=$(id -g) \
+ website
+$ docker-compose run website
+```
+
+In addition to customising the container build, you can customise the environment variables used inside the container. These can be used to influence Antora's rending of the site. For example, Antora will generate documentation for all Cassandra releases starting at 3.11.5. Hence, if you have a specific branch in your fork of Cassandra that you would like to render, you can override the variable the specifies the branches to render.
+
+TODO: Explain that you need an env.dev file with variables in it and to run `docker-compose run website-dev`
+
+Below is a select list of environment variables that can be overridden to control Antora's rendering of the site and why you would want to override them.
+
+TODO: Add reason why you would override these
+`CASSANDRA_REPOSITORY_URL`
+`CASSANDRA_VERSIONS`
+`CASSANDRA_WEBSITE_REPOSITORY_URL`
+`CASSANDRA_WEBSITE_VERSIONS`
+`UI_BUNDLE_ZIP_URL`
-Previewing the site
--------------------
+
+
+# WARNING: From here on is all wrong!!!!
+
+### Previewing the Website Build
The fastest way to preview the site is to run the following:
@@ -106,3 +141,40 @@ Updating the main website, after verifying the staged website, involves copying
git switch asf-site
git reset --hard origin/asf-staging
git push -f origin asf-site
+
+
+## Building the Site UI
+
+### Executing Tasks
+
+To get a list of the tasks that can be executed run the following commands:
+
+```
+$ export BUILD_USER=build
+$ docker-compose build website-ui:
+$ docker-compose run website-ui
+```
+
+A task can be executed using the following commands:
+
+```bash
+$ export BUILD_USER=build
+$ docker-compose build website-ui:
+$ docker-compose run website-ui <task>
+```
+
+### Building Bundle
+
+TODO: Add info
+
+### Releasing Bundle
+
+TODO: Add info
+
+### Preview UI
+
+```bash
+$ export BUILD_USER=build
+$ docker-compose build website-ui-preview
+$ docker-compose up website-ui-preview
+```
\ No newline at end of file
diff --git a/docker-compose.yaml b/docker-compose.yaml
new file mode 100644
index 0000000..0d18bb6
--- /dev/null
+++ b/docker-compose.yaml
@@ -0,0 +1,15 @@
+version: "3.8"
+services:
+ website-ui:
+ build: ./site-ui/
+ volumes:
+ - ./site-ui/:/home/site-ui
+
+ website-ui-preview:
+ build: ./site-ui/
+ ports:
+ - "5252:5252"
+ volumes:
+ - ./site-ui/:/home/site-ui
+ command:
+ - preview
diff --git a/docker-entrypoint.sh b/docker-entrypoint.sh
deleted file mode 100755
index e6b9d5f..0000000
--- a/docker-entrypoint.sh
+++ /dev/null
@@ -1,51 +0,0 @@
-#!/usr/bin/env bash
-
-set -xe
-
-export CASSANDRA_SITE_DIR="${BUILD_DIR}/cassandra-site"
-export CASSANDRA_DIR="${BUILD_DIR}/cassandra"
-
-jekyll --version
-
-cd ${CASSANDRA_SITE_DIR}/src
-
-# Now make the docs for the latest version
-pushd ${CASSANDRA_DIR}
-git clean -xdff
-git checkout trunk
-git pull --rebase --prune
-popd
-make .build-doc .latest-doc-link
-
-
-# Uncomment for building docs for each of the released versions on the 3.11 branch, from 3.11.5 (previous don't work)
-#
-#versions="cassandra-3.11.5 cassandra-3.11.6 cassandra-3.11.7 cassandra-3.11.8"
-#
-#for v in $versions ; do
-# pushd ${CASSANDRA_DIR}
-# git clean -xdff
-# git checkout $v
-# popd
-# make .build-doc
-#done
-
-# Now make the docs for 3.11
-pushd ${CASSANDRA_DIR}
-git clean -xdff
-git checkout cassandra-3.11
-git pull --rebase --prune
-popd
-make .build-doc .stable-doc-link
-
-# Relink the 3.11 version
-LATEST_VERSION=$(basename $(find ./doc -iname 3.11* -type d | sort | tail -n 1))
-rm -f doc/3.11
-ln -s -f ${LATEST_VERSION} doc/3.11
-
-# Generate the rest of the site
-make build
-
-# Fix the links in the resource paths for the landing pages of each version in the publish directory
-cd ${CASSANDRA_SITE_DIR}
-sed -i 's/\.\/\.\.\//\.\/\.\.\/\.\.\//g' ./publish/doc/*/index.html
diff --git a/site-content/Dockerfile b/site-content/Dockerfile
new file mode 100644
index 0000000..73d9755
--- /dev/null
+++ b/site-content/Dockerfile
@@ -0,0 +1,124 @@
+FROM ubuntu:18.04
+# Set up non-root user, 'build', with default uid:gid
+# This allows passing --build-arg to use localhost username, and uid:gid:
+# $ docker build \
+# -t cassandra-website:latest \
+# --build-arg BUILD_USER_ARG=$(whoami) \
+# --build-arg UID_ARG=$(id -u) \
+# --build-arg GID_ARG=$(id -g) \
+# .
+#
+# Other container parameters can be overridden at build time as well:
+# - NODE_VERSION_ARG: Version of node to use.
+# - ENTR_VERSION_ARG: Version of entr to use.
+ARG BUILD_USER_ARG="build"
+ARG UID_ARG=1000
+ARG GID_ARG=1000
+ARG NODE_VERSION_ARG="v12.16.2"
+ARG ENTR_VERSION_ARG="4.6"
+
+ENV BUILD_USER=${BUILD_USER_ARG}
+
+RUN echo "Building with arguments:" \
+ && echo " - BUILD_USER_ARG=${BUILD_USER_ARG}" \
+ && echo " - UID_ARG=${UID_ARG}" \
+ && echo " - GID_ARG=${GID_ARG}" \
+ && echo " - NODE_VERSION_ARG=${NODE_VERSION_ARG}" \
+ && echo " - ENTR_VERSION_ARG=${ENTR_VERSION_ARG}"
+
+RUN echo "Setting up user '${BUILD_USER}'"
+RUN groupadd --gid ${GID_ARG} --non-unique ${BUILD_USER}
+RUN useradd --create-home --shell /bin/bash \
+ --uid ${UID_ARG} --gid ${GID_ARG} --non-unique ${BUILD_USER}
+
+# INSTALL wget, python3, java11, and other tools required to build the docs
+RUN apt-get update && \
+ apt-get install -y \
+ wget \
+ gpg \
+ python3 \
+ python3-pip \
+ openjdk-11-jdk \
+ git \
+ make \
+ ant \
+ ant-optional \
+ vim
+
+RUN pip3 install jinja2 requests
+
+# INSTALL nodejs and nvm
+ENV NODE_PACKAGE="node-${NODE_VERSION_ARG}-linux-x64.tar.gz"
+RUN wget https://nodejs.org/download/release/${NODE_VERSION_ARG}/${NODE_PACKAGE} && \
+ tar -C /usr/local --strip-components 1 -xzf ${NODE_PACKAGE} && \
+ rm ${NODE_PACKAGE}
+
+# Use npm to install Antora globally, and antora-lunr for site search, and js-yaml to load YAML files
+RUN npm i -g @antora/cli@2.3 @antora/site-generator-default@2.3 @djencks/asciidoctor-openblock
+RUN npm i -g antora-lunr antora-site-generator-lunr
+RUN npm i -g live-server
+
+# Setup directories for building the docs
+# Give the build user rw access to everything in the build directory,
+# neccessary for the ASF 'website'.
+ENV BUILD_DIR="/home/${BUILD_USER}"
+ENV ENTR_PACKAGE="${ENTR_VERSION_ARG}.tar.gz"
+WORKDIR ${BUILD_DIR}
+RUN wget https://github.com/eradman/entr/archive/${ENTR_PACKAGE} && \
+ mkdir entr && \
+ tar -C ${BUILD_DIR}/entr --strip-components 1 -xzf ${ENTR_PACKAGE} && \
+ rm ${ENTR_PACKAGE}
+
+WORKDIR ${BUILD_DIR}/entr
+RUN ./configure && \
+ make test && \
+ make install
+
+WORKDIR ${BUILD_DIR}
+RUN mkdir -p ${BUILD_DIR}/cassandra-website && \
+ mkdir -p ${BUILD_DIR}/cassandra && \
+ chmod -R a+rw ${BUILD_DIR} && \
+ chown -R ${BUILD_USER}:${BUILD_USER} ${BUILD_DIR}
+
+# Set defaults for site build environment variables.
+ENV GIT_EMAIL_ADDRESS="${BUILD_USER}@apache.org"
+ENV GIT_USER_NAME="${BUILD_USER}"
+
+ENV ANTORA_SITE_TITLE="Apache Cassandra Documentation"
+ENV ANTORA_SITE_URL="/"
+ENV ANTORA_SITE_START_PAGE="_"
+
+# Build from 3.11.5 as document generation for previous versions is broken.
+ENV ANTORA_CONTENT_SOURCES_CASSANDRA_URL="https://github.com/apache/cassandra.git"
+ENV ANTORA_CONTENT_SOURCES_CASSANDRA_BRANCHES="trunk cassandra-4.0 cassandra-3.11"
+ENV ANTORA_CONTENT_SOURCES_CASSANDRA_TAGS=""
+ENV ANTORA_CONTENT_SOURCES_CASSANDRA_START_PATH="doc"
+
+ENV ANTORA_CONTENT_SOURCES_CASSANDRA_WEBSITE_URL="https://github.com/apache/cassandra-website.git"
+ENV ANTORA_CONTENT_SOURCES_CASSANDRA_WEBSITE_BRANCHES="HEAD"
+ENV ANTORA_CONTENT_SOURCES_CASSANDRA_WEBSITE_TAGS=""
+ENV ANTORA_CONTENT_SOURCES_CASSANDRA_WEBSITE_START_PATH="site-content/source"
+
+ENV ANTORA_UI_BUNDLE_URL="https://github.com/ianjevans/antora-ui-datastax/releases/download/v0.1oss/ui-bundle.zip"
+
+ENV CASSANDRA_DOWNLOADS_URL="https://downloads.apache.org/cassandra/"
+
+ENV INCLUDE_VERSION_DOCS_WHEN_GENERATING_WEBSITE="disabled"
+ENV COMMIT_GENERATED_VERSION_DOCS_TO_REPOSITORY="enabled"
+
+EXPOSE 5151/tcp
+
+# Run as build user from here
+USER ${BUILD_USER}
+WORKDIR ${BUILD_DIR}
+COPY docker-entrypoint.sh /usr/local/bin/
+ENTRYPOINT ["docker-entrypoint.sh"]
+
+# Possible commands are listed below. The entrypoint will accept any combination of these commands.
+#
+# generate-docs - Generate Cassandra documentation using Antora
+# build-site - Build site using Antora
+# preview - Run container in preview mode where Antora is called to regenerate the site everytime content files are changed
+#
+# By default we will only generate the docs and build the site.
+CMD ["generate-docs","build-site"]
diff --git a/site-content/bin/site_yaml_generator.py b/site-content/bin/site_yaml_generator.py
new file mode 100755
index 0000000..69e5048
--- /dev/null
+++ b/site-content/bin/site_yaml_generator.py
@@ -0,0 +1,158 @@
+#!/usr/bin/env python3
+
+import abc
+import argparse
+import html.parser
+import jinja2
+import json
+import requests
+import re
+
+
+class SiteYAML:
+ def __init__(self):
+ self._site_dict = {}
+ self._source_list = []
+ self._ui_bundle_dict = {}
+ self._asciidoc_attributes = {}
+
+ def set_site_setting(self, site_info):
+ self._site_dict = json.loads(site_info)
+
+ def set_content_source_setting(self, content_source_list):
+ for source in content_source_list:
+ self._source_list.append(json.loads(source))
+
+ def set_ui_bundle_setting(self, ui_bundle_url):
+ self._ui_bundle_dict["url"] = ui_bundle_url
+
+ def set_asciidoc_attributes(self, release_download_url):
+ response = requests.get(release_download_url)
+
+ if not response.ok:
+ raise IOError("Failed to retrieve download information from: {}".format(release_download_url))
+
+ parser = DownloadsHTMLParser()
+ parser.feed(response.text)
+
+ for version in parser.version_list:
+ version_key = version.key
+
+ # Add the latest version as "latest" as well as its version number.
+ if parser.latest == version.release_number:
+ self._asciidoc_attributes["latest"] = {"name": version.name, "date": version.date}
+
+ self._asciidoc_attributes[version_key] = {"name": version.name, "date": version.date}
+ self._asciidoc_attributes['url_downloads_cassandra'] = release_download_url
+
+ def generate_file(self):
+ template_loader = jinja2.FileSystemLoader(searchpath="./")
+ template_env = jinja2.Environment(loader=template_loader)
+ template = template_env.get_template(args.file_path)
+ template.stream(
+ site=self._site_dict,
+ source_list=self._source_list,
+ ui_bundle=self._ui_bundle_dict,
+ asciidoc_attributes=self._asciidoc_attributes
+ ).dump("site.yaml")
+
+
+class Version:
+ def __init__(self, version):
+ v_components = version.split(".")
+ v_major = v_components[0]
+ v_minor = v_components[1].split("-")[0]
+
+ self.key = "v{}{}".format(v_major, v_minor)
+ self.name = version
+ self.date = ""
+ self.release_number = float("{}.{}".format(v_major, v_minor))
+
+
+class DownloadsHTMLParser(html.parser.HTMLParser, abc.ABC):
+ def __init__(self):
+ super().__init__()
+
+ self._current_version = None
+ self._previous_tag = None
+
+ self._version_pattern = re.compile(r"^\d+\.\d+[.\-\w]+\d+")
+ self._date_pattern = re.compile(r"^\d+-\d+-\d+")
+
+ self.version_list = []
+ self.latest = 0.0
+
+ def handle_starttag(self, tag, attrs):
+ if tag == "a" and len(attrs) > 0:
+ for attr in attrs:
+ if attr[0] == "href":
+ regex_match = self._version_pattern.match(attr[1])
+
+ if regex_match:
+ self._current_version = Version(regex_match.group())
+
+ if self._current_version.release_number > self.latest:
+ self.latest = self._current_version.release_number
+
+ def handle_endtag(self, tag):
+ self._previous_tag = tag
+
+ def handle_data(self, data):
+ if self._previous_tag == "a" and self._current_version:
+ regex_match = self._date_pattern.match(data.strip())
+
+ if regex_match:
+ self._current_version.date = regex_match.group()
+ self.version_list.append(self._current_version)
+ self._current_version = None
+
+ self._previous_tag = None
+
+
+def build_arg_parser():
+ parser = argparse.ArgumentParser(description="Generate the site.yml using the site.yml.template.")
+
+ parser.add_argument(
+ "file_path", metavar="FILE_PATH", help="Path to site.template to use to generate site.yaml")
+ parser.add_argument(
+ "-s",
+ "--site-info",
+ metavar="JSON",
+ required=True,
+ dest="site_info",
+ help="Information about the site.")
+ parser.add_argument(
+ "-c"
+ "--content-source",
+ metavar="JSON",
+ required=True,
+ action="append",
+ dest="content_source_list",
+ help="JSON object containing the url, branches, tags, and start_path of a source for the website.")
+ parser.add_argument(
+ "-u",
+ "--ui-bundle-zip-url",
+ metavar="URL",
+ required=True,
+ dest="ui_bundle_url",
+ help="Local path or URL to UI bundle.zip.")
+ parser.add_argument(
+ "-r",
+ "--release-download-url",
+ metavar="URL",
+ required=True,
+ dest="release_download_url",
+ help="URL to the page listing all the available downloads.")
+
+ return parser
+
+# --- main ---
+
+
+args = build_arg_parser().parse_args()
+site_yaml = SiteYAML()
+site_yaml.set_site_setting(args.site_info)
+site_yaml.set_content_source_setting(args.content_source_list)
+site_yaml.set_ui_bundle_setting(args.ui_bundle_url)
+site_yaml.set_asciidoc_attributes(args.release_download_url)
+site_yaml.generate_file()
diff --git a/site-content/docker-entrypoint.sh b/site-content/docker-entrypoint.sh
new file mode 100755
index 0000000..ebc8604
--- /dev/null
+++ b/site-content/docker-entrypoint.sh
@@ -0,0 +1,299 @@
+#!/bin/bash
+
+# Abort script if a command fails
+set -e
+
+export CASSANDRA_USE_JDK11=true
+export CASSANDRA_WEBSITE_DIR="${BUILD_DIR}/cassandra-website"
+export CASSANDRA_DIR="${BUILD_DIR}/cassandra"
+export CASSANDRA_DOC="${CASSANDRA_DIR}/doc"
+GIT_USER_SETUP="false"
+
+setup_git_user() {
+ if [ "${GIT_USER_SETUP}" = "false" ]
+ then
+ # Setup git so we can commit back to the Cassandra repository locally
+ git config --global user.email "${GIT_EMAIL_ADDRESS}"
+ git config --global user.name "${GIT_USER_NAME}"
+ GIT_USER_SETUP="true"
+ fi
+}
+
+generate_cassandra_versioned_docs() {
+ if [ "$(find "${CASSANDRA_DIR}" -mindepth 1 -type f | wc -l)" -eq 0 ]
+ then
+ git clone "${ANTORA_CONTENT_SOURCES_CASSANDRA_URL}" "${BUILD_DIR}"/cassandra
+
+ # Once the repository has been cloned set the Antora Cassandra source URL to be the local copy we have cloned in
+ # the container. This is so it will be used as the version documentation source by Antora if we are generating the
+ # website HTML.
+ ANTORA_CONTENT_SOURCES_CASSANDRA_URL="${BUILD_DIR}"/cassandra
+ fi
+
+ # If we are generating the website HTML as well, make sure the versioned documentation is part of the output.
+ INCLUDE_VERSION_DOCS_WHEN_GENERATING_WEBSITE="enabled"
+
+ mkdir -p "${CASSANDRA_DIR}"/cassandra/doc/build_gen
+
+ local commit_changes_to_branch=""
+ if [ "$(wc -w <<< "${GENERATE_CASSANDRA_VERSIONS}")" -gt 1 ] || [ "${COMMIT_GENERATED_VERSION_DOCS_TO_REPOSITORY}" = "enabled" ]
+ then
+ commit_changes_to_branch="enabled"
+ else
+ commit_changes_to_branch="disabled"
+ fi
+
+ for version in ${GENERATE_CASSANDRA_VERSIONS}
+ do
+ echo "Checking out '${version}'"
+ pushd "${CASSANDRA_DIR}" > /dev/null
+ git clean -xdff
+ git checkout "${version}"
+ git pull --rebase --prune
+
+ echo "Building JAR files"
+ # Nodetool docs are autogenerated, but that needs nodetool to be built
+ ant jar
+ local doc_version=""
+ doc_version=$(ant echo-base-version | grep "\[echo\]" | tr -s ' ' | cut -d' ' -f3)
+ popd > /dev/null
+
+ pushd "${CASSANDRA_DOC}" > /dev/null
+ # cassandra-3.11 is missing gen-nodetool-docs.py, ref: CASSANDRA-16093
+ gen_nodetool_docs=$(find . -iname gen-nodetool-docs.py | head -n 1)
+ if [ ! -f "${gen_nodetool_docs}" ]
+ then
+ echo "Unable to find ${gen_nodetool_docs}, so I will download it from the Cassandra repository using commit a47be7e."
+ wget \
+ -nc \
+ -O ./gen-nodetool-docs.py \
+ https://raw.githubusercontent.com/apache/cassandra/a47be7eddd5855fc7723d4080ca1a63c611efdab/doc/gen-nodetool-docs.py
+ fi
+
+ echo "Generating asciidoc for version ${doc_version}"
+ # generate the nodetool docs
+ python3 "${gen_nodetool_docs}"
+
+ # generate cassandra.yaml docs
+ local convert_yaml_to_adoc=$(find . -iname convert_yaml_to_adoc.py | head -n 1)
+ if [ -f "${gen_nodetool_docs}" ]
+ then
+ YAML_INPUT="${CASSANDRA_DIR}/conf/cassandra.yaml"
+ YAML_OUTPUT="${CASSANDRA_DOC}/modules/cassandra/pages/configuration/cass_yaml_file.adoc"
+ python3 "${convert_yaml_to_adoc}" "${YAML_INPUT}" "${YAML_OUTPUT}"
+ fi
+
+ if [ "${commit_changes_to_branch}" = "enabled" ]
+ then
+ git add .
+ git commit -m "Generated nodetool and configuration documentation for ${doc_version}." || echo "No new changes to commit."
+ fi
+ popd > /dev/null
+ done
+}
+
+string_to_json() {
+ local key="${1}"
+ local value="${2}"
+
+ echo -e "\"${key}\":\"${value}\""
+}
+
+list_to_json() {
+ local key="${1}"
+ local value="${2}"
+
+ echo -e "\"${key}\":[$(echo \""${value}"\" | sed 's~\ ~\",\"~g')]"
+}
+
+generate_json() {
+ local json_output
+ local count
+
+ json_output="{"
+ count=1
+ while true
+ do
+ local arg
+ local json_type
+ local key
+ local value
+
+ arg="${!count}"
+
+ if [ -z "${arg}" ]
+ then
+ break
+ fi
+
+ json_type="$(cut -d':' -f1 <<< ${arg})"
+ key="$(cut -d':' -f2 <<< ${arg})"
+ value=${arg//${json_type}:${key}:/}
+ if [ -n "${value}" ]
+ then
+ json_obj=$("${json_type}_to_json" "${key}" "${value}")
+
+ if [ "${json_output}" = "{" ]
+ then
+ json_output="${json_output}${json_obj}"
+ else
+ json_output="${json_output},${json_obj}"
+ fi
+ fi
+ count=$((count + 1))
+ done
+ json_output="${json_output}}"
+
+ echo -e "${json_output}"
+}
+
+generate_site_yaml() {
+ pushd "${CASSANDRA_WEBSITE_DIR}/site-content" > /dev/null
+
+ if [ "${INCLUDE_VERSION_DOCS_WHEN_GENERATING_WEBSITE}" = "enabled" ]
+ then
+ ANTORA_CONTENT_SOURCE_REPOSITORIES+=(CASSANDRA)
+ fi
+
+ local repository_url=""
+ local start_path=""
+ local branches=""
+ local tags=""
+ local content_source_options=()
+ for repo in ${ANTORA_CONTENT_SOURCE_REPOSITORIES[*]}
+ do
+ repository_url=$(eval echo "$"ANTORA_CONTENT_SOURCES_${repo}_URL"")
+ start_path=$(eval echo "$"ANTORA_CONTENT_SOURCES_${repo}_START_PATH"")
+ branches=$(eval echo "$"ANTORA_CONTENT_SOURCES_${repo}_BRANCHES"")
+ tags=$(eval echo "$"ANTORA_CONTENT_SOURCES_${repo}_TAGS"")
+
+ if [ -n "${repository_url}" ] && [ -n "${start_path}" ] && { [ -n "${branches}" ] || [ -n "${tags}" ]; }
+ then
+ content_source_options+=("-c")
+ content_source_options+=("$(generate_json \
+ "string:url:${repository_url}" \
+ "string:start_path:${start_path}" \
+ "list:branches:${branches}" \
+ "list:tags:${tags}")")
+ fi
+ done
+
+ echo "Building site.yaml"
+ rm -f site.yaml
+ python3 ./bin/site_yaml_generator.py \
+ -s "$(generate_json \
+ "string:title:${ANTORA_SITE_TITLE}" \
+ "string:url:${ANTORA_SITE_URL}" \
+ "string:start_page:${ANTORA_SITE_START_PAGE}") "\
+ "${content_source_options[@]}" \
+ -u "${ANTORA_UI_BUNDLE_URL}" \
+ -r "${CASSANDRA_DOWNLOADS_URL}" \
+ site.template.yaml
+ popd > /dev/null
+}
+
+render_site_content_to_html() {
+ pushd "${CASSANDRA_WEBSITE_DIR}/site-content" > /dev/null
+ echo "Building the site HTML content."
+ antora --generator antora-site-generator-lunr site.yaml
+ echo "Rendering complete!"
+ popd > /dev/null
+}
+
+run_preview_mode() {
+ echo "Entering preview mode!"
+
+ export -f render_site_content_to_html
+
+ local on_change_functions="render_site_content_to_html"
+ local find_paths="${CASSANDRA_WEBSITE_DIR}/${ANTORA_CONTENT_SOURCES_CASSANDRA_WEBSITE_START_PATH}"
+
+ if [ "${COMMAND_GENERATE_DOCS}" = "run" ]
+ then
+ on_change_functions="generate_cassandra_versioned_docs && ${on_change_functions}"
+ find_paths="${find_paths} ${CASSANDRA_DIR}/${ANTORA_CONTENT_SOURCES_CASSANDRA_START_PATH}"
+
+ export -f generate_cassandra_versioned_docs
+
+ # Ensure we only have one branch to generate docs for
+ GENERATE_CASSANDRA_VERSIONS=$(cut -d' ' -f1 <<< "${GENERATE_CASSANDRA_VERSIONS}")
+ fi
+
+ if [ "${COMMAND_BUILD_SITE}" != "run" ]
+ then
+ generate_site_yaml
+
+ export DOCSEARCH_ENABLED=true
+ export DOCSEARCH_ENGINE=lunr
+ export NODE_PATH="$(npm -g root)"
+ export DOCSEARCH_INDEX_VERSION=latest
+
+ render_site_content_to_html
+ fi
+
+ pushd "${CASSANDRA_WEBSITE_DIR}/site-content/build/html" > /dev/null
+ live-server --port=5151 --host=0.0.0.0 --no-browser --no-css-inject --wait=2000 &
+ popd > /dev/null
+
+ find "${find_paths}" -type f | entr /bin/bash -c "${on_change_functions}"
+}
+
+
+# ============ MAIN ============
+
+GENERATE_CASSANDRA_VERSIONS=$(sed 's/^[[:space:]]]*//' <<< "${ANTORA_CONTENT_SOURCES_CASSANDRA_BRANCHES} ${ANTORA_CONTENT_SOURCES_CASSANDRA_TAGS}")
+export GENERATE_CASSANDRA_VERSIONS
+
+ANTORA_CONTENT_SOURCE_REPOSITORIES=(
+ CASSANDRA_WEBSITE
+)
+
+# Initialise commands and assume none of them will run
+COMMAND_GENERATE_DOCS="skip"
+COMMAND_BUILD_SITE="skip"
+COMMAND_PREVIEW="skip"
+
+# Work out which commands the caller has requested. We need to do this first as the commands should be run in a certain order
+while [ "$1" != "" ]
+do
+ case $1 in
+ "generate-docs")
+ COMMAND_GENERATE_DOCS="run"
+ ;;
+ "build-site")
+ COMMAND_BUILD_SITE="run"
+ ;;
+ "preview")
+ COMMAND_PREVIEW="run"
+ ;;
+ *)
+ echo "Skipping unrecognised command '$1'."
+ ;;
+ esac
+
+ shift
+done
+
+# Execute the commands as requested by the caller.
+if [ "${COMMAND_GENERATE_DOCS}" = "run" ]
+then
+ setup_git_user
+ generate_cassandra_versioned_docs
+fi
+
+if [ "${COMMAND_BUILD_SITE}" = "run" ]
+then
+ generate_site_yaml
+
+ export DOCSEARCH_ENABLED=true
+ export DOCSEARCH_ENGINE=lunr
+ export NODE_PATH="$(npm -g root)"
+ export DOCSEARCH_INDEX_VERSION=latest
+
+ render_site_content_to_html
+fi
+
+if [ "${COMMAND_PREVIEW}" = "run" ]
+then
+ run_preview_mode
+fi
\ No newline at end of file
diff --git a/site-content/site.template.yaml b/site-content/site.template.yaml
new file mode 100644
index 0000000..93bbe29
--- /dev/null
+++ b/site-content/site.template.yaml
@@ -0,0 +1,80 @@
+site:
+ title: {{ site.title }}
+ url: {{ site.url }}
+ start_page: {{ site.start_page }}::index.adoc
+
+content:
+ sources:
+ {%- for source in source_list %}
+ - url: {{ source.url }}
+ {%- if 'branches' in source %}
+ branches:
+ {%- for branch in source.branches %}
+ - '{{ branch }}'
+ {%- endfor %}
+ {%- endif %}
+ {%- if 'tags' in source %}
+ tags:
+ {%- for tag in source.tags %}
+ - '{{ tag }}'
+ {%- endfor %}
+ {%- endif %}
+ start_path: {{ source.start_path }}
+ {%- endfor %}
+
+ui:
+ bundle:
+ url: {{ ui_bundle.url }}
+ snapshot: true
+ output_dir: assets
+
+output:
+ clean: true
+ dir: ./build/html
+
+asciidoc:
+ attributes:
+ idprefix: ''
+ idseparator: '-'
+ experimental: ''
+ source-language: asciidoc
+ current-version: 4.0
+ latest-version: 4.0
+ previous-version: 3.11
+ 40_version: '4.0'
+ 3x_version: '3.11'
+ example-caption: ~
+ hide-uri-scheme: ''
+ linkattrs: ''
+ table-caption: ~
+ tabs: tabs
+ page-pagination: true
+ latest-name: '{{ asciidoc_attributes.latest.name }}'
+ latest-date: '{{ asciidoc_attributes.latest.date }}'
+ 3_11-name: '{{ asciidoc_attributes.v311.name }}'
+ 3_11-date: '{{ asciidoc_attributes.v311.date }}'
+ 3_0-name: '{{ asciidoc_attributes.v30.name }}'
+ 3_0-date: '{{ asciidoc_attributes.v30.date }}'
+ 2_2-name: '{{ asciidoc_attributes.v22.name }}'
+ 2_2-date: '{{ asciidoc_attributes.v22.date }}'
+ 2_1-name: '{{ asciidoc_attributes.v21.name }}'
+ 2_1-date: '{{ asciidoc_attributes.v21.date }}'
+ url-downloads-cassandra: {{ asciidoc_attributes.url_downloads_cassandra }}
+ url-apache-closer: https://www.apache.org/dyn/closer.lua/cassandra
+ url-project: https://asciidoctor.org
+ url-org: https://github.com/asciidoctor
+ url-exten-lab: https://github.com/asciidoctor/asciidoctor-extensions-lab
+ url-rubygem: https://rubygems.org/gems/asciidoctor
+ url-tilt: https://github.com/rtomayko/tilt
+ url-foundation: https://foundation.zurb.com
+ url-jruby: https://jruby.org
+ url-highlightjs: https://highlightjs.org/
+ url-highlightjs-lang: https://highlightjs.org/download/
+ url-prettify: https://code.google.com/p/google-code-prettify
+ url-pygments: http://pygments.org
+ url-pygments-lang: http://pygments.org/languages/
+ url-pygments-gem: https://rubygems.org/gems/pygments.rb
+ url-python: https://www.python.org
+ extensions:
+ - ./lib/tabs-block.js
+ - "@djencks/asciidoctor-openblock"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org