You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ja...@apache.org on 2019/06/20 12:45:21 UTC
[lucene-solr] branch master updated: LUCENE-8852 ReleaseWizard tool
(#710)
This is an automated email from the ASF dual-hosted git repository.
janhoy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucene-solr.git
The following commit(s) were added to refs/heads/master by this push:
new 87c131b LUCENE-8852 ReleaseWizard tool (#710)
87c131b is described below
commit 87c131baa739f591f2585ba1666b7d98768a5450
Author: Jan Høydahl <ja...@users.noreply.github.com>
AuthorDate: Thu Jun 20 14:45:17 2019 +0200
LUCENE-8852 ReleaseWizard tool (#710)
---
.gitignore | 3 +-
dev-tools/idea/dev-tools/scripts/scripts.iml | 9 +
dev-tools/scripts/README.md | 211 +++
dev-tools/scripts/addBackcompatIndexes.py | 2 +
dev-tools/scripts/addVersion.py | 6 +-
dev-tools/scripts/buildAndPushRelease.py | 2 +
dev-tools/scripts/releaseWizard.py | 2011 ++++++++++++++++++++++++++
dev-tools/scripts/releaseWizard.yaml | 1735 ++++++++++++++++++++++
dev-tools/scripts/releasedJirasRegex.py | 2 +
dev-tools/scripts/requirements.txt | 6 +
lucene/CHANGES.txt | 2 +
11 files changed, 3987 insertions(+), 2 deletions(-)
diff --git a/.gitignore b/.gitignore
index 5f462f7..11d7f83 100644
--- a/.gitignore
+++ b/.gitignore
@@ -26,4 +26,5 @@ pom.xml
/nbproject
/nb-build
.pydevproject
-__pycache__
\ No newline at end of file
+__pycache__
+/dev-tools/scripts/scripts.iml
diff --git a/dev-tools/idea/dev-tools/scripts/scripts.iml b/dev-tools/idea/dev-tools/scripts/scripts.iml
new file mode 100644
index 0000000..88ad541
--- /dev/null
+++ b/dev-tools/idea/dev-tools/scripts/scripts.iml
@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<module type="PYTHON_MODULE" version="4">
+ <component name="NewModuleRootManager" inherit-compiler-output="true">
+ <exclude-output />
+ <content url="file://$MODULE_DIR$" />
+ <orderEntry type="jdk" jdkName="Python 3.7" jdkType="Python SDK" />
+ <orderEntry type="sourceFolder" forTests="false" />
+ </component>
+</module>
\ No newline at end of file
diff --git a/dev-tools/scripts/README.md b/dev-tools/scripts/README.md
new file mode 100644
index 0000000..8ecbee2
--- /dev/null
+++ b/dev-tools/scripts/README.md
@@ -0,0 +1,211 @@
+# Developer Scripts
+
+This folder contains various useful scripts for developers, mostly related to
+releasing new versions of Lucene/Solr and testing those.
+
+Python scripts require Python 3. To install necessary python modules, please run:
+
+ pip3 install -r requirements.txt
+
+## Scripts description
+
+### smokeTestRelease.py
+
+Used to validate new release candidates (RC). The script downloads an RC from a URL
+or local folder, then runs a number of sanity checks on the artifacts, and then runs
+the full tests.
+
+ usage: smokeTestRelease.py [-h] [--tmp-dir PATH] [--not-signed]
+ [--local-keys PATH] [--revision REVISION]
+ [--version X.Y.Z(-ALPHA|-BETA)?]
+ [--test-java12 JAVA12_HOME] [--download-only]
+ url ...
+
+ Utility to test a release.
+
+ positional arguments:
+ url Url pointing to release to test
+ test_args Arguments to pass to ant for testing, e.g.
+ -Dwhat=ever.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --tmp-dir PATH Temporary directory to test inside, defaults to
+ /tmp/smoke_lucene_$version_$revision
+ --not-signed Indicates the release is not signed
+ --local-keys PATH Uses local KEYS file instead of fetching from
+ https://archive.apache.org/dist/lucene/KEYS
+ --revision REVISION GIT revision number that release was built with,
+ defaults to that in URL
+ --version X.Y.Z(-ALPHA|-BETA)?
+ Version of the release, defaults to that in URL
+ --test-java12 JAVA12_HOME
+ Path to Java12 home directory, to run tests with if
+ specified
+ --download-only Only perform download and sha hash check steps
+
+ Example usage:
+ python3 -u dev-tools/scripts/smokeTestRelease.py https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.0.1-RC2-revc7510a0...
+
+### releaseWizard.py
+
+The Release Wizard guides the Release Manager through the release process step
+by step, helping you to to run the right commands in the right order, generating
+e-mail templates with the correct texts, versions, paths etc, obeying
+the voting rules and much more. It also serves as a documentation of all the
+steps, with timestamps, preserving log files from each command etc, showing only
+the steps and commands required for a major/minor/bugfix release. It also lets
+you generate a full Asciidoc guide for the release. The wizard will execute many
+of the other tools in this folder.
+
+ usage: releaseWizard.py [-h] [--dry-run] [--init]
+
+ Script to guide a RM through the whole release process
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --dry-run Do not execute any commands, but echo them instead. Display
+ extra debug info
+ --init Re-initialize root and version
+
+ Go push that release!
+
+### buildAndPushRelease.py
+
+ usage: buildAndPushRelease.py [-h] [--no-prepare] [--local-keys PATH]
+ [--push-local PATH] [--sign KEYID]
+ [--rc-num NUM] [--root PATH] [--logfile PATH]
+
+ Utility to build, push, and test a release.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-prepare Use the already built release in the provided checkout
+ --local-keys PATH Uses local KEYS file to validate presence of RM's gpg key
+ --push-local PATH Push the release to the local path
+ --sign KEYID Sign the release with the given gpg key
+ --rc-num NUM Release Candidate number. Default: 1
+ --root PATH Root of Git working tree for lucene-solr. Default: "."
+ (the current directory)
+ --logfile PATH Specify log file path (default /tmp/release.log)
+
+ Example usage for a Release Manager:
+ python3 -u dev-tools/scripts/buildAndPushRelease.py --push-local /tmp/releases/6.0.1 --sign 6E68DA61 --rc-num 1
+
+### addBackcompatIndexes.py
+
+ usage: addBackcompatIndexes.py [-h] [--force] [--no-cleanup] [--temp-dir DIR]
+ version
+
+ Add backcompat index and test for new version. See:
+ http://wiki.apache.org/lucene-java/ReleaseTodo#Generate_Backcompat_Indexes
+
+ positional arguments:
+ version Version to add, of the form X.Y.Z
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --force Redownload the version and rebuild, even if it already
+ exists
+ --no-cleanup Do not cleanup the built indexes, so that they can be reused
+ for adding to another branch
+ --temp-dir DIR Temp directory to build backcompat indexes within
+
+### addVersion.py
+
+ usage: addVersion.py [-h] version
+
+ Add a new version to CHANGES, to Version.java, lucene/version.properties and
+ solrconfig.xml files
+
+ positional arguments:
+ version
+
+ optional arguments:
+ -h, --help show this help message and exit
+
+### releasedJirasRegex.py
+
+Pulls out all JIRAs mentioned at the beginning of bullet items
+under the given version in the given CHANGES.txt file
+and prints a regular expression that will match all of them
+
+ usage: releasedJirasRegex.py [-h] version changes
+
+ Prints a regex matching JIRAs fixed in the given version by parsing the given
+ CHANGES.txt file
+
+ positional arguments:
+ version Version of the form X.Y.Z
+ changes CHANGES.txt file to parse
+
+ optional arguments:
+ -h, --help show this help message and exit
+
+### reproduceJenkinsFailures.py
+
+ usage: reproduceJenkinsFailures.py [-h] [--no-git] [--iters N] URL
+
+ Must be run from a Lucene/Solr git workspace. Downloads the Jenkins
+ log pointed to by the given URL, parses it for Git revision and failed
+ Lucene/Solr tests, checks out the Git revision in the local workspace,
+ groups the failed tests by module, then runs
+ 'ant test -Dtest.dups=%d -Dtests.class="*.test1[|*.test2[...]]" ...'
+ in each module of interest, failing at the end if any of the runs fails.
+ To control the maximum number of concurrent JVMs used for each module's
+ test run, set 'tests.jvms', e.g. in ~/lucene.build.properties
+
+ positional arguments:
+ URL Points to the Jenkins log to parse
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-git Do not run "git" at all
+ --iters N Number of iterations per test suite (default: 5)
+
+### poll-mirrors.py
+
+ usage: poll-mirrors.py [-h] [-version VERSION] [-path PATH]
+ [-interval INTERVAL] [-details] [-once]
+
+ Periodically checks that all Lucene/Solr mirrors contain either a copy of a
+ release or a specified path
+
+ optional arguments:
+ -h, --help show this help message and exit
+ -version VERSION, -v VERSION
+ Lucene/Solr version to check
+ -path PATH, -p PATH instead of a versioned release, check for
+ some/explicit/path
+ -interval INTERVAL, -i INTERVAL
+ seconds to wait before re-querying mirrors
+ -details, -d print missing mirror URLs
+ -once, -o run only once
+
+### githubPRs.py
+
+ usage: githubPRs.py [-h] [--json] [--token TOKEN]
+
+ Find open Pull Requests that need attention
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --json Output as json
+ --token TOKEN Github access token in case you query too often anonymously
+
+### gitignore-gen.sh
+
+TBD
+
+### publish-solr-ref-guide.sh
+
+TBD
+
+### prep-solr-ref-guide-rc.sh
+
+TBD
+
+## Tools to be removed
+
+* svnBranchToGit.py
+* createPatch.py (svn)
\ No newline at end of file
diff --git a/dev-tools/scripts/addBackcompatIndexes.py b/dev-tools/scripts/addBackcompatIndexes.py
old mode 100644
new mode 100755
index 9ca26b3..e885531
--- a/dev-tools/scripts/addBackcompatIndexes.py
+++ b/dev-tools/scripts/addBackcompatIndexes.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
diff --git a/dev-tools/scripts/addVersion.py b/dev-tools/scripts/addVersion.py
old mode 100644
new mode 100755
index e73b990..64f0398
--- a/dev-tools/scripts/addVersion.py
+++ b/dev-tools/scripts/addVersion.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
@@ -174,7 +176,7 @@ def check_solr_version_tests():
print('ok')
def read_config(current_version):
- parser = argparse.ArgumentParser(description='Add a new version')
+ parser = argparse.ArgumentParser(description='Add a new version to CHANGES, to Version.java, lucene/version.properties and solrconfig.xml files')
parser.add_argument('version', type=Version.parse)
newconf = parser.parse_args()
@@ -209,6 +211,8 @@ def get_solr_init_changes():
''' % parse_properties_file('lucene/ivy-versions.properties'))
def main():
+ if not os.path.exists('lucene/version.properties'):
+ sys.exit("Tool must be run from the root of a source checkout.")
current_version = Version.parse(find_current_version())
newconf = read_config(current_version)
diff --git a/dev-tools/scripts/buildAndPushRelease.py b/dev-tools/scripts/buildAndPushRelease.py
old mode 100644
new mode 100755
index 0667be9..f801e32
--- a/dev-tools/scripts/buildAndPushRelease.py
+++ b/dev-tools/scripts/buildAndPushRelease.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
diff --git a/dev-tools/scripts/releaseWizard.py b/dev-tools/scripts/releaseWizard.py
new file mode 100755
index 0000000..3ec18d0
--- /dev/null
+++ b/dev-tools/scripts/releaseWizard.py
@@ -0,0 +1,2011 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This script is a wizard that aims to (some day) replace the todoList at https://wiki.apache.org/lucene-java/ReleaseTodo
+# It will walk you through the steps of the release process, asking for decisions or input along the way
+# CAUTION: This is an alpha version, please read the HELP section in the main menu.
+#
+# Requirements:
+# Install requirements with this command:
+# pip3 install -r requirements.txt
+#
+# Usage:
+# releaseWizard.py [-h] [--dry-run] [--root PATH]
+#
+# optional arguments:
+# -h, --help show this help message and exit
+# --dry-run Do not execute any commands, but echo them instead. Display
+# extra debug info
+# --root PATH Specify different root folder than ~/.lucene-releases
+
+import argparse
+import copy
+import fcntl
+import json
+import os
+import platform
+import re
+import shlex
+import shutil
+import subprocess
+import sys
+import textwrap
+import time
+import urllib
+from collections import OrderedDict
+from datetime import datetime
+from datetime import timedelta
+
+try:
+ import holidays
+ import yaml
+ from ics import Calendar, Event
+ from jinja2 import Environment
+except:
+ print("You lack some of the module dependencies to run this script.")
+ print("Please run 'pip3 install -r requirements.txt' and try again.")
+ sys.exit(1)
+
+import scriptutil
+from consolemenu import ConsoleMenu
+from consolemenu.items import FunctionItem, SubmenuItem, ExitItem
+from consolemenu.screen import Screen
+from scriptutil import BranchType, Version, check_ant, download, run
+
+# Solr-to-Java version mapping
+java_versions = {6: 8, 7: 8, 8: 8, 9: 11}
+
+
+# Edit this to add other global jinja2 variables or filters
+def expand_jinja(text, vars=None):
+ global_vars = OrderedDict({
+ 'script_version': state.script_version,
+ 'release_version': state.release_version,
+ 'release_version_underscore': state.release_version.replace('.', '_'),
+ 'release_date': state.get_release_date(),
+ 'ivy2_folder': os.path.expanduser("~/.ivy2/"),
+ 'config_path': state.config_path,
+ 'rc_number': state.rc_number,
+ 'script_branch': state.script_branch,
+ 'release_folder': state.get_release_folder(),
+ 'git_checkout_folder': state.get_git_checkout_folder(),
+ 'dist_url_base': 'https://dist.apache.org/repos/dist/dev/lucene',
+ 'm2_repository_url': 'https://repository.apache.org/service/local/staging/deploy/maven2',
+ 'dist_file_path': state.get_dist_folder(),
+ 'rc_folder': state.get_rc_folder(),
+ 'base_branch': state.get_base_branch_name(),
+ 'release_branch': state.release_branch,
+ 'stable_branch': state.get_stable_branch_name(),
+ 'minor_branch': state.get_minor_branch_name(),
+ 'release_type': state.release_type,
+ 'is_feature_release': state.release_type in ['minor', 'major'],
+ 'release_version_major': state.release_version_major,
+ 'release_version_minor': state.release_version_minor,
+ 'release_version_bugfix': state.release_version_bugfix,
+ 'state': state,
+ 'epoch': unix_time_millis(datetime.utcnow()),
+ 'get_next_version': state.get_next_version(),
+ 'current_git_rev': state.get_current_git_rev(),
+ 'keys_downloaded': keys_downloaded(),
+ 'editor': get_editor(),
+ 'rename_cmd': 'ren' if is_windows() else 'mv',
+ 'vote_close_72h': vote_close_72h_date().strftime("%Y-%m-%d %H:00 UTC"),
+ 'vote_close_72h_epoch': unix_time_millis(vote_close_72h_date()),
+ 'lucene_highlights_file': lucene_highlights_file,
+ 'solr_highlights_file': solr_highlights_file,
+ 'tlp_news_draft': tlp_news_draft,
+ 'lucene_news_draft': lucene_news_draft,
+ 'solr_news_draft': solr_news_draft,
+ 'tlp_news_file': tlp_news_file,
+ 'lucene_news_file': lucene_news_file,
+ 'solr_news_file': solr_news_file,
+ 'load_lines': load_lines,
+ 'set_java_home': set_java_home,
+ 'latest_version': state.get_latest_version(),
+ 'latest_lts_version': state.get_latest_lts_version(),
+ 'master_version': state.get_master_version(),
+ 'mirrored_versions': state.get_mirrored_versions(),
+ 'mirrored_versions_to_delete': state.get_mirrored_versions_to_delete(),
+ 'home': os.path.expanduser("~")
+ })
+ global_vars.update(state.get_todo_states())
+ if vars:
+ global_vars.update(vars)
+
+ filled = replace_templates(text)
+
+ try:
+ env = Environment(lstrip_blocks=True, keep_trailing_newline=False, trim_blocks=True)
+ env.filters['path_join'] = lambda paths: os.path.join(*paths)
+ env.filters['expanduser'] = lambda path: os.path.expanduser(path)
+ env.filters['formatdate'] = lambda date: (datetime.strftime(date, "%-d %B %Y") if date else "<date>" )
+ template = env.from_string(str(filled), globals=global_vars)
+ filled = template.render()
+ except Exception as e:
+ print("Exception while rendering jinja template %s: %s" % (str(filled)[:10], e))
+ return filled
+
+
+def replace_templates(text):
+ tpl_lines = []
+ for line in text.splitlines():
+ if line.startswith("(( template="):
+ match = re.search(r"^\(\( template=(.+?) \)\)", line)
+ name = match.group(1)
+ tpl_lines.append(replace_templates(templates[name].strip()))
+ else:
+ tpl_lines.append(line)
+ return "\n".join(tpl_lines)
+
+
+def getScriptVersion():
+ topLevelDir = os.path.join(os.path.abspath("%s/" % script_path), os.path.pardir, os.path.pardir)
+ reBaseVersion = re.compile(r'version\.base\s*=\s*(\d+\.\d+\.\d+)')
+ return reBaseVersion.search(open('%s/lucene/version.properties' % topLevelDir).read()).group(1)
+
+
+def get_editor():
+ return os.environ['EDITOR'] if 'EDITOR' in os.environ else 'notepad.exe' if is_windows() else 'vi'
+
+
+def check_prerequisites(todo=None):
+ if sys.version_info < (3, 4):
+ sys.exit("Script requires Python v3.4 or later")
+ try:
+ gpg_ver = run("gpg --version").splitlines()[0]
+ except:
+ sys.exit("You will need gpg installed")
+ if not check_ant().startswith('1.8'):
+ print("WARNING: This script will work best with ant 1.8. The script buildAndPushRelease.py may have problems with PGP password input under ant 1.10")
+ if not 'JAVA8_HOME' in os.environ or not 'JAVA11_HOME' in os.environ:
+ sys.exit("Please set environment variables JAVA8_HOME and JAVA11_HOME")
+ try:
+ asciidoc_ver = run("asciidoctor -V").splitlines()[0]
+ except:
+ asciidoc_ver = ""
+ print("WARNING: In order to export asciidoc version to HTML, you will need asciidoctor installed")
+ try:
+ git_ver = run("git --version").splitlines()[0]
+ except:
+ sys.exit("You will need git installed")
+ if not 'EDITOR' in os.environ:
+ print("WARNING: Environment variable $EDITOR not set, using %s" % get_editor())
+
+ if todo:
+ print("%s\n%s\n%s\n" % (gpg_ver, asciidoc_ver, git_ver))
+ return True
+
+
+epoch = datetime.utcfromtimestamp(0)
+
+
+def unix_time_millis(dt):
+ return int((dt - epoch).total_seconds() * 1000.0)
+
+
+def bootstrap_todos(todo_list):
+ # Establish links from commands to to_do for finding todo vars
+ for tg in todo_list:
+ if dry_run:
+ print("Group %s" % tg.id)
+ for td in tg.get_todos():
+ if dry_run:
+ print(" Todo %s" % td.id)
+ cmds = td.commands
+ if cmds:
+ if dry_run:
+ print(" Commands")
+ cmds.todo_id = td.id
+ for cmd in cmds.commands:
+ if dry_run:
+ print(" Command %s" % cmd.cmd)
+ cmd.todo_id = td.id
+
+ print("Loaded TODO definitions from releaseWizard.yaml")
+ return todo_list
+
+
+def maybe_remove_rc_from_svn():
+ todo = state.get_todo_by_id('import_svn')
+ if todo and todo.is_done():
+ print("import_svn done")
+ Commands(state.get_git_checkout_folder(),
+ """Looks like you uploaded artifacts for {{ build_rc.git_rev | default("<git_rev>", True) }} to svn which needs to be removed.""",
+ [Command(
+ """svn -m "Remove cancelled Lucene/Solr {{ release_version }} RC{{ rc_number }}" rm {{ dist_url }}""",
+ logfile="svn_rm.log",
+ tee=True,
+ vars={
+ 'dist_folder': """lucene-solr-{{ release_version }}-RC{{ rc_number }}-rev{{ build_rc.git_rev | default("<git_rev>", True) }}""",
+ 'dist_url': "{{ dist_url_base }}/{{ dist_folder }}"
+ }
+ )],
+ enable_execute=True, confirm_each_command=False).run()
+
+
+# To be able to hide fields when dumping Yaml
+class SecretYamlObject(yaml.YAMLObject):
+ hidden_fields = []
+ @classmethod
+ def to_yaml(cls,dumper,data):
+ print("Dumping object %s" % type(data))
+
+ new_data = copy.deepcopy(data)
+ for item in cls.hidden_fields:
+ if item in new_data.__dict__:
+ del new_data.__dict__[item]
+ for item in data.__dict__:
+ if item in new_data.__dict__ and new_data.__dict__[item] is None:
+ del new_data.__dict__[item]
+ return dumper.represent_yaml_object(cls.yaml_tag, new_data, cls,
+ flow_style=cls.yaml_flow_style)
+
+
+def str_presenter(dumper, data):
+ if len(data.split('\n')) > 1: # check for multiline string
+ return dumper.represent_scalar('tag:yaml.org,2002:str', data, style='|')
+ return dumper.represent_scalar('tag:yaml.org,2002:str', data)
+
+
+class ReleaseState:
+ def __init__(self, config_path, release_version, script_version):
+ self.script_version = script_version
+ self.config_path = config_path
+ self.todo_groups = None
+ self.todos = None
+ self.latest_version = None
+ self.previous_rcs = {}
+ self.rc_number = 1
+ self.start_date = unix_time_millis(datetime.utcnow())
+ self.script_branch = run("git rev-parse --abbrev-ref HEAD").strip()
+ self.mirrored_versions = None
+ try:
+ self.script_branch_type = scriptutil.find_branch_type()
+ except:
+ print("WARNING: This script shold (ideally) run from the release branch, not a feature branch (%s)" % self.script_branch)
+ self.script_branch_type = 'feature'
+ self.set_release_version(release_version)
+
+ def set_release_version(self, version):
+ self.validate_release_version(self.script_branch_type, self.script_branch, version)
+ self.release_version = version
+ v = Version.parse(version)
+ self.release_version_major = v.major
+ self.release_version_minor = v.minor
+ self.release_version_bugfix = v.bugfix
+ self.release_branch = "branch_%s_%s" % (v.major, v.minor)
+ if v.is_major_release():
+ self.release_type = 'major'
+ elif v.is_minor_release():
+ self.release_type = 'minor'
+ else:
+ self.release_type = 'bugfix'
+
+ def is_released(self):
+ return self.get_todo_by_id('announce_lucene').is_done()
+
+ def get_release_date(self):
+ publish_task = self.get_todo_by_id('publish_maven')
+ if publish_task.is_done():
+ return unix_to_datetime(publish_task.get_state()['done_date'])
+ else:
+ return None
+
+ def get_latest_version(self):
+ if self.latest_version is None:
+ versions = self.get_mirrored_versions()
+ latest = versions[0]
+ for ver in versions:
+ if Version.parse(ver).gt(Version.parse(latest)):
+ latest = ver
+ self.latest_version = latest
+ self.save()
+ return state.latest_version
+
+ def get_mirrored_versions(self):
+ if state.mirrored_versions is None:
+ releases_str = load("https://projects.apache.org/json/foundation/releases.json", "utf-8")
+ releases = json.loads(releases_str)['lucene']
+ state.mirrored_versions = [ r for r in list(map(lambda y: y[7:], filter(lambda x: x.startswith('lucene-'), list(releases.keys())))) ]
+ return state.mirrored_versions
+
+ def get_mirrored_versions_to_delete(self):
+ versions = self.get_mirrored_versions()
+ to_keep = versions
+ if state.release_type == 'major':
+ to_keep = [self.release_version, self.get_latest_version()]
+ if state.release_type == 'minor':
+ to_keep = [self.release_version, self.get_latest_lts_version()]
+ if state.release_type == 'bugfix':
+ if Version.parse(state.release_version).major == Version.parse(state.get_latest_version()).major:
+ to_keep = [self.release_version, self.get_latest_lts_version()]
+ elif Version.parse(state.release_version).major == Version.parse(state.get_latest_lts_version()).major:
+ to_keep = [self.get_latest_version(), self.release_version()]
+ else:
+ raise Exception("Release version %s must have same major version as current minor or lts release")
+ return [ver for ver in versions if ver not in to_keep]
+
+ def get_master_version(self):
+ v = Version.parse(self.get_latest_version())
+ return "%s.%s.%s" % (v.major + 1, 0, 0)
+
+ def get_latest_lts_version(self):
+ versions = self.get_mirrored_versions()
+ latest = self.get_latest_version()
+ lts_prefix = "%s." % (Version.parse(latest).major - 1)
+ lts_versions = list(filter(lambda x: x.startswith(lts_prefix), versions))
+ latest_lts = lts_versions[0]
+ for ver in lts_versions:
+ if Version.parse(ver).gt(Version.parse(latest_lts)):
+ latest_lts = ver
+ return latest_lts
+
+ def validate_release_version(self, branch_type, branch, release_version):
+ ver = Version.parse(release_version)
+ # print("release_version=%s, ver=%s" % (release_version, ver))
+ if branch_type == BranchType.release:
+ if not branch.startswith('branch_'):
+ sys.exit("Incompatible branch and branch_type")
+ if not ver.is_bugfix_release():
+ sys.exit("You can only release bugfix releases from an existing release branch")
+ elif branch_type == BranchType.stable:
+ if not branch.startswith('branch_') and branch.endswith('x'):
+ sys.exit("Incompatible branch and branch_type")
+ if not ver.is_minor_release():
+ sys.exit("You can only release minor releases from an existing stable branch")
+ elif branch_type == BranchType.unstable:
+ if not branch == 'master':
+ sys.exit("Incompatible branch and branch_type")
+ if not ver.is_major_release():
+ sys.exit("You can only release a new major version from master branch")
+ if not getScriptVersion() == release_version:
+ print("WARNING: Expected release version %s when on branch %s, but got %s" % (
+ getScriptVersion(), branch, release_version))
+
+ def get_base_branch_name(self):
+ v = Version.parse(self.release_version)
+ if v.is_major_release():
+ return 'master'
+ elif v.is_minor_release():
+ return self.get_stable_branch_name()
+ elif v.major == Version.parse(self.get_latest_version()).major:
+ return self.get_minor_branch_name()
+ else:
+ return self.release_branch
+
+ def clear_rc(self):
+ if ask_yes_no("Are you sure? This will clear and restart RC%s" % self.rc_number):
+ maybe_remove_rc_from_svn()
+ dict = {}
+ for g in list(filter(lambda x: x.in_rc_loop(), self.todo_groups)):
+ for t in g.get_todos():
+ t.clear()
+ print("Cleared RC TODO state")
+ try:
+ shutil.rmtree(self.get_rc_folder())
+ print("Cleared folder %s" % self.get_rc_folder())
+ except Exception as e:
+ print("WARN: Failed to clear %s, please do it manually with higher privileges" % self.get_rc_folder())
+ self.save()
+
+ def new_rc(self):
+ if ask_yes_no("Are you sure? This will abort current RC"):
+ maybe_remove_rc_from_svn()
+ dict = {}
+ for g in list(filter(lambda x: x.in_rc_loop(), self.todo_groups)):
+ for t in g.get_todos():
+ if t.applies(self.release_type):
+ dict[t.id] = copy.deepcopy(t.state)
+ t.clear()
+ self.previous_rcs["RC%d" % self.rc_number] = dict
+ self.rc_number += 1
+ self.save()
+
+ def to_dict(self):
+ tmp_todos = {}
+ for todo_id in self.todos:
+ t = self.todos[todo_id]
+ tmp_todos[todo_id] = copy.deepcopy(t.state)
+ dict = {
+ 'script_version': self.script_version,
+ 'release_version': self.release_version,
+ 'start_date': self.start_date,
+ 'rc_number': self.rc_number,
+ 'script_branch': self.script_branch,
+ 'todos': tmp_todos,
+ 'previous_rcs': self.previous_rcs
+ }
+ if self.latest_version:
+ dict['latest_version'] = self.latest_version
+ return dict
+
+ def restore_from_dict(self, dict):
+ self.script_version = dict['script_version']
+ assert dict['release_version'] == self.release_version
+ if 'start_date' in dict:
+ self.start_date = dict['start_date']
+ if 'latest_version' in dict:
+ self.latest_version = dict['latest_version']
+ else:
+ self.latest_version = None
+ self.rc_number = dict['rc_number']
+ self.script_branch = dict['script_branch']
+ self.previous_rcs = copy.deepcopy(dict['previous_rcs'])
+ for todo_id in dict['todos']:
+ if todo_id in self.todos:
+ t = self.todos[todo_id]
+ for k in dict['todos'][todo_id]:
+ t.state[k] = dict['todos'][todo_id][k]
+ else:
+ print("Warning: Could not restore state for %s, Todo definition not found" % todo_id)
+
+ def load(self):
+ if os.path.exists(os.path.join(self.config_path, self.release_version, 'state.yaml')):
+ state_file = os.path.join(self.config_path, self.release_version, 'state.yaml')
+ with open(state_file, 'r') as fp:
+ try:
+ dict = yaml.load(fp, Loader=yaml.Loader)
+ self.restore_from_dict(dict)
+ print("Loaded state from %s" % state_file)
+ except Exception as e:
+ print("Failed to load state from %s: %s" % (state_file, e))
+
+ def save(self):
+ print("Saving")
+ if not os.path.exists(os.path.join(self.config_path, self.release_version)):
+ print("Creating folder %s" % os.path.join(self.config_path, self.release_version))
+ os.makedirs(os.path.join(self.config_path, self.release_version))
+
+ with open(os.path.join(self.config_path, self.release_version, 'state.yaml'), 'w') as fp:
+ yaml.dump(self.to_dict(), fp, sort_keys=False, default_flow_style=False)
+
+ def clear(self):
+ self.previous_rcs = {}
+ self.rc_number = 1
+ for t_id in self.todos:
+ t = self.todos[t_id]
+ t.state = {}
+ self.save()
+
+ def get_rc_number(self):
+ return self.rc_number
+
+ def get_current_git_rev(self):
+ try:
+ return run("git rev-parse HEAD", cwd=self.get_git_checkout_folder()).strip()
+ except:
+ return "<git-rev>"
+
+ def get_group_by_id(self, id):
+ lst = list(filter(lambda x: x.id == id, self.todo_groups))
+ if len(lst) == 1:
+ return lst[0]
+ else:
+ return None
+
+ def get_todo_by_id(self, id):
+ lst = list(filter(lambda x: x.id == id, self.todos.values()))
+ if len(lst) == 1:
+ return lst[0]
+ else:
+ return None
+
+ def get_todo_state_by_id(self, id):
+ lst = list(filter(lambda x: x.id == id, self.todos.values()))
+ if len(lst) == 1:
+ return lst[0].state
+ else:
+ return {}
+
+ def get_release_folder(self):
+ folder = os.path.join(self.config_path, self.release_version)
+ if not os.path.exists(folder):
+ print("Creating folder %s" % folder)
+ os.makedirs(folder)
+ return folder
+
+ def get_rc_folder(self):
+ folder = os.path.join(self.get_release_folder(), "RC%d" % self.rc_number)
+ if not os.path.exists(folder):
+ print("Creating folder %s" % folder)
+ os.makedirs(folder)
+ return folder
+
+ def get_dist_folder(self):
+ folder = os.path.join(self.get_rc_folder(), "dist")
+ return folder
+
+ def get_git_checkout_folder(self):
+ folder = os.path.join(self.get_release_folder(), "lucene-solr")
+ return folder
+
+ def get_minor_branch_name(self):
+ latest = state.get_latest_version()
+ if latest is not None:
+ v = Version.parse(latest)
+ return "branch_%s_%s" % (v.major, v.minor)
+ else:
+ raise Exception("Cannot find latest version")
+
+ def get_stable_branch_name(self):
+ v = Version.parse(self.get_latest_version())
+ return "branch_%sx" % v.major
+
+ def get_next_version(self):
+ if self.release_type == 'major':
+ return "%s.0" % (self.release_version_major + 1)
+ if self.release_type == 'minor':
+ return "%s.%s" % (self.release_version_major, self.release_version_minor + 1)
+ if self.release_type == 'bugfix':
+ return "%s.%s.%s" % (self.release_version_major, self.release_version_minor, self.release_version_bugfix + 1)
+
+ def get_java_home(self):
+ return self.get_java_home_for_version(self.release_version)
+
+ def get_java_home_for_version(self, version):
+ v = Version.parse(version)
+ java_ver = java_versions[v.major]
+ java_home_var = "JAVA%s_HOME" % java_ver
+ if java_home_var in os.environ:
+ return os.environ.get(java_home_var)
+ else:
+ raise Exception("Script needs environment variable %s" % java_home_var )
+
+ def get_java_cmd_for_version(self, version):
+ return os.path.join(self.get_java_home_for_version(version), "bin", "java")
+
+ def get_java_cmd(self):
+ return os.path.join(self.get_java_home(), "bin", "java")
+
+ def get_todo_states(self):
+ states = {}
+ if self.todos:
+ for todo_id in self.todos:
+ t = self.todos[todo_id]
+ states[todo_id] = copy.deepcopy(t.state)
+ return states
+
+ def init_todos(self, groups):
+ self.todo_groups = groups
+ self.todos = {}
+ for g in self.todo_groups:
+ for t in g.get_todos():
+ self.todos[t.id] = t
+
+
+class TodoGroup(SecretYamlObject):
+ yaml_tag = u'!TodoGroup'
+ hidden_fields = []
+ def __init__(self, id, title, description, todos, is_in_rc_loop=None, depends=None):
+ self.id = id
+ self.title = title
+ self.description = description
+ self.depends = depends
+ self.is_in_rc_loop = is_in_rc_loop
+ self.todos = todos
+
+ @classmethod
+ def from_yaml(cls, loader, node):
+ fields = loader.construct_mapping(node, deep = True)
+ return TodoGroup(**fields)
+
+ def num_done(self):
+ return sum(1 for x in self.todos if x.is_done() > 0)
+
+ def num_applies(self):
+ count = sum(1 for x in self.todos if x.applies(state.release_type))
+ # print("num_applies=%s" % count)
+ return count
+
+ def is_done(self):
+ # print("Done=%s, applies=%s" % (self.num_done(), self.num_applies()))
+ return self.num_done() >= self.num_applies()
+
+ def get_title(self):
+ # print("get_title: %s" % self.is_done())
+ prefix = ""
+ if self.is_done():
+ prefix = "✓ "
+ return "%s%s (%d/%d)" % (prefix, self.title, self.num_done(), self.num_applies())
+
+ def get_submenu(self):
+ menu = UpdatableConsoleMenu(title=self.title, subtitle=self.get_subtitle, prologue_text=self.get_description(),
+ screen=MyScreen())
+ menu.exit_item = CustomExitItem("Return")
+ for todo in self.get_todos():
+ if todo.applies(state.release_type):
+ menu.append_item(todo.get_menu_item())
+ return menu
+
+ def get_menu_item(self):
+ item = UpdatableSubmenuItem(self.get_title, self.get_submenu())
+ return item
+
+ def get_todos(self):
+ return self.todos
+
+ def in_rc_loop(self):
+ return self.is_in_rc_loop is True
+
+ def get_description(self):
+ desc = self.description
+ if desc:
+ return expand_jinja(desc)
+ else:
+ return None
+
+ def get_subtitle(self):
+ if self.depends:
+ ret_str = ""
+ for dep in ensure_list(self.depends):
+ g = state.get_group_by_id(dep)
+ if not g:
+ g = state.get_todo_by_id(dep)
+ if g and not g.is_done():
+ ret_str += "NOTE: Please first complete '%s'\n" % g.title
+ return ret_str.strip()
+ return None
+
+
+class Todo(SecretYamlObject):
+ yaml_tag = u'!Todo'
+ hidden_fields = ['state']
+ def __init__(self, id, title, description=None, post_description=None, done=None, types=None, links=None,
+ commands=None, user_input=None, depends=None, vars=None, asciidoc=None, persist_vars=None,
+ function=None):
+ self.id = id
+ self.title = title
+ self.description = description
+ self.asciidoc = asciidoc
+ self.types = types
+ self.depends = depends
+ self.vars = vars
+ self.persist_vars = persist_vars
+ self.function = function
+ self.user_input = user_input
+ self.commands = commands
+ self.post_description = post_description
+ self.links = links
+ self.state = {}
+
+ self.set_done(done)
+ if self.types:
+ self.types = ensure_list(self.types)
+ for t in self.types:
+ if not t in ['minor', 'major', 'bugfix']:
+ sys.exit("Wrong Todo config for '%s'. Type needs to be either 'minor', 'major' or 'bugfix'" % self.id)
+ if commands:
+ self.commands.todo_id = self.id
+ for c in commands.commands:
+ c.todo_id = self.id
+
+ @classmethod
+ def from_yaml(cls, loader, node):
+ fields = loader.construct_mapping(node, deep = True)
+ return Todo(**fields)
+
+ def get_vars(self):
+ myvars = {}
+ if self.vars:
+ for k in self.vars:
+ val = self.vars[k]
+ if callable(val):
+ myvars[k] = expand_jinja(val(), vars=myvars)
+ else:
+ myvars[k] = expand_jinja(val, vars=myvars)
+ return myvars
+
+ def set_done(self, is_done):
+ if is_done:
+ self.state['done_date'] = unix_time_millis(datetime.utcnow())
+ if self.persist_vars:
+ for k in self.persist_vars:
+ self.state[k] = self.get_vars()[k]
+ else:
+ self.state.clear()
+ self.state['done'] = is_done
+
+ def applies(self, type):
+ if self.types:
+ return type in self.types
+ return True
+
+ def is_done(self):
+ return 'done' in self.state and self.state['done'] is True
+
+ def get_title(self):
+ prefix = ""
+ if self.is_done():
+ prefix = "✓ "
+ return expand_jinja("%s%s" % (prefix, self.title), self.get_vars_and_state())
+
+ def display_and_confirm(self):
+ try:
+ if self.depends:
+ ret_str = ""
+ for dep in ensure_list(self.depends):
+ g = state.get_group_by_id(dep)
+ if not g:
+ g = state.get_todo_by_id(dep)
+ if not g.is_done():
+ print("This step depends on '%s'. Please complete that first\n" % g.title)
+ return
+ desc = self.get_description()
+ if desc:
+ print("%s" % desc)
+ try:
+ if self.function and not self.is_done():
+ if not eval(self.function)(self):
+ return
+ except Exception as e:
+ print("Function call to %s for todo %s failed: %s" % (self.function, self.id, e))
+ raise e
+ if self.user_input and not self.is_done():
+ ui_list = ensure_list(self.user_input)
+ for ui in ui_list:
+ ui.run(self.state)
+ print()
+ if self.links:
+ print("\nLinks:\n")
+ for link in self.links:
+ print("- %s" % expand_jinja(link, self.get_vars_and_state()))
+ print()
+ cmds = self.get_commands()
+ if cmds:
+ if not self.is_done():
+ if not cmds.logs_prefix:
+ cmds.logs_prefix = self.id
+ cmds.run()
+ else:
+ print("This step is already completed. You have to first set it to 'not completed' in order to execute commands again.")
+ print()
+ if self.post_description:
+ print("%s" % self.get_post_description())
+ todostate = self.get_state()
+ if self.is_done() and len(todostate) > 2:
+ print("Variables registered\n")
+ for k in todostate:
+ if k == 'done' or k == 'done_date':
+ continue
+ print("* %s = %s" % (k, todostate[k]))
+ print()
+ completed = ask_yes_no("Mark task '%s' as completed?" % self.get_title())
+ self.set_done(completed)
+ state.save()
+ except Exception as e:
+ print("ERROR while executing todo %s (%s)" % (self.get_title(), e))
+
+ def get_menu_item(self):
+ return UpdatableFunctionItem(self.get_title, self.display_and_confirm)
+
+ def clone(self):
+ clone = Todo(self.id, self.title, description=self.description)
+ clone.state = copy.deepcopy(self.state)
+ return clone
+
+ def clear(self):
+ self.state.clear()
+ self.set_done(False)
+
+ def get_state(self):
+ return self.state
+
+ def get_description(self):
+ desc = self.description
+ if desc:
+ return expand_jinja(desc, vars=self.get_vars_and_state())
+ else:
+ return None
+
+ def get_post_description(self):
+ if self.post_description:
+ return expand_jinja(self.post_description, vars=self.get_vars_and_state())
+ else:
+ return None
+
+ def get_commands(self):
+ cmds = self.commands
+ return cmds
+
+ def get_asciidoc(self):
+ if self.asciidoc:
+ return expand_jinja(self.asciidoc, vars=self.get_vars_and_state())
+ else:
+ return None
+
+ def get_vars_and_state(self):
+ d = self.get_vars().copy()
+ d.update(self.get_state())
+ return d
+
+
+def get_release_version():
+ v = str(input("Which version are you releasing? (x.y.z) "))
+ try:
+ version = Version.parse(v)
+ except:
+ print("Not a valid version %s" % v)
+ return get_release_version()
+
+ return str(version)
+
+
+def get_subtitle():
+ applying_groups = list(filter(lambda x: x.num_applies() > 0, state.todo_groups))
+ done_groups = sum(1 for x in applying_groups if x.is_done())
+ return "Please complete the below checklist (Complete: %s/%s)" % (done_groups, len(applying_groups))
+
+
+def get_todo_menuitem_title():
+ return "Go to checklist (RC%d)" % (state.rc_number)
+
+
+def get_releasing_text():
+ return "Releasing Lucene/Solr %s RC%d" % (state.release_version, state.rc_number)
+
+
+def get_start_new_rc_menu_title():
+ return "Abort RC%d and start a new RC%d" % (state.rc_number, state.rc_number + 1)
+
+
+def start_new_rc():
+ state.new_rc()
+ print("Started RC%d" % state.rc_number)
+
+
+def reset_state():
+ global state
+ if ask_yes_no("Are you sure? This will erase all current progress"):
+ maybe_remove_rc_from_svn()
+ shutil.rmtree(os.path.join(state.config_path, state.release_version))
+ state.clear()
+
+
+def template(name, vars=None):
+ return expand_jinja(templates[name], vars=vars)
+
+
+def help():
+ print(template('help'))
+ pause()
+
+
+def ensure_list(o):
+ if o is None:
+ return []
+ if not isinstance(o, list):
+ return [o]
+ else:
+ return o
+
+
+def open_file(filename):
+ print("Opening file %s" % filename)
+ if platform.system().startswith("Win"):
+ run("start %s" % filename)
+ else:
+ run("open %s" % filename)
+
+
+def expand_multiline(cmd_txt, indent=0):
+ return re.sub(r' +', " %s\n %s" % (Commands.cmd_continuation_char, " "*indent), cmd_txt)
+
+
+def unix_to_datetime(unix_stamp):
+ return datetime.utcfromtimestamp(unix_stamp / 1000)
+
+
+def generate_asciidoc():
+ base_filename = os.path.join(state.get_release_folder(),
+ "lucene_solr_release_%s"
+ % (state.release_version.replace("\.", "_")))
+
+ filename_adoc = "%s.adoc" % base_filename
+ filename_html = "%s.html" % base_filename
+ fh = open(filename_adoc, "w")
+
+ fh.write("= Lucene/Solr Release %s\n\n" % state.release_version)
+ fh.write("(_Generated by releaseWizard.py v%s ALPHA at %s_)\n\n"
+ % (getScriptVersion(), datetime.utcnow().strftime("%Y-%m-%d %H:%M UTC")))
+ fh.write(":numbered:\n\n")
+ fh.write("%s\n\n" % template('help'))
+ for group in state.todo_groups:
+ if group.num_applies() == 0:
+ continue
+ fh.write("== %s\n\n" % group.get_title())
+ fh.write("%s\n\n" % group.get_description())
+ for todo in group.get_todos():
+ if not todo.applies(state.release_type):
+ continue
+ fh.write("=== %s\n\n" % todo.get_title())
+ if todo.is_done():
+ fh.write("_Completed %s_\n\n" % unix_to_datetime(todo.state['done_date']).strftime(
+ "%Y-%m-%d %H:%M UTC"))
+ if todo.get_asciidoc():
+ fh.write("%s\n\n" % todo.get_asciidoc())
+ else:
+ desc = todo.get_description()
+ if desc:
+ fh.write("%s\n\n" % desc)
+ state_copy = copy.deepcopy(todo.state)
+ state_copy.pop('done', None)
+ state_copy.pop('done_date', None)
+ if len(state_copy) > 0 or todo.user_input is not None:
+ fh.write(".Variables collected in this step\n")
+ fh.write("|===\n")
+ fh.write("|Variable |Value\n")
+ mykeys = set()
+ for e in ensure_list(todo.user_input):
+ mykeys.add(e.name)
+ for e in state_copy.keys():
+ mykeys.add(e)
+ for key in mykeys:
+ val = "(not set)"
+ if key in state_copy:
+ val = state_copy[key]
+ fh.write("\n|%s\n|%s\n" % (key, val))
+ fh.write("|===\n\n")
+ cmds = todo.get_commands()
+ if cmds:
+ if cmds.commands_text:
+ fh.write("%s\n\n" % cmds.get_commands_text())
+ fh.write("[source,sh]\n----\n")
+ if cmds.env:
+ for key in cmds.env:
+ val = cmds.env[key]
+ if is_windows():
+ fh.write("SET %s=%s\n" % (key, val))
+ else:
+ fh.write("export %s=%s\n" % (key, val))
+ fh.write(abbreviate_homedir("cd %s\n" % cmds.get_root_folder()))
+ cmds2 = ensure_list(cmds.commands)
+ for c in cmds2:
+ for line in c.display_cmd():
+ fh.write("%s\n" % line)
+ fh.write("----\n\n")
+ if todo.post_description and not todo.get_asciidoc():
+ fh.write("\n%s\n\n" % todo.get_post_description())
+ if todo.links:
+ fh.write("Links:\n\n")
+ for l in todo.links:
+ fh.write("* %s\n" % expand_jinja(l))
+ fh.write("\n")
+
+ fh.close()
+ print("Wrote file %s" % os.path.join(state.get_release_folder(), filename_adoc))
+ print("Running command 'asciidoctor %s'" % filename_adoc)
+ run_follow("asciidoctor %s" % filename_adoc)
+ if os.path.exists(filename_html):
+ open_file(filename_html)
+ else:
+ print("Failed generating HTML version, please install asciidoctor")
+ pause()
+
+
+def load_rc():
+ lucenerc = os.path.expanduser("~/.lucenerc")
+ try:
+ with open(lucenerc, 'r') as fp:
+ return json.load(fp)
+ except:
+ return None
+
+
+def store_rc(release_root, release_version=None):
+ lucenerc = os.path.expanduser("~/.lucenerc")
+ dict = {}
+ dict['root'] = release_root
+ if release_version:
+ dict['release_version'] = release_version
+ with open(lucenerc, "w") as fp:
+ json.dump(dict, fp, indent=2)
+
+
+def release_other_version():
+ if not state.is_released():
+ maybe_remove_rc_from_svn()
+ store_rc(state.config_path, None)
+ print("Please restart the wizard")
+ sys.exit(0)
+
+def file_to_string(filename):
+ with open(filename, encoding='utf8') as f:
+ return f.read().strip()
+
+def download_keys():
+ download('KEYS', "https://archive.apache.org/dist/lucene/KEYS", state.config_path)
+
+def keys_downloaded():
+ return os.path.exists(os.path.join(state.config_path, "KEYS"))
+
+
+def dump_yaml():
+ file = open(os.path.join(script_path, "releaseWizard.yaml"), "w")
+ yaml.add_representer(str, str_presenter)
+ yaml.Dumper.ignore_aliases = lambda *args : True
+ dump_obj = {'templates': templates,
+ 'groups': state.todo_groups}
+ yaml.dump(dump_obj, width=180, stream=file, sort_keys=False, default_flow_style=False)
+
+
+def parse_config():
+ description = 'Script to guide a RM through the whole release process'
+ parser = argparse.ArgumentParser(description=description, epilog="Go push that release!",
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ parser.add_argument('--dry-run', dest='dry', action='store_true', default=False,
+ help='Do not execute any commands, but echo them instead. Display extra debug info')
+ parser.add_argument('--init', action='store_true', default=False,
+ help='Re-initialize root and version')
+ config = parser.parse_args()
+
+ return config
+
+
+def load(urlString, encoding="utf-8"):
+ try:
+ content = urllib.request.urlopen(urlString).read().decode(encoding)
+ except Exception as e:
+ print('Retrying download of url %s after exception: %s' % (urlString, e))
+ content = urllib.request.urlopen(urlString).read().decode(encoding)
+ return content
+
+
+def configure_pgp(gpg_todo):
+ print("Based on your Apache ID we'll lookup your key online\n"
+ "and through this complete the 'gpg' prerequisite task.\n")
+ gpg_state = gpg_todo.get_state()
+ id = str(input("Please enter your Apache id: (ENTER=skip) "))
+ if id.strip() == '':
+ return False
+ all_keys = load('https://home.apache.org/keys/group/lucene.asc')
+ lines = all_keys.splitlines()
+ keyid_linenum = None
+ for idx, line in enumerate(lines):
+ if line == 'ASF ID: %s' % id:
+ keyid_linenum = idx+1
+ break
+ if keyid_linenum:
+ keyid_line = lines[keyid_linenum]
+ assert keyid_line.startswith('LDAP PGP key: ')
+ gpg_id = keyid_line[14:].replace(" ", "")[-8:]
+ print("Found gpg key id %s on file at Apache (https://home.apache.org/keys/group/lucene.asc)" % gpg_id)
+ else:
+ print(textwrap.dedent("""\
+ Could not find your GPG key from Apache servers.
+ Please make sure you have registered your key ID in
+ id.apache.org, see links for more info."""))
+ gpg_id = str(input("Enter your key ID manually, 8 last characters (ENTER=skip): "))
+ if gpg_id.strip() == '':
+ return False
+ elif len(gpg_id) != 8:
+ print("gpg id must be the last 8 characters of your key id")
+ gpg_id = gpg_id.upper()
+ try:
+ res = run("gpg --list-secret-keys %s" % gpg_id)
+ print("Found key %s on your private gpg keychain" % gpg_id)
+ # Check rsa and key length >= 4096
+ match = re.search(r'^sec +((rsa|dsa)(\d{4})) ', res)
+ type = "(unknown)"
+ length = -1
+ if match:
+ type = match.group(2)
+ length = int(match.group(3))
+ else:
+ match = re.search(r'^sec +((\d{4})([DR])/.*?) ', res)
+ if match:
+ type = 'rsa' if match.group(3) == 'R' else 'dsa'
+ length = int(match.group(2))
+ else:
+ print("Could not determine type and key size for your key")
+ print("%s" % res)
+ if not ask_yes_no("Is your key of type RSA and size >= 2048 (ideally 4096)? "):
+ print("Sorry, please generate a new key, add to KEYS and register with id.apache.org")
+ return False
+ if not type == 'rsa':
+ print("We strongly recommend RSA type key, your is '%s'. Consider generating a new key." % type.upper())
+ if length < 2048:
+ print("Your key has key length of %s. Cannot use < 2048, please generate a new key before doing the release" % length)
+ return False
+ if length < 4096:
+ print("Your key length is < 4096, Please generate a stronger key.")
+ print("Alternatively, follow instructions in http://www.apache.org/dev/release-signing.html#note")
+ if not ask_yes_no("Have you configured your gpg to avoid SHA-1?"):
+ print("Please either generate a strong key or reconfigure your client")
+ return False
+ print("Validated that your key is of type RSA and has a length >= 2048 (%s)" % length)
+ except:
+ print(textwrap.dedent("""\
+ Key not found on your private gpg keychain. In order to sign the release you'll
+ need to fix this, then try again"""))
+ return False
+ try:
+ lines = run("gpg --check-signatures %s" % gpg_id).splitlines()
+ sigs = 0
+ apache_sigs = 0
+ for line in lines:
+ if line.startswith("sig") and not gpg_id in line:
+ sigs += 1
+ if '@apache.org' in line:
+ apache_sigs += 1
+ print("Your key has %s signatures, of which %s are by committers (@apache.org address)" % (sigs, apache_sigs))
+ if apache_sigs < 1:
+ print(textwrap.dedent("""\
+ Your key is not signed by any other committer.
+ Please review http://www.apache.org/dev/openpgp.html#apache-wot
+ and make sure to get your key signed until next time.
+ You may want to run 'gpg --refresh-keys' to refresh your keychain."""))
+ uses_apacheid = is_code_signing_key = False
+ for line in lines:
+ if line.startswith("uid") and "%s@apache" % id in line:
+ uses_apacheid = True
+ if 'CODE SIGNING KEY' in line.upper():
+ is_code_signing_key = True
+ if not uses_apacheid:
+ print("WARNING: Your key should use your apache-id email address, see http://www.apache.org/dev/release-signing.html#user-id")
+ if not is_code_signing_key:
+ print("WARNING: You code signing key should be labeled 'CODE SIGNING KEY', see http://www.apache.org/dev/release-signing.html#key-comment")
+ except Exception as e:
+ print("Could not check signatures of your key: %s" % e)
+
+ download_keys()
+ keys_text = file_to_string(os.path.join(state.config_path, "KEYS"))
+ if gpg_id in keys_text or "%s %s" % (gpg_id[:4], gpg_id[-4:]) in keys_text:
+ print("Found your key ID in official KEYS file. KEYS file is not cached locally.")
+ else:
+ print(textwrap.dedent("""\
+ Could not find your key ID in official KEYS file.
+ Please make sure it is added to https://dist.apache.org/repos/dist/release/lucene/KEYS
+ and committed to svn. Then re-try this initialization"""))
+ if not ask_yes_no("Do you want to continue without fixing KEYS file? (not recommended) "):
+ return False
+
+ gpg_state['apache_id'] = id
+ gpg_state['gpg_key'] = gpg_id
+ return True
+
+
+def pause(fun=None):
+ if fun:
+ fun()
+ input("\nPress ENTER to continue...")
+
+
+# Custom classes for ConsoleMenu, to make menu texts dynamic
+# Needed until https://github.com/aegirhall/console-menu/pull/25 is released
+# See https://pypi.org/project/console-menu/ for other docs
+
+class UpdatableConsoleMenu(ConsoleMenu):
+
+ def __repr__(self):
+ return "%s: %s. %d items" % (self.get_title(), self.get_subtitle(), len(self.items))
+
+ def draw(self):
+ """
+ Refreshes the screen and redraws the menu. Should be called whenever something changes that needs to be redrawn.
+ """
+ self.screen.printf(self.formatter.format(title=self.get_title(), subtitle=self.get_subtitle(), items=self.items,
+ prologue_text=self.get_prologue_text(), epilogue_text=self.get_epilogue_text()))
+
+ # Getters to get text in case method reference
+ def get_title(self):
+ return self.title() if callable(self.title) else self.title
+
+ def get_subtitle(self):
+ return self.subtitle() if callable(self.subtitle) else self.subtitle
+
+ def get_prologue_text(self):
+ return self.prologue_text() if callable(self.prologue_text) else self.prologue_text
+
+ def get_epilogue_text(self):
+ return self.epilogue_text() if callable(self.epilogue_text) else self.epilogue_text
+
+
+class UpdatableSubmenuItem(SubmenuItem):
+ def __init__(self, text, submenu, menu=None, should_exit=False):
+ """
+ :ivar ConsoleMenu self.submenu: The submenu to be opened when this item is selected
+ """
+ super(SubmenuItem, self).__init__(text=text, menu=menu, should_exit=should_exit)
+
+ self.submenu = submenu
+ if menu:
+ self.get_submenu().parent = menu
+
+ def show(self, index):
+ return "%2d - %s" % (index + 1, self.get_text())
+
+ # Getters to get text in case method reference
+ def get_text(self):
+ return self.text() if callable(self.text) else self.text
+
+ def set_menu(self, menu):
+ """
+ Sets the menu of this item.
+ Should be used instead of directly accessing the menu attribute for this class.
+
+ :param ConsoleMenu menu: the menu
+ """
+ self.menu = menu
+ self.get_submenu().parent = menu
+
+ def action(self):
+ """
+ This class overrides this method
+ """
+ self.get_submenu().start()
+
+ def clean_up(self):
+ """
+ This class overrides this method
+ """
+ self.get_submenu().join()
+ self.menu.clear_screen()
+ self.menu.resume()
+
+ def get_return(self):
+ """
+ :return: The returned value in the submenu
+ """
+ return self.get_submenu().returned_value
+
+ def get_submenu(self):
+ """
+ We unwrap the submenu variable in case it is a reference to a method that returns a submenu
+ """
+ return self.submenu if not callable(self.submenu) else self.submenu()
+
+
+class UpdatableFunctionItem(FunctionItem):
+ def show(self, index):
+ return "%2d - %s" % (index + 1, self.get_text())
+
+ # Getters to get text in case method reference
+ def get_text(self):
+ return self.text() if callable(self.text) else self.text
+
+
+class MyScreen(Screen):
+ def clear(self):
+ return
+
+
+class CustomExitItem(ExitItem):
+ def show(self, index):
+ return super(ExitItem, self).show(index)
+
+ def get_return(self):
+ return ""
+
+
+def main():
+ global state
+ global dry_run
+ global templates
+
+ print("Lucene/Solr releaseWizard v%s" % getScriptVersion())
+ c = parse_config()
+
+ if c.dry:
+ print("Entering dry-run mode where all commands will be echoed instead of executed")
+ dry_run = True
+
+ release_root = os.path.expanduser("~/.lucene-releases")
+ if not load_rc() or c.init:
+ print("Initializing")
+ dir_ok = False
+ root = str(input("Choose root folder: [~/.lucene-releases] "))
+ if os.path.exists(root) and (not os.path.isdir(root) or not os.access(root, os.W_OK)):
+ sys.exit("Root %s exists but is not a directory or is not writable" % root)
+ if not root == '':
+ if root.startswith("~/"):
+ release_root = os.path.expanduser(root)
+ else:
+ release_root = os.path.abspath(root)
+ if not os.path.exists(release_root):
+ try:
+ print("Creating release root %s" % release_root)
+ os.makedirs(release_root)
+ except Exception as e:
+ sys.exit("Error while creating %s: %s" % (release_root, e))
+ release_version = get_release_version()
+ else:
+ conf = load_rc()
+ release_root = conf['root']
+ if 'release_version' in conf:
+ release_version = conf['release_version']
+ else:
+ release_version = get_release_version()
+ store_rc(release_root, release_version)
+
+
+ check_prerequisites()
+
+ try:
+ y = yaml.load(open(os.path.join(script_path, "releaseWizard.yaml"), "r"), Loader=yaml.Loader)
+ templates = y.get('templates')
+ todo_list = y.get('groups')
+ state = ReleaseState(release_root, release_version, getScriptVersion())
+ state.init_todos(bootstrap_todos(todo_list))
+ state.load()
+ except Exception as e:
+ sys.exit("Failed initializing. %s" % e)
+
+ state.save()
+
+ # Smoketester requires JAVA_HOME to point to JAVA8 and JAVA11_HOME to point ot Java11
+ os.environ['JAVA_HOME'] = state.get_java_home()
+ os.environ['JAVACMD'] = state.get_java_cmd()
+
+ global tlp_news_draft
+ global lucene_news_draft
+ global solr_news_draft
+ global lucene_highlights_file
+ global solr_highlights_file
+ global website_folder
+ global tlp_news_file
+ global lucene_news_file
+ global solr_news_file
+ lucene_highlights_file = os.path.join(state.get_release_folder(), 'lucene_highlights.txt')
+ solr_highlights_file = os.path.join(state.get_release_folder(), 'solr_highlights.txt')
+ tlp_news_draft = os.path.join(state.get_release_folder(), 'tlp_news.md')
+ lucene_news_draft = os.path.join(state.get_release_folder(), 'lucene_news.md')
+ solr_news_draft = os.path.join(state.get_release_folder(), 'solr_news.md')
+ website_folder = os.path.join(state.get_release_folder(), 'website-source')
+ tlp_news_file = os.path.join(website_folder, 'content', 'mainnews.mdtext')
+ lucene_news_file = os.path.join(website_folder, 'content', 'core', 'corenews.mdtext')
+ solr_news_file = os.path.join(website_folder, 'content', 'solr', 'news.mdtext')
+
+ main_menu = UpdatableConsoleMenu(title="Lucene/Solr ReleaseWizard",
+ subtitle=get_releasing_text,
+ prologue_text="Welcome to the release wizard. From here you can manage the process including creating new RCs. "
+ "All changes are persisted, so you can exit any time and continue later. Make sure to read the Help section.",
+ epilogue_text="® 2019 The Lucene/Solr project. Licensed under the Apache License 2.0\nScript version v%s ALPHA)" % getScriptVersion(),
+ screen=MyScreen())
+
+ todo_menu = UpdatableConsoleMenu(title=get_releasing_text,
+ subtitle=get_subtitle,
+ prologue_text=None,
+ epilogue_text=None,
+ screen=MyScreen())
+ todo_menu.exit_item = CustomExitItem("Return")
+
+ for todo_group in state.todo_groups:
+ if todo_group.num_applies() >= 0:
+ menu_item = todo_group.get_menu_item()
+ menu_item.set_menu(todo_menu)
+ todo_menu.append_item(menu_item)
+
+ main_menu.append_item(UpdatableSubmenuItem(get_todo_menuitem_title, todo_menu, menu=main_menu))
+ main_menu.append_item(UpdatableFunctionItem(get_start_new_rc_menu_title, start_new_rc))
+ main_menu.append_item(UpdatableFunctionItem('Clear and restart current RC', state.clear_rc))
+ main_menu.append_item(UpdatableFunctionItem("Clear all state, restart the %s release" % state.release_version, reset_state))
+ main_menu.append_item(UpdatableFunctionItem('Start release for a different version', release_other_version))
+ main_menu.append_item(UpdatableFunctionItem('Generate Asciidoc guide for this release', generate_asciidoc))
+ # main_menu.append_item(UpdatableFunctionItem('Dump YAML', dump_yaml))
+ main_menu.append_item(UpdatableFunctionItem('Help', help))
+
+ main_menu.show()
+
+
+sys.path.append(os.path.dirname(__file__))
+current_git_root = os.path.abspath(
+ os.path.join(os.path.abspath(os.path.dirname(__file__)), os.path.pardir, os.path.pardir))
+
+dry_run = False
+
+major_minor = ['major', 'minor']
+script_path = os.path.dirname(os.path.realpath(__file__))
+os.chdir(script_path)
+
+
+def git_checkout_folder():
+ return state.get_git_checkout_folder()
+
+
+def tail_file(file, lines):
+ bufsize = 8192
+ fsize = os.stat(file).st_size
+ with open(file) as f:
+ if bufsize >= fsize:
+ bufsize = fsize
+ idx = 0
+ while True:
+ idx += 1
+ seek_pos = fsize - bufsize * idx
+ if seek_pos < 0:
+ seek_pos = 0
+ f.seek(seek_pos)
+ data = []
+ data.extend(f.readlines())
+ if len(data) >= lines or f.tell() == 0 or seek_pos == 0:
+ if not seek_pos == 0:
+ print("Tailing last %d lines of file %s" % (lines, file))
+ print(''.join(data[-lines:]))
+ break
+
+
+def run_with_log_tail(command, cwd, logfile=None, tail_lines=10, tee=False, live=False, shell=None):
+ fh = sys.stdout
+ if logfile:
+ logdir = os.path.dirname(logfile)
+ if not os.path.exists(logdir):
+ os.makedirs(logdir)
+ fh = open(logfile, 'w')
+ rc = run_follow(command, cwd, fh=fh, tee=tee, live=live, shell=shell)
+ if logfile:
+ fh.close()
+ if not tee and tail_lines and tail_lines > 0:
+ tail_file(logfile, tail_lines)
+ return rc
+
+
+def ask_yes_no(text):
+ answer = None
+ while answer not in ['y', 'n']:
+ answer = str(input("\nQ: %s (y/n): " % text))
+ print("\n")
+ return answer == 'y'
+
+
+def abbreviate_line(line, width):
+ line = line.rstrip()
+ if len(line) > width:
+ line = "%s.....%s" % (line[:(width / 2 - 5)], line[-(width / 2):])
+ else:
+ line = "%s%s" % (line, " " * (width - len(line) + 2))
+ return line
+
+
+def print_line_cr(line, linenum, stdout=True, tee=False):
+ if not tee:
+ if not stdout:
+ print("[line %s] %s" % (linenum, abbreviate_line(line, 80)), end='\r')
+ else:
+ if line.endswith("\r"):
+ print(line.rstrip(), end='\r')
+ else:
+ print(line.rstrip())
+
+
+def run_follow(command, cwd=None, fh=sys.stdout, tee=False, live=False, shell=None):
+ doShell = '&&' in command or '&' in command or shell is not None
+ if not doShell and not isinstance(command, list):
+ command = shlex.split(command)
+ process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=cwd,
+ universal_newlines=True, bufsize=0, close_fds=True, shell=doShell)
+ lines_written = 0
+
+ fl = fcntl.fcntl(process.stdout, fcntl.F_GETFL)
+ fcntl.fcntl(process.stdout, fcntl.F_SETFL, fl | os.O_NONBLOCK)
+
+ flerr = fcntl.fcntl(process.stderr, fcntl.F_GETFL)
+ fcntl.fcntl(process.stderr, fcntl.F_SETFL, flerr | os.O_NONBLOCK)
+
+ endstdout = endstderr = False
+ errlines = []
+ while not (endstderr and endstdout):
+ lines_before = lines_written
+ if not endstdout:
+ try:
+ if live:
+ chars = process.stdout.read()
+ if chars == '' and process.poll() is not None:
+ endstdout = True
+ else:
+ fh.write(chars)
+ fh.flush()
+ if '\n' in chars:
+ lines_written += 1
+ else:
+ line = process.stdout.readline()
+ if line == '' and process.poll() is not None:
+ endstdout = True
+ else:
+ fh.write("%s\n" % line.rstrip())
+ fh.flush()
+ lines_written += 1
+ print_line_cr(line, lines_written, stdout=(fh == sys.stdout), tee=tee)
+
+ except Exception as ioe:
+ pass
+ if not endstderr:
+ try:
+ if live:
+ chars = process.stderr.read()
+ if chars == '' and process.poll() is not None:
+ endstderr = True
+ else:
+ fh.write(chars)
+ fh.flush()
+ if '\n' in chars:
+ lines_written += 1
+ else:
+ line = process.stderr.readline()
+ if line == '' and process.poll() is not None:
+ endstderr = True
+ else:
+ errlines.append("%s\n" % line.rstrip())
+ lines_written += 1
+ print_line_cr(line, lines_written, stdout=(fh == sys.stdout), tee=tee)
+ except Exception as e:
+ pass
+
+ if not lines_written > lines_before:
+ # if no output then sleep a bit before checking again
+ time.sleep(0.1)
+
+ print(" " * 80)
+ rc = process.poll()
+ if len(errlines) > 0:
+ for line in errlines:
+ fh.write("%s\n" % line.rstrip())
+ fh.flush()
+ return rc
+
+
+def is_windows():
+ return platform.system().startswith("Win")
+
+
+class Commands(SecretYamlObject):
+ yaml_tag = u'!Commands'
+ hidden_fields = ['todo_id']
+ cmd_continuation_char = "^" if is_windows() else "\\"
+ def __init__(self, root_folder, commands_text=None, commands=None, logs_prefix=None, run_text=None, enable_execute=None,
+ confirm_each_command=None, env=None, vars=None, todo_id=None, remove_files=None):
+ self.root_folder = root_folder
+ self.commands_text = commands_text
+ self.vars = vars
+ self.env = env
+ self.run_text = run_text
+ self.remove_files = remove_files
+ self.todo_id = todo_id
+ self.logs_prefix = logs_prefix
+ self.enable_execute = enable_execute
+ self.confirm_each_command = confirm_each_command
+ self.commands = commands
+ for c in self.commands:
+ c.todo_id = todo_id
+
+ @classmethod
+ def from_yaml(cls, loader, node):
+ fields = loader.construct_mapping(node, deep = True)
+ return Commands(**fields)
+
+ def run(self):
+ root = self.get_root_folder()
+
+ if self.commands_text:
+ print(self.get_commands_text())
+ if self.env:
+ for key in self.env:
+ val = self.jinjaify(self.env[key])
+ os.environ[key] = val
+ if is_windows():
+ print("\n SET %s=%s" % (key, val))
+ else:
+ print("\n export %s=%s" % (key, val))
+ print(abbreviate_homedir("\n cd %s" % root))
+ commands = ensure_list(self.commands)
+ for cmd in commands:
+ for line in cmd.display_cmd():
+ print(" %s" % line)
+ print()
+ if not self.enable_execute is False:
+ if self.run_text:
+ print("\n%s\n" % self.get_run_text())
+ if len(commands) > 1:
+ if not self.confirm_each_command is False:
+ print("You will get prompted before running each individual command.")
+ else:
+ print(
+ "You will not be prompted for each command but will see the ouput of each. If one command fails the execution will stop.")
+ success = True
+ if ask_yes_no("Do you want me to run these commands now?"):
+ if self.remove_files:
+ for _f in ensure_list(self.get_remove_files()):
+ f = os.path.join(root, _f)
+ if os.path.exists(f):
+ filefolder = "File" if os.path.isfile(f) else "Folder"
+ if ask_yes_no("%s %s already exists. Shall I remove it now?" % (filefolder, f)) and not dry_run:
+ if os.path.isdir(f):
+ shutil.rmtree(f)
+ else:
+ os.remove(f)
+ index = 0
+ log_folder = self.logs_prefix if len(commands) > 1 else None
+ for cmd in commands:
+ index += 1
+ if len(commands) > 1:
+ log_prefix = "%02d_" % index
+ else:
+ log_prefix = self.logs_prefix if self.logs_prefix else ''
+ if not log_prefix[-1:] == '_':
+ log_prefix += "_"
+ cwd = root
+ if cmd.cwd:
+ cwd = os.path.join(root, cmd.cwd)
+ folder_prefix = ''
+ if cmd.cwd:
+ folder_prefix = cmd.cwd + "_"
+ if self.confirm_each_command is False or len(commands) == 1 or ask_yes_no("Shall I run '%s' in folder '%s'" % (cmd, cwd)):
+ if self.confirm_each_command is False:
+ print("------------\nRunning '%s' in folder '%s'" % (cmd, cwd))
+ logfilename = cmd.logfile
+ logfile = None
+ cmd_to_run = "%s%s" % ("echo Dry run, command is: " if dry_run else "", cmd.get_cmd())
+ if cmd.redirect:
+ try:
+ out = run(cmd_to_run, cwd=cwd)
+ mode = 'a' if cmd.redirect_append is True else 'w'
+ with open(os.path.join(root, cwd, cmd.get_redirect()), mode) as outfile:
+ outfile.write(out)
+ outfile.flush()
+ print("Wrote %s bytes to redirect file %s" % (len(out), cmd.get_redirect()))
+ except Exception as e:
+ print("Command %s failed: %s" % (cmd_to_run, e))
+ success = False
+ break
+ else:
+ if not cmd.stdout:
+ if not log_folder:
+ log_folder = os.path.join(state.get_rc_folder(), "logs")
+ elif not os.path.isabs(log_folder):
+ log_folder = os.path.join(state.get_rc_folder(), "logs", log_folder)
+ if not logfilename:
+ logfilename = "%s.log" % re.sub(r"\W", "_", cmd.get_cmd())
+ logfile = os.path.join(log_folder, "%s%s%s" % (log_prefix, folder_prefix, logfilename))
+ if cmd.tee:
+ print("Output of command will be printed (logfile=%s)" % logfile)
+ elif cmd.live:
+ print("Output will be shown live byte by byte")
+ logfile = None
+ else:
+ print("Wait until command completes... Full log in %s\n" % logfile)
+ if cmd.comment:
+ print("# %s\n" % cmd.get_comment())
+ start_time = time.time()
+ returncode = run_with_log_tail(cmd_to_run, cwd, logfile=logfile, tee=cmd.tee, tail_lines=25,
+ live=cmd.live, shell=cmd.shell)
+ elapsed = time.time() - start_time
+ if not returncode == 0:
+ if cmd.should_fail:
+ print("Command failed, which was expected")
+ success = True
+ else:
+ print("WARN: Command %s returned with error" % cmd.get_cmd())
+ success = False
+ break
+ else:
+ if cmd.should_fail and not dry_run:
+ print("Expected command to fail, but it succeeded.")
+ success = False
+ break
+ else:
+ if elapsed > 30:
+ print("Command completed in %s seconds" % elapsed)
+ if not success:
+ print("WARNING: One or more commands failed, you may want to check the logs")
+ return success
+
+ def get_root_folder(self):
+ return self.jinjaify(self.root_folder)
+
+ def get_commands_text(self):
+ return self.jinjaify(self.commands_text)
+
+ def get_run_text(self):
+ return self.jinjaify(self.run_text)
+
+ def get_remove_files(self):
+ return self.jinjaify(self.remove_files)
+
+ def get_vars(self):
+ myvars = {}
+ if self.vars:
+ for k in self.vars:
+ val = self.vars[k]
+ if callable(val):
+ myvars[k] = expand_jinja(val(), vars=myvars)
+ else:
+ myvars[k] = expand_jinja(val, vars=myvars)
+ return myvars
+
+ def jinjaify(self, data, join=False):
+ if not data:
+ return None
+ v = self.get_vars()
+ if self.todo_id:
+ v.update(state.get_todo_by_id(self.todo_id).get_vars())
+ if isinstance(data, list):
+ if join:
+ return expand_jinja(" ".join(data), v)
+ else:
+ res = []
+ for rf in data:
+ res.append(expand_jinja(rf, v))
+ return res
+ else:
+ return expand_jinja(data, v)
+
+
+def abbreviate_homedir(line):
+ if is_windows():
+ if 'HOME' in os.environ:
+ return re.sub(r'([^/]|\b)%s' % os.path.expanduser('~'), "\\1%HOME%", line)
+ elif 'USERPROFILE' in os.environ:
+ return re.sub(r'([^/]|\b)%s' % os.path.expanduser('~'), "\\1%USERPROFILE%", line)
+ else:
+ return re.sub(r'([^/]|\b)%s' % os.path.expanduser('~'), "\\1~", line)
+
+
+class Command(SecretYamlObject):
+ yaml_tag = u'!Command'
+ hidden_fields = ['todo_id']
+ def __init__(self, cmd, cwd=None, stdout=None, logfile=None, tee=None, live=None, comment=None, vars=None,
+ todo_id=None, should_fail=None, redirect=None, redirect_append=None, shell=None):
+ self.cmd = cmd
+ self.cwd = cwd
+ self.comment = comment
+ self.logfile = logfile
+ self.vars = vars
+ self.tee = tee
+ self.live = live
+ self.stdout = stdout
+ self.should_fail = should_fail
+ self.shell = shell
+ self.todo_id = todo_id
+ self.redirect_append = redirect_append
+ self.redirect = redirect
+ if tee and stdout:
+ self.stdout = None
+ print("Command %s specifies 'tee' and 'stdout', using only 'tee'" % self.cmd)
+ if live and stdout:
+ self.stdout = None
+ print("Command %s specifies 'live' and 'stdout', using only 'live'" % self.cmd)
+ if live and tee:
+ self.tee = None
+ print("Command %s specifies 'tee' and 'live', using only 'live'" % self.cmd)
+ if redirect and (tee or stdout or live):
+ self.tee = self.stdout = self.live = None
+ print("Command %s specifies 'redirect' and other out options at the same time. Using redirect only" % self.cmd)
+
+ @classmethod
+ def from_yaml(cls, loader, node):
+ fields = loader.construct_mapping(node, deep = True)
+ return Command(**fields)
+
+ def get_comment(self):
+ return self.jinjaify(self.comment)
+
+ def get_redirect(self):
+ return self.jinjaify(self.redirect)
+
+ def get_cmd(self):
+ return self.jinjaify(self.cmd, join=True)
+
+ def get_vars(self):
+ myvars = {}
+ if self.vars:
+ for k in self.vars:
+ val = self.vars[k]
+ if callable(val):
+ myvars[k] = expand_jinja(val(), vars=myvars)
+ else:
+ myvars[k] = expand_jinja(val, vars=myvars)
+ return myvars
+
+ def __str__(self):
+ return self.get_cmd()
+
+ def jinjaify(self, data, join=False):
+ v = self.get_vars()
+ if self.todo_id:
+ v.update(state.get_todo_by_id(self.todo_id).get_vars())
+ if isinstance(data, list):
+ if join:
+ return expand_jinja(" ".join(data), v)
+ else:
+ res = []
+ for rf in data:
+ res.append(expand_jinja(rf, v))
+ return res
+ else:
+ return expand_jinja(data, v)
+
+ def display_cmd(self):
+ lines = []
+ pre = post = ''
+ if self.comment:
+ if is_windows():
+ lines.append("REM %s" % self.get_comment())
+ else:
+ lines.append("# %s" % self.get_comment())
+ if self.cwd:
+ lines.append("pushd %s" % self.cwd)
+ redir = "" if self.redirect is None else " %s %s" % (">" if self.redirect_append is None else ">>" , self.get_redirect())
+ line = "%s%s" % (expand_multiline(self.get_cmd(), indent=2), redir)
+ # Print ~ or %HOME% rather than the full expanded homedir path
+ line = abbreviate_homedir(line)
+ lines.append(line)
+ if self.cwd:
+ lines.append("popd")
+ return lines
+
+class UserInput(SecretYamlObject):
+ yaml_tag = u'!UserInput'
+
+ def __init__(self, name, prompt, type=None):
+ self.type = type
+ self.prompt = prompt
+ self.name = name
+
+ @classmethod
+ def from_yaml(cls, loader, node):
+ fields = loader.construct_mapping(node, deep = True)
+ return UserInput(**fields)
+
+ def run(self, dict=None):
+ correct = False
+ while not correct:
+ try:
+ result = str(input("%s : " % self.prompt))
+ if self.type and self.type == 'int':
+ result = int(result)
+ correct = True
+ except Exception as e:
+ print("Incorrect input: %s, try again" % e)
+ continue
+ if dict:
+ dict[self.name] = result
+ return result
+
+
+def create_ical(todo):
+ if ask_yes_no("Do you want to add a Calendar reminder for the close vote time?"):
+ c = Calendar()
+ e = Event()
+ e.name = "Lucene/Solr %s vote ends" % state.release_version
+ e.begin = vote_close_72h_date()
+ e.description = "Remember to sum up votes and continue release :)"
+ c.events.add(e)
+ ics_file = os.path.join(state.get_rc_folder(), 'vote_end.ics')
+ with open(ics_file, 'w') as my_file:
+ my_file.writelines(c)
+ open_file(ics_file)
+ return True
+
+
+today = datetime.utcnow().date()
+weekends = {(today + timedelta(days=x)): 'Saturday' for x in range(10) if (today + timedelta(days=x)).weekday() == 5}
+weekends.update({(today + timedelta(days=x)): 'Sunday' for x in range(10) if (today + timedelta(days=x)).weekday() == 6})
+y = datetime.utcnow().year
+years = [y, y+1]
+non_working = holidays.CA(years=years) + holidays.US(years=years) + holidays.England(years=years) \
+ + holidays.DE(years=years) + holidays.NO(years=years) + holidays.SE(years=years) + holidays.RU(years=years)
+
+
+def vote_close_72h_date():
+ working_days = 0
+ day_offset = -1
+ # Require voting open for 3 working days, not counting todays date
+ # Working day is defined as saturday, sunday or a public holiday observed by 3 or more [CA, US, EN, DE, NO, SE, RU]
+ while working_days < 4:
+ day_offset += 1
+ d = today + timedelta(days=day_offset)
+ if not (d in weekends or (d in non_working and len(non_working[d]) >= 3)):
+ working_days += 1
+ return datetime.utcnow() + timedelta(days=day_offset) + timedelta(hours=1)
+
+
+def website_javadoc_redirect(todo):
+ htfile = os.path.join(website_folder, 'content', '.htaccess')
+ latest = state.get_latest_version()
+ if Version.parse(state.release_version).gt(Version.parse(latest)):
+ print("We are releasing the latest version ")
+ htaccess = file_to_string(htfile)
+ print("NOT YET IMPLEMENTED")
+ return False
+ else:
+ print("Task not necessary since %s is not the latest release version" % state.release_version)
+ return True
+
+
+def prepare_highlights(todo):
+ if not os.path.exists(lucene_highlights_file):
+ with open(lucene_highlights_file, 'w') as fp:
+ fp.write("* New cool Lucene feature\n* Important bugfix")
+ if not os.path.exists(solr_highlights_file):
+ with open(solr_highlights_file, 'w') as fp:
+ fp.write("* New cool Solr feature\n* Important bugfix")
+ return True
+
+
+def prepare_announce(todo):
+ if not os.path.exists(tlp_news_draft):
+ tlp_text = expand_jinja("(( template=announce_tlp ))")
+ with open(tlp_news_draft, 'w') as fp:
+ fp.write(tlp_text)
+ # print("Wrote TLP announce draft to %s" % tlp_news_file)
+
+ lucene_text = expand_jinja("(( template=announce_lucene ))")
+ with open(lucene_news_draft, 'w') as fp:
+ fp.write(lucene_text)
+ # print("Wrote Lucene announce draft to %s" % lucene_news_file)
+
+ solr_text = expand_jinja("(( template=announce_solr ))")
+ with open(solr_news_draft, 'w') as fp:
+ fp.write(solr_text)
+ # print("Wrote Solr announce draft to %s" % solr_news_file)
+ else:
+ print("Drafts already exist, not re-generating")
+ return True
+
+
+def patch_news_file(orig, draft, sticky_lines):
+ orig_lines = open(orig).readlines()
+ draft_lines = open(draft).readlines()
+ lines = orig_lines[0:sticky_lines]
+ lines.extend(draft_lines)
+ lines.append('\n')
+ lines.append('\n')
+ lines.extend(orig_lines[sticky_lines:])
+ with open(orig, 'w') as fp:
+ fp.writelines(lines)
+ print("Added news to %s" % orig)
+
+
+def update_news(todo):
+ touch_file = os.path.join(state.get_release_folder(), 'news_updated')
+ if not os.path.exists(touch_file):
+ patch_news_file(tlp_news_file, tlp_news_draft, 2)
+ patch_news_file(lucene_news_file, lucene_news_draft, 2)
+ patch_news_file(solr_news_file, solr_news_draft, 4)
+
+ latest = state.get_latest_version()
+ if Version.parse(state.release_version).gt(Version.parse(latest)):
+ print("We are releasing the latest version, updating latestversion.mdtext")
+ with open(os.path.join(website_folder, 'content', 'latestversion.mdtext'), 'w') as fp:
+ fp.write(state.release_version)
+
+ with open(touch_file, 'w') as fp:
+ fp.write("true")
+ print("News files in website folder updated with draft announcements")
+ else:
+ print("News files not changed, already patched earlier. Please edit by hand")
+
+ return True
+
+
+def set_java_home(version):
+ os.environ['JAVA_HOME'] = state.get_java_home_for_version(version)
+ os.environ['JAVACMD'] = state.get_java_cmd_for_version(version)
+
+
+def load_lines(file):
+ if os.path.exists(file):
+ with open(file, 'r') as fp:
+ return fp.readlines()
+ else:
+ return ['* foo', '* bar']
+
+
+if __name__ == '__main__':
+ try:
+ main()
+ except KeyboardInterrupt:
+ print('Keyboard interrupt...exiting')
diff --git a/dev-tools/scripts/releaseWizard.yaml b/dev-tools/scripts/releaseWizard.yaml
new file mode 100644
index 0000000..0827cdf
--- /dev/null
+++ b/dev-tools/scripts/releaseWizard.yaml
@@ -0,0 +1,1735 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# =====================================================================
+# This file contains the definition TODO steps and commands to run for the
+# releaseWizard.py script. It also contains Jinja2 templates for use in those
+# definitions. See documentation for "groups" below the templates.
+# Edit this file with an editor with YAML support, such as Sublime Text
+# for syntax highlighting and context sensitive tag suggestion
+# =====================================================================
+#
+# Templates may be included in any text by starting a line with this syntax:
+# (( template=my_template_name ))
+# Templates may contain other templates for easy re-use of snippets
+# And of course all Jinja2 syntax for inclusion of variables etc is supported.
+# See http://jinja.pocoo.org/docs/2.10/templates/ for details
+# To add new global variables or functions/filters, edit releaseWizard.py
+#
+templates:
+ help: |
+ Welcome to the role as Release Manager for Lucene/Solr, and the releaseWizard!
+
+ The Release Wizard aims to walk you through the whole release process step by step,
+ helping you to to run the right commands in the right order, generating
+ e-mail templates for you with the correct texts, versions, paths etc, obeying
+ the voting rules and much more. It also serves as a documentation of all the
+ steps, with timestamps, preserving log files from each command etc.
+
+ As you complete each step the tool will ask you if the task is complete, making
+ it easy for you to know what is done and what is left to do. If you need to
+ re-spin a Release Candidata (RC) the Wizard will also help.
+
+ The Lucene project has automated much of the release process with various scripts,
+ and this wizard is the glue that binds it all together.
+
+ In the first TODO step in the checklist you will be asked to read up on the
+ Apache release policy and other relevant documents before you start the release.
+
+ NOTE: Even if we have great tooling and some degree of automation, there are
+ still many manual steps and it is also important that the RM validates
+ and QAs the process, validating that the right commands are run, and that
+ the output from scripts are correct before proceeding.
+
+ DISCLAIMER: This is an alpha version. The wizard may be buggy and may generate
+ faulty commands, commands that won't work on your OS or with all
+ versions of tooling etc. So please keep the old ReleaseTODO handy
+ for cross-checking for now :)
+ vote_logic: |
+ {% set passed = plus_binding >= 3 and minus < plus_binding %}
+ {% set too_few = plus_binding < 3 %}
+ {% set veto = plus_binding < minus %}
+ {% set reason = 'too few binding votes' if too_few else 'too many negative votes' if veto else 'unknown' %}
+ vote_macro: |
+ {% macro end_vote_result(plus_binding, plus_other, zero, minus) -%}
+ (( template=vote_logic ))
+ .Mail template {% if passed %}successful{% else %}failed{% endif %} vote
+ ----
+ To: dev@lucene.apache.org
+ Subject: [{% if passed %}RESULT{% else %}FAILED{% endif %}] [VOTE] Release Lucene/Solr {{ release_version }} RC{{ rc_number }}
+
+ It's been >72h since the vote was initiated and the result is:
+
+ +1 {{ plus_binding + plus_other }} ({{ plus_binding }} binding)
+ 0 {{ zero }}
+ -1 {{ minus }}
+
+ {% if not passed %}
+ Reason for fail is {{ reason }}.
+
+ {% endif %}
+ This vote has {% if passed %}PASSED{% else %}FAILED{% endif %}
+ ----
+ {%- endmacro %}
+ announce_tlp: |
+ ## {{ release_date | formatdate | default('<date>') }}, Apache Lucene {{ release_version }} and Apache Solr {{ release_version }} available
+
+ The Lucene PMC is pleased to announce the release of Apache® Lucene™ {{ release_version }} and Apache® Solr™ {{ release_version }}.
+
+ Lucene can be downloaded from <https://lucene.apache.org/core/downloads.html>
+ and Solr can be downloaded from <https://lucene.apache.org/solr/downloads.html>
+
+ ### Highlights of this Lucene release include:
+
+ {% for line in load_lines(lucene_highlights_file) %}
+ {{ line }}
+ {%- endfor %}
+
+
+ ### Highlights of this Solr release include:
+
+ {% for line in load_lines(solr_highlights_file) %}
+ {{ line }}
+ {%- endfor %}
+ announce_lucene: |
+ ## {{ release_date | formatdate | default('<date>') }}, Apache Lucene™ {{ release_version }} available
+
+ The Lucene PMC is pleased to announce the release of Apache Lucene {{ release_version }}.
+
+ Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform.
+
+ This release contains numerous bug fixes, optimizations, and improvements, some of which are highlighted below. The release is available for immediate download at:
+
+ <https://lucene.apache.org/core/downloads.html>
+
+ ### Lucene {{ release_version }} Release Highlights:
+
+ {% for line in load_lines(lucene_highlights_file) %}
+ {{ line }}
+ {%- endfor %}
+
+
+ Please read CHANGES.txt for a full list of {% if is_feature_release %}new features and {% endif %}changes:
+
+ <https://lucene.apache.org/core/{{ release_version_underscore }}/changes/Changes.html>
+ announce_solr: |
+ ## {{ release_date | formatdate | default('<date>') }}, Apache Solr™ {{ release_version }} available
+
+ The Lucene PMC is pleased to announce the release of Apache Solr {{ release_version }}.
+
+ Solr is the popular, blazing fast, open source NoSQL search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, rich document handling, and geospatial search. Solr is highly scalable, providing fault tolerant distributed search and indexing, and powers the search and navigation features of many of the world's largest internet sites.
+
+ Solr {{ release_version }} is available for immediate download at:
+
+ <https://lucene.apache.org/solr/downloads.html>
+
+ ### Solr {{ release_version }} Release Highlights:
+
+ {% for line in load_lines(solr_highlights_file) %}
+ {{ line }}
+ {%- endfor %}
+
+
+ Please read CHANGES.txt for a full list of {% if is_feature_release %}new features and {% endif %}changes:
+
+ <https://lucene.apache.org/solr/{{ release_version_underscore }}/changes/Changes.html>
+
+ Solr {{ release_version }} also includes {% if is_feature_release %}features, optimizations {% endif %} and bugfixes in the corresponding Apache Lucene release:
+
+ <https://lucene.apache.org/core/{{ release_version_underscore }}/changes/Changes.html>
+ announce_lucene_mail: |
+ The template below can be used to announce the Lucene release to the
+ internal mailing lists.
+
+ .Mail template
+ ----
+ To: dev@lucene.apache.org, general@lucene.apache.org, java-user@lucene.apache.org
+ Subject: [ANNOUNCE] Apache Lucene {{ release_version }} released
+
+ (( template=announce_lucene_mail_body ))
+ ----
+ announce_lucene_sign_mail: |
+ The template below can be used to announce the Lucene release to the
+ `announce@apache.org` mailing list. The mail *should be signed with PGP.*
+ and sent *from your `@apache.org` account*.
+
+ .Mail template
+ ----
+ From: {{ gpg.apache_id }}@apache.org
+ To: announce@apache.org
+ Subject: [ANNOUNCE] Apache Lucene {{ release_version }} released
+
+ (( template=announce_lucene_mail_body ))
+ ----
+ announce_lucene_mail_body: |
+ {% for line in load_lines(lucene_news_draft) -%}
+ {{ line }}
+ {%- endfor %}
+
+
+ Note: The Apache Software Foundation uses an extensive mirroring network for
+ distributing releases. It is possible that the mirror you are using may not have
+ replicated the release yet. If that is the case, please try another mirror.
+ This also applies to Maven access.
+ announce_solr_mail: |
+ The template below can be used to announce the Solr release to the
+ internal mailing lists.
+
+ .Mail template
+ ----
+ To: dev@lucene.apache.org, general@lucene.apache.org, solr-user@lucene.apache.org
+ Subject: [ANNOUNCE] Apache Solr {{ release_version }} released
+
+ (( template=announce_solr_mail_body ))
+ ----
+ announce_solr_sign_mail: |
+ The template below can be used to announce the Solr release to the
+ `announce@apache.org` mailing list. The mail *should be signed with PGP.*
+ and sent *from your `@apache.org` account*.
+
+ .Mail template
+ ----
+ From: {{ gpg.apache_id }}@apache.org
+ To: announce@apache.org
+ Subject: [ANNOUNCE] Apache Solr {{ release_version }} released
+
+ (( template=announce_solr_mail_body ))
+ ----
+ announce_solr_mail_body: |
+ {% for line in load_lines(solr_news_draft) -%}
+ {{ line }}
+ {%- endfor %}
+
+
+ Note: The Apache Software Foundation uses an extensive mirroring network for
+ distributing releases. It is possible that the mirror you are using may not have
+ replicated the release yet. If that is the case, please try another mirror.
+ This also applies to Maven access.
+# TODOs belong to groups for easy navigation in menus. Todo objects may contain asciidoc
+# descriptions, a number of commands to execute, some links to display, user input to gather
+# etc. Here is the documentation of each type of object. For further details, please consult
+# the corresponding Python object in releaseWizard.py, as these are parsed 1:1 from yaml.
+#
+# - !TodoGroup
+# id: unique_id
+# title: Title which will appear in menu
+# description: Longer description that will appear in sub-menu
+# depends: ['group1_id', 'group2_id'] # Explicit dependencies for groups
+# is_in_rc_loop: Tells that a group is thrown away on RC re-psin (default=False)
+# todos: # Array of !Todo objects beloning to the group
+# !Todo
+# id: todo_id
+# title: Short title that will appear in menu and elsewhere
+# description: |
+# The main description being printed when selecing the todo item. Here
+# you should introduce the task in more detail. You can use {{ jinja_var }} to
+# reference variables. See `releaseWizard.py` for list of global vars supported.
+# You can reference state saved from earlier TODO items using syntax
+# {{ todi_id.var_name }}
+# with `var_name` being either fetched from user_input or persist_vars
+# depends: # One or more dependencies which will bar execution
+# - todo_id1
+# - todo_id2
+# vars: # Dictionary of jinja2 variables local to this TODO, e.g.
+# logfile_path: "{{ [rc_folder, 'logs'] | path_join }}"
+# # Vars can contain global jinja vars or local vars earlier defined (ordered dict)
+# persist_vars: ['var_name', 'var_name'] # List of variables to persist in TODO state
+# asciidoc: |
+# Some `asciidoc` text to be included in asciidoc guide
+# *instead of* description/post_description
+# function: my_python_function # Will call the named function for complex tasks
+# commands: !Commands # A !Commands object holding commands to execute for this todo
+# root_folder: '{{ git_checkout_folder }}' # path to where commands will run
+# commands_text: Introduction text to be displayed just before the commands
+# enable_execute: true # Set to false to never offer to run commands automatically
+# confirm_each_command: true # Set to false to run all commands without prompting
+# remove_files: ['file1', 'folder2'] # List of files or folders that must be gone
+# logs_prefix: prefix # Lets you prefix logs file names with this string
+# commands: # List of !Commands to execute
+# - !Command # One single command
+# cmd: "ls {{ folder_to_ls }}" # A single command. May reference jinja vars
+# # Double spaces in a cmd will trigger multi-line display with continuation char \
+# cwd: relative_path # Where to run command, relative to root_folder
+# comment: # Will display a # or REM comment above the command in printouts
+# vars: {} # Possible to define local vars for this command only
+# logfile: my.og # Overrides log file name which may grow very long :)
+# tee: false # If true, sends output to console and file
+# stdout: false # if true, sends output only to console, not log file
+# live: false # If true, sends output live byte-by-byte to console
+# redirect: file.txt # Send output to file. Use instead of >
+# redirect_append: false # Will cause output to be appended, like >>
+# shell: false $ Set to true to use built-in shell commands
+# user_input: # An array of !UserInput objects for requesting input from user
+# - !UserInput
+# prompt: Please enter your gpg key ID, e.g. 0D8D0B93
+# name: gpg_id # This will be stored in todo state and can be referenced as {{ todo_id.name }}
+# type: int # if no type is given, a string is stored. Supported types are 'int'
+# post_description: |
+# Some `asciidoc` text (with jinja template support)
+# to be printed *after* commands and user_input is done.
+# links:
+# - http://example.com/list/of/links?to&be&displayed
+groups:
+- !TodoGroup
+ id: prerequisites
+ title: Prerequisites
+ description: |
+ Releasing software requires thorough understanding of the process and careful execution,
+ as it is easy to make mistakes. It also requires an environtment and tools such as gpg
+ correctly setup. This section makes sure you're in good shape for the job!
+ todos:
+ - !Todo
+ id: read_up
+ title: Read up on the release process
+ description: |-
+ As a Release Manager (RM) you should be familiar with Apache's release policy,
+ voting rules, create a PGP/GPG key for use with signing and more. Please familiarise
+ yourself with the resources listed below.
+ links:
+ - http://www.apache.org/dev/release-publishing.html
+ - http://www.apache.org/legal/release-policy.html
+ - http://www.apache.org/dev/release-signing.html
+ - https://wiki.apache.org/lucene-java/ReleaseTodo
+ - !Todo
+ id: tools
+ title: Necessary tools are installed
+ description: |
+ You will need these tools:
+
+ * Python v3.4 or later, with dependencies listed in requirements.txt
+ * Java 8 in $JAVA8_HOME and Java 11 in $JAVA11_HOME
+ * Apache Ant 1.8 or later. (Known issue with 1.10 and GPG password entry)
+ * gpg
+ * git
+ * svn
+ * asciidoctor (to generate HTML version)
+
+ You should also set the $EDITOR environment variable, else we'll fallback to
+ `vi` on Linux and `notepad.exe` on Windows, and you don't want that :)
+ function: check_prerequisites
+ links:
+ - https://gnupg.org/download/index.html
+ - https://asciidoctor.org
+ - !Todo
+ id: gpg
+ title: GPG key id is configured
+ description: |-
+ To sign the release you need to provide your GPG key ID. This must be
+ the same key ID that you have registered in your Apache account.
+ The ID is the last 8 bytes of your key fingerprint, e.g. 0D8D0B93.
+
+ * Make sure your gpg key is 4096 bits key or larger
+ * Upload your key to the MIT key server, pgp.mit.edu
+ * Put you GPG key's fingerprint in the `OpenPGP Public Key Primary Fingerprint`
+ field in your id.apache.org profile
+ * The tests will complain if your GPG key has not been signed by another Lucene
+ committer. This makes you a part of the GPG "web of trust" (WoT). Ask a committer
+ that you know personally to sign your key for you, providing them with the
+ fingerprint for the key.
+ function: configure_pgp
+ links:
+ - http://www.apache.org/dev/release-signing.html
+ - http://www.apache.org/dev/openpgp.html#apache-wot
+ - https://id.apache.org
+ - https://dist.apache.org/repos/dist/release/lucene/KEYS
+- !TodoGroup
+ id: preparation
+ title: Prepare for the release
+ description: Work with the community to decide when the release will happen and what work must be completed before it can happen
+ todos:
+ - !Todo
+ id: decide_jira_issues
+ title: Select JIRA issues to be included
+ description: Set the appropriate "Fix Version" in JIRA for the issues that should be included in the release.
+ - !Todo
+ id: decide_branch_date
+ title: Decide the date for branching
+ types:
+ - major
+ - minor
+ user_input: !UserInput
+ prompt: Enter date (YYYY-MM-DD)
+ name: branch_date
+ - !Todo
+ id: decide_freeze_length
+ title: Decide the lenght of feature freeze
+ types:
+ - major
+ - minor
+ user_input: !UserInput
+ prompt: Enter end date of feature freeze (YYYY-MM-DD)
+ name: feature_freeze_date
+- !TodoGroup
+ id: branching_versions
+ title: Create branch (if needed) and update versions
+ description: Here you'll do all the branching and version updates needed to prepare for the new release version
+ todos:
+ - !Todo
+ id: clean_git_checkout
+ title: Do a clean git clone to do the release from.
+ description: This eliminates the risk of a dirty checkout
+ commands: !Commands
+ root_folder: '{{ release_folder }}'
+ commands_text: Run these commands to make a fresh clone in the release folder
+ remove_files:
+ - '{{ git_checkout_folder }}'
+ commands:
+ - !Command
+ cmd: git clone --progress https://gitbox.apache.org/repos/asf/lucene-solr.git lucene-solr
+ logfile: git_clone.log
+ - !Todo
+ id: ant_precommit
+ title: Run ant precommit and fix issues
+ depends: clean_git_checkout
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: |-
+ From the base branch {{ base_branch }} we'll run precommit tests.
+ Fix any problems that are found by pushing fixes to the branch
+ and then running this task again. This task will always do `git pull`
+ before `ant precommit` so it will catch changes to your branch :)
+ confirm_each_command: false
+ commands:
+ - !Command
+ cmd: git checkout {{ base_branch }}
+ stdout: true
+ - !Command
+ cmd: git clean -df && git checkout -- .
+ comment: Make sure checkout is clean and up to date
+ logfile: git_clean.log
+ tee: true
+ - !Command
+ cmd: git pull
+ stdout: true
+ - !Command
+ cmd: ant clean precommit
+ - !Todo
+ id: create_stable_branch
+ title: Create a new stable branch, off from master
+ description: In our case we'll create {{ stable_branch }}
+ types:
+ - major
+ depends: clean_git_checkout
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: Run these commands to create a stable branch
+ commands:
+ - !Command
+ cmd: git checkout master
+ tee: true
+ - !Command
+ cmd: git update
+ tee: true
+ - !Command
+ cmd: git ls-remote --exit-code --heads origin {{ stable_branch }}
+ stdout: true
+ should_fail: true
+ comment: We expect error code 2 since {{ stable_branch }} does not already exist
+ - !Command
+ cmd: git checkout -b {{ stable_branch }}
+ tee: true
+ - !Command
+ cmd: git push origin {{ stable_branch }}
+ tee: true
+ - !Todo
+ id: create_minor_branch
+ title: Create a new minor branch off the stable branch
+ description: In our case we'll create {{ release_branch }}
+ types:
+ - major
+ - minor
+ depends: clean_git_checkout
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: Run these commands to create a release branch
+ commands:
+ - !Command
+ cmd: git checkout {{ stable_branch }}
+ tee: true
+ - !Command
+ cmd: git update
+ tee: true
+ - !Command
+ cmd: git ls-remote --exit-code --heads origin {{ release_branch }}
+ stdout: true
+ should_fail: true
+ comment: This command should fail with exit code 2 to verify branch {{ release_branch }} does not already exist
+ - !Command
+ cmd: git checkout -b {{ release_branch }}
+ tee: true
+ - !Command
+ cmd: git push origin {{ release_branch }}
+ tee: true
+ - !Todo
+ id: add_version_major
+ title: Add a new major version on master branch
+ types:
+ - major
+ depends: clean_git_checkout
+ vars:
+ next_version: "{{ release_version_major + 1 }}.0.0"
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: Run these commands to add the new major version {{ next_version }} to the master branch
+ commands:
+ - !Command
+ cmd: git checkout master
+ tee: true
+ - !Command
+ cmd: python3 -u dev-tools/scripts/addVersion.py {{ next_version }}
+ tee: true
+ post_description: |
+ Make sure to follow the manual instructions printed by the script:
+
+ * Move backcompat oldIndexes to unsupportedIndexes in TestBackwardsCompatibility
+ * Update IndexFormatTooOldException throw cases
+
+ There may be other steps needed as well
+ - !Todo
+ id: add_version_minor
+ title: Add a new minor version on stable branch
+ types:
+ - major
+ - minor
+ depends: clean_git_checkout
+ vars:
+ next_version: "{{ release_version_major }}.{{ release_version_minor + 1 }}.0"
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: Run these commands to add the new major version {{ next_version }} to the stable branch
+ commands:
+ - !Command
+ cmd: git checkout {{ stable_branch }}
+ tee: true
+ - !Command
+ cmd: python3 -u dev-tools/scripts/addVersion.py {{ next_version }}
+ tee: true
+ - !Todo
+ id: sanity_check_doap
+ title: Sanity check the DOAP files
+ description: |-
+ Sanity check the DOAP files under `dev-tools/doap/`.
+ Do they contain all releases less than the one in progress?
+
+ TIP: The buildAndPushRelease script run later will check this automatically
+ links:
+ - https://projects.apache.org/doap.html
+ - !Todo
+ id: jenkins_builds
+ title: Add Jenkins task for the release branch
+ description: '...so that builds run for the new branch. Consult the JenkinsReleaseBuilds page.'
+ types:
+ - major
+ - minor
+ links:
+ - https://wiki.apache.org/lucene-java/JenkinsReleaseBuilds
+ - !Todo
+ id: inform_devs
+ title: Inform Devs of the new Release Branch
+ description: |-
+ Send a note to dev@ to inform the committers that the branch
+ has been created and the feature freeze phase has started.
+
+ This is an e-mail template you can use as a basis for
+ announcing the new branch and feature freeze.
+
+ .Mail template
+ ----
+ To: dev@lucene.apache.org
+ Subject: New branch and feature freeze for Lucene/Solr {{ release_version }}
+
+ NOTICE:
+
+ Branch {{ release_branch }} has been cut and versions updated to {{ release_version_major }}.{{ release_version_minor + 1 }} on stable branch.
+
+ Please observe the normal rules:
+
+ * No new features may be committed to the branch.
+ * Documentation patches, build patches and serious bug fixes may be
+ committed to the branch. However, you should submit all patches you
+ want to commit to Jira first to give others the chance to review
+ and possibly vote against the patch. Keep in mind that it is our
+ main intention to keep the branch as stable as possible.
+ * All patches that are intended for the branch should first be committed
+ to the unstable branch, merged into the stable branch, and then into
+ the current release branch.
+ * Normal unstable and stable branch development may continue as usual.
+ However, if you plan to commit a big change to the unstable branch
+ while the branch feature freeze is in effect, think twice: can't the
+ addition wait a couple more days? Merges of bug fixes into the branch
+ may become more difficult.
+ * Only Jira issues with Fix version {{ release_version_major }}.{{ release_version_minor }} and priority "Blocker" will delay
+ a release candidate build.
+ ----
+ types:
+ - major
+ - minor
+ - !Todo
+ id: inform_devs_bugfix
+ title: Inform Devs about the planned release
+ description: |-
+ Send a note to dev@ to inform the committers about the rules for committing to the branch.
+
+ This is an e-mail template you can use as a basis for
+ announcing the rules for committing to the release branch
+
+ .Mail template
+ ----
+ To: dev@lucene.apache.org
+ Subject: Bugfix release Lucene/Solr {{ release_version }}
+
+ NOTICE:
+
+ I am now preparing for a bugfix release from branch {{ release_branch }}
+
+ Please observe the normal rules for committing to this branch:
+
+ * Before committing to the branch, reply to this thread and argue
+ why the fix needs backporting and how long it will take.
+ * All issues accepted for backporting should be marked with {{ release_version }}
+ in JIRA, and issues that should delay the release must be marked as Blocker
+ * All patches that are intended for the branch should first be committed
+ to the unstable branch, merged into the stable branch, and then into
+ the current release branch.
+ * Only Jira issues with Fix version {{ release_version }} and priority "Blocker" will delay
+ a release candidate build.
+ ----
+ types:
+ - bugfix
+ - !Todo
+ id: draft_release_notes
+ title: Get a draft of the release notes in place
+ description: |-
+ These are typically edited on the Wiki.
+
+ Clone a page for a previous version as a starting point for your release notes.
+ You will need two pages, one for Lucene and another for Solr, see links.
+ Edit the contents of `CHANGES.txt` into a more concise format for public consumption.
+ Ask on dev@ for input. Ideally the timing of this request mostly coincides with the
+ release branch creation. It's a good idea to remind the devs of this later in the release too.
+ links:
+ - https://wiki.apache.org/lucene-java/ReleaseNote77
+ - https://wiki.apache.org/solr/ReleaseNote77
+ - !Todo
+ id: new_jira_versions
+ title: Add a new version in JIRA for the next release
+ description: |-
+ Go to the JIRA "Manage Versions" Administration pages and add the new version:
+
+ {% if release_type == 'major' -%}
+ . Change name of version `master ({{ release_version_major }}.0)` into `{{ release_version_major }}.0`
+ {%- endif %}
+ . Create a new (unreleased) version `{{ get_next_version }}`
+
+ This needs to be done both for Lucene and Solr JIRAs, see links.
+ types:
+ - major
+ - minor
+ links:
+ - https://issues.apache.org/jira/plugins/servlet/project-config/LUCENE/versions
+ - https://issues.apache.org/jira/plugins/servlet/project-config/SOLR/versions
+- !TodoGroup
+ id: artifacts
+ title: Build the release artifacts
+ description: |-
+ If after the last day of the feature freeze phase no blocking issues are
+ in JIRA with "Fix Version" {{ release_version }}, then it's time to build the
+ release artifacts, run the smoke tester and stage the RC in svn
+ depends:
+ - test
+ - prerequisites
+ is_in_rc_loop: true
+ todos:
+ - !Todo
+ id: run_tests
+ title: Run javadoc tests
+ depends: clean_git_checkout
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: Run some tests not ran by `buildAndPublishRelease.py`
+ confirm_each_command: false
+ commands:
+ - !Command
+ cmd: git checkout {{ release_branch }}
+ stdout: true
+ - !Command
+ cmd: ant javadocs
+ cwd: lucene
+ - !Command
+ cmd: ant javadocs
+ cwd: solr
+ post_description: Check that both tests pass. If they fail, commit fixes for the failures before proceeding.
+ - !Todo
+ id: clear_ivy_cache
+ title: Clear the ivy cache
+ description: |
+ It is recommended to clean your Ivy cache before building the artifacts.
+ This ensures that all Ivy dependencies are freshly downloaded,
+ so we emulate a user that never used the Lucene build system before.
+ commands: !Commands
+ root_folder: '{{ home }}'
+ remove_files: .ivy2/cache_bak
+ commands_text: These commands will help you rename the folder so you can get it back later if you wish
+ commands:
+ - !Command
+ cmd: "{{ rename_cmd }} cache cache_bak"
+ cwd: .ivy2
+ stdout: true
+ - !Todo
+ id: build_rc
+ title: Build the release candidate
+ depends:
+ - gpg
+ - run_tests
+ vars:
+ logfile: '{{ [rc_folder, ''logs'', ''buildAndPushRelease.log''] | path_join }}'
+ git_rev: '{{ current_git_rev }}' # Note, git_rev will be recorded in todo state AFTER completion of commands
+ local_keys: '{% if keys_downloaded %} --local-keys "{{ [config_path, ''KEYS''] | path_join }}"{% endif %}'
+ persist_vars:
+ - git_rev
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: |-
+ In this step we will build the RC using python script `buildAndPushRelease.py`
+ We have tried to compile the correct command below, and you can choose whether
+ to let this script kick it off or execute them in another Terminal window yourself
+
+ Note that the build will take a long time. To follow the detailed build
+ log, you can tail the log file {{ logfile | default("<logfile>") }}.
+ confirm_each_command: false
+ remove_files:
+ - '{{ dist_file_path }}'
+ commands:
+ - !Command
+ cmd: git checkout {{ release_branch }}
+ tee: true
+ - !Command
+ cmd: git clean -df && git checkout -- .
+ comment: Make sure checkout is clean and up to date
+ logfile: git_clean.log
+ tee: true
+ - !Command
+ cmd: git pull
+ tee: true
+ - !Command
+ cmd: python3 -u dev-tools/scripts/buildAndPushRelease.py {{ local_keys }} --logfile {{ logfile }} --push-local "{{ dist_file_path }}" --rc-num {{ rc_number }} --sign {{ gpg.gpg_id | default("<gpg_key_id>", True) }}
+ comment: "NOTE: Remember to type your GPG pass-phrase at the prompt!"
+ logfile: build_rc.log
+ tee: true
+ - !Todo
+ id: smoke_tester
+ title: Run the smoke tester
+ depends: build_rc
+ vars:
+ dist_folder: lucene-solr-{{ release_version }}-RC{{ rc_number }}-rev{{ build_rc.git_rev | default("<git_rev>", True) }}
+ dist_path: '{{ [dist_file_path, dist_folder] | path_join }}'
+ tmp_dir: '{{ [rc_folder, ''smoketest''] | path_join }}'
+ local_keys: '{% if keys_downloaded %} --local-keys "{{ [config_path, ''KEYS''] | path_join }}"{% endif %}'
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: Here we'll smoke test the release by 'downloading' the artifacts, running the tests, validating GPG signatures etc.
+ remove_files:
+ - '{{ tmp_dir }}'
+ commands:
+ - !Command
+ cmd: python3 -u dev-tools/scripts/smokeTestRelease.py {{ local_keys }} --tmp-dir "{{ tmp_dir }}" file://{{ dist_path | expanduser }}
+ logfile: smoketest.log
+ - !Todo
+ id: import_svn
+ title: Import artifacts into SVN
+ description: |
+ Here we'll import the artifacts into Subversion.
+ depends: smoke_tester
+ vars:
+ dist_folder: lucene-solr-{{ release_version }}-RC{{ rc_number }}-rev{{ build_rc.git_rev | default("<git_rev>", True) }}
+ dist_path: '{{ [dist_file_path, dist_folder] | path_join }}'
+ dist_url: https://dist.apache.org/repos/dist/dev/lucene/{{ dist_folder}}
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: Have your Apache credentials handy, you'll be prompted for your password
+ commands:
+ - !Command
+ cmd: svn -m "Lucene/Solr {{ release_version }} RC{{ rc_number }}" import {{ dist_path }} {{ dist_url }}
+ logfile: import_svn.log
+ tee: true
+ - !Todo
+ id: verify_staged
+ title: Verify staged artifacts
+ description: |
+ A lightweight smoke testing which downloads the artifacts from stage
+ area and checks hash and signatures, but does not re-run all tests.
+ depends: import_svn
+ vars:
+ dist_folder: lucene-solr-{{ release_version }}-RC{{ rc_number }}-rev{{ build_rc.git_rev | default("<git_rev>", True) }}
+ dist_url: https://dist.apache.org/repos/dist/dev/lucene/{{ dist_folder}}
+ tmp_dir: '{{ [rc_folder, ''smoketest_staged''] | path_join }}'
+ local_keys: '{% if keys_downloaded %} --local-keys "{{ [config_path, ''KEYS''] | path_join }}"{% endif %}'
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: Here we'll verify that the staged artifacts are downloadable and hash/signatures match.
+ remove_files:
+ - '{{ tmp_dir }}'
+ commands:
+ - !Command
+ cmd: python3 -u dev-tools/scripts/smokeTestRelease.py {{ local_keys }} --download-only --tmp-dir "{{ tmp_dir }}" {{ dist_url }}
+ logfile: smoketest_staged.log
+- !TodoGroup
+ id: voting
+ title: Hold the vote and sum up the results
+ description: These are the steps necessary for the voting process. Read up on the link first!
+ is_in_rc_loop: true
+ todos:
+ - !Todo
+ id: initiate_vote
+ title: Initiate the vote
+ depends: verify_staged
+ description: |
+ If the smoke test passes against the staged artifacts, send an email to the dev mailing list announcing the release candidate.
+
+ .Mail template
+ ----
+ To: dev@lucene.apache.org
+ Subject: [VOTE] Release Lucene/Solr {{ release_version }} RC{{ rc_number }}
+
+ Please vote for release candidate {{ rc_number }} for Lucene/Solr {{ release_version }}
+
+ The artifacts can be downloaded from:
+ https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-{{ release_version }}-RC{{ rc_number }}-rev{{ build_rc.git_rev | default("<git_rev>", True) }}
+
+ You can run the smoke tester directly with this command:
+
+ python3 -u dev-tools/scripts/smokeTestRelease.py \
+ https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-{{ release_version }}-RC{{ rc_number }}-rev{{ build_rc.git_rev | default("<git_rev>", True) }}
+
+ The vote will be open for at least 3 working days, i.e. until {{ vote_close }}.
+
+ [ ] +1 approve
+ [ ] +0 no opinion
+ [ ] -1 disapprove (and reason why)
+
+ Here is my +1
+ ----
+ vars:
+ vote_close: '{{ vote_close_72h }}'
+ vote_close_epoch: '{{ vote_close_72h_epoch }}'
+ persist_vars:
+ - vote_close
+ - vote_close_epoch
+ function: create_ical
+ links:
+ - https://www.apache.org/foundation/voting.html
+ - !Todo
+ id: end_vote
+ title: End vote
+ depends: initiate_vote
+ description: |
+ At the end of the voting deadline, count the votes and send RESULT message to the mailing list.
+
+ {% set vote_close_epoch = initiate_vote.vote_close_epoch | int %}
+ {% if epoch < vote_close_epoch %}
+ WARNING: The planned voting deadline {{ initiate_vote.vote_close }} has not yet passed
+ {% else %}
+ The planned 72h voting deadline {{ initiate_vote.vote_close }} has passed.
+ {% endif %}
+ asciidoc: |
+ (( template=vote_macro ))
+ Note down how many votes were cast, summing as:
+
+ * Binding PMC-member +1 votes
+ * Non-binding +1 votes
+ * Neutral +/-0 votes
+ * Negative -1 votes
+
+ You need 3 binding +1 votes and more +1 than -1 votes for the release to happen.
+ A release cannot be vetoed, see more in provided links.
+
+ Here are some mail templates for successful and failed vote results with sample numbers:
+
+ {{ end_vote_result(3,1,0,2) }}
+
+ {{ end_vote_result(3,1,0,4) }}
+
+ {{ end_vote_result(2,9,0,0) }}
+ user_input:
+ - !UserInput
+ type: int
+ prompt: Number of binding +1 votes (PMC members)
+ name: plus_binding
+ - !UserInput
+ type: int
+ prompt: Number of other +1 votes
+ name: plus_other
+ - !UserInput
+ type: int
+ prompt: Number of 0 votes
+ name: zero
+ - !UserInput
+ type: int
+ prompt: Number of -1 votes
+ name: minus
+ post_description: |
+ (( template=vote_logic ))
+ (( template=vote_macro ))
+ {% if passed -%}
+ Congratulations! The vote has passed.
+
+ {% if minus > 0 %}
+ However, there were negative votes. A release cannot be vetoed, and as long as
+ there are more positive than negative votes you can techically release
+ the software. However, please review the negative votes and consider
+ a re-spin.
+
+ {% endif %}
+ {%- endif %}
+ {{ end_vote_result(plus_binding,plus_other,zero,minus) }}
+ links:
+ - https://www.apache.org/foundation/voting.html
+- !TodoGroup
+ id: publish
+ title: Publishing to the ASF Mirrors
+ description: Once the vote has passed, the release may be published to the ASF Mirrors and to Maven Central.
+ todos:
+ - !Todo
+ id: tag_release
+ title: Tag the release
+ description: Tag the release from the same revision from which the passing release candidate's was built
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: This will tag the release in git
+ logs_prefix: tag_release
+ commands:
+ - !Command
+ cmd: git tag -a releases/lucene-solr/{{ release_version }} -m "Lucene/Solr {{ release_version }} release" {{ build_rc.git_rev | default("<git_rev>", True) }}
+ logfile: git_tag.log
+ tee: true
+ - !Command
+ cmd: git push origin releases/lucene-solr/{{ release_version }}
+ logfile: git_push_tag.log
+ tee: true
+ - !Todo
+ id: rm_staged_mvn
+ title: Delete mvn artifacts from staging repo
+ vars:
+ dist_folder: lucene-solr-{{ release_version }}-RC{{ rc_number }}-rev{{ build_rc.git_rev | default("<git_rev>", True) }}
+ dist_path: '{{ [dist_file_path, dist_folder] | path_join }}'
+ dist_stage_url: https://dist.apache.org/repos/dist/dev/lucene/{{ dist_folder}}
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ confirm_each_command: false
+ commands_text: This will remove maven artifacts so they do not end up in the mirrors
+ commands:
+ - !Command
+ cmd: svn rm -m "Delete the lucene maven artifacts" {{ dist_stage_url }}/lucene/maven
+ logfile: svn_rm_mvn_lucene.log
+ tee: true
+ - !Command
+ cmd: svn rm -m "Delete the solr maven artifacts" {{ dist_stage_url }}/solr/maven
+ logfile: svn_rm_mvn_solr.log
+ tee: true
+ - !Todo
+ id: mv_to_release
+ title: Move release artifacts to release repo
+ vars:
+ dist_folder: lucene-solr-{{ release_version }}-RC{{ rc_number }}-rev{{ build_rc.git_rev | default("<git_rev>", True) }}
+ dist_stage_url: https://dist.apache.org/repos/dist/dev/lucene/{{ dist_folder}}
+ dist_release_url: https://dist.apache.org/repos/dist/release/lucene
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ confirm_each_command: false
+ commands_text: This will move the new release artifacts from staging repo to the release repo
+ commands:
+ - !Command
+ cmd: svn move -m "Move Lucene {{ release_version }} RC{{ rc_number }} to release repo" {{ dist_stage_url }}/lucene {{ dist_release_url }}/java/{{ release_version }}
+ logfile: svn_mv_lucene.log
+ tee: true
+ - !Command
+ cmd: svn move -m "Move Solr {{ release_version }} RC{{ rc_number }} to release repo" {{ dist_stage_url }}/solr {{ dist_release_url }}/solr/{{ release_version }}
+ logfile: svn_mv_solr.log
+ tee: true
+ - !Command
+ cmd: svn rm -m "Clean up the RC folder for {{ release_version }} RC{{ rc_number }}" https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-{{ release_version }}-RC{{ rc_number }}-rev{{ build_rc.git_rev | default("<git_rev>", True) }}
+ logfile: svn_rm_containing.log
+ comment: Clean up containing folder on the staging repo
+ tee: true
+ post_description: 'Note at this point you will see the Jenkins job "Lucene-Solr-SmokeRelease-master" begin to fail, until you run the "Generate Backcompat Indexes" '
+ - !Todo
+ id: publish_maven
+ title: Publish maven artifacts
+ vars:
+ dist_folder: lucene-solr-{{ release_version }}-RC{{ rc_number }}-rev{{ build_rc.git_rev | default("<git_rev>", True) }}
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ confirm_each_command: true
+ commands_text: In the source checkout do the following (note that this step will prompt you for your Apache LDAP credentials)
+ commands:
+ - !Command
+ cmd: ant clean stage-maven-artifacts -Dmaven.dist.dir={{ [dist_file_path, dist_folder, 'lucene', 'maven'] | path_join }} -Dm2.repository.id=apache.releases.https -Dm2.repository.url={{ m2_repository_url }}
+ logfile: publish_lucene_maven.log
+ - !Command
+ cmd: ant clean stage-maven-artifacts -Dmaven.dist.dir={{ [dist_file_path, dist_folder, 'solr', 'maven'] | path_join }} -Dm2.repository.id=apache.releases.https -Dm2.repository.url={{ m2_repository_url }}
+ logfile: publish_solr_maven.log
+ post_description: |
+ Once you have transferred all maven artifacts to repository.apache.org,
+ you will need to:
+
+ * Close the staging repository
+ . Log in to https://repository.apache.org/ with your ASF credentials
+ . Select "Staging Repositories" under "Build Promotion" from the navigation bar on the left
+ . Select the staging repository containing the Lucene artifacts
+ . Click on the "Close" button above the repository list, then enter a description when prompted, e.g. "Lucene/Solr {{ release_version }} RC{{ rc_number }}"
+ * The system will not spend some time validating the artifacts. Grab a coke and come back.
+ * Release the Lucene and/or Solr artifacts
+ . Wait and keep clicking refresh until the "Release" button becomes available
+ . Click on the "Release" button above the repository list, then enter a description when prompted, e.g. "Lucene/Solr {{ release_version }}".
+
+ Maven central should show the release after a short while, but you need to
+ wait 24 hours to give the Apache mirrors a chance to copy the new release.
+ links:
+ - https://wiki.apache.org/lucene-java/PublishMavenArtifacts
+ - https://repository.apache.org/index.html
+ - !Todo
+ id: check_mirroring
+ title: Check state of mirroring so far
+ description: Mark this as complete once a good spread is confirmed
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: Run this script to check the number and percentage of mirrors (and Maven Central) that have the release
+ commands:
+ - !Command
+ cmd: python3 -u dev-tools/scripts/poll-mirrors.py -version {{ release_version }} -o
+ live: true
+- !TodoGroup
+ id: website
+ title: Update the website
+ description: |
+ For every release we publish docs on the website, we need to update the
+ download pages etc.
+ todos:
+ - !Todo
+ id: website_expaths
+ title: Update extpaths.txt
+ description: |
+ The file extpaths.txt lists paths in the svn production tree, relative to the project website's root directory,
+ that are allowed to be out of sync with the staging tree. We need to update this in order to push generated
+ javadocs directly to production SVN and thus avoid breaking the CMS staging capability.
+ links:
+ - http://www.apache.org/dev/cmsref
+ - http://www.apache.org/dev/cmsref#extpaths
+ - http://www.apache.org/dev/cmsref#generated-docs
+ commands: !Commands
+ root_folder: '{{ release_folder }}'
+ confirm_each_command: true
+ commands:
+ - !Command
+ cmd: svn co https://svn.apache.org/repos/asf/lucene/cms/trunk website-source
+ logfile: svn-checkout-website.log
+ tee: true
+ - !Command
+ cmd: echo core/{{ release_version_underscore }}
+ cwd: website-source/content
+ redirect: extpaths.txt
+ redirect_append: true
+ comment: Add Lucene javadocs dir to extpaths
+ - !Command
+ cmd: echo solr/{{ release_version_underscore }}
+ cwd: website-source/content
+ redirect: extpaths.txt
+ redirect_append: true
+ comment: Add Solr javadocs dir to extpaths
+ - !Command
+ cmd: svn commit -m "Update CMS production sync exceptions for {{ release_version_underscore }} javadocs" extpaths.txt
+ cwd: website-source/content
+ logfile: svn_commit_website.log
+ tee: true
+ - !Todo
+ id: website_docs
+ title: Publish docs, changes and javadocs
+ description: |
+ Ensure your refridgerator has at least 2 beers - the svn import operation can take a while,
+ depending on your upload bandwidth. We'll publish this directly to the production tree.
+ At the end of the task, the two links below shall work.
+ links:
+ - http://lucene.apache.org/core/{{ version }}
+ - http://lucene.apache.org/solr/{{ version }}
+ vars:
+ release_tag: releases/lucene-solr/{{ release_version }}
+ version: "{{ release_version_major }}_{{ release_version_minor }}_{{ release_version_bugfix }}"
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: Check out the CMS content folder
+ commands:
+ - !Command
+ cmd: git fetch && git checkout {{ release_tag }}
+ comment: Checkout the release branch
+ logfile: checkout-release-tag.log
+ tee: true
+ - !Command
+ cmd: ant documentation -Dversion={{ release_version }}
+ comment: Build documentation
+ - !Command
+ cmd: svn -m "Add docs, changes and javadocs for Lucene {{ release_version }}" import {{ git_checkout_folder }}/lucene/build/docs https://svn.apache.org/repos/infra/websites/production/lucene/content/core/{{ version }}
+ logfile: add-docs-lucene.log
+ comment: Add docs for Lucene
+ - !Command
+ cmd: svn -m "Add docs, changes and javadocs for Solr {{ release_version }}" import {{ git_checkout_folder }}/solr/build/docs https://svn.apache.org/repos/infra/websites/production/lucene/content/solr/{{ version }}
+ logfile: add-docs-solr.log
+ comment: Add docs for Solr
+ - !Todo
+ id: website_javadoc_redirect
+ title: Update redirect to latest Javadoc
+ depends: website_expaths
+ vars:
+ release_tag: releases/lucene-solr/{{ release_version }}
+ version: "{{ release_version_major }}_{{ release_version_minor }}_{{ release_version_bugfix }}"
+ description: |
+ We make it possible to link to latest javadoc by providing redirect links for e.g.
+ http://lucene.apache.org/solr/api/solr-core/ which will auto redirect to whatever
+ is the latest released version, i.e. http://lucene.apache.org/solr/{{ version }}/solr-core/<version>.
+ This is handled in `.htaccess`
+
+ *NOT YET IMPLEMENTED*
+ function: website_javadoc_redirect
+ asciidoc: |
+ We make it possible to link to latest javadoc by providing redirect links for e.g.
+ http://lucene.apache.org/solr/api/solr-core/ which will auto redirect to whatever
+ is the latest released version, i.e. http://lucene.apache.org/solr/{{ version }}/solr-core/<version>.
+
+ If we are releasing the latest version, this task will offer to patch the `.htaccess`
+ file with latest redirect info and commit this file back to website svn.
+ post_description: After the task is done, please test that the links below redirect to the correct version
+ links:
+ - http://lucene.apache.org/solr/api/solr-core/
+ - http://lucene.apache.org/core/api/core/
+ - !Todo
+ id: prepare_highlights
+ title: Edit the release highlights for Lucene and Solr
+ description: |
+ You will edit the release highlights for Lucene and Solr.
+ This will be done in two separate files, one highlight per line,
+ each line starting with a '*'.
+ function: prepare_highlights
+ commands: !Commands
+ root_folder: '{{ release_folder }}'
+ commands_text: |
+ Edit the highlights files, one highlight per line.
+ You have to exit the editor after edit to continue.
+ commands:
+ - !Command
+ cmd: "{{ editor }} {{ lucene_highlights_file }}"
+ comment: Edit the Lucene highlights
+ - !Command
+ cmd: "{{ editor }} {{ solr_highlights_file }}"
+ comment: Edit the Solr highlights
+ - !Todo
+ id: prepare_announce
+ title: Edit the drafts for the website news
+ description: |
+ Edit draft news texts for the TLP site and for Lucene and Solr sites.
+ These texts will also be used as a starting point for the release email later.
+ function: prepare_announce
+ commands: !Commands
+ root_folder: '{{ release_folder }}'
+ commands_text: |
+ Proof read the draft announcements and edit as you see fit.
+ You have to exit the editor after edit to continue.
+ commands:
+ - !Command
+ cmd: "{{ editor }} {{ tlp_news_draft }}"
+ comment: Edit the draft for TLP news announcement
+ - !Command
+ cmd: "{{ editor }} {{ lucene_news_draft }}"
+ comment: Edit the draft for Lucene announcement
+ - !Command
+ cmd: "{{ editor }} {{ solr_news_draft }}"
+ comment: Edit the draft for Solr announcement
+ - !Todo
+ id: update_news
+ title: Publish news to the web site
+ depends: prepare_announce
+ description: |
+ Update the news sections of the TLP, Lucene and Solr sites
+ vars:
+ website_source: "{{ [release_folder, 'website-source'] | path_join }}"
+ function: update_news
+ commands: !Commands
+ root_folder: '{{ website_source }}'
+ commands_text: |
+ Add news to the site. We'll first add the {{ release_version }} text to the file,
+ then ask you to edit the file to verify the end result. You will now only edit
+ the files, we'll commit all changes in a later step.
+
+ You have to exit the editor after edit to continue.
+ commands:
+ - !Command
+ cmd: "{{ editor }} {{ tlp_news_file }}"
+ comment: Edit the complete website file for TLP news
+ - !Command
+ cmd: "{{ editor }} {{ lucene_news_file }}"
+ comment: Edit the complete website file for Lucene news
+ - !Command
+ cmd: "{{ editor }} {{ solr_news_file }}"
+ comment: Edit the complete website file for Solr news
+ post_description: |
+ You will review and commit all changes later
+ - !Todo
+ id: update_download_page
+ title: Update the download pages
+ depends: prepare_announce
+ description: |
+ Update the download pages
+ vars:
+ website_source: "{{ [release_folder, 'website-source'] | path_join }}"
+ commands: !Commands
+ root_folder: '{{ website_source }}'
+ commands_text: |
+ Update download pages to point to the latest release.
+ You have to exit the editor after edit to continue.
+ commands:
+ - !Command
+ cmd: "{{ editor }} content/solr/downloads.mdtext"
+ comment: Edit the Solr downloads page
+ stdout: true
+ - !Command
+ cmd: "{{ editor }} content/core/downloads.mdtext"
+ comment: Edit the Lucene downloads page
+ stdout: true
+ post_description: |
+ You will review and commit all changes later
+ links:
+ - https://www.apache.org/dev/release-download-pages.html
+ - !Todo
+ id: update_other
+ title: Update rest of webpage
+ depends: prepare_announce
+ description: |
+ Update the rest of the web page. Please review all files in the checkout
+ and consider if any need change based on what changes there are in the
+ release you are doing. Things to consider:
+
+ * System requirements
+ * Quickstart and tutorial?
+ vars:
+ website_source: "{{ [release_folder, 'website-source'] | path_join }}"
+ commands: !Commands
+ root_folder: '{{ website_source }}'
+ commands_text: |
+ We'll open an editor on the root folder of the site checkout
+ You have to exit the editor after edit to continue.
+ commands:
+ - !Command
+ cmd: "{{ editor }} ."
+ comment: Open an editor on the root folder
+ stdout: true
+ - !Todo
+ id: publish_website
+ title: Publish the website changes
+ depends:
+ - update_news
+ - update_download_page
+ description: |
+ Publish the website changes
+ vars:
+ website_source: "{{ [release_folder, 'website-source'] | path_join }}"
+ commands: !Commands
+ root_folder: '{{ website_source }}'
+ commands_text: |
+ Verify that changes look good, and then publish.
+ You have to exit the editor after review to continue.
+ commands:
+ - !Command
+ cmd: svn st
+ stdout: true
+ - !Command
+ cmd: svn diff
+ redirect: "{{ [release_folder, 'website.diff'] | path_join }}"
+ comment: Make a diff of all edits. Will open in next step
+ - !Command
+ cmd: "{{ editor }} {{ [release_folder, 'website.diff'] | path_join }}"
+ comment: View the diff of the website changes
+ stdout: true
+ - !Command
+ cmd: svn commit -m "Update website for {{ release_version }} release"
+ comment: Commit changes if all looks good
+ logfile: commit-website.log
+ post_description: |
+ * Go to https://ci.apache.org/builders/lucene-site-staging and see that site is built
+ * Open http://lucene.staging.apache.org and see that everything looks fine
+ * Publish the site at https://cms.apache.org/lucene/publish
+
+ Wait for these changes to appear on both of Apache's main webservers
+ (US: http://lucene.us.apache.org, EU: http://lucene.eu.apache.org, http://lucene.apache.org
+ is dependent on your own geographic location, so the other mirror may still be outdated)
+ before doing the next steps (see http://www.apache.org/dev/project-site.html for details on
+ how the site is mirrored to Apache's main web servers). Once they appear, verify all links
+ are correct in your changes!
+ links:
+ - https://ci.apache.org/builders/lucene-site-staging
+ - http://lucene.staging.apache.org
+ - https://svn.apache.org/repos/asf/lucene/cms/trunk/
+ - https://cms.apache.org/lucene/publish
+ - !Todo
+ id: update_doap
+ title: Update the DOAP files
+ description: |
+ Update the Core & Solr DOAP RDF files on the unstable, stable and release branches to
+ reflect the new versions (note that the website .htaccess file redirects from their
+ canonical URLs to their locations in the Lucene/Solr Git source repository - see
+ dev-tools/doap/README.txt for more info)
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: Edit DOAP files
+ commands:
+ - !Command
+ cmd: git checkout master && git pull
+ stdout: true
+ comment: Goto master branch
+ - !Command
+ cmd: "{{ editor }} dev-tools/doap/lucene.rdf"
+ comment: Edit Lucene DOAP, add version {{ release_version }}
+ stdout: true
+ - !Command
+ cmd: "{{ editor }} dev-tools/doap/solr.rdf"
+ comment: Edit Solr DOAP, add version {{ release_version }}
+ stdout: true
+ - !Command
+ cmd: git add dev-tools/doap/lucene.rdf dev-tools/doap/solr.rdf && git commit -m "DOAP changes for release {{ release_version }}"
+ logfile: commit.log
+ stdout: true
+ - !Command
+ cmd: git push origin
+ logfile: push.log
+ stdout: true
+- !TodoGroup
+ id: announce
+ title: Announce the release
+ description: |
+ For feature releases, your announcement should describe the main features included
+ in the release. *Send the announce as Plain-text email, not HTML.*
+
+ NOTE: Copy-pasting from the release notes might sometimes make cause it to be sent
+ as HTML formatted, which can break at the mailing list bot.
+ Better to copy-paste the raw/edit text.
+ todos:
+ - !Todo
+ id: announce_lucene
+ title: Announce the Lucene release (@l.a.o)
+ description: |
+ (( template=announce_lucene_mail ))
+ - !Todo
+ id: announce_solr
+ title: Announce the Solr release (@l.a.o)
+ description: |
+ (( template=announce_solr_mail ))
+ - !Todo
+ id: setup_pgp_mail
+ title: Setup your mail client for PGP
+ description: |
+ The announce mail to `announce@apache.org` should be cryptographically signed.
+ Make sure you have a PGP enabled email client with your apache ket installed.
+ There are plugins for popular email programs, as well as browser plugins for webmail.
+ See links for help on how to setup your email client for PGP.
+ links:
+ - https://www.openpgp.org/software/
+ - https://ssd.eff.org/en/module/how-use-pgp-mac-os-x
+ - https://ssd.eff.org/en/module/how-use-pgp-linux
+ - https://ssd.eff.org/en/module/how-use-pgp-windows
+ - https://www.openpgp.org/software/mailvelope/
+ - !Todo
+ id: announce_lucene_sig
+ title: Announce the Lucene release (announce@a.o)
+ description: |
+ (( template=announce_solr_sign_mail ))
+ - !Todo
+ id: announce_solr_sig
+ title: Announce the Solr release (announce@a.o)
+ description: |
+ (( template=announce_solr_sign_mail ))
+ - !Todo
+ id: add_to_wikipedia
+ title: Add the new version to Wikipedia
+ description: |
+ Go to Wikipedia and edit the page to include the new release.
+ Major versions should have a small new paragraph under 'History'.
+ If you know other languages than English, edit those as well.
+ links:
+ - https://en.wikipedia.org/wiki/Apache_Lucene
+ - https://en.wikipedia.org/wiki/Apache_Solr
+- !TodoGroup
+ id: post_release
+ title: Tasks to do after release
+ description: There are many more tasks to do now that the new version is out there, so hang in there for a few more hours :)
+ todos:
+ - !Todo
+ id: add_version_bugfix
+ title: Add a new bugfix version to stable and unstable branches
+ types:
+ - bugfix
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: |
+ Update versions on master and stable branch.
+ You may have to hand-edit some files before commit, so go slowly :)
+ confirm_each_command: true
+ commands:
+ - !Command
+ cmd: git checkout master && git pull && git clean -df && git checkout -- .
+ comment: Go to master branch
+ logfile: checkout-master.log
+ - !Command
+ cmd: python3 -u dev-tools/scripts/addVersion.py {{ release_version }}
+ logfile: addversion-master.log
+ - !Command
+ cmd: git diff
+ logfile: diff-master.log
+ tee: true
+ - !Command
+ cmd: git add -u . && git commit -m "Add bugfix version {{ release_version }}" && git push
+ logfile: commit-master.log
+ - !Command
+ cmd: git checkout {{ stable_branch }} && git pull && git clean -df && git checkout -- .
+ logfile: checkout-stable.log
+ comment: Now the same for the stable branch
+ - !Command
+ cmd: python3 -u dev-tools/scripts/addVersion.py {{ release_version }}
+ logfile: addversion-stable.log
+ - !Command
+ cmd: git diff
+ logfile: diff-stable.log
+ tee: true
+ - !Command
+ cmd: git add -u . && git commit -m "Add bugfix version {{ release_version }}" && git push
+ logfile: commit-stable.log
+ - !Todo
+ id: synchronize_changes
+ title: Synchronize CHANGES.txt
+ description: |
+ Copy the CHANGES.txt section for this release back to the stable and unstable branches'
+ CHANGES.txt files, removing any duplicate entries, but only from sections for as-yet
+ unreleased versions; leave intact duplicate entries for already-released versions.
+
+ There is a script to generate a regex that will match JIRAs fixed in a release:
+ `releasedJirasRegex.py`. The following examples will print regexes matching all JIRAs
+ fixed in {{ release_version }}, which can then be used to find duplicates in unreleased
+ version sections of the corresponding CHANGES.txt files.
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: Synchronize CHANGES.txt
+ commands:
+ - !Command
+ cmd: git checkout {{ release_branch }}
+ comment: Go to release branch
+ logfile: checkout-release.log
+ stdout: true
+ - !Command
+ cmd: python3 -u -B dev-tools/scripts/releasedJirasRegex.py {{ release_version }} lucene/CHANGES.txt && python3 -u -B dev-tools/scripts/releasedJirasRegex.py {{ release_version }} solr/CHANGES.txt
+ tee: true
+ comment: Find version regexes
+ - !Command
+ cmd: git checkout master && git pull && git clean -df && git checkout -- .
+ comment: Go to master branch
+ logfile: checkout-master.log
+ - !Command
+ cmd: "{{ editor }} solr/CHANGES.txt"
+ comment: Edit Solr CHANGES, do necessary changes
+ stdout: true
+ - !Command
+ cmd: "{{ editor }} lucene/CHANGES.txt"
+ comment: Edit Lucene CHANGES, do necessary changes
+ stdout: true
+ - !Command
+ cmd: git add -u . && git commit -m "Sync CHANGES for {{ release_version }}" && git push
+ logfile: commit-master.log
+ - !Command
+ cmd: git checkout {{ stable_branch }} && git pull && git clean -df && git checkout -- .
+ comment: Go to stable branch
+ logfile: checkout-stable.log
+ - !Command
+ cmd: "{{ editor }} solr/CHANGES.txt"
+ comment: Edit Solr CHANGES, do necessary changes
+ stdout: true
+ - !Command
+ cmd: "{{ editor }} lucene/CHANGES.txt"
+ comment: Edit Lucene CHANGES, do necessary changes
+ stdout: true
+ - !Command
+ cmd: git add -u . && git commit -m "Sync CHANGES for {{ release_version }}" && git push
+ logfile: commit-stable.log
+ - !Todo
+ id: increment_release_version
+ title: Add the next version on release branch
+ description: Add the next version after the just-released version on the release branch
+ depends: publish_maven
+ vars:
+ next_version: "{{ release_version_major }}.{{ release_version_minor }}.{{ release_version_bugfix + 1 }}"
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: Run these commands to add the new bugfix version {{ next_version }} to the release branch
+ commands:
+ - !Command
+ cmd: git checkout {{ release_branch }}
+ tee: true
+ - !Command
+ cmd: python3 -u dev-tools/scripts/addVersion.py {{ next_version }}
+ tee: true
+ - !Command
+ cmd: git diff
+ logfile: diff.log
+ comment: Check the git diff before committing. Do any edits if necessary
+ tee: true
+ - !Command
+ cmd: git add -u . && git commit -m "Add next bugfix version {{ next_version }}" && git push
+ logfile: commit-stable.log
+ - !Todo
+ id: backcompat_release
+ title: Generate Backcompat Indexes for release branch
+ description: |
+ After each version of Lucene is released, compressed CFS, non-CFS, and sorted indexes created with
+ the newly released version are added to `lucene/backwards-codecs/src/test/org/apache/lucene/index/`,
+ for use in testing backward index compatibility via org.apache.lucene.index.TestBackwardsCompatibility,
+ which is also located under the `backwards-codecs/` module. There are also three indexes created only
+ with major Lucene versions: moreterms, empty, and dvupdates. These indexes are created via methods
+ on `TestBackwardsCompatibility` itself - see comments in the source for more information.
+
+ There is a script (`dev-tools/scripts/addBackcompatIndexes.py`) that automates most of the process.
+ It downloads the source for the specified release; generates indexes for the current release using
+ `TestBackwardsCompatibility`; compresses the indexes and places them in the correct place in the source
+ tree; modifies TestBackwardsCompatibility.java to include the generated indexes in the list of indexes
+ to test; and then runs `TestBackwardsCompatibility`.
+
+ In this and the next two steps we'll guide you through using this tool on each of the branches.
+ depends:
+ - increment_release_version
+ vars:
+ temp_dir: "{{ [release_folder, 'backcompat'] | path_join }}"
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: Run these commands to add back-compat indices to release branch
+ commands:
+ - !Command
+ cmd: git checkout {{ release_branch }} && git pull && git clean -df && git checkout -- .
+ tee: true
+ logfile: checkout.log
+ - !Command
+ cmd: ant clean
+ - !Command
+ cmd: python3 -u dev-tools/scripts/addBackcompatIndexes.py --no-cleanup --temp-dir {{ temp_dir }} {{ release_version }} && git add lucene/backward-codecs/src/test/org/apache/lucene/index/
+ logfile: add-bakccompat.log
+ - !Command
+ cmd: git diff
+ comment: Check the git diff before committing
+ tee: true
+ - !Command
+ cmd: git add -u . && git commit -m "Add back-compat indices for {{ release_version }}" && git push
+ logfile: commit.log
+ - !Todo
+ id: backcompat_stable
+ title: Generate Backcompat Indexes for stable branch
+ description: |
+ Now generate back-compat for stable branch ({{ stable_branch }})
+ depends:
+ - increment_release_version
+ vars:
+ temp_dir: "{{ [release_folder, 'backcompat'] | path_join }}"
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: Run these commands to add back-compat indices to {{ stable_branch }}
+ commands:
+ - !Command
+ cmd: git checkout {{ stable_branch }} && git pull && git clean -df && git checkout -- .
+ tee: true
+ logfile: checkout.log
+ - !Command
+ cmd: ant clean
+ - !Command
+ cmd: python3 -u dev-tools/scripts/addBackcompatIndexes.py --no-cleanup --temp-dir {{ temp_dir }} {{ release_version }} && git add lucene/backward-codecs/src/test/org/apache/lucene/index/
+ logfile: add-bakccompat.log
+ - !Command
+ cmd: git diff
+ comment: Check the git diff before committing
+ tee: true
+ - !Command
+ cmd: git add -u . && git commit -m "Add back-compat indices for {{ release_version }}" && git push
+ logfile: commit.log
+ - !Todo
+ id: backcompat_master
+ title: Generate Backcompat Indexes for unstable branch
+ description: |
+ Now generate back-compat for unstable (master) branch.
+ Note that this time we do not specify `--no-cleanup` meaning the tmp folder will be deleted
+ depends:
+ - increment_release_version
+ vars:
+ temp_dir: "{{ [release_folder, 'backcompat'] | path_join }}"
+ version: "{{ set_java_home(master_version) }}"
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: Run these commands to add back-compat indices to master
+ commands:
+ - !Command
+ cmd: git checkout master && git pull && git clean -df && git checkout -- .
+ tee: true
+ logfile: checkout.log
+ - !Command
+ cmd: ant clean
+ - !Command
+ cmd: python3 -u dev-tools/scripts/addBackcompatIndexes.py --temp-dir {{ temp_dir }} {{ release_version }} && git add lucene/backward-codecs/src/test/org/apache/lucene/index/
+ logfile: add-bakccompat.log
+ - !Command
+ cmd: git diff
+ comment: Check the git diff before committing
+ tee: true
+ - !Command
+ cmd: git add -u . && git commit -m "Add back-compat indices for {{ release_version }}" && git push
+ logfile: commit.log
+ post_description: |
+ When doing a major version release, eg. 8.0.0, you might also need to reenable some
+ backward compatibility tests for corner cases. To find them, run grep -r assume
+ lucene/backward-codecs/, which should find tests that have been disabled on master
+ because there was no released Lucene version to test against.
+ {{ set_java_home(release_version) }}
+ - !Todo
+ id: jira_release
+ title: Mark version as released in JIRA
+ description: |-
+ Go to the JIRA "Manage Versions" Administration pages.
+
+ . Next to version {{ release_version }}, click the gear pop-up menu icon and choose "Release"
+ . Fill in the release date ({{ release_date | formatdate }})
+ . It will give the option of transitioning issues marked fix-for the released version to the
+ next version, but do not do this as it will send an email for each issue :)
+
+ This needs to be done both for Lucene and Solr JIRAs, see links.
+ links:
+ - https://issues.apache.org/jira/plugins/servlet/project-config/LUCENE/versions
+ - https://issues.apache.org/jira/plugins/servlet/project-config/SOLR/versions
+ - !Todo
+ id: jira_close_resolved
+ title: Close all issues resolved in the release
+ description: |-
+ Go to JIRA search in both Solr and Lucene and find all issues that were fixed in the release
+ you just made, whose Status is Resolved.
+
+ . Go to https://issues.apache.org/jira/issues/?jql=project+in+(LUCENE,SOLR)+AND+status=Resolved+AND+fixVersion={{ release_version }}
+ . Do a bulk change (Under Tools... menu) to close all of these issues. This is a workflow transition task
+ . In the 'Comment' box type `Closing after the {{ release_version }} release`
+ . *Uncheck* the box that says `Send mail for this update`
+
+ This needs to be done both for Lucene and Solr JIRAs, see links.
+ links:
+ - https://issues.apache.org/jira/issues/?jql=project+in+(LUCENE,SOLR)+AND+status=Resolved+AND+fixVersion={{ release_version }}
+ - !Todo
+ id: jira_change_unresolved
+ title: Remove fixVersion for unresolved
+ description: |-
+ Do another JIRA search to find all issues with Resolution=_Unresolved_ and fixVersion=_{{ release_version }}_.
+ You need to do this separately for Lucene and Solr.
+
+ *Lucene*
+
+ . Open https://issues.apache.org/jira/issues/?jql=project+=+LUCENE+AND+resolution=Unresolved+AND+fixVersion={{ release_version }}
+ . In the `Tools` menu, start a bulk change - operation="Edit issues"
+ . Identify issues that *are included* in the release, but are unresolved e.g. due to being REOPENED. These shall *not* be bulk changed!
+ . Check the box next to `Change Fix Version/s` and in the dropdown `Find and remove these`, selecting v {{ release_version }}
+ . On the bottom of the form, uncheck the box that says `Send mail for this update`
+ . Click `Next`, review the changes and click `Confirm`
+
+ *Solr*
+
+ . Open https://issues.apache.org/jira/issues/?jql=project+=+LUCENE+AND+resolution=Unresolved+AND+fixVersion={{ release_version }}
+ . In the `Tools` menu, start a bulk change - operation="Edit issues"
+ . Identify issues that *are included* in the release, but are unresolved e.g. due to being REOPENED. These shall *not* be bulk changed!
+ . Check the box next to `Change Fix Version/s` and in the dropdown `Find and remove these`, selecting v {{ release_version }}
+ . On the bottom of the form, uncheck the box that says `Send mail for this update`
+ . Click `Next`, review the changes and click `Confirm`
+ links:
+ - https://issues.apache.org/jira/issues/?jql=project+=+LUCENE+AND+resolution=Unresolved+AND+fixVersion={{ release_version }}
+ - https://issues.apache.org/jira/issues/?jql=project+=+LUCENE+AND+resolution=Unresolved+AND+fixVersion={{ release_version }}
+ - !Todo
+ id: jira_clear_security
+ title: Clear Security Level of Public Solr JIRA Issues
+ description: |-
+ ASF JIRA has a deficiency in which issues that have a security level of "Public" are nonetheless not searchable.
+ As a maintenance task, we'll clear the security flag for all public Solr JIRAs, even if it is not a task directly
+ related to the release:
+
+ . Open in browser: https://issues.apache.org/jira/issues/?jql=project+=+SOLR+AND+level+=+%22Public%22
+ . In the `Tools` menu, start a bulk change, select all issues and click `Next`
+ . Select operation="Edit issues" and click `Next`
+ . Click the checkbox next to `Change security level` and choose `None` in the dropdown.
+ . On the bottom of the form, uncheck the box that says `Send mail for this update`
+ . Click `Next`, review the changes and click `Confirm`
+ links:
+ - https://issues.apache.org/jira/issues/?jql=project+=+SOLR+AND+level+=+%22Public%22
+ - !Todo
+ id: new_jira_versions_bugfix
+ title: Add a new version in JIRA for the next release
+ description: |-
+ Go to the JIRA "Manage Versions" Administration pages and add the new version:
+
+ . Create a new (unreleased) version `{{ get_next_version }}`
+
+ This needs to be done both for Lucene and Solr JIRAs, see links.
+ types:
+ - bugfix
+ links:
+ - https://issues.apache.org/jira/plugins/servlet/project-config/LUCENE/versions
+ - https://issues.apache.org/jira/plugins/servlet/project-config/SOLR/versions
+ - !Todo
+ id: stop_mirroring
+ title: Stop mirroring old releases
+ description: |
+ Shortly after new releases are first mirrored, they are automatically copied to the archives.
+ Only the latest point release from each active branch should be kept under the Lucene PMC
+ svnpubsub areas `dist/releases/lucene/` and `dist/releases/solr/`. Older releases can be
+ safely deleted, since they are already backed up in the archives.
+
+ Currenlty these versions are in the mirrors:
+
+ *{{ mirrored_versions|join(', ') }}*
+
+ The commands below will remove old versions automatically. If this suggestion is wrong,
+ please do *not* execute the commands automatically, but edit the command and run manually.
+ Versions to be deleted from the mirrors are:
+
+ *{{ mirrored_versions_to_delete|join(', ') }}*
+
+ commands: !Commands
+ root_folder: '{{ git_checkout_folder }}'
+ commands_text: |
+ Run these commands to delete proposed versions from mirrors.
+
+ WARNING: Validate that the proposal is correct!
+ commands:
+ - !Command
+ cmd: |
+ svn rm -m "Stop mirroring old Lucene releases"{% for ver in mirrored_versions_to_delete %} https://dist.apache.org/repos/dist/release/lucene/java/{{ ver }}{% endfor %}
+ logfile: svn-rm-lucene.log
+ - !Command
+ cmd: |
+ svn rm -m "Stop mirroring old Solr releases"{% for ver in mirrored_versions_to_delete %} https://dist.apache.org/repos/dist/release/lucene/solr/{{ ver }}{% endfor %}
+ logfile: svn-rm-lucene.log
+ - !Todo
+ id: update_wiki
+ title: Update old WIKI
+ description: |
+ The Solr WIKI has a page for every version which is often linked to from WIKI pages to
+ indicate differences between versions, example: http://wiki.apache.org/solr/Solr4.3.
+
+ Do the following:
+
+ . Update the page for the released version with release date and link to release statement
+ . Create a new placeholder page for the "next" version, if it does not exist
diff --git a/dev-tools/scripts/releasedJirasRegex.py b/dev-tools/scripts/releasedJirasRegex.py
old mode 100644
new mode 100755
index dc5e1b3..7de0b05
--- a/dev-tools/scripts/releasedJirasRegex.py
+++ b/dev-tools/scripts/releasedJirasRegex.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
diff --git a/dev-tools/scripts/requirements.txt b/dev-tools/scripts/requirements.txt
index a65679d..b8a124b 100644
--- a/dev-tools/scripts/requirements.txt
+++ b/dev-tools/scripts/requirements.txt
@@ -1,2 +1,8 @@
+six>=1.11.0
+Jinja2>=2.10.1
+PyYAML>=5.1
+holidays>=0.9.10
+ics>=0.4
+console-menu>=0.5.1
PyGithub
jira
\ No newline at end of file
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index d3e1883..2216d69 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -130,6 +130,8 @@ Other
* LUCENE-8861: Script to find open Github PRs that needs attention (janhoy)
+* LUCENE-8852: ReleaseWizard tool for release managers (janhoy)
+
======================= Lucene 8.1.1 =======================
(No Changes)