Month: October 2019

How can I test a release process?

I’m improving the scripting of the Apache Subversion release process. Preparing and publishing a release involves things like making a branch in the source repository, updating web pages, updating buildbot configurations, and (not yet automated) sending emails.

When I’m modifying the release scripts, I needed a way to test without making bogus “live” releases.

The source code, web pages, and buildbot config are all stored in Subversion repositories, so that was a good place to start.

Where I’ve got to so far:

• scripted the creation of a set of local repositories that contain a minimum viable contents that the script expects to find;
• pass the release.py script a set of alternative URLs for the various repositories it is going to read and modify;
• manually inspect what the script changed in those repositories.

My initial script to set up local dummy repos is ugly and slow. But that’s OK; the important thing is it’s enough to proceed with verifying the automation while I improve it.

#!/bin/bash
# Make a local svn repo suitable for testing our release procedures
# ('release.py' etc.)

set -e

SRC_SVN_REPO_URL=https://svn.apache.org/repos/asf
SRC_DIST_REPO_URL=https://dist.apache.org/repos/dist
SRC_TRUNK_WC=$HOME/src/subversion-c
SRC_BRANCHES_WC=$HOME/src/svn/branches
SRC_SITE_WC=$HOME/src/svn/site

DUMMY_REPOS_DIR=/opt/svn/dummy-asf-repos
DUMMY_DIST_REPO_DIR=$DUMMY_REPOS_DIR/dist-repo
DUMMY_DIST_REPO_URL=file://$DUMMY_DIST_REPO_DIR
DUMMY_SVN_REPO_DIR=$DUMMY_REPOS_DIR/svn-repo
DUMMY_SVN_REPO_URL=file://$DUMMY_SVN_REPO_DIR
SVN_WC_DIR=$DUMMY_REPOS_DIR/svn-wc
#DIST_WC_DIR=$DUMMY_REPOS_DIR/dist-wc

# export_with_props SRC_WC DST_WC
function export_with_props() {
  SRC_WC="$1"
  DST_WC="$2"
  svn export $SRC_WC $DST_WC
  svn add --force --no-auto-props $DST_WC
  # remove automatically added mime-type props as some of them are not an exact replica
  svn pd -R svn:mime-type $DST_WC
  # add an exact replica of source props
  PROPS_TO_ADD=$PWD/props-to-add
  (cd $SRC_WC && svn diff --properties-only --old=^/@0 --new=.) > $PROPS_TO_ADD
  (cd $DST_WC && svn patch $PROPS_TO_ADD)
  # rm props-to-del props-to-add
}

set -x

mkdir "$DUMMY_REPOS_DIR"


### the 'dist.a.o' repo ###

if ! [ -d $DUMMY_DIST_REPO_DIR ]; then

  # create a repo
  svnadmin create $DUMMY_DIST_REPO_DIR

  # create skeleton dirs
  svn -m "Init" mkdir --parents $DUMMY_DIST_REPO_URL/{dev,release}/subversion

fi


### the 'svn.a.o' repo

if ! [ -d $DUMMY_SVN_REPO_DIR ]; then

  # create a repo
  svnadmin create $DUMMY_SVN_REPO_DIR

  # create skeleton dirs
  svn -m "Init" mkdir --parents $DUMMY_SVN_REPO_URL/subversion/{trunk,tags,branches,site}

  # check out
  svn co $DUMMY_SVN_REPO_URL $SVN_WC_DIR
  cd $SVN_WC_DIR

  # populate trunk
  rmdir subversion/trunk
  export_with_props $SRC_TRUNK_WC subversion/trunk
  svn revert -R subversion/trunk/{contrib,notes}/*
  svn ci -m "Add trunk" subversion/trunk

  # populate site
  export_with_props $SRC_SITE_WC/tools subversion/site/tools
  export_with_props $SRC_SITE_WC/publish subversion/site/publish
  svn revert -R subversion/site/publish/docs/{api,javahl}/*
  svn ci -m "Add site" subversion/site
  svn cp -m "Add site staging branch" ^/subversion/site/publish ^/subversion/site/staging

  # a mod on trunk
  echo hello > subversion/trunk/hello.txt
  svn add subversion/trunk/hello.txt
  svn ci -m "Trunk mod." subversion/trunk/hello.txt

  # make 1.13.x branch
  svn cp -m "Branch 1.13.x" ^/subversion/trunk ^/subversion/branches/1.13.x
  svn up --depth=immediates subversion/branches/1.13.x/
  sed 's/1\.12\.[0-9]*/1.13.0/' < $SRC_BRANCHES_WC/1.12.x/STATUS > subversion/branches/1.13.x/STATUS
  svn add subversion/branches/1.13.x/STATUS
  svn ci -m "Add 'STATUS' file."

fi

Make the dummy repos:

$ cd /opt/svn
$ rm -rf dummy-asf-repos/ && svn-mk-dummy-asf-repo.sh
[...]

Tell ‘release.py’ where the dummy repositories are:

$ export SVN_RELEASE_SVN_REPOS=file:///opt/svn/dummy-asf-repos/svn-repo/subversion
$ export SVN_RELEASE_DIST_REPOS=file:///opt/svn/dummy-asf-repos/dist-repo
$ export SVN_RELEASE_BUILDBOT_REPOS=file:///opt/svn/dummy-asf-repos/buildbot-repo

Start testing:

$ mkdir -p test-releasing/ && cd test-releasing/
$ release.py build-env 1.13.0-alpha1
INFO:root:Creating release environment
[...]
$ ./release.py roll 1.13.0-alpha1 7
INFO:root:Rolling release 1.13.0-alpha1 from branch branches/1.13.x@7
[...]
$ ./release.py sign-candidates 1.13.0-alpha1
INFO:root:Signing /opt/svn/test-releasing/deploy/subversion-1.13.0-alpha1.zip
[...]
$ ./release.py create-tag 1.13.0-alpha1 7
INFO:root:Creating tag for 1.13.0-alpha1
[...]
$ ./release.py post-candidates 1.13.0-alpha1 
INFO:root:Importing tarballs to file:///opt/svn/dummy-asf-repos/dist-repo/dev/subversion
[...]
# ... and so on

Inspect what the script changed in the dummy repositories:

$ svn log -v --limit=1 $SVN_RELEASE_SVN_REPOS 

r8 | julianfoad | 2019-10-02 15:40:18 +0100 (Wed, 02 Oct 2019) | 1 line
Changed paths:
   A /subversion/tags/1.13.0-alpha1 (from /subversion/branches/1.13.x:7)
   M /subversion/tags/1.13.0-alpha1/subversion/include/svn_version.h
Tagging release 1.13.0-alpha1
------------------------------------------------------------------------

$ svn log -v --limit=1 $SVN_RELEASE_DIST_REPOS 

r2 | julianfoad | 2019-10-02 15:40:56 +0100 (Wed, 02 Oct 2019) | 1 line
Changed paths:
   A /dev/subversion/subversion-1.13.0-alpha1.tar.bz2
   A /dev/subversion/subversion-1.13.0-alpha1.tar.bz2.asc
   A /dev/subversion/subversion-1.13.0-alpha1.tar.bz2.sha512
   A /dev/subversion/subversion-1.13.0-alpha1.tar.gz
   A /dev/subversion/subversion-1.13.0-alpha1.tar.gz.asc
   A /dev/subversion/subversion-1.13.0-alpha1.tar.gz.sha512
   A /dev/subversion/subversion-1.13.0-alpha1.zip
   A /dev/subversion/subversion-1.13.0-alpha1.zip.asc
   A /dev/subversion/subversion-1.13.0-alpha1.zip.sha512
   A /dev/subversion/svn_version.h.dist-1.13.0-alpha1
Add Subversion 1.13.0-alpha1 candidate release artifacts
------------------------------------------------------------------------

and so on.

The Subversion release process had too many manual steps in it.

The manual effort required was becoming a burden, now we’re doing a “regular” release every six months, as well as a source of inconsistency and errors.

As part of an effort to improve stability and availability of Subversion releases, I’m scripting some of the manual steps to bring us closer to automated releases.

The headings below link to the descriptions in our community guide. An automated step or set of steps is indicated as “release.py <command>“; the rest are manual. Steps in strikethrough were previously manual, now automated.

Creating a new minor release branch

• release.py create-release-branch
  • Create the new release branch with a server-side copy
  • increment version in svn_version.h, NativeResources.java, main.py
  • add a section in CHANGES
  • Create a new STATUS file on the release branch
  • add the new branch to the buildbot config
• release.py write-release-notes
  • Create a template release-notes document
  • Commit it
• Ask someone with appropriate access to add the A.B.x branch to the backport merge bot.

Rolling a release (repeat for both RC and final release)

• Merge CHANGES file from trunk to release branch; set the release date in it.
• Check get-deps.sh works on the release branch.
• release.py roll
• Test one or both of the tarballs
• release.py sign-candidates
• release.py create-tag
• release.py post-candidates
• send an email to the dev@ list
• adjust the topic on #svn-dev to mention “X.Y.Z is up for testing/signing”
• Update the issue tracker to have appropriate versions/milestones

The actual releasing (repeat for both RC and final release)

• Uploading the release
  • release.py move-to-dist
  • wait 24 hours
  • release.py clean-dist
  • Submit the new release version number on reporter.apache.org
• Announcing the release
  • release.py write-announcement
    • send the announcement email
  • Update the topics in IRC channels #svn, #svn-dev
• Update the website, any release:
  • release.py write-downloads
    • edit the result in download.html
  • release.py write-news … news.html
    
  • release.py write-news … index.html
    • check date, text, URL; remove old news from index.html
• Update the website, stable release X.Y.Z (not alpha/beta/rc):
  • List the new release in doap.rdf
  • List the new release in release-history.html
• Update the website, new minor release X.Y.0:
  • Update the support levels in release-notes/index.html
  • Update supported_release_lines in release.py
  • Remove “draft” warning from release-notes/X.Y.html
  • Create/update API docs: docs/api/X.Y, docs/javahl/X.Y
    • an example script is given in the doc
    • Update the links to the API docs in docs/index.html
  • Publish (commit or merge) these modifications

So quite a bit still to do…