How can I test a release process?

I’m improving the scripting of the Apache Subversion release process. Preparing and publishing a release involves things like making a branch in the source repository, updating web pages, updating buildbot configurations, and (not yet automated) sending emails.

When I’m modifying the release scripts, I needed a way to test without making bogus “live” releases.

The source code, web pages, and buildbot config are all stored in Subversion repositories, so that was a good place to start.

Where I’ve got to so far:

• scripted the creation of a set of local repositories that contain a minimum viable contents that the script expects to find;
• pass the release.py script a set of alternative URLs for the various repositories it is going to read and modify;
• manually inspect what the script changed in those repositories.

My initial script to set up local dummy repos is ugly and slow. But that’s OK; the important thing is it’s enough to proceed with verifying the automation while I improve it.

#!/bin/bash
# Make a local svn repo suitable for testing our release procedures
# ('release.py' etc.)

set -e

SRC_SVN_REPO_URL=https://svn.apache.org/repos/asf
SRC_DIST_REPO_URL=https://dist.apache.org/repos/dist
SRC_TRUNK_WC=$HOME/src/subversion-c
SRC_BRANCHES_WC=$HOME/src/svn/branches
SRC_SITE_WC=$HOME/src/svn/site

DUMMY_REPOS_DIR=/opt/svn/dummy-asf-repos
DUMMY_DIST_REPO_DIR=$DUMMY_REPOS_DIR/dist-repo
DUMMY_DIST_REPO_URL=file://$DUMMY_DIST_REPO_DIR
DUMMY_SVN_REPO_DIR=$DUMMY_REPOS_DIR/svn-repo
DUMMY_SVN_REPO_URL=file://$DUMMY_SVN_REPO_DIR
SVN_WC_DIR=$DUMMY_REPOS_DIR/svn-wc
#DIST_WC_DIR=$DUMMY_REPOS_DIR/dist-wc

# export_with_props SRC_WC DST_WC
function export_with_props() {
  SRC_WC="$1"
  DST_WC="$2"
  svn export $SRC_WC $DST_WC
  svn add --force --no-auto-props $DST_WC
  # remove automatically added mime-type props as some of them are not an exact replica
  svn pd -R svn:mime-type $DST_WC
  # add an exact replica of source props
  PROPS_TO_ADD=$PWD/props-to-add
  (cd $SRC_WC && svn diff --properties-only --old=^/@0 --new=.) > $PROPS_TO_ADD
  (cd $DST_WC && svn patch $PROPS_TO_ADD)
  # rm props-to-del props-to-add
}

set -x

mkdir "$DUMMY_REPOS_DIR"


### the 'dist.a.o' repo ###

if ! [ -d $DUMMY_DIST_REPO_DIR ]; then

  # create a repo
  svnadmin create $DUMMY_DIST_REPO_DIR

  # create skeleton dirs
  svn -m "Init" mkdir --parents $DUMMY_DIST_REPO_URL/{dev,release}/subversion

fi


### the 'svn.a.o' repo

if ! [ -d $DUMMY_SVN_REPO_DIR ]; then

  # create a repo
  svnadmin create $DUMMY_SVN_REPO_DIR

  # create skeleton dirs
  svn -m "Init" mkdir --parents $DUMMY_SVN_REPO_URL/subversion/{trunk,tags,branches,site}

  # check out
  svn co $DUMMY_SVN_REPO_URL $SVN_WC_DIR
  cd $SVN_WC_DIR

  # populate trunk
  rmdir subversion/trunk
  export_with_props $SRC_TRUNK_WC subversion/trunk
  svn revert -R subversion/trunk/{contrib,notes}/*
  svn ci -m "Add trunk" subversion/trunk

  # populate site
  export_with_props $SRC_SITE_WC/tools subversion/site/tools
  export_with_props $SRC_SITE_WC/publish subversion/site/publish
  svn revert -R subversion/site/publish/docs/{api,javahl}/*
  svn ci -m "Add site" subversion/site
  svn cp -m "Add site staging branch" ^/subversion/site/publish ^/subversion/site/staging

  # a mod on trunk
  echo hello > subversion/trunk/hello.txt
  svn add subversion/trunk/hello.txt
  svn ci -m "Trunk mod." subversion/trunk/hello.txt

  # make 1.13.x branch
  svn cp -m "Branch 1.13.x" ^/subversion/trunk ^/subversion/branches/1.13.x
  svn up --depth=immediates subversion/branches/1.13.x/
  sed 's/1\.12\.[0-9]*/1.13.0/' < $SRC_BRANCHES_WC/1.12.x/STATUS > subversion/branches/1.13.x/STATUS
  svn add subversion/branches/1.13.x/STATUS
  svn ci -m "Add 'STATUS' file."

fi

Make the dummy repos:

$ cd /opt/svn
$ rm -rf dummy-asf-repos/ && svn-mk-dummy-asf-repo.sh
[...]

Tell ‘release.py’ where the dummy repositories are:

$ export SVN_RELEASE_SVN_REPOS=file:///opt/svn/dummy-asf-repos/svn-repo/subversion
$ export SVN_RELEASE_DIST_REPOS=file:///opt/svn/dummy-asf-repos/dist-repo
$ export SVN_RELEASE_BUILDBOT_REPOS=file:///opt/svn/dummy-asf-repos/buildbot-repo

Start testing:

$ mkdir -p test-releasing/ && cd test-releasing/
$ release.py build-env 1.13.0-alpha1
INFO:root:Creating release environment
[...]
$ ./release.py roll 1.13.0-alpha1 7
INFO:root:Rolling release 1.13.0-alpha1 from branch branches/1.13.x@7
[...]
$ ./release.py sign-candidates 1.13.0-alpha1
INFO:root:Signing /opt/svn/test-releasing/deploy/subversion-1.13.0-alpha1.zip
[...]
$ ./release.py create-tag 1.13.0-alpha1 7
INFO:root:Creating tag for 1.13.0-alpha1
[...]
$ ./release.py post-candidates 1.13.0-alpha1 
INFO:root:Importing tarballs to file:///opt/svn/dummy-asf-repos/dist-repo/dev/subversion
[...]
# ... and so on

Inspect what the script changed in the dummy repositories:

$ svn log -v --limit=1 $SVN_RELEASE_SVN_REPOS 

r8 | julianfoad | 2019-10-02 15:40:18 +0100 (Wed, 02 Oct 2019) | 1 line
Changed paths:
   A /subversion/tags/1.13.0-alpha1 (from /subversion/branches/1.13.x:7)
   M /subversion/tags/1.13.0-alpha1/subversion/include/svn_version.h
Tagging release 1.13.0-alpha1
------------------------------------------------------------------------

$ svn log -v --limit=1 $SVN_RELEASE_DIST_REPOS 

r2 | julianfoad | 2019-10-02 15:40:56 +0100 (Wed, 02 Oct 2019) | 1 line
Changed paths:
   A /dev/subversion/subversion-1.13.0-alpha1.tar.bz2
   A /dev/subversion/subversion-1.13.0-alpha1.tar.bz2.asc
   A /dev/subversion/subversion-1.13.0-alpha1.tar.bz2.sha512
   A /dev/subversion/subversion-1.13.0-alpha1.tar.gz
   A /dev/subversion/subversion-1.13.0-alpha1.tar.gz.asc
   A /dev/subversion/subversion-1.13.0-alpha1.tar.gz.sha512
   A /dev/subversion/subversion-1.13.0-alpha1.zip
   A /dev/subversion/subversion-1.13.0-alpha1.zip.asc
   A /dev/subversion/subversion-1.13.0-alpha1.zip.sha512
   A /dev/subversion/svn_version.h.dist-1.13.0-alpha1
Add Subversion 1.13.0-alpha1 candidate release artifacts
------------------------------------------------------------------------

and so on.

The Subversion release process had too many manual steps in it.

The manual effort required was becoming a burden, now we’re doing a “regular” release every six months, as well as a source of inconsistency and errors.

As part of an effort to improve stability and availability of Subversion releases, I’m scripting some of the manual steps to bring us closer to automated releases.

The headings below link to the descriptions in our community guide. An automated step or set of steps is indicated as “release.py <command>“; the rest are manual. Steps in strikethrough were previously manual, now automated.

Creating a new minor release branch

• release.py create-release-branch
  • Create the new release branch with a server-side copy
  • increment version in svn_version.h, NativeResources.java, main.py
  • add a section in CHANGES
  • Create a new STATUS file on the release branch
  • add the new branch to the buildbot config
• release.py write-release-notes
  • Create a template release-notes document
  • Commit it
• Ask someone with appropriate access to add the A.B.x branch to the backport merge bot.

Rolling a release (repeat for both RC and final release)

• Merge CHANGES file from trunk to release branch; set the release date in it.
• Check get-deps.sh works on the release branch.
• release.py roll
• Test one or both of the tarballs
• release.py sign-candidates
• release.py create-tag
• release.py post-candidates
• send an email to the dev@ list
• adjust the topic on #svn-dev to mention “X.Y.Z is up for testing/signing”
• Update the issue tracker to have appropriate versions/milestones

The actual releasing (repeat for both RC and final release)

• Uploading the release
  • release.py move-to-dist
  • wait 24 hours
  • release.py clean-dist
  • Submit the new release version number on reporter.apache.org
• Announcing the release
  • release.py write-announcement
    • send the announcement email
  • Update the topics in IRC channels #svn, #svn-dev
• Update the website, any release:
  • release.py write-downloads
    • edit the result in download.html
  • release.py write-news … news.html
    
  • release.py write-news … index.html
    • check date, text, URL; remove old news from index.html
• Update the website, stable release X.Y.Z (not alpha/beta/rc):
  • List the new release in doap.rdf
  • List the new release in release-history.html
• Update the website, new minor release X.Y.0:
  • Update the support levels in release-notes/index.html
  • Update supported_release_lines in release.py
  • Remove “draft” warning from release-notes/X.Y.html
  • Create/update API docs: docs/api/X.Y, docs/javahl/X.Y
    • an example script is given in the doc
    • Update the links to the API docs in docs/index.html
  • Publish (commit or merge) these modifications

So quite a bit still to do…

Today I pushed the final “go” button, announcing Apache Subversion 1.12.2, 1.10.6, and 1.9.12. Thank-you, everyone who helped with reporting, developing, reviewing and testing the changes that went into these releases.

Release-managing was a bit of a struggle this time around. While Subversion has moved into the “mature” phase of its life with possiblly the most users ever, the number of active contributors has been dwindling, which is understandable. Recently it reached the point where the development community could no longer meet its self-imposed requirements for three members to approve each back-ported change. As the release was stalled even though we had fixes waiting that were otherwise ready to release, we — those of us who are still around to discuss such things — agreed to lower those requirements as a pragmatic move. Then I was able to merge in all the outstanding fixes in the queue. As a result, these releases have a few more changes in than they otherwise would have, albeit small changes as they are only patch releases.

Looking to future releases, it seems to me one of the most important issues we need to address at this stage of the project’s life is to streamline the release process, in two different ways — from the inside, making it easier for us to produce a release — and more importantly from the outside perspective, making it easier for end users and packagers to receive an upgrade. We are currently discussing this and other aspects of Subversion’s “community health”.

And when I say “we” — the Subversion project — that’s you too, if you want to be included. It might surprise some users to hear that there is no team dedicated to producing Subversion for you. There is just me, using a bit of my employer‘s time, and a very few others using a bit of their personal time, and … potentially you or a bit of your employer’s time? If you care about Subversion continuing to be available, and you can offer any kind of help — perhaps with building and releasing it in one of the many forms it’s distributed these days — you could make all the difference.

Could you be a part of true, free, libre, open source software development? Please come and say hello in the #svn channel (on IRC or on Matrix) or on the mailing lists.

I’m happy to announce the release of Apache Subversion 1.12.0.
Please choose the mirror closest to you by visiting:

https://subversion.apache.org/download.cgi#supported-releases

This is a stable feature release of the Apache Subversion open source
version control system.

SHA-512 checksums are available at:

https://www.apache.org/dist/subversion/subversion-1.12.0.tar.bz2.sha512
https://www.apache.org/dist/subversion/subversion-1.12.0.tar.gz.sha512
https://www.apache.org/dist/subversion/subversion-1.12.0.zip.sha512

PGP Signatures are available at:

https://www.apache.org/dist/subversion/subversion-1.12.0.tar.bz2.asc
https://www.apache.org/dist/subversion/subversion-1.12.0.tar.gz.asc
https://www.apache.org/dist/subversion/subversion-1.12.0.zip.asc

For this release, the following people have provided PGP signatures:

Julian Foad [4096R/1FB064B84EECC493] with fingerprint:
6011 63CF 9D49 9FD7 18CF  582D 1FB0 64B8 4EEC C493
Stefan Sperling [2048R/4F7DBAA99A59B973] with fingerprint:
8BC4 DAE0 C5A4 D65F 4044  0107 4F7D BAA9 9A59 B973
Johan Corveleyn [4096R/B59CE6D6010C8AAD] with fingerprint:
8AA2 C10E EAAD 44F9 6972  7AEA B59C E6D6 010C 8AAD
Stefan Fuhrmann [4096R/99EC741B57921ACC] with fingerprint:
056F 8016 D9B8 7B1B DE41  7467 99EC 741B 5792 1ACC

Release notes for the 1.12.x release series may be found at:

https://subversion.apache.org/docs/release-notes/1.12.html

You can find the list of changes between 1.12.0 and earlier versions at:

https://svn.apache.org/repos/asf/subversion/tags/1.12.0/CHANGES

Questions, comments, and bug reports to users@subversion.apache.org .

---

— from the announcement email

This week I took a break from coding to write about some non-coding ways I’ve been thinking about to modernize the Subversion project’s communications and reach, and encourage its community.

Here’s a snapshot of “What’s In My Head” copied from Subversion’s Wiki.

In a later post I write about some potential svn code developments in my head.

Community

Ways to modernize the Subversion project’s communications and reach, and encourage its community.

Communication technology

General

• keep using Open technologies
• keep plain email the baseline standard of communication
• integrate ‘forum’ and ‘mailing list’ forms of access
• integrate long-form and chat-form if possible
• integrate archives / logs, permalinks, searching

Email: Forums and Mailing Lists

Problem: Simple old mailing lists are inconvenient for new and occasional users. They have weak or no integration with their own archives, our issue tracker, etc.

Some proprietary forums e.g. Google groups already attempt to mirror and integrate with our lists, with some degree of success, but we are mainly ignoring it.

We should seek to integrate better with forums. For a start: let users know that it’s an option for them (on our ‘mailing-lists’ web page); and see if we can make some better steps of integration (such as unified permalinks).

For a longer term solution, I wonder if any suitable open source software exists, or if an ASF group might consider developing/adapting something.

• Pony Mail can substitute as a simple forum interface. See, for example, https://lists.apache.org/list.html?users@subversion.apache.org If you log in, you can post messages directly from the web UI.

Chat (IRC, Matrix)

Problem: The IRC chat systems we use are inconvenient for new and occasional users. They have weak integration with archives, issue tracker, etc.

Use Matrix as an upgrade path from IRC. That is my strong recommendation.

Matrix is Open, bridges to existing IRC channels, has good UI on mobile and desktop, is simple enough for newbies, has permalinks and can act as an archive, can be self-hosted.

Matrix is ready for immediate ad-hoc use by individual participants in our IRC channels, through the Freenode bridge operated by matrix.org, and is being used in this way by at least two svn-dev members.

In future the ASF should run its own Matrix infrastructure: server, bridges, etc. The ASF would then control its own data, its own user accounts, and its own integrations with archives, commits, issue trackers, etc. Usability advantages: use ASF single-sign-on; install specialist bridges/integrations.

Examples of migration to Matrix in other IRC-based communities: Wikimedia, Drupal

Tasks:

• encourage users to consider using Matrix to access #svn-dev and #svn
  
• share with the rest of the ASF – on users@infra.apache.org or dev@community.apache.org?

Integrate archives / logs, permalinks, searching

Problem: Past communications are scattered across systems and storage locations, with no consistent archives or permalinks, so cross-referencing is difficult and non-permanent. Our issue tracker and wiki provide only links that are tied to their current provider technology.

The types of information include:

• Short-form chat (IRC) archives: Matrix-Static, irclogger, ASF Wilderness IRC logs.
• Email/forum archives: haxx.se, ASF Pony Mail, the old Tigris lists. (The old Tigris mailing-lists system used to provide a permalink to the archived version at the bottom of each mail delivered.)
• Issue Tracker: the current Jira issues, the old issuezilla.
• Wiki pages: the current Confluence, the old MoinMoin.

An important step is to develop a URL “permalink” scheme to refer to our various resources. These would be technology-ignorant URLs, all under subversion.apache.org, like “/issue/1234“.

A baby step is the ‘.message-ids.tsv’ file in our web site directory, holding a mapping from haxx archive URLs used in our web pages to email message ids, with (in the commit log message) a script to generate it. There is, as yet, no automation to use the mapping in any way.

Initial tasks:

• start documenting a URL-space map for our resources
• populate one entry, e.g. “/issue/<number> → issue <number>”
• implement some simple automated handling (e.g. redirects) for that
  
  • well, well… we already have this in our .htaccess which covers that exact case along with some aliases:
  • "RedirectMatch ^/issue[^A-Za-z0-9]?(\d+)$ https://issues.apache.org/jira/browse/SVN-$1"
• start using it: update existing direct links to point here instead; publicize it

Deeper integration: A permalink URL should not merely redirect the user to its technology-specific target URL, but present the target in such a way that other inbound and outbound URLs also use the permalink form. With a big third-party system like Jira or Confluence the feasibility of that is going to depend entirely on whether the system has built-in support for that usage.

Software Distribution

Problem: Subversion packages are outdated or unavailable for many platforms, especially server/cloud environments (e.g. Docker) and mobile (e.g. Android).

Examples:

•  For Ubuntu trusty (14.04LTS): 1.8.8; for xenial (16.04LTS): 1.9.3; for bionic (18.04LTS): 1.9.7
• For OpenSUSE Tumbleweed: 1.7.4; for Leap 15.0: 1.10.0; for Leap 42.3: 1.9.7
• For Android: OAsvn, based on svnkit 1.7.5
  
• NAS: Synology svn server pkg 1.9.7; QNAP svn server pkg 1.7

What could we do?

• reach out to individual maintainers and ask if they need any help or just a ping?
• encourage companies to take on packaging (Assembla?)
• make dedicated efforts to establish builds that may be important

Traditional OS / Desktop

(Windows, Linux, BSD, Solaris)

pkgs.org lists the current package versions for many traditional Linux/BSD distributions.

Some companies maintain up to date package builds for several platforms, notably WANdisco & CollabNet.

Cloud / Server / Containers

(Docker, SNAP, VM, etc.)

On Docker Hub the most comprehensive svn server seems to be elleflorio/svn-server (http + svnserve). Next is garethflowers/svn-server (very simple; svnserve only). None seem to be an enterprise-grade installation.

There are no ‘subversion’ or ‘svn’ packages in the SNAP store.

Mobile (client)

(Android, IOS)

We should be able to have functional svn client apps on Android and IOS. The libs and bindings might be the best focus. The command-line client won’t so often be wanted, but no reason it should not be available.

For Android, the only open-source client is OAsvn, based on svnkit 1.7.5. It works, but is unmaintained and primitive.

Small / Personal Servers

(Home NAS; Raspberry Pi, etc.; also personal server software platforms like Sandstorm, UBOS, Yunohost, etc.)

Integrations with IDEs etc.

(Visual Studio, Netbeans, IntelliJ, XCode, etc.)

Documentation

The Svn Book

The authors of the Svn Book no longer maintain it have recently been offering to hand its ownership to the Subversion project.

What should we do with it?

Man Pages and Help Text

The built-in help is half way between a summary and a detailed reference. The Book contains an index which is a variation on this. We never got around to generating man pages or other formatted help.

Tasks:

• Generate svn help and man pages from common source: there is an old patch as a starting point