Testing Svn Release Scripts

How can I test a release process?

I’m improving the scripting of the Apache Subversion release process. Preparing and publishing a release involves things like making a branch in the source repository, updating web pages, updating buildbot configurations, and (not yet automated) sending emails.

When I’m modifying the release scripts, I needed a way to test without making bogus “live” releases.

The source code, web pages, and buildbot config are all stored in Subversion repositories, so that was a good place to start.

Where I’ve got to so far:

  • scripted the creation of a set of local repositories that contain a minimum viable contents that the script expects to find;
  • pass the release.py script a set of alternative URLs for the various repositories it is going to read and modify;
  • manually inspect what the script changed in those repositories.

My initial script to set up local dummy repos is ugly and slow. But that’s OK; the important thing is it’s enough to proceed with verifying the automation while I improve it.

#!/bin/bash
# Make a local svn repo suitable for testing our release procedures
# ('release.py' etc.)

set -e

SRC_SVN_REPO_URL=https://svn.apache.org/repos/asf
SRC_DIST_REPO_URL=https://dist.apache.org/repos/dist
SRC_TRUNK_WC=$HOME/src/subversion-c
SRC_BRANCHES_WC=$HOME/src/svn/branches
SRC_SITE_WC=$HOME/src/svn/site

DUMMY_REPOS_DIR=/opt/svn/dummy-asf-repos
DUMMY_DIST_REPO_DIR=$DUMMY_REPOS_DIR/dist-repo
DUMMY_DIST_REPO_URL=file://$DUMMY_DIST_REPO_DIR
DUMMY_SVN_REPO_DIR=$DUMMY_REPOS_DIR/svn-repo
DUMMY_SVN_REPO_URL=file://$DUMMY_SVN_REPO_DIR
SVN_WC_DIR=$DUMMY_REPOS_DIR/svn-wc
#DIST_WC_DIR=$DUMMY_REPOS_DIR/dist-wc

# export_with_props SRC_WC DST_WC
function export_with_props() {
  SRC_WC="$1"
  DST_WC="$2"
  svn export $SRC_WC $DST_WC
  svn add --force --no-auto-props $DST_WC
  # remove automatically added mime-type props as some of them are not an exact replica
  svn pd -R svn:mime-type $DST_WC
  # add an exact replica of source props
  PROPS_TO_ADD=$PWD/props-to-add
  (cd $SRC_WC && svn diff --properties-only --old=^/@0 --new=.) > $PROPS_TO_ADD
  (cd $DST_WC && svn patch $PROPS_TO_ADD)
  # rm props-to-del props-to-add
}

set -x

mkdir "$DUMMY_REPOS_DIR"


### the 'dist.a.o' repo ###

if ! [ -d $DUMMY_DIST_REPO_DIR ]; then

  # create a repo
  svnadmin create $DUMMY_DIST_REPO_DIR

  # create skeleton dirs
  svn -m "Init" mkdir --parents $DUMMY_DIST_REPO_URL/{dev,release}/subversion

fi


### the 'svn.a.o' repo

if ! [ -d $DUMMY_SVN_REPO_DIR ]; then

  # create a repo
  svnadmin create $DUMMY_SVN_REPO_DIR

  # create skeleton dirs
  svn -m "Init" mkdir --parents $DUMMY_SVN_REPO_URL/subversion/{trunk,tags,branches,site}

  # check out
  svn co $DUMMY_SVN_REPO_URL $SVN_WC_DIR
  cd $SVN_WC_DIR

  # populate trunk
  rmdir subversion/trunk
  export_with_props $SRC_TRUNK_WC subversion/trunk
  svn revert -R subversion/trunk/{contrib,notes}/*
  svn ci -m "Add trunk" subversion/trunk

  # populate site
  export_with_props $SRC_SITE_WC/tools subversion/site/tools
  export_with_props $SRC_SITE_WC/publish subversion/site/publish
  svn revert -R subversion/site/publish/docs/{api,javahl}/*
  svn ci -m "Add site" subversion/site
  svn cp -m "Add site staging branch" ^/subversion/site/publish ^/subversion/site/staging

  # a mod on trunk
  echo hello > subversion/trunk/hello.txt
  svn add subversion/trunk/hello.txt
  svn ci -m "Trunk mod." subversion/trunk/hello.txt

  # make 1.13.x branch
  svn cp -m "Branch 1.13.x" ^/subversion/trunk ^/subversion/branches/1.13.x
  svn up --depth=immediates subversion/branches/1.13.x/
  sed 's/1\.12\.[0-9]*/1.13.0/' < $SRC_BRANCHES_WC/1.12.x/STATUS > subversion/branches/1.13.x/STATUS
  svn add subversion/branches/1.13.x/STATUS
  svn ci -m "Add 'STATUS' file."

fi

Make the dummy repos:

$ cd /opt/svn
$ rm -rf dummy-asf-repos/ && svn-mk-dummy-asf-repo.sh
[...]

Tell ‘release.py’ where the dummy repositories are:

$ export SVN_RELEASE_SVN_REPOS=file:///opt/svn/dummy-asf-repos/svn-repo/subversion
$ export SVN_RELEASE_DIST_REPOS=file:///opt/svn/dummy-asf-repos/dist-repo
$ export SVN_RELEASE_BUILDBOT_REPOS=file:///opt/svn/dummy-asf-repos/buildbot-repo

Start testing:

$ mkdir -p test-releasing/ && cd test-releasing/
$ release.py build-env 1.13.0-alpha1
INFO:root:Creating release environment
[...]
$ ./release.py roll 1.13.0-alpha1 7
INFO:root:Rolling release 1.13.0-alpha1 from branch branches/1.13.x@7
[...]
$ ./release.py sign-candidates 1.13.0-alpha1
INFO:root:Signing /opt/svn/test-releasing/deploy/subversion-1.13.0-alpha1.zip
[...]
$ ./release.py create-tag 1.13.0-alpha1 7
INFO:root:Creating tag for 1.13.0-alpha1
[...]
$ ./release.py post-candidates 1.13.0-alpha1 
INFO:root:Importing tarballs to file:///opt/svn/dummy-asf-repos/dist-repo/dev/subversion
[...]
# ... and so on

Inspect what the script changed in the dummy repositories:

$ svn log -v --limit=1 $SVN_RELEASE_SVN_REPOS 

r8 | julianfoad | 2019-10-02 15:40:18 +0100 (Wed, 02 Oct 2019) | 1 line
Changed paths:
   A /subversion/tags/1.13.0-alpha1 (from /subversion/branches/1.13.x:7)
   M /subversion/tags/1.13.0-alpha1/subversion/include/svn_version.h
Tagging release 1.13.0-alpha1
------------------------------------------------------------------------

$ svn log -v --limit=1 $SVN_RELEASE_DIST_REPOS 

r2 | julianfoad | 2019-10-02 15:40:56 +0100 (Wed, 02 Oct 2019) | 1 line
Changed paths:
   A /dev/subversion/subversion-1.13.0-alpha1.tar.bz2
   A /dev/subversion/subversion-1.13.0-alpha1.tar.bz2.asc
   A /dev/subversion/subversion-1.13.0-alpha1.tar.bz2.sha512
   A /dev/subversion/subversion-1.13.0-alpha1.tar.gz
   A /dev/subversion/subversion-1.13.0-alpha1.tar.gz.asc
   A /dev/subversion/subversion-1.13.0-alpha1.tar.gz.sha512
   A /dev/subversion/subversion-1.13.0-alpha1.zip
   A /dev/subversion/subversion-1.13.0-alpha1.zip.asc
   A /dev/subversion/subversion-1.13.0-alpha1.zip.sha512
   A /dev/subversion/svn_version.h.dist-1.13.0-alpha1
Add Subversion 1.13.0-alpha1 candidate release artifacts
------------------------------------------------------------------------

and so on.

Scripting Svn Releases

The Subversion release process had too many manual steps in it.

The manual effort required was becoming a burden, now we’re doing a “regular” release every six months, as well as a source of inconsistency and errors.

As part of an effort to improve stability and availability of Subversion releases, I’m scripting some of the manual steps to bring us closer to automated releases.

The headings below link to the descriptions in our community guide. An automated step or set of steps is indicated as “release.py <command>“; the rest are manual. Steps in strikethrough were previously manual, now automated.

Creating a new minor release branch

  • release.py create-release-branch
    • Create the new release branch with a server-side copy
    • increment version in svn_version.h, NativeResources.java, main.py
    • add a section in CHANGES
    • Create a new STATUS file on the release branch
    • add the new branch to the buildbot config
  • release.py write-release-notes
    • Create a template release-notes document
    • Commit it
  • Ask someone with appropriate access to add the A.B.x branch to the backport merge bot.

Rolling a release (repeat for both RC and final release)

  • Merge CHANGES file from trunk to release branch; set the release date in it.
  • Check get-deps.sh works on the release branch.
  • release.py roll
  • Test one or both of the tarballs
  • release.py sign-candidates
  • release.py create-tag
  • release.py post-candidates
  • send an email to the dev@ list
  • adjust the topic on #svn-dev to mention “X.Y.Z is up for testing/signing”
  • Update the issue tracker to have appropriate versions/milestones

The actual releasing (repeat for both RC and final release)

  • Uploading the release
    • release.py move-to-dist
    • wait 24 hours
    • release.py clean-dist
    • Submit the new release version number on reporter.apache.org
  • Announcing the release
    • release.py write-announcement
      • send the announcement email
    • Update the topics in IRC channels #svn, #svn-dev
  • Update the website, any release:
    • release.py write-downloads
      • edit the result in download.html
    • release.py write-news … news.html
    • release.py write-news … index.html
      • check date, text, URL; remove old news from index.html
  • Update the website, stable release X.Y.Z (not alpha/beta/rc):
    • List the new release in doap.rdf
    • List the new release in release-history.html
  • Update the website, new minor release X.Y.0:
    • Update the support levels in release-notes/index.html
    • Update supported_release_lines in release.py
    • Remove “draft” warning from release-notes/X.Y.html
    • Create/update API docs: docs/api/X.Y, docs/javahl/X.Y
      • an example script is given in the doc
      • Update the links to the API docs in docs/index.html
    • Publish (commit or merge) these modifications

So quite a bit still to do…

Apache Subversion 1.12.2, 1.10.6, 1.9.12 released

Today I pushed the final “go” button, announcing Apache Subversion 1.12.2, 1.10.6, and 1.9.12. Thank-you, everyone who helped with reporting, developing, reviewing and testing the changes that went into these releases.

Release-managing was a bit of a struggle this time around. While Subversion has moved into the “mature” phase of its life with possiblly the most users ever, the number of active contributors has been dwindling, which is understandable. Recently it reached the point where the development community could no longer meet its self-imposed requirements for three members to approve each back-ported change. As the release was stalled even though we had fixes waiting that were otherwise ready to release, we — those of us who are still around to discuss such things — agreed to lower those requirements as a pragmatic move. Then I was able to merge in all the outstanding fixes in the queue. As a result, these releases have a few more changes in than they otherwise would have, albeit small changes as they are only patch releases.

Looking to future releases, it seems to me one of the most important issues we need to address at this stage of the project’s life is to streamline the release process, in two different ways — from the inside, making it easier for us to produce a release — and more importantly from the outside perspective, making it easier for end users and packagers to receive an upgrade. We are currently discussing this and other aspects of Subversion’s “community health”.

And when I say “we” — the Subversion project — that’s you too, if you want to be included. It might surprise some users to hear that there is no team dedicated to producing Subversion for you. There is just me, using a bit of my employer‘s time, and a very few others using a bit of their personal time, and … potentially you or a bit of your employer’s time? If you care about Subversion continuing to be available, and you can offer any kind of help — perhaps with building and releasing it in one of the many forms it’s distributed these days — you could make all the difference.

Could you be a part of true, free, libre, open source software development? Please come and say hello in the #svn channel (on IRC or on Matrix) or on the mailing lists.

Subversion 1.12 Released

I’m happy to announce the release of Apache Subversion 1.12.0.
Please choose the mirror closest to you by visiting:

https://subversion.apache.org/download.cgi#supported-releases

This is a stable feature release of the Apache Subversion open source
version control system.

SHA-512 checksums are available at:

https://www.apache.org/dist/subversion/subversion-1.12.0.tar.bz2.sha512
https://www.apache.org/dist/subversion/subversion-1.12.0.tar.gz.sha512
https://www.apache.org/dist/subversion/subversion-1.12.0.zip.sha512

PGP Signatures are available at:

https://www.apache.org/dist/subversion/subversion-1.12.0.tar.bz2.asc
https://www.apache.org/dist/subversion/subversion-1.12.0.tar.gz.asc
https://www.apache.org/dist/subversion/subversion-1.12.0.zip.asc

For this release, the following people have provided PGP signatures:

Julian Foad [4096R/1FB064B84EECC493] with fingerprint:
6011 63CF 9D49 9FD7 18CF  582D 1FB0 64B8 4EEC C493
Stefan Sperling [2048R/4F7DBAA99A59B973] with fingerprint:
8BC4 DAE0 C5A4 D65F 4044  0107 4F7D BAA9 9A59 B973
Johan Corveleyn [4096R/B59CE6D6010C8AAD] with fingerprint:
8AA2 C10E EAAD 44F9 6972  7AEA B59C E6D6 010C 8AAD
Stefan Fuhrmann [4096R/99EC741B57921ACC] with fingerprint:
056F 8016 D9B8 7B1B DE41  7467 99EC 741B 5792 1ACC

Release notes for the 1.12.x release series may be found at:

https://subversion.apache.org/docs/release-notes/1.12.html

You can find the list of changes between 1.12.0 and earlier versions at:

https://svn.apache.org/repos/asf/subversion/tags/1.12.0/CHANGES

Questions, comments, and bug reports to users@subversion.apache.org .


— from the announcement email

Svn Code Developments in my Head

Thoughts on where Subversion should be going, code-wise.

A companion page to What’s In My Head which is about community and project management aspects of Subversion.

General remarks:

  • Progress is slow: attempting to do anything with svn code is slow going. Reasons?
  • Technical debt: low level C code doing business logic, compatibility (almost bug-for-bug) trumps rewrites.
  • Functionality that may not suit all users (diff, patch, merge, externals, keywords, eol-style, WC storage, repo storage, …) has usually been built in, where pluggable interfaces would encourage modularity. Pluggable “diff” is crude, incomplete. Pluggable RA layers are our best example.
  • Writing third-party software that uses Svn is not as easy as it should be. See: Bindings; etc.
  • Companies writing commercial systems do not interact much; how can we improve that?
  • Rewrite clients (svn) and high level libs (libsvn_client) in a more suitable language?

Internal improvements needed:

  • WC APIs. The WC layer needs API that reflects operational units. For example, “a tree in the repository” should function interchangeably as the base of a copy or of a checkout; and “the modifications to a tree”; but not “merge” and “update” which are client level concepts; thus both higher and lower level than current APIs. See: SVN-4786 “WC working-mods editor”, for a start.
  • Diff in particular shows up many inconsistencies due to bitty implementation. Once generic changes APIs are in place, rewrite ‘diff’ to use them: diff(WC, repo): {a=tree_open(WC), b=tree_open(repo); diff_trees(a,b)}.
  • Diff: distinguish “diff a commit” from “diff two trees”.
  • Bindings. State is poor, inconsistent. What’s needed is an OO generic interface, implemented in several languages. An implementation technique: write C++ OO layer on top of existing C API, then wrap it in other languages. Problem: underlying svn API design is not OO. Opportunity: design the APIs we would like to provide; reimplement libs to provide them. See Brane’s current “svncxx” work.
  • Repository API: Eliminate no-op changes.
  • Repository API: Copy content without recording ‘copy-from’.

Feature improvements:

  • Server-side “pull request” merging.
  • Renames. TODO: Write up how my strict element-id scheme (see svnmover) can improve handling renames in update and merge, even in a primitive form without server-side support. To integrate it in existing svn, depends on: WC APIs; rewrite of merge code. Develop a variation where directories are ephemeral, for better real-life semantics (like splitting and joining dirs).
  • Query language (Hg revsets; Paul Hammanthttps://graphql.org/learn/GraphQL).
  • Repo syncing (revprops and locks as versioned/sequenced events; Merkle trees).

General Remarks

Subversion’s client side code is particularly obtuse.

The server side is considerably cleaner. Its user is the network protocols, it is focused on doing its job quickly and accurately, and its API surface is relatively compact and stable. The API semantics are not perfectly clean and the implementation is complicated by a lot of optimization and caching which sometimes introduces bugs, but overall it is a lot better than the client.

Rewrite clients (svn) and high level libs (libsvn_client) in a more suitable language? This code consists of simple logic, filters, configuration, hashes, composition, etc., which in C is tedious, error-prone, inconsistent, contributing to buggy inconsistent UX. Python would make sense for the client itself. For the libs, difficulty is how to provide bindings; C is good base language for that. Rewrite might not make a lot of sense for the existing client, because it has too many idiosyncrasies which would have to be either recreated in excruciating detail or smoothed out losing compatibility. However, a rewrite dropping a lot of oddities and little-used features such as externals might make make a lot of sense if we want a new client for e.g. Android and IOS.

WC APIs

The WC layer needs an API that reflects the granularity and shape of the operations we want to perform on it. Usually these operations are reading or modifying a given subtree to a given depth, with criteria such as selecting items with a certain set of statuses.

When the client commit code is scanning a WC, looking for items to commit according to the user’s given criteria, it should not itself have to iterate over a list of children, deciding which ones fit the criteria, looking up the child’s schedule status and deciding whether to recurse into a subdirectory based on its schedule and other options. Instead it should be able to invoke a suitable tree walker which does all that, and just perform commit processing on each item found. This same tree walker should be shared with other client functions such as diff and shelve that also want to operate on a similar selection.

The base of a checkout and the base of a copy are both “a tree in the repository”, more or less. There should be APIs to put such a tree into the WC storage, read it out, use it as the base of a checkout, use it as the base of a copy, use it as the base of a diff, and so on, and these operations should all use a common, interchangeable “tree” object. At present there is no such “tree” object concept in the APIs, and each of these operations is implemented implicitly by code that is specific to each use case, all different from each other, so that there is no chance of obtaining consistent results and no easy way to spin up a new instance of such behaviour in the implementation of shelving.

To improve this, I plan to create a set of interfaces for accessing the WC and other WC-like data such as shelves. One part of this interface is dedicated to accessing the local modifications that can be committed; this must be compatible with the interface used to commit changes to a repository and read them out. Another part is dedicated to accessing the base state of the WC; this will have something in common with the repository access interfaces too, but also will have notions such as “switched”. A third part is dedicated to accessing the uncommittable WC state such as changelists, conflicts, obstructed, missing, and unversioned items.

For the “committable changes” interface, svn_delta_editor_t is a good starting point as by definition it must support the necessary operations for a commit.

For the “uncommittable changes” interface, something new must be invented, perhaps starting with the WC “status” APIs.

For the “base state” API, the recent experimental “viewspec” output feature built on top of the WC “reporter” shows how we can describe the base state. To that must be added some way of transferring the necessary content. The present implementation directly contacts the repository URL through the RA layer when it needs to fetch base content; this needs to be abstracted out to the new interface.

Subversion’s “copy” operation comprises not just local changes but also a request for a new base subtree to be fetched. Therefore there is some dependency between the “committable changes” interface and the “base state” interface. Either the “committable changes” interface will need to depend directly on “base state” in order to fetch more base content when it needs to, or the caller will need to ensure that the necessary base content has been provided in advance and the “copy” method will need to be able to find it.

WC Tree-of-Changes Walker

(May be needed by: “Revert”, “unshelve”, …)

Need a tree-of-changes walker that visits nodes in a depth-first order with: (1) visit a dir before its children and also afterwards; and (2) visit a replaced subtree twice, once for the delete and then again for the add. Useful especially but not only for WC.

Diff a Commit vs. Diff Two Trees

Repository API: Eliminate No-op Changes

A curse of inconsistency. Some operations in the commit API, such as “open a file”, leave a trace in the FS even if no change in content is made, which then manifests in a similar API method call sequence when the commit is replayed. We often call these “no-op changes”, though “touches” may be more descriptive.

Arguing from a written description of the issue, without concrete details, it has proven hard to convince people that this is a bug: they can argue that storing and retrieving an extra flag that “the object was touched” is a feature. Using two approaches I am confident we can show it is a bug. (1) Show that the behaviour is inconsistent. (2) Argue that the Subversion data model is supposed to be based on stored state, and this amounts to “hidden state”.

  1. There are different potential cases where a no-op change might be considered: open a file and close it, open and apply a null text-delta, copy and apply a null text-delta, change existing property to its same value, delete nonexistent property, etc. TODO: test many cases and report the inconsistencies between them.
  2. There are different methods for information retrieval: list changes, replay revision, delta-dirs, update editor, etc. There are different API levels to try: FS, repos, RA. TODO: test retrieval methods and report the inconsistencies between them.

(2) Data model. In the general model of versioning a series of snapshots of user’s data, we should be able to extract any two stored states and calculate the difference between them, but a “touched” flag does not work like that; it is a function of some hidden state.

Repository API: Copy Content Without ‘Copy-From’

Should be able to:

  • commit a reintegrate merge from WC to repo efficiently, in O(1) time and size.
  • perform a reintegrate merge server-side, in O(1) time and size (new feature)

When committing changes, it is sometimes the case that the target content (of a file, subtree, or branch) is a content that already exists somewhere else in the repository. This is common in merging, where the merged content of a file (or subtree) on the target branch is identical to a content that exists on the source branch. In a reintegrate (or “copy up”) merge, the entire content of the target branch becomes equal to the current content of the source branch.

It is inefficient for the client to transmit changes against its working copy base to recreate the desired content on the server, when in principle it could instead tell the server where else that content may be found.

The existing “copy” method in the editor API copies the referenced content. At the same time it rewrites the history pointer to point to the “copied from” location. What we need is a variant of “copy” which keeps the default “natural predecessor” history.

(There is a “link” method in the FS API, which is not exposed in the RA API. It appears to be somehow related to this. TODO: investigate whether that is of any use.)

Server-side “pull request” merging

Thoughts about Svn Design and Code

A companion page to What’s In My Head.

Some specific code enhancements

  • WC subtree read/write APIs. The WC layer needs API that reflects operational units such as “a tree in the repository” to function consistently as the base of a copy or of a checkout; and “the modifications to a tree”; but not “merge” and “update” which are client level concepts; thus both higher and lower level than current APIs. See: SVN-4786, for a start.
  • Diff in particular shows up many inconsistencies due to bitty implementation. Once generic changes APIs are in place, rewrite ‘diff’ to use them: diff(WC, repo) := {a=open(WC), b=open(repo); diff(a,b)}.

System-level topics

  • Renames
    • My strict element-id scheme (see svnmover) can improve handling renames in update and merge. Write it up, esp. how it can be used primitively without server-side support. To integrate it in existing svn, depends on: WC subtree read/write APIs. Develop a variant where directories are ephemeral, for better real-life semantics (like splitting and joining dirs).
  • Query language
  • Repo syncing
    • The design of repo syncing (svnsync) is incomplete: especially revprops and locks
    • revprops and locks as versioned/sequenced events; Merkle trees?
  • Rewrite high level libs (libsvn_client) and clients (svn) in Python
    • They consist of simple logic, filters, config, hashes, composition, etc., which in C is tedious, error-prone, inconsistent, contributing to buggy inconsistent UX. For libs, difficulty is how to provide bindings; C is good base language for that.
  • OO Bindings
    • Bindings state is poor, inconsistent. What’s needed is an OO generic interface, implemented in several languages. An implementation technique: write C++ OO layer on top of existing C API, then wrap it in other languages. Problem: underlying svn API design is not OO. Opportunity: design the APIs we would like to provide; reimplement libs to provide them.
  • Plug-ins
    • Functionality is built-in first, where pluggable interfaces would be better (diff, patch, merge, externals, keywords, eol-style, WC storage, repo storage, …)
    • example: ‘diff’ plug-in support exists but is weak: want different diff tools for different file types, want to configure an external tool for tree diffs
  • A new ‘svn’ client
    • Consider writing an alternative ‘svn’ client from scratch with consistent functionality built from layers of blocks, not attempting to emulate CVS, and taking inspiration from git/hg/bzr. For example it should provide a function to output a tree (or file) which should encompass all of the existing ‘ svn list’, ‘svn cat’, ‘svn proplist’, ‘svn export’, ‘svn diff -r0:REV’, ‘svnrdump dump’; and a function to input a tree (or file) which should provide the inverse of all of those output modes, encompassing ‘svn propset’, ‘svnmucc put’, the hypothetical ‘svn addremove’, etc.

General observations

Progress seems to be stifled. Why?

  • It’s a “maturing” project with decreasing volunteer activity.
  • Technical debt: low level C code doing business logic, compatibility (almost bug-for-bug) trumps rewrites.
  • Writing third-party software that uses Svn is not as easy as it should be. See: Bindings; etc.
  • Companies writing commercial systems do not interact much; how can we improve that?

Svn: What’s in my Head

This week I took a break from coding to write about some non-coding ways I’ve been thinking about to modernize the Subversion project’s communications and reach, and encourage its community.

Here’s a snapshot of “What’s In My Head” copied from Subversion’s Wiki.

In a later post I write about some potential svn code developments in my head.

Community

Ways to modernize the Subversion project’s communications and reach, and encourage its community.

Communication technology

General

  • keep using Open technologies
  • keep plain email the baseline standard of communication
  • integrate ‘forum’ and ‘mailing list’ forms of access
  • integrate long-form and chat-form if possible
  • integrate archives / logs, permalinks, searching

Email: Forums and Mailing Lists

Problem: Simple old mailing lists are inconvenient for new and occasional users. They have weak or no integration with their own archives, our issue tracker, etc.

Some proprietary forums e.g. Google groups already attempt to mirror and integrate with our lists, with some degree of success, but we are mainly ignoring it.

We should seek to integrate better with forums. For a start: let users know that it’s an option for them (on our ‘mailing-lists’ web page); and see if we can make some better steps of integration (such as unified permalinks).

For a longer term solution, I wonder if any suitable open source software exists, or if an ASF group might consider developing/adapting something.

Chat (IRC, Matrix)

Problem: The IRC chat systems we use are inconvenient for new and occasional users. They have weak integration with archives, issue tracker, etc.

Use Matrix as an upgrade path from IRC. That is my strong recommendation.

Matrix is Open, bridges to existing IRC channels, has good UI on mobile and desktop, is simple enough for newbies, has permalinks and can act as an archive, can be self-hosted.

Matrix is ready for immediate ad-hoc use by individual participants in our IRC channels, through the Freenode bridge operated by matrix.org, and is being used in this way by at least two svn-dev members.

In future the ASF should run its own Matrix infrastructure: server, bridges, etc. The ASF would then control its own data, its own user accounts, and its own integrations with archives, commits, issue trackers, etc. Usability advantages: use ASF single-sign-on; install specialist bridges/integrations.

Examples of migration to Matrix in other IRC-based communities: Wikimedia, Drupal

Tasks:

Integrate archives / logs, permalinks, searching

Problem: Past communications are scattered across systems and storage locations, with no consistent archives or permalinks, so cross-referencing is difficult and non-permanent. Our issue tracker and wiki provide only links that are tied to their current provider technology.

The types of information include:

An important step is to develop a URL “permalink” scheme to refer to our various resources. These would be technology-ignorant URLs, all under subversion.apache.org, like “/issue/1234“.

A baby step is the ‘.message-ids.tsv’ file in our web site directory, holding a mapping from haxx archive URLs used in our web pages to email message ids, with (in the commit log message) a script to generate it. There is, as yet, no automation to use the mapping in any way.

Initial tasks:

  • start documenting a URL-space map for our resources
  • populate one entry, e.g. “/issue/<number> → issue <number>”
  • implement some simple automated handling (e.g. redirects) for that
    • well, well… we already have this in our .htaccess which covers that exact case along with some aliases:
    • "RedirectMatch ^/issue[^A-Za-z0-9]?(\d+)$ https://issues.apache.org/jira/browse/SVN-$1"
  • start using it: update existing direct links to point here instead; publicize it

Deeper integration: A permalink URL should not merely redirect the user to its technology-specific target URL, but present the target in such a way that other inbound and outbound URLs also use the permalink form. With a big third-party system like Jira or Confluence the feasibility of that is going to depend entirely on whether the system has built-in support for that usage.

Software Distribution

Problem: Subversion packages are outdated or unavailable for many platforms, especially server/cloud environments (e.g. Docker) and mobile (e.g. Android).

Examples:

What could we do?

  • reach out to individual maintainers and ask if they need any help or just a ping?
  • encourage companies to take on packaging (Assembla?)
  • make dedicated efforts to establish builds that may be important

Traditional OS / Desktop

(Windows, Linux, BSD, Solaris)

pkgs.org lists the current package versions for many traditional Linux/BSD distributions.

Some companies maintain up to date package builds for several platforms, notably WANdisco & CollabNet.

Cloud / Server / Containers

(Docker, SNAP, VM, etc.)

On Docker Hub the most comprehensive svn server seems to be elleflorio/svn-server (http + svnserve). Next is garethflowers/svn-server (very simple; svnserve only). None seem to be an enterprise-grade installation.

There are no ‘subversion’ or ‘svn’ packages in the SNAP store.

Mobile (client)

(Android, IOS)

We should be able to have functional svn client apps on Android and IOS. The libs and bindings might be the best focus. The command-line client won’t so often be wanted, but no reason it should not be available.

For Android, the only open-source client is OAsvn, based on svnkit 1.7.5. It works, but is unmaintained and primitive.

Small / Personal Servers

(Home NAS; Raspberry Pi, etc.; also personal server software platforms like Sandstorm, UBOS, Yunohost, etc.)

Integrations with IDEs etc.

(Visual Studio, Netbeans, IntelliJ, XCode, etc.)

Documentation

The Svn Book

The authors of the Svn Book no longer maintain it have recently been offering to hand its ownership to the Subversion project.

What should we do with it?

Man Pages and Help Text

The built-in help is half way between a summary and a detailed reference. The Book contains an index which is a variation on this. We never got around to generating man pages or other formatted help.

Tasks:

  • Generate svn help and man pages from common source: there is an old patch as a starting point

Subversion Hackathon 2017

in Aachen, Germany

Today I am going to meet up with other Subversion developers to hack, talk, invent, fix, inspire.

For those who are volunteer contributors, who may have little time to devote to Subversion usually, the hackathon provides a few days to concentrate on their favourite improvements and fixes, and to bring their ideas together.

For me, employed by Assembla to improve Subversion’s support for modern cloud work flows, this gives me a chance to gather ideas from other developers, bounce my ideas off them and see where they lead, and find new inspirations and enthusiasm.

Shelving and Checkpointing is my focus right now — the ability to quickly save and restore work in progress, without having to commit it to the repository. My initial version of shelving was merged to trunk last week, and Assembla is helping to make it available for users to try. (Download it here — and please let us know what you want it to become.)

Now I have started on checkpointing, discussing the design on the dev mailing list and implementing it on the ‘shelve-checkpoint’ branch. One of the next bits of coding, that I may try to complete during the hackathon, is when restoring or ‘unshelving’ a saved change to make Subversion first check whether any of the files it will touch is already modified in the working copy, so it can warn me that applying the saved changes may conflict.

More important, though, is to come up with longer term directions. One high on my list is the ability to send a locally prepared change to a code review system, and later to commit a change that has been reviewed, like the popular ‘merge request’ or ‘pull request’ work flows. Systems built on top of Subversion including RhodeCode, Rietveld, and Assembla provide these merge-request work flows. What we have now is the opportunity to streamline and standardize how these work, making them more generally available and interoperable.

 

I wonder how my list of topics will have changed by the end of the week.