Versioning filesystems Part 1: how it works in OpenVMS

Most computer users manually save versions of files with names like Presentation-2027-Jan-11, followed later by Presentation-2027-Jan-11-UPDATED and Presentation-2027-For-Sally-Only, perhaps with Presentation-panic-edits-before-event. This is manual versioning, and even the most organised of people who stick rigidly to a single convention still get confused at times. Some applications append .bak or prefix ~ to create their own backups, and then sometimes a TMP or $tmp$ file can really save the day, so on. And then when it comes time to clean up and organise a filesystem, everyone hesitates. Am I really sure there was not some useful work in each of those ten versions? Which ones do I need to keep? So we usually keep them all.

There are many problems around this area, and some solutions too. The instant-search feature which constantly indexes filesystems will be familiar to most people, but that still the problem of multiple versions. The instant search bar has to decide whether or not to suppress almost-identical results from different versions of files, when some of those small differences might be vital. Similarly, the common idea of content addressable storage ↗ is very useful but still doesn’t address the question of versions.

OpenVMS deals with versioning in a way that has been user-friendly and effective since 1977. In OpenVMS, saving a file does not destroy the file it replaces. The old one is still there with the same name, with a version number after it. Here is how it works:

$ DIR MYFILE.TXT

MYFILE.TXT;1

$ EDIT MYFILE.TXT  ! Make some changes to MYFILE.TXT
$ DIR MYFILE.TXT

MYFILE.TXT;2        MYFILE.TXT;1

The second file appears automatically when the editor closed the first one, without the editor’s knowledge. COPY behaves the same way, this time with the default directory output left in:

$ DIR *.TXT

Directory DISK$USER:[SALLYSMITH]

ANOTHERFILE.TXT;1   MYFILE.TXT;1

Total of 2 files.

$ COPY MYFILE.TXT ANOTHERFILE.TXT
$ DIR *.TXT

Directory DISK$USER:[SALLYSMITH]

ANOTHERFILE.TXT;2   ANOTHERFILE.TXT;1   MYFILE.TXT;1

Total of 3 files.

(OpenVMS users will notice that this time I did not remove the standard OpenVMS DIR output.)

Anything that opens a file for writing on OpenVMS has this effect, and accessing MYFILE.TXT without a version number will always return the highest-numbered version. TYPE MYFILE.TXT will display the contents of version 2 on the screen (like the Unix cat command), while TYPE MYFILE.TXT;1 explicitly gives the first version. At any time you can use the PURGE command to delete all versions except for the highest numbered, which retains its number because the version numbers just keep climbing. The maximum possible version number is 32,767, but most OpenVMS installations default to 30 or so, a number commonly found to be good among the massive community of OpenVMS administrators.

Other versioning systems

In closed source operating systems, TENEX ↗ had versioning in 1969 and later TOPS-20 ↗ took it to market, both dead by 1990. Similarly for Digital Equipment Corporation (DEC)’s RSX-11 ↗ and RSTS/E ↗ operating systems (available today under hobbyist licenses). DEC remembered about versioning and kept it in OpenVMS, which is how I came to use it years later. In open source, research literature describes the Elephant Filesystem, CVFS, VersionFS ↗ and ext3cow ↗ , none of which ever had a userbase.

What has been very successful in production is whole filesystem snapshots: open source has Btrfs ↗ and ZFS ↗ , with WAFL and Windows Shadow Copy also widely used. A snapshot freezes an entire volume at one moment, and that snapshot can be backed up. Snapshots answer a different question from what did this one file look like three saves ago.

How it feels to use

Mostly you can ignore the versing altogether. You know that the newest version is the one you get by default, and that you can’t accidentally overwrite data, and that PURGE tidies up.

When you do want an older version, you don’t even necessarily need the version number. As the VSI wiki ↗ explains:

So TYPE MYFILE.TXT;-1 gives you the file as it was before your last save, without your having to look anything up first.

Commands differ in what they do when you leave the version off, according to the user’s likely requirement:

$ DELETE MYFILE.TXT
%DELETE-E-DELVER, explicit version number or wild card required

$ DELETE MYFILE.TXT;3      ! that one
$ DELETE MYFILE.TXT;       ! the highest
$ DELETE MYFILE.TXT;*      ! the lot
$ PURGE/KEEP=2 MYFILE.TXT  ! everything below the top two

Version numbers wrap around

Left alone, versions accumulate until the limit, and when a new version would exceed the limit, the lowest surviving one is first deleted:

$ CREATE/DIRECTORY/VERSION_LIMIT=3 [.THREE]
$ SET DEFAULT [.THREE]
$ ! four rounds of editing REMY.DAT later...
$ DIR REMY.DAT

REMY.DAT;4          REMY.DAT;3          REMY.DAT;2

Version 1 vanished when version 4 was created, fully explainewd in Raymii’s walkthrough ↗ . The default limit is zero, meaning no limit. A system where every directory reports 30 or some other number means the administrator has set this limit in site policy, or, it has been set somewhere up the tree and it propagated, because a new directory inherits its parent’s default.

Versioning lives below RMS

The Record Management Services (RMS) is where a program on OpenVMS goes to open a file, so from an application’s point of view it looks like the filesystem, and it is where you would expect versioning to live. RMS presents a file as a logical collection of records in one of three organisations: sequential, relative, or indexed with keys. A COBOL program gets keyed lookup on a data file as an operating system service, where on Unix you get a stream of bytes and add the structure yourself with your choice of library abstraction.

Disk block layout, protection and versioning are all below RMS, in the Files-11 filesystem. RMS parses the ;3 off the end of a file specification and passes it down, and the number is assigned by IO$_CREATE, which is also what enforces the directory’s version limit. A program that bypasses RMS and issues $QIO directly still gets versioning, which is what makes this a filesystem feature rather than a convention that well-behaved programs follow.

In Files-11, each version is a separate file with its own File ID holding a full copy of the data, so FOO.DAT;4 has no shared disk blocks with FOO.DAT;5 even if they are substantially the same file. OpenVMS does not do delta storage or compression.

OpenVMS versioning documentation and code

1. VSI OpenVMS I/O User’s Reference Manual ↗ , chapter 1, the ACP-QIO interface (PDF ↗ ). This is the canonical, comprehensive documentation. The version-limit algorithm is given in the discussion of IO$_CREATE and IO$_ACCESS.
2. VSI wiki: File version ↗ . Brief practical document covering the number range, per-command defaults, relative versions and limits.
3. David Deley, VMS/RMS Internal Data Structures ↗ . A byte-level walk through the on-disk format, with the DUMP commands shown as he goes so you can follow along.
4. VMS File System Internals ↗ , Kirby McCoy, Digital Press 1990. The persistent structures are in chapter 2. It describes VMS 5.2, so it predates ODS-5 and isn’t quite a complete on-disk specification, but it’s close.
5. For working code, Paul Nankervis’s ODS2 reader ↗ parses directory records to produce listings on non-VMS hosts, and Carl Lydick’s ODS-2 reader ↗ does the same thing in a single C file.