Paying off technical debt

In recent months, as we wait for the prolonged controversy affecting our shareholder to settle down, at the OER Foundation we've had the opportunity to focus on consolidating our position. And that means providing technological refinements like a cross-service 'Single Sign-On' solution (a work-in-progress for which I'll write an update in future) and paying off technical debt which starts mounting the instant any computing infrastructure or software service (on that infrastructure) is implemented. Technical debt, left unpaid, has been the bane of all modern technologists (and businesses or organisations dependent on them) since the era of the transistor began.

The ice floe analogy

I liken working in computer system and software administration to be like standing on a chunk of ice in a spring thaw ice floe (in, say, the Arctic ocean at some time in the past given that there's not much ice there these days).

The ice I'm standing on is melting, so I have to be on the look out for a suitable chunk of ice that's going to come near enough that I can jump off the check I'm on, onto the slightly large one in the floe that's generally going in the direction I want to head. Sometimes ice chunk melt faster than expected, or the other chunk doesn't pass very close and we get a bit wet while jumping between them. Other times, the Fates smile on us and the chunks actually come into contact and we can step across in a dignified manner. Sometimes, no chunk comes near enough, and we go down with the 'ship'. The key message: never stop moving and keep your eyes on the horizon looking for the next suitable chunk!

In this technical debt analogy, each chunk of ice in the floe is a technology platform... They all have a limited life - some last longer than others, and sometimes the ones you think will last don't, and visa versa... so it's a black art making predictions. That said, I think our (OER Foundation's) ability to choose durable libre technologies has been remarkably good thus far. Long may it continue.

Maintaining VPSs

Under the auspices of the OER Foundation, we maintain quite a few 'Virtual Private Servers' (VPSs) - fourteen, to be exact - that each host one or more of our crucial services. Nowadays most of our commodity cloud Linux hosting services are provided by Hetzner, a German-owned cloud services company with hosting facilities in Europe and North America, which are 'aging' from the moment they're commissioned. Our normal practice is to deploy the most recent 'Long-Term Support' (LTS) version of Ubuntu Linux on the VPSs - this provides us with a minimum 5 years of support from Canonical and the Ubuntu community, built on the brilliant packages created by the broader Debian community. With the launch of the latest Ubuntu LTS release in April 2024 (the 24.04 release, known as 'Noble Numbat'), a whole new round of 'paying technical debt' can begin. Any new server we commission will be built on top of it.

Existing servers, assuming they're not on Ubuntu 18.04 (we decommissioned our last one last year) or 20.4 (which will go out of support next year, giving us a year to migrate - there're only four of those), have Ubuntu-community-provided updates applied weekly. That helps minimise the vulnerability of any of our VPSs to attacks via known vulnerabilities - which is what affects most businesses and organisations that get 'pwned' or have known vulnerabilities in their systems exploited, often the detriment of their customers or users and can sometimes be ruinous. They haven't been managing their technical debt - few businesses and organisations do.

The libre advantage over proprietary

It is worth noting that because all of our software and services are libre (i.e. Free and Open Source) software, we have few (if any) technical - and no cost - barriers to keeping our software up-to-date.

In the proprietary world there're conflicting pressures of the cost of new versions of software vs. the older software still being serviceable. In most instances where businesses and organisations depend on proprietary software, a quiet rot sets in due to unmanaged technical debt, where a cascade of system dependencies, usually triggered by the fiscal priority of retaining an 'old' version of paid-for software long after its vendor has stopped supporting it, leads to a need for those organisations to stay on old outdated software platforms that have well known security vulnerabilities.

For example, consider all the companies and institutions that still use old versions of Microsoft Windows because some crucial bespoke business application was too expensive to upgrade to support a newer version of Windows. These are the sorts of organisations that get raked over the coals by malware and ransomware creators.

Keeping services current

A messier and more pressing issue thank keeping our VPSs on the latest security updates is keeping the array or services - running on top of the Ubuntu operating system - up-to-date. Each of the many services we run is being developed by its own community, and, in turn, has its own software dependencies (like software libraries, database engines, caching systems, authentication frameworks, interface libraries, content management systems, etc.) upon which it relies. Each one of these (a complex service might have hundreds of discrete software dependencies) is being developed by its own community of interested developers (or, in some cases, a single developer) a some rate. Some are blindingly fast as they're in active development, some barely move because they're mature code that does its job well, and others move in fits and starts as developers come and go or have greater or lesser availability due to other life commitments or pressures.

Most of our services can be usefully upgraded every few weeks, others every few months, and a few only perhaps every year or two. It depends on the cadence of their development communities and on how crucial they are to our priorities.

For example, every week or so, I apply updates to each of our VPSs. Every couple weeks or so, I apply updates to our many WordPress sites, minor version upgrades to our Drupal sites, and updates to our Gitlab instance. Every month or two I apply upgrades to our Discoursen, RocketChats, and our Moodle instances. Every three or four months I update our Mastodon instance, our VaultWarden instances, and other sites like our PixelFed, PeerTube, OwnCast, Mobilizon, and Matrix instances. Every six months to a year, I look at a few of our more obscure instances, like our Mautic, WriteFreely, HedgeDoc, and RustDesk instances, and various other less rapidly changing or less critical services.

Laggards

Our MediaWiki systems, the English and Spanish-language versions of WikiEducator have thus far resisted upgrading due to their substantial historical customisation and very peculiar requirements... that's a longer-term work-in-progress.

Our Drupal-based platforms periodically have 'major-version' upgrades available for them. In the past year, the Drupal developer community urged people to migrate from the Drupal 8 platform to Drupal 9, which was a relatively complex process, and then, quite quickly, decided to 'end-of-life' Drupal 9 (i.e. providing no ongoing security updates) to provide a strong incentive for those on Drupal 9 to upgrade to Drupal 10, which the community had engineered with longer-term support in mind (as well as less wrenching upgrades to Drupal 11).

Inspiration and living up to cultural norms

This article was actually spurred on by my thoughts as I was upgrading the OER Foundation's two Drupal sites (recently upgraded from Drupal 8 to Drupal 9, and now upgraded to Drupal 10), namely our H5P Studio site this very Tech Blog. Each of those Drupal upgrades too more than a day to achieve, but, having been through Drupal 6 to Drupal 7 and Drupal 7 to Drupal 8 upgrades in the past, these were a breeze by comparison - but I didn't know that for sure until I finished them. Prior to that, I was dreading them. I'm pleased I prioritised and finished them.

I occurred to me during that process that managing 'technical debt' is a 'practice'. The more we do it, and the more we take ownership of that problem (and avoid/deny it, like so many people and organisations do, putting their heads in the sand to their inevitable regret) and build that practice into our daily, weekly, and monthly routines, the better we get at managing it. After a while, it just becomes a positive habit. And goodness knows it's nice to have some habits we can be proud of. If it's a habit long enough, it becomes an expectation and even part of our organisational culture. That's what we're after. Even though our technical debt 'balance' never goes down to zero, it's gratifying to see it drop sharply every now and again, as it did this past couple weeks, and now it's also culturally sensitive and appropriate within our Foundation!