free & open source

Introducing the OERu Tech Blog

dave — Thu, 08 Sep 2016 01:00:54 +0000

Introducing the OERu Tech Blog

Blog tags

free & open source

technology

kanban

devops

foss

dave Thu 08/09/2016 - 13:00

The Open Education Resource universitas (OERu) is an open organisation from top to bottom. Our entire technological infrastructure (with a couple exceptions) is built with and on Free and Open Source Software to which I normally refer as "FOSS".

We are committed to using FOSS for our infrastructure because:

it is consistent with our "open" philosophy for education - if our educational materials and all our OERu planning processes are "open on principle", then delivering them on "closed" (or, more precisely, "proprietary") technologies would be a clanging bit of cognitive dissonance. There's even a word for it: "prefigurative" - use means that are consistent with your desired end.
it allows us to use "best-of-breed" technologies (in the web-space, most innovation happens as FOSS first), combining many single-purpose solutions together in a modular way - via the open standards, open data formats, and open APIs they all support - to form a robust, feature-rich and secure communications and collaboration infrastructure. If a better solution to one of our requirements emerges, we can rapidly adapt our systems to adopt it, ensuring we are responsive and offer a fresh, state-of-the-art, well-supported and technically excellent platform.
our partner institutions collaborators become familiar with these tools, and if they find any of them valuable, they can champion their adoption by their home institutions, helped by the instructions provided by this blog or our FOSS code repositories. The money saved and flexibility gained by our partner institutions by doing so are likely to be of substantially greater value than their annual OERu membership fees!
Perhaps most importantly, none of our collaborators or learners are ever compelled to accept dubious (and usually exploitative) proprietary software Terms and Conditions to be part of our community. We respect our community's right to privacy and personal data sovereignty from the ground up.

Physical Infrastructure

Our infrastructure is currently hosted on FOSS Ubuntu and Debian Linux systems provides by four IaaS (Infrastructure as a Service) providers:

our MediaWiki infrastructure, the star of which is Wikieducator.org, is hosted on Amazon's "Elastic Compute" EC technology (located in the US, in our case), running Ubuntu Linux 14.04.
the rest of our self-hosted infrastructure, hosted on a high-specification virtual machine, "Hetzner", also running Ubuntu Linux 14.04 (based in Germany). More on the many services provided by Hetzner below.
more recently, we have created hosting for some of our new services on Azure, taking advantage of an annual grant given to qualifying non-profit organisations by Microsoft. We run Ubuntu Linux 16.04 hosts there. We ensure we do not make use of any proprietary Azure capabilities in any of our automated scripts so that we can shift hosting providers with minimal cost and inconvenience if/when Microsoft changes their free-hosting policy.
in 2018, we also received sponsored hosting capacity from NZ's own cloud service provider - whose cloud offering is entirely open source software built on open stack - Catalyst Cloud. We host our code repositories on their infrastructure. Many thanks to Catalyst!

Incidentally, the FOSS operating system, Linux, is the most widely used hosting platform on the Internet today, and Ubuntu is the most widely used "distribution" of Linux.

Externally hosted

We also make use of some externally hosted commercial FOSS services (we pay them for their services) to provide all the functionality we require:

OnlineGroups.Net for our family of mailing lists.
Mautic for our newsletter and user-engagement needs (Update 2017-06-16: due to substantial price increases for the hosted Mautic service, we are moving to a self-hosted version set up like this).
Update 2017-06-16: we have adopted a new open source Kanban planning tool, Kanboard, and we're supporting the developer by paying for the hosted service.

Web Applications

We host and maintain a number of websites "that do stuff", otherwise known as web applications. These include:

Our Course website which acts as a platform for per-course and per-cohort course websites, generated automatically via an OERu innovation: our course "snapshot" process from learning materials formulated on Wikieducator that are transformed into fully-formed, partner-institution-branded websites. Built on the WordPress blog platform, running in "multisite" mode.
Our main OERu Website - which provides information about the organisation relevant to both learners and partners. It is built on the Silverstripe Content Management System (CMS).
This Technology Blog... which is built on the Drupal CMS (version 8).

Web Services

To maintain control and flexibility, we self-host a myriad of useful web-based resources and services. These include:

our data sharing/digital artefact-storage site, comparable to having our own "Dropbox", is NextCloud.
our collaborative document editing platform is Etherpad-Lite.
our two "next generation" online forums, Community (for educators and OERu collaborators) and Forums (for learners), built on Discourse.
our chat system, a Rocket.Chat instance, similar to the proprietary Slack platform, replaces our venerable geeks-only platform, IRC (Internet Relay Chat).
our planning system, an instance of the Wekan "virtual Kanban board" (Update 2017-06-16: although we still use this a bit, we have found we prefer Kanboard)
our link shortening service is an instance of YourLS.
our issue tracking service is an instance of Mantis Bug Tracker (Update 2017-06-16: we've retired this as it wasn't quite the right fit for our users)
our integrated link sharing "course resource bank" is Semantic Scuttle.
our website usage tracking system built with Piwik.
Update 2017-06-16: we have recently set up a Mastodon instance to facilitate training our learners in the use of social networking without having to resort to a proprietary freedom-compromised platform. Here's how we did it.
Update 2017-06-16: we have a NextCloud instance, linked to a Collabora Online office suite instance, a concurrent editing application similar to Google Docs/Sheets. Howto coming soon!
Update 2017-09-22: we have a Lime Survey instance, replacing our use of Google Forms for conducting web-based surveys and polls.

Code Repositories

We have a convention of documenting (including instructions, source code, and configuration examples) all of our individual implementations FOSS implementations. We have multiple projects for public reference stored at both Bitbucket (we have repositories in both the Wikieducator and the OER Foundation projects - Bitbucket is a source code "forge" run by Atlassian in Australia) and Github (a forge run by Github in the US).

Over time, a few of us will be writing up some blog posts on specific technologies to introduce them in a gentle way to those for whom terms like "git" and "pull request" do not yet have a respectable technology-related connotation.

Our Toolbox

To build and maintain our infrastructure, we use a cornucopia of additional FOSS tools. Among these are text editors, monitoring systems, backup tools, debugging environments, "devops" platforms, container technologies, and many others. We'll no doubt cover some of these in future blog posts.

A couple noteworthy tools that all of our institutional partners should know about include:

we get all of our SSL (Secure Sockets Layer) certificates, that help keep our users' information secure and private by providing browser-to-server end-to-end encryption from Let's Encrypt, at no cost. We encourage everyone else to do likewise! Here's our Let's Encrypt howto.
OpenSSH (Secure SHell) - which ships with all Linux systems of which we're aware - it's the way we access all of our remote systems securely from anywhere.

Add new comment

Creating strong random passwords

dave — Mon, 13 Sep 2021 00:26:04 +0000

Creating strong random passwords

Blog tags

ubuntu linux

random passwords

free & open source

password keeper

dave Mon 13/09/2021 - 12:26

Throughout our Free and Open Source Software tutorials, we need to specify passwords for things. Creating random passwords is surprisingly hard, but we've found a method that's very serviceable and makes it easy to do as we all should: ensure every separate identity or service has a strong password that is unique to that identity and application (i.e. never use the same password in more than one place). We also strongly encourage you all to track your passwords using a password manager!

To generate decent random passwords, we use pwgen, which you can easily use on your server (log in - via SSH or using your hosting provider's console to your server - as root or, even better as a non-root user with sudo privileges) by typing at the terminal command prompt:

sudo apt-get install pwgen

I don't tend to use special characters in passwords stored in configuration files because they can lead to syntax parsing issues when put into configuration files. Instead, I just make them fairly long. To create a random password, I use this

pwgen -s 19 1

which returns a single 19 character-long password with a mixture of letters (lower and uppercase) and digits. A few examples: HxF0GAyS1jw63Dy3T5K avZ5qj4xt0tTS0ONyLo 43IJZbZxLrKJSegZhyR.

Note: your passwords are likely to appear, in clear text, in your terminal window after you've created them, which is a temporary security threat. Once you've got them entered where they need to be, I recommend running CTRL-L in your terminal window which will clear the visible text from past commands and give you a default command prompt (your terminal session will still remember past commands if you click up arrow).

You can create all the passwords you need to follow one of our tutorials up front and then copy and paste them somewhere useful, like into a text editor on your desktop, from which you can easily copy and paste them and track where they belong. The ones you use to log into remote services yourself should be stored in a password manager (as recommended above!).

Add new comment

Configuring a Linux server to send email via the Postfix SMTP server using an external authenticating SMTP host

dave — Mon, 02 Aug 2021 02:08:28 +0000

Configuring a Linux server to send email via the Postfix SMTP server using an external authenticating SMTP host

Blog tags

ubuntu linux

postfix

smtp

18.04

free & open source

foss

20.04

22.04

dave Mon 02/08/2021 - 14:08

Just about any and every server needs to be able to send email - whether it's end-user-email, like password recovery services for a website to emails to system administrators reporting on the status of system backups and errors. The problem is that it's non trivial (understatement) to set up a mail server properly.

This howto assumes you have a Linux server (these instructions are for Ubuntu 22.04 and 20.04, although it should work on earlier versions of Ubuntu server and Debian Linux with minor changes, and the concepts will be very similar on other Linuxen) with a static IP address, with one or more fully-qualified-domain-names (fdqn) pointing at that address, and you have SSH-based access to it. I've previously provided tips on how to get to this stage.

Authenticating SMTP

To send email, you need access to a server, somewhere on the Internet, that provides the Simple Mail Transfer Protocol (SMTP) service. It's an open standard, and for most of the history of the Internet, email services have been mostly provided by Free and Open Source Software (FOSS) tools - the first SMTP was called "Sendmail" and it was fully FOSS, and it's still in use today (although it has mostly been superseded by faster, more secure systems, the best of which are also FOSS).

At the OERu, we use the Docker-based installation of the amazing, completely FOSS MailCow project to provide our organisational email services. I might cover that set up in a future tutorial here, because MailCow makes an otherwise almost intractable problem - hosting your own email service - much more tractable. Having a MailCow set up means we can offer "full service" email for any number of domains and users and aliases with all the bells and whistles including incoming and outgoing mail with all the virus scanning (we don't really need it because we use Linux desktops, but for other folks it's useful) and dynamic spam filtering services you'd expect from a much larger operation: Team MailCow have done an amazing job in pulling together a comprehensive set of FOSS applications to provide all the conceivable requirements of a full-fledged, multi-domain email system, including shared calendaring, contacts, and webmail. A great companion to your organisation's MailCow server would be a BitWarden password safe server (also FOSS)... just sayin'.

So, now, assuming that we have a MailCow server or some other functionally equivalent SMTP service available (apparently you can do this with Gmail, if you're a paying using although because of Google's terms of use, we recommend finding a more trustworthy solution), we have the option of "authenticated SMTP" for outgoing email using credentials we can set up. For example, in MailCow, we can specify a domain we host, like say oeru.org (and for which we've defined an MX record and a few other relevant records as guided by MailCow administrative web interface). On top of that, we can specify a mailbox for a dedicated "send stuff from remote relay hosts" email address using that domain, like smtp@oeru.org, with a strong password. With that, we can securely send email using that email address as the username and that password from anywhere we have access to the Internet.

The only tricky part is that we have to ensure that whatever "reply to" email address we specify from our applications, say notifications@tech.oeru.org, is using a domain we also host on the same server, and that there's an email alias of that email address defined and set as "allow to send from smtp@oeru.org" in the MailCow interface. If we haven't made sure of that, our mail server is likely to reject sending emails with that "mismatching" email address. This is a basic spam deterrence measure, which is for the best, despite sometimes making a email system administrator's life harder.

Once we've got that (and it's easy once you've done it once or twice - I'm mostly writing this down now so I don't have to try to re-remember every time I need to set up a new server - and I hope it helps others, too), we can set up any server we control to send secure (and spam-filter-resilient) email. For what it's worth, too, MailCow uses Postfix as its SMTP server component (there're a bunch of other components, too).

Postfix SMTP with SmartHost

The first thing you need to do to create a postfix smarthost is to install the postfix application on a new server (this assumes you're logged in with a user who has "sudo" - aka admin - permissions):

sudo apt update && sudo apt install postfix bsd-mailx

During the install, you'll be asked to select a bunch of configuration parameters. Select the defaults except:

Select "Internet Site with Smarthost",
fill in the domain name for your server,
the domain name and port (in the form [smtp server domain]:[port], e.g. smtp.oeru.org:587 ) of your "smarthost" who'll be doing the authenticating SMTP for you, and
the email address to which you want to receive system-related messages.

After that's done, you can proceed.

Next Steps

For the rest of this tutorial, you'll need to do the following. First, select your text editor. I use vim, but if you're new to the command line, I recommend using nano - it's more straightforward:

EDIT=`which nano` or EDIT=`which vim`

sudo $EDIT /etc/aliases

We need to make sure the "root" user points to a real email address. Add a line at the bottom which says (replacing [your email] with your email :) )

root: [your email]

After which you'll need to convert the aliases file into a form that postfix can process, simply by running this:

sudo newaliases

Then we have to define the authentication credentials required to convince your mail server that you're you!

sudo $EDIT /etc/postfix/relay_password

The resulting file only needs one line with three bits of information:

[smtp server domain] [user name]:[password]

for example:

smtp.oeru.org smtp@oeru.org:SomeObscurePassw0rd

Then save the file and, like the aliases file, run the conversion process (which uses a slightly different mechanism):

sudo postmap /etc/postfix/relay_password

Finally, we'll edit the main configuration file for Postfix to tell it about all this stuff:

sudo $EDIT /etc/postfix/main.cf

If your SMTP server uses port 25 (the default for unencrypted SMTP) you don't have to change anything, although most people nowadays prefer to use StartTLS or otherwise encrypted transport to at least ensure that your SMTP authentication details (at least) are transferred encrypted. That means using port 587 or 465. If you're using either of those ports, find the "relayhost = [your server name]" line... and add your port number after a colon, like this

relayhost = [your server name]:[server port]

or, for example:

relayhost = smtp.oeru.org:465

Next, add the following lines at the bottom of the file:

# added to configure accessing the relay host via authenticating SMTP smtp_sasl_auth_enable = yes smtp_sasl_password_maps = hash:/etc/postfix/relay_password smtp_sasl_security_options = noanonymous
smtp_tls_security_level = encrypt

# if you're using Ubuntu prior to 20.04, uncomment (remove the #) the
# earlier line smtp_tls_security_level = may to save errors in 'postfix check' # and comment this line (by adding a # at the start) smtp_tls_wrappermode = yes

And, finally, comment out the line smtp_tls_security_level = may higher in the file - careful not to confuse it with the very similar smtpd_tls_security_level variable (note the extra '''d''' in `smtpd...`) line.

Save the file, and then check that your syntax is correct:

sudo postfix check

If it is (running the command returns no errors, and it might not return anything at all - that's a good thing!), then you can run

sudo postfix reload

to get postfix to reload its configurations and you can test out your new smarthost-configured SMTP server!

If not, the output of the check command will usually give you a helpful insight into what is wrong with your configuration... you'll also find that looking at the mail log is very helpful and offers great insights:

sudo less +G /var/log/mail.log

and if you're not able to fix it based on those, you'll find postfix is widely documented and has rich set of easily discoverable resources out there on the web - a search engine is your best resource!

Testing your outgoing email

By default, a command line application called "mail" is installed as part of the bsd-mailx package we installed alongside postfix. You can use it to send test email from the command line on your host to verify you've got things working correctly! The stuff in <> are the keys to hit at the end of the line...

$ mail you@email.domain<ENTER>

Subject: Testing from your.relay.server.domain<ENTER> Testing postfix remote host<ENTER> <CTRL-D> Cc:<ENTER>

Typing <CTRL-D> (hold down the Control or Ctrl key on your keyboard and press the "d" key) will finish your message, showing you a "CC:" field, in which you can type in other email addresses if you want to test sending to multiple addresses. When you then hit <ENTER>, it will attempt to send this email. It might take a few minutes to work its way through to the receiving email system (having to run the gauntlet of spam and virus filters on the way).

You can also always check the postfix system logs to see what postfix thinks about it using the command above. Hit <SHIFT-F> to have the log update in real time.

Done

Now you've got working outgoing email from your server. That means many higher-level web applications you might install on your infrastructure will work out-of-the-box, because what you've set up, for example, enables the default PHP email service and that used by other stacks.

Sending from Docker Containers

You can configure your server so you can reference it from services you run from Docker containers on your host. You do this by referencing the host, like via an ad hoc SMTP server on your container like msmtp, and you can just reference it as 172.17.0.1, which is the default base IP for Docker hosts from the perspective of Docker containers. You might find it's different on your particular install. In that case, you have to make your Postfix SmartHost accept email for sending from the Docker containers on that server. There're quite a few examples of that among my Docker recipes on the OERu's git repository.

Add new comment

The OERu Blog Feed Finder

dave — Sun, 14 Oct 2018 22:40:06 +0000

The OERu Blog Feed Finder

Blog tags

lida101

wenotes

blog feed finder

free & open source

rss

atom

json

dave Mon 15/10/2018 - 11:40

One of the distributed tools that the OERu makes available as part of its open source distributed digital learning environment is WEnotes. I've described what it is and how it works previously. One of the most powerful distributed tools has also been the most complicated to achieve: the ability for learners to complete assignments and reflect on their learning journeys on their own blogs, tagging posts with relevant OERu course ids (e.g. lida101 - see the Learning in a Digital Age 101 course feed for example) and then having our blog feed scanner:

know to look at the learner's blog in a timely manner,
identify the newly published appropriate blog post (i.e. with the relevant tag), and
create a WEnotes message with a reference to the blog post

for the other learners participating in the course to see.

There are a lot of moving parts in achieving this ambition, but the most unexpectedly difficult (from my perspective as a technologist) was solving the first problem: knowing how to find the learner blog feeds to monitor!

The mysterious blog feed

It turns out few people who use the web, even those who blog, are familiar with "feeds". A "feed" is a web accessible file, adhering to a well-known and publicly defined "open standard" machine readable format, which summarises and references (links to), the content on a website, in this case a blog. Most blog platforms enable them right out-of-the-box, by default. This fact isn't widely appreciated, even among active bloggers!

If you haven't seen a feed, here're some examples and an explanation of the two most common types: RSS (which stands for "Really Simple Syndication") and Atom, both of which support "tagging" content.

When we (here at the OER Foundation) first looked into offering the monitoring of learner (or educator) blogs for suitably tagged posts to include in course interaction feeds, I thought it would be a simple process of allowing learners to nominate a "Blog Feed URL" when they registered an account on our Course Site. I was quite mistaken.

Over the course of six or so months after we made a "Blog Feed URL" field available in learner profiles, entered during the registration process, we had perhaps a hundred or so learners nominate a URL. (N.B. our model allows a learner to nominate a different blog feed URL for each course in which they're participating, allowing for the possibility that they might create a new blog site for each course. Naturally, they can nominate the same blog URL for multiple courses) Of those, perhaps one in 50 was an actual valid blog feed URL. Most were not even valid blog addresses. The most common entries were either facebook.com, google.com, a wikieducator.org user profile, or the course's own url - none of which are blogs, much less blog feeds. So this was useful market research: the market does not, in general, know what a blog is or what its feed might be, if it has one.

Among other problems this creates is the fact that our automated feed monitoring code could not effectively sift the wheat from the chaff to find valid blog feeds to monitor. Clearly, a different approach was required.

Back to the drawing board

Turns out that if a learner has a blog, they generally know its web address. If provided clear instructions, they can fairly reliably go to it, and copy and paste the URL into a text field. So I decided to use this as a starting point for creating our "Blog Feed Finder" - a simple tool that helps us side step the issue of requiring a learner to find their blog feed URL - by finding it for them. Our approach was to provide a clear, well documented interface that embraces "elaboration theory", providing enough information at a glance to guide a learner, but offers further information to those who want to understand more.

Here's what it looks like:

Here's the default view of the BFF - you can enter your blog URL as per the instructions, or seek further instruction if you're not sure how to do that.

I'll copy in the URL of this very blog (and I won't even bother with the "https://" that appears at the start of it):

The URL is pasted in. Now push the "Submit" button...

After a few seconds of behind-the-scenes whirring and grinding...

We have a result from the BFF. Note that the BFF also determines the type of feed, typically RSS or Atom. Some blog platforms offer both, in which case the BFF lets you choose which one to use.

If we want to know more about what the BFF had to do to find that blog feed, we can review its output, and even ask for more information by hovering over the info icons, fo example:

Showing an elaborating pop-up, triggered by hovering over (or clicking on) an information icon.

And having found a feed, we can now update any of the courses for which we're registered to use that feed as the designated "feed to scan periodically" for blog posts tagged with the relevant course code (in brackets next to each course title, e.g. analyticsdev, lida101, mun).

Here's what it looks like after I've replaced all of my blog feed associations with the new feed URL:

This is what the interface looks like after I've elected to "Replace" each of my blog feed URLs with the newly found URL.

And, what if, instead of tech.oeru.org, I'd mistakenly entered a URL like one of those I mentioned above? Here's how the BFF responds if someone enters, say, Facebook.com as their blog URL:

This is what the BFF says to a learner if they mistakenly enter one of a handful of frequently entered non-blog URLs like facebook.com, google.com, or even course.oeru.org.

Try it now, or adapt it! It's free (and open source)

Of course, you can try it yourself right now without even needing to log in, although you get additional functionality if you're logged in and enrolled in one or more courses, like the ability to assign blog feed URLs to any courses in which you're enrolled.

Even though we're only on our first iteration of it (any software developer will tell you: no software application is ever finished) our Blog Feed Finder (BFF for short) has also been assessed by others in the online learning realm, who have given it a good going over and their seal of approval.

If anyone is curious about how it works, the entire source code is available under the terms of the GNU Affero General Public License (same as what's used by the Linux kernel, WordPress, Drupal, MediaWiki, and thousands of other well know free and open source projects) to make its review and reuse convenient and reliable!

OERu Blog Scanner is Go!

As of a few weeks ago, following the introduction of the BFF (and a bit of manual tidy-up of previously entered blog feed URL where OERu administrators used the BFF to find legitimate feed URLs where-ever possible) we now have a working scan which periodically checks all registered learner blog feeds and, if a suitably tagged post is found on any of them, incorporates a link to the post in the relevant course feed!

The course interaction feed (also linked above) includes some posts with a "blog" designator, which indicates they are references to blog posts which have been scanned by our WEnotes blog scanner (which is a different piece of software).

Loose Ends

As with any first iteration software release, there are quite a few known issues (a whole itemised, prioritised list, actually) we'd like to address for the next release. The first among them in the BFF's case is making the interface properly "responsive" (i.e. friendly for mobile device users)... At present, it doesn't do very well at that.

If you have a go with the BFF and run into trouble related to the software itself, we encourage you to let us know in the project's Issue Queue.

Final note: we will be moving our source code repositories to a properly open source code repository (our own Gitlab instance) in the coming weeks, so you may find you're being redirected from Github to a different site. Please don't be alarmed! It's by design.