foss http://tech.oeru.org/ en Introducing the OERu Tech Blog http://tech.oeru.org/intro <span class="field field--name-title field--type-string field--label-hidden">Introducing the OERu Tech Blog</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--free--open-source"> <span class="field__item-wrapper"><a href="/taxonomy/term/6" hreflang="en">free &amp; open source</a></span> </div> <div class="field__item field__item--technology"> <span class="field__item-wrapper"><a href="/taxonomy/term/7" hreflang="en">technology</a></span> </div> <div class="field__item field__item--kanban"> <span class="field__item-wrapper"><a href="/taxonomy/term/8" hreflang="en">kanban</a></span> </div> <div class="field__item field__item--devops"> <span class="field__item-wrapper"><a href="/taxonomy/term/9" hreflang="en">devops</a></span> </div> <div class="field__item field__item--foss"> <span class="field__item-wrapper"><a href="/taxonomy/term/10" hreflang="en">foss</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Thu 08/09/2016 - 13:00</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>The <a href="http://oeru.org">Open Education Resource universitas</a> (OERu) is an open organisation from top to bottom. Our entire technological infrastructure (with a <a href="/node/1" target="_blank">couple exceptions</a>) is built with and on Free and Open Source Software to which I normally refer as "FOSS".</p> <p>We are committed to using FOSS for our infrastructure because:</p> <ul><li>it is consistent with our "open" philosophy for education - if our educational materials and all our OERu planning processes are "open on principle", then delivering them on "closed" (or, more precisely, "proprietary") technologies would be a clanging bit of <a href="https://en.wikipedia.org/wiki/Cognitive_dissonance">cognitive dissonance</a>. There's even a word for it: "prefigurative" - use means that are consistent with your desired end.</li> <li>it allows us to use "best-of-breed" technologies (in the web-space, most innovation happens as FOSS first), combining many single-purpose solutions together in a modular way - via the open standards, open data formats, and open APIs they all support - to form a robust, feature-rich and secure communications and collaboration infrastructure. If a better solution to one of our requirements emerges, we can rapidly adapt our systems to adopt it, ensuring we are responsive and offer a fresh, state-of-the-art, well-supported and technically excellent platform.</li> <li>our partner institutions collaborators become familiar with these tools, and if they find any of them valuable, they can champion their adoption by their home institutions, helped by the instructions provided by this blog or our FOSS code repositories. The money saved and flexibility gained by our partner institutions by doing so are likely to be of substantially greater value than their annual OERu membership fees!</li> <li>Perhaps most importantly, none of our collaborators or learners are <em>ever</em> compelled to accept dubious (and usually exploitative) proprietary software Terms and Conditions to be part of our community. We respect our community's right to privacy and personal data sovereignty from the ground up.</li> </ul><h2>Physical Infrastructure</h2> <p>Our infrastructure is currently hosted on FOSS <a href="https://ubuntu.com">Ubuntu</a> and <a href="https://debian.org">Debian</a> Linux systems provides by four IaaS (Infrastructure as a Service) providers:</p> <ol><li>our <a href="https://www.mediawiki.org">MediaWiki</a> infrastructure, the star of which is <a href="https://wikieducator.org">Wikieducator.org</a>, is hosted on Amazon's "Elastic Compute" EC technology (located in the US, in our case), running Ubuntu Linux 14.04.</li> <li>the rest of our self-hosted infrastructure, hosted on a high-specification virtual machine, "Hetzner", also running Ubuntu Linux 14.04 (based in Germany). More on the many services provided by Hetzner below.</li> <li>more recently, we have created hosting for some of our new services on <a href="http://azure.com" title="Microsoft's Infrastructure-as-a-service Platform.">Azure</a>, taking advantage of an annual grant given to qualifying non-profit organisations by Microsoft. We run Ubuntu Linux 16.04 hosts there. We ensure we do not make use of any proprietary Azure capabilities in any of our automated scripts so that we can shift hosting providers with minimal cost and inconvenience if/when Microsoft changes their free-hosting policy.</li> <li>in 2018, we also received sponsored hosting capacity from NZ's own cloud service provider - whose cloud offering is entirely open source software built on open stack - <a href="https://catalystcloud.nz">Catalyst Cloud</a>. We host our <a href="https://git.oeru.org">code repositories</a> on their infrastructure. Many thanks to Catalyst!</li> </ol><p>Incidentally, the FOSS operating system, Linux, <a href="https://w3techs.com/technologies/details/os-unix/all/all">is the most widely used hosting platform on the Internet </a>today, and <a href="https://w3techs.com/blog/entry/ubuntu_became_the_most_popular_linux_distribution_for_web_servers">Ubuntu is the most widely used</a> "distribution" of Linux.</p> <h2>Externally hosted</h2> <p>We also make use of some externally hosted commercial FOSS services (we pay them for their services) to provide all the functionality we require:</p> <ul><li><a href="http://onlinegroups.net">OnlineGroups.Net</a> for <a href="http://groups.oeru.org">our family of mailing lists</a>.</li> <li><a href="https://mautic.net">Mautic</a> for our newsletter and user-engagement needs (Update 2017-06-16: due to substantial price increases for the hosted Mautic service, we are moving to a self-hosted version <a href="/installing-mautic-php7-fpm-docker-nginx-and-mariadb-ubuntu-1604">set up like this</a>).</li> <li>Update 2017-06-16: we have adopted a new open source Kanban planning tool, <a href="http://kanboard.net">Kanboard</a>, and we're supporting the developer by paying for the hosted service.</li> </ul><h2>Web Applications</h2> <p>We host and maintain a number of websites "that do stuff", otherwise known as web applications. These include:</p> <ul><li>Our <a href="https://course.oeru.org">Course website</a> which acts as a platform for per-course and per-cohort course websites, generated automatically via an OERu innovation: our <a href="/oeru-mediawiki-wordpress-snapshot-toolchain">course "snapshot" process</a> from learning materials formulated on Wikieducator that are transformed into fully-formed, partner-institution-branded websites. Built on the <a href="https://wordpress.org">WordPress</a> blog platform, running in "multisite" mode.</li> <li>Our main <a href="https://oeru.org">OERu Website</a> - which provides information about the organisation relevant to both learners and partners. It is built on the <a href="https://silverstripe.org">Silverstripe</a> Content Management System (CMS).</li> <li>This Technology Blog... which is built on the <a href="https://drupal.org">Drupal</a> CMS (version 8).</li> </ul><h2>Web Services</h2> <p>To maintain control and flexibility, we self-host a myriad of useful web-based resources and services. These include:</p> <ul><li>our <a href="https://cloud.oeru.org">data sharing/digital artefact-storage site</a>, comparable to having our own "Dropbox", is <a href="https://nextcloud.org">NextCloud</a>.</li> <li>our <a href="https://etherpad.oerfoundation.org">collaborative document editing</a> platform is <a href="https://github.com/ether/etherpad-lite">Etherpad-Lite</a>.</li> <li>our two "next generation" online forums, <a href="https://community.oeru.org">Community</a> (for educators and OERu collaborators) and <a href="https://forums.oeru.org">Forums</a> (for learners), built on <a href="http://www.discourse.org/">Discourse</a>.</li> <li>our <a href="https://chat.oeru.org">chat system</a>, a <a href="https://rocket.chat">Rocket.Chat</a> instance, similar to the proprietary Slack platform, replaces our venerable geeks-only platform, IRC (Internet Relay Chat).</li> <li>our <a href="https://plan.oeru.org">planning system</a>, an instance of the <a href="http://wekan.org">Wekan</a> "virtual <a href="https://en.wikipedia.org/wiki/Kanban_board">Kanban board</a>" (Update 2017-06-16: although we still use this a bit, we have found we prefer <a href="http://kanboard.net">Kanboard</a>)</li> <li>our <a href="https://oer.nz/admin">link shortening</a> service is an instance of <a href="http://yourls.org/">YourLS</a>.</li> <li>our <a href="https://mantis.oeru.org">issue tracking</a> service is an instance of <a href="https://www.mantisbt.org">Mantis Bug Tracker</a> (Update 2017-06-16: we've retired this as it wasn't quite the right fit for our users)</li> <li>our integrated <a href="https://links.oeru.org">link sharing "course resource bank"</a> is <a href="https://semanticscuttle.sourceforge.net">Semantic Scuttle</a>.</li> <li>our <a href="https://stats.oeru.org">website usage tracking</a> system built with <a href="http://piwik.org">Piwik</a>.</li> <li>Update 2017-06-16: we have recently set up a <a href="https://github.com/tootsuite/mastodon" title="The Mastodon Federated Social Network">Mastodon</a> instance to facilitate training our learners in the use of social networking without having to resort to a proprietary freedom-compromised platform. Here's <a href="/installing-mastodon-docker-compose-ubuntu-1604">how we did it</a>.</li> <li>Update 2017-06-16: we have a <a href="https://nextcloud.com" title="Open source, self-hosted web-based multi-tenented file store similar to Dropbox, but with more freedom.">NextCloud</a> instance, linked to a <a href="https://www.collaboraoffice.com/">Collabora Online office suite</a> instance, a concurrent editing application similar to Google Docs/Sheets. Howto coming soon!</li> <li>Update 2017-09-22: we have a <a href="https://limesurvey.org" title="Comprehensive open source online survey and polling tool.">Lime Survey</a> instance, replacing our use of Google Forms for conducting web-based surveys and polls.</li> </ul><h2>Code Repositories</h2> <p>We have a convention of documenting (including instructions, source code, and configuration examples) all of our individual implementations FOSS implementations. We have multiple projects for public reference stored at both Bitbucket (we have repositories in both the <a href="https://bitbucket.org/oerf/">Wikieducator</a> and the <a href="https://bitbucket.org/oerf/">OER Foundation</a> projects - Bitbucket is a source code "<a href="https://en.wikipedia.org/wiki/Forge_(software)">forge</a>" run by Atlassian in Australia) and<a href="github.com/oeru"> </a><a href="https://bitbucket.org/wikieducator/">Github</a><a href="github.com/oeru"> </a>(a forge run by Github in the US).</p> <p>Over time, a few of us will be writing up some blog posts on specific technologies to introduce them in a gentle way to those for whom terms like "git" and "pull request" do not yet have a respectable technology-related connotation.</p> <h2>Our Toolbox</h2> <p>To build and maintain our infrastructure, we use a cornucopia of additional FOSS tools. Among these are text editors, monitoring systems, backup tools, debugging environments, "<a href="https://en.wikipedia.org/wiki/DevOps">devops</a>" platforms, container technologies, and many others. We'll no doubt cover some of these in future blog posts.</p> <p>A couple noteworthy tools that all of our institutional partners should know about include:</p> <ul><li>we get all of our SSL (Secure Sockets Layer) certificates, that help keep our users' information secure and private by providing browser-to-server end-to-end encryption from <a href="https://letsencrypt.org/">Let's Encrypt</a>, at no cost. We encourage everyone else to do likewise! Here's our <a href="/protecting-your-users-lets-encrypt-ssl-certs">Let's Encrypt howto</a>.</li> <li><a href="https://en.wikipedia.org/wiki/OpenSSH">OpenSSH</a> (Secure SHell) - which ships with all Linux systems of which we're aware - it's the way we access all of our remote systems securely from anywhere.</li> </ul></div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=2&amp;2=field_blog_comments&amp;3=comment" token="CdJY_SzH-b9OZT5uqHtipOPQseJjEJj5aRpmwEa4k8M"></drupal-render-placeholder> </div> </section> Thu, 08 Sep 2016 01:00:54 +0000 dave 2 at http://tech.oeru.org OERF FOSS Tech Hosting Conventions http://tech.oeru.org/oerf-foss-tech-hosting-conventions <span class="field field--name-title field--type-string field--label-hidden">OERF FOSS Tech Hosting Conventions</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--overview"> <span class="field__item-wrapper"><a href="/taxonomy/term/76" hreflang="en">overview</a></span> </div> <div class="field__item field__item--docker"> <span class="field__item-wrapper"><a href="/taxonomy/term/16" hreflang="en">docker</a></span> </div> <div class="field__item field__item--docker-compose"> <span class="field__item-wrapper"><a href="/taxonomy/term/49" hreflang="en">docker-compose</a></span> </div> <div class="field__item field__item--mariadb"> <span class="field__item-wrapper"><a href="/taxonomy/term/48" hreflang="en">mariadb</a></span> </div> <div class="field__item field__item--nginx"> <span class="field__item-wrapper"><a href="/taxonomy/term/30" hreflang="en">nginx</a></span> </div> <div class="field__item field__item--ubuntu-linux"> <span class="field__item-wrapper"><a href="/taxonomy/term/12" hreflang="en">ubuntu linux</a></span> </div> <div class="field__item field__item--ssh"> <span class="field__item-wrapper"><a href="/taxonomy/term/77" hreflang="en">ssh</a></span> </div> <div class="field__item field__item--restic"> <span class="field__item-wrapper"><a href="/taxonomy/term/78" hreflang="en">restic</a></span> </div> <div class="field__item field__item--lets-encrypt"> <span class="field__item-wrapper"><a href="/taxonomy/term/17" hreflang="en">let&#039;s encrypt</a></span> </div> <div class="field__item field__item--git"> <span class="field__item-wrapper"><a href="/taxonomy/term/79" hreflang="en">git</a></span> </div> <div class="field__item field__item--gitlab"> <span class="field__item-wrapper"><a href="/taxonomy/term/80" hreflang="en">gitlab</a></span> </div> <div class="field__item field__item--foss"> <span class="field__item-wrapper"><a href="/taxonomy/term/10" hreflang="en">foss</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Mon 30/08/2021 - 11:44</span> <div class="field field-node--field-image field-name-field-image field-type-image field-label-hidden has-multiple"> <figure class="field-type-image__figure image-count-1"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2021-08/SSH_Session_Screenshot_about.oerfoundation.org-20210830.png?itok=yu7PbTtx" title="A snapshot of an SSH session on about.oerfoundation.org, located in San Francisco, CA, USA, and its performance stats as viewed from Christchurch, New Zealand. " data-colorbox-gallery="gallery-field_image-B_Ihf5bSVNQ" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;A snapshot of an SSH session on about.oerfoundation.org, located in San Francisco, CA, USA, and its performance stats as viewed from Christchurch, New Zealand. &quot;}"><img src="/sites/default/files/styles/medium/public/2021-08/SSH_Session_Screenshot_about.oerfoundation.org-20210830.png?itok=6cfu9o9O" width="220" height="105" alt="A snapshot of an SSH session on about.oerfoundation.org, located in San Francisco, CA, USA, and its performance stats as viewed from Christchurch, New Zealand. " typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-2"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2021-08/Editing_tech.oeru_.org_on_about.oerfoundation.org-20210830.png?itok=-EfYK9sU" title="Editing the docker-compose.yml file for this site using the Vim editor on about.oerfoundation.org via SSH." data-colorbox-gallery="gallery-field_image-B_Ihf5bSVNQ" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Editing the docker-compose.yml file for this site using the Vim editor on about.oerfoundation.org via SSH.&quot;}"><img src="/sites/default/files/styles/medium/public/2021-08/Editing_tech.oeru_.org_on_about.oerfoundation.org-20210830.png?itok=npeVOBsh" width="220" height="176" alt="Editing the docker-compose.yml file for this site using the Vim editor on about.oerfoundation.org via SSH." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-3"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2021-08/Screenshot%202021-08-30%20at%2016-21-09%20OERu.png?itok=TwAuGCsB" title="A snapshot of OERu FOSS software projects in our Gitlab instance. All are visible to anyone and we invite contributions." data-colorbox-gallery="gallery-field_image-B_Ihf5bSVNQ" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;A snapshot of OERu FOSS software projects in our Gitlab instance. All are visible to anyone and we invite contributions.&quot;}"><img src="/sites/default/files/styles/medium/public/2021-08/Screenshot%202021-08-30%20at%2016-21-09%20OERu.png?itok=3FmBKhhs" width="202" height="220" alt="A snapshot of OERu FOSS software projects in our Gitlab instance. All are visible to anyone and we invite contributions." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> </div> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>At the <a href="https://oerfoundation.org">OER Foundation</a> we host <a href="/node/34">an array</a> of Free and Open Source Software (FOSS) services. To keep things manageable, we have established a set of conventions to which we adhere, and I'm describing them here in case it's helpful to others who can either learn from, or improve on, our experience.</p> <p>To begin with, we use <a href="https://ubuntu.com">Ubuntu Linux</a> for all of our hosts. We deploy the current Long Term Support (LTS) version of Ubuntu (at the time of this writing, it's 20.04) on commodity Linux hosting services, like Digital Ocean and Hetzner.</p> <h2><a id="user-content-host-access" href="#host-access" name="host-access" class="heading-permalink" aria-hidden="true" title="Permalink"></a>Host access</h2> <p>We access all our hosts on the command line, via secure, end-to-end encrypted connections, using <a href="https://openssh.com">OpenSSH</a>. We don't use a server management tool (neither an open source one, like ISPConfig or Webmin, nor proprietary ones like Plex or CPanel). They would just slow us down. We prefer the efficiency and power afforded us by SSH. At any given time, I'm logged into a half a dozen remote systems, issuing updates, deploying code, diagnosing issues, checking resource usage, shoring up disk space, or tweaking Docker configurations.</p> <h2><a id="user-content-software-on-the-host" href="#software-on-the-host" name="software-on-the-host" class="heading-permalink" aria-hidden="true" title="Permalink"></a>Software on the host</h2> <p>On the host, we run the following:</p> <ol><li>We run <a href="https://etckeeper.branchable.com/">etckeeper</a> on each server to track changes in configuration. In general, we have these changes pushed to a central server to provide a reference in the event of a disaster (e.g. the server is compromised and needs to be 'nuked from orbit'...).</li> <li>The <a href="https://nginx.org">Nginx</a> web server - it's our reverse proxy of choice. All of our <a href="/node/11">Let's Encrypt</a> certificates terminate at the host server, greatly simplifying the deployment of the individual services running on it.</li> <li>The <a href="https://postfix.org">Postfix</a> SMTP server - it's our outgoing mailserver of choice, and we <a href="/node/28">set it up as a smarthost</a>, so that the server can send status and administrative messages to us via our <a href="https://mailcow.email">MailCow</a> email infrastructure using authenticating secure SMTP.</li> <li>The <a href="https://docs.docker.com/get-started/overview/">Docker</a> containerisation framework using the <a href="https://docs.docker.com/compose/">Docker Compose</a> system to coordinate collections of containers that work together to provide specific services. Docker allows us to run a bunch of effectively independent Linux systems, each with potentially different sets of complex software dependencies, efficiently and reliably on a single host.</li> <li>For each service, with an individual domain or subdomain name, we get a <a href="https://letsencrypt.org">Let's Encrypt</a> <a href="https://en.wikipedia.org/wiki/Transport_Layer_Security#SSL_1.0,_2.0,_and_3.0">SSL certificate</a> using the letsencrypt scripting toolchain. It is free of cost, and <a href="/node/11">easy to build into our workflow</a>, so it's a 'no-brainer' as far as we're concerned, ensuring all of our sites protect the data of their users with full encrypted links!</li> <li>For applications we deploy that depend on the MySQL database, we run a single instance of the fully MySQL-compatible <a href="https://mariadb.org/">MariaDB</a> on our host, and the Docker containers connect to it via the internal network. We do it this way to easy the process of backing up the various MariaDB databases, which would be much more fiddly if we were running a bunch of individual MariaDB or MySQL instances in Docker containers. We use MariaDB because we prefer its development model, design decisions, and the fact that it's not being run by the Oracle Corporation, which owns the MySQL project.</li> <li>We use <a href="https://restic.net/">Restic</a> to perform remote encrypted incremental file backups for the filesystems on each server. We send them to a development server we have which has oodles of disk space (<a href="https://en.wikipedia.org/wiki/Btrfs">BTRFS</a> <a href="https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_1">RAID1</a> if you're curious) for safekeeping.</li> </ol><p>When each new Ubuntu LTS release comes out, we tend to create an install image for our commodity hosting provider (most recently, Digital Ocean), which has all these things pre-installed.</p> <h2><a id="user-content-docker-deployments" href="#docker-deployments" name="docker-deployments" class="heading-permalink" aria-hidden="true" title="Permalink"></a>Docker deployments</h2> <p>For the various services we deploy using Docker - more specifically, using the very handy Docker Compose scripting toolchain, we have a bunch of conventional practices.</p> <ol><li>We put all our Docker Composer recipes in a common directory: <code>/home/docker</code> - each service goes in a directory using the domain name of that service. For example, this Tech blog, tech.oeru.org is in the directory <code>/home/docker/tech.oeru.org</code> on our server about.oerfoundation.org. Each service directory has a <code>docker-compose.yml</code> file which defines the collection of Docker containers making up each service. It also specifies the local directors in which data we want to make persistent, even surviving the removal of relevant Docker containers. In the case of this site, the containers include an Nginx webserver container which submits requests delivered to it by our hosts's Nginx reverse proxy to a PHP scripting engine container which, in turn, consults the Drupal 8 source code making up the site, and the MariaDB on the host which stores the data. There is also a PHP container which automatically runs the behind-the-scenes automated 'cron' tasks that every Drupal site requires.</li> <li>We store that per-service persistent data in a similarly named directory under <code>/home/data</code>, so this site's data is stored in <code>/home/data/tech.oeru.org</code>. That data includes, in the case of this site, an Nginx directory, with a generic Drupal website configuration, and a directory to contain all the Drupal 8 core source code and that for theme, module, and library dependencies. In a few cases, where we're using tools with specialised Docker deployment practices, like Mailcow, BigBlueButton, and Discourse, the persistent data for the site is stored under the <code>/home/docker</code> directory to avoid unnecessarily complicating our use of those tools. So our conventions are just that - not hard and fast, if there's a good reason to compromise them.</li> </ol><h2><a id="user-content-keeping-track" href="#keeping-track" name="keeping-track" class="heading-permalink" aria-hidden="true" title="Permalink"></a>Keeping track</h2> <p>For all of our service configurations, and, in particular our Docker deployments, we use <a href="https://git-scm.com">Git</a> to provide source code versioning and management. We also use it to deploy code that we have developed ourselves. Where possible, we use (and contribute back to!) upstream git repositories supplied by the communities surrounding many of the FOSS services we offer. We see that as doing our part to be good, contributing FOSS community members.</p> <p>All of our git repositories are held on our own, self-hosted <a href="https://about.gitlab.com/community/">Gitlab</a> instance: <a href="https://git.oeru.org/explore/projects">https://git.oeru.org</a> - anyone is welcome to peruse the repositories, and we invite anyone interested in contributing to <a href="/contact">request a an account</a> (give us an idea of what you're interested in doing and how you'd like to participate!).</p></div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=39&amp;2=field_blog_comments&amp;3=comment" token="mc3_yQFNlAQZsoRapw9jke6XUslZiwMrWV52Y56Nyas"></drupal-render-placeholder> </div> </section> Sun, 29 Aug 2021 23:44:43 +0000 dave 39 at http://tech.oeru.org Configuring a Linux server to send email via the Postfix SMTP server using an external authenticating SMTP host http://tech.oeru.org/configuring-linux-server-send-email-postfix-smtp-server-using-external-authenticating-smtp-host <span class="field field--name-title field--type-string field--label-hidden">Configuring a Linux server to send email via the Postfix SMTP server using an external authenticating SMTP host</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--ubuntu-linux"> <span class="field__item-wrapper"><a href="/taxonomy/term/12" hreflang="en">ubuntu linux</a></span> </div> <div class="field__item field__item--postfix"> <span class="field__item-wrapper"><a href="/taxonomy/term/66" hreflang="en">postfix</a></span> </div> <div class="field__item field__item--smtp"> <span class="field__item-wrapper"><a href="/taxonomy/term/67" hreflang="en">smtp</a></span> </div> <div class="field__item field__item--_804"> <span class="field__item-wrapper"><a href="/taxonomy/term/68" hreflang="en">18.04</a></span> </div> <div class="field__item field__item--free--open-source"> <span class="field__item-wrapper"><a href="/taxonomy/term/6" hreflang="en">free &amp; open source</a></span> </div> <div class="field__item field__item--foss"> <span class="field__item-wrapper"><a href="/taxonomy/term/10" hreflang="en">foss</a></span> </div> <div class="field__item field__item--_004"> <span class="field__item-wrapper"><a href="/taxonomy/term/75" hreflang="en">20.04</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Mon 02/08/2021 - 14:08</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>Just about any and every server needs to be able to send email - whether it's end-user-email, like password recovery services for a website to emails to system administrators reporting on the status of system backups and errors. The problem is that it's <em>non trivial</em> (understatement) to set up a mail server properly.</p> <p>This howto assumes you have a Linux server (these instructions are for Ubuntu 18.04 and 20.04, although it should work on earlier versions of Ubuntu server and Debian Linux with minor changes, and the concepts will be very similar on other Linuxen) with a static IP address, with one or more fully-qualified-domain-names (fdqn) pointing at that address, and you have SSH-based access to it. I've <a href="/setting-your-own-bitwarden-password-keeper-and-sync-server">previously provided tips</a> on how to get to this stage.</p> <h2>Authenticating SMTP</h2> <p>To send email, you need access to a server, somewhere on the Internet, that provides the <a href="https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol">Simple Mail Transfer Protocol</a> (SMTP) service. It's an open standard, and for most of the history of the Internet, email services have been mostly provided by Free and Open Source Software (FOSS) tools - the first SMTP was called "<a href="https://en.wikipedia.org/wiki/Sendmail">Sendmail</a>" and it was fully FOSS, and it's still in use today (although it has mostly been superseded by faster, more secure systems, the best of which are also FOSS).</p> <p>At the OERu, we use the <a href="https://mailcow.github.io/mailcow-dockerized-docs/" title="Dockerised MailCow">Docker-based installation of the amazing, completely FOSS MailCow project</a> to provide our organisational email services. I might cover that set up in a future tutorial here, because MailCow makes an otherwise almost intractable problem - hosting your own email service - much more tractable. Having a MailCow set up means we can offer "full service" email for any number of domains and users and aliases with all the bells and whistles including incoming and outgoing mail with all the virus scanning (we don't really need it because we use Linux desktops, but for other folks it's useful) and dynamic spam filtering services you'd expect from a much larger operation: <a href="https://mailcow.email/">Team MailCow</a> have done an amazing job in pulling together a comprehensive set of FOSS applications to provide all the conceivable requirements of a full-fledged, multi-domain email system, including shared calendaring, contacts, and webmail. A great companion to your organisation's MailCow server would be a <a href="/setting-your-own-bitwarden-password-keeper-and-sync-server">BitWarden password safe</a> server (also FOSS)... just sayin'.</p> <p>So, now, assuming that we have a MailCow server or some other functionally equivalent SMTP service available (apparently you can <a href="https://support.google.com/a/answer/2956491">do this with Gmail</a>, if you're a paying using although because of Google's terms of use, we recommend finding a more trustworthy solution), we have the option of "authenticated SMTP" for outgoing email using credentials we can set up. For example, in MailCow, we can specify a domain we host, like say <strong>oeru.org</strong> (and for which we've defined an MX record and a few other relevant records as guided by MailCow administrative web interface). On top of that, we can specify a mailbox for a dedicated "send stuff from remote relay hosts" email address using that domain, like <strong><a href="mailto:smtp@oeru.org">smtp@oeru.org</a></strong>, with a strong password. With that, we can <em>securely </em>send email using that email address as the username and that password from <em>anywhere we have access to the Internet</em>.</p> <p>The <strong>only tricky part</strong> is that we have to ensure that whatever "reply to" email address we specify from our applications, say <strong><a href="mailto:notifications@tech.oeru.org">notifications@tech.oeru.org</a></strong>, is using a domain we <em>also host on the same server, </em>and that there's an <em>email alias</em> of that email address defined and set as "allow to send from <a href="mailto:smtp@oeru.org">smtp@oeru.org</a>" in the MailCow interface. If we haven't made sure of that, our mail server is likely to reject sending emails with that "mismatching" email address. This is a basic spam deterrence measure, which is for the best, despite sometimes making a email system administrator's life harder.</p> <p>Once we've got that (and it's easy once you've done it once or twice - I'm mostly writing this down now so I don't have to try to re-remember every time I need to set up a new server - and I hope it helps others, too), we can set up any server we control to send secure (and spam-filter-resilient) email. For what it's worth, too, MailCow uses Postfix as its SMTP server component (there're a bunch of other components, too).</p> <h2>Postfix SMTP with SmartHost</h2> <p>The first thing you need to do to create a postfix <a href="https://en.wikipedia.org/wiki/Smart_host">smarthost</a> is to install the postfix application on a new server (this assumes you're logged in with a user who has "sudo" - aka admin - permissions):</p> <p><code>sudo apt update &amp;&amp; sudo apt install postfix bsd-mailx</code></p> <p>During the install, you'll be asked to select a bunch of configuration parameters. Select the defaults except:</p> <ul><li>Select "Internet Site with Smarthost",</li> <li>fill in the domain name for your server,</li> <li>the domain name and port (in the form <code>[smtp server domain]:[port]</code>, e.g. <code>smtp.oeru.org:587</code> ) of your "smarthost" who'll be doing the authenticating SMTP for you, and</li> <li>the email address to which you want to receive system-related messages.</li> </ul><p>After that's done, you can proceed.</p> <h2>Next Steps</h2> <p>For the rest of this tutorial, you'll need to do the following. First, select your text editor. I use vim, but if you're new to the command line, I recommend using nano - it's more straightforward:</p> <p><code>EDIT=`which nano`</code> or <code>EDIT=`which vim`</code></p> <p><code>sudo $EDIT /etc/aliases</code></p> <p>We need to make sure the "root" user points to a real email address. Add a line at the bottom which says (replacing [your email] with <em>your email :) </em>)</p> <p><code>root: [your email]</code></p> <p>After which you'll need to convert the aliases file into a form that postfix can process, simply by running this:</p> <p><code>sudo newaliases</code></p> <p>Then we have to define the authentication credentials required to convince your mail server that you're you!</p> <p><code>sudo $EDIT /etc/postfix/relay_password</code></p> <p>The resulting file only needs one line with three bits of information:</p> <p><code>[smtp server domain] [user name]:[password]</code></p> <p>for example:</p> <p><code>smtp.oeru.org smtp@oeru.org:SomeObscurePassw0rd</code></p> <p>Then save the file and, like the aliases file, run the conversion process (which uses a slightly different mechanism):</p> <p><code>sudo postmap /etc/postfix/relay_password</code></p> <p>Finally, we'll edit the main configuration file for Postfix to tell it about all this stuff:</p> <p><code>sudo $EDIT /etc/postfix/main.cf</code></p> <p>If your SMTP server uses port 25 (the default for <em>unencrypted</em> SMTP) you don't have to change anything, although most people nowadays prefer to use StartTLS or otherwise encrypted transport to at least ensure that your SMTP authentication details (<em>at least</em>) are transferred encrypted. That means using port 587 or 465. If you're using either of those ports, find the "relayhost = [your server name]" line... and add your port number after a colon, like this</p> <p><code>relayhost = [your server name]:[server port] </code></p> <p>or, for example:</p> <p><code>relayhost = smtp.oeru.org:465 </code></p> <p>Next, add the following lines at the bottom of the file:</p> <p><code># added to configure accessing the relay host via authenticating SMTP<br /> smtp_sasl_auth_enable = yes<br /> smtp_sasl_password_maps = hash:/etc/postfix/relay_password<br /> smtp_sasl_security_options =</code><br /><code>smtp_tls_security_level = encrypt</code></p> <p><code># add this if you're using Ubuntu 20.04, and comment out (with a "#") the </code><br /><code># earlier line smtp_tls_security_level = may to save errors in 'postfix check'<br /> # and uncomment this line (by removing the #)<br /> #smtp_tls_wrappermode = yes</code><br />  </p> <p>Save the file, and then check that your syntax is correct:</p> <p><code>sudo postfix check</code></p> <p>If it is (running the command returns no errors, and it might not return anything at all - that's a good thing!), then you can run</p> <p><code>sudo postfix reload</code></p> <p>to get postfix to reload its configurations and you can test out your new smarthost-configured SMTP server!</p> <p>If not, the output of the check command will usually give you a helpful insight into what is wrong with your configuration... you'll also find that looking at the mail log is very helpful and offers great insights:</p> <p><code>sudo less +G /var/log/mail.log</code></p> <p>and if you're not able to fix it based on those, you'll find postfix is widely documented and has rich set of easily discoverable resources out there on the web - a search engine is your best resource!</p> <h2>Testing your outgoing email</h2> <p>By default, a command line application called "mail" is installed as part of the bsd-mailx package we installed alongside postfix. You can use it to send test email from the command line on your host to verify you've got things working correctly! The stuff in &lt;&gt; are the keys to hit at the end of the line...</p> <p><code>$ mail you@email.domain&lt;ENTER&gt;</code></p> <p><code>Subject: Testing from your.relay.server.domain&lt;ENTER&gt;<br /> Testing postfix remote host&lt;ENTER&gt;<br /> &lt;CTRL-D&gt;<br /> Cc:&lt;ENTER&gt;</code></p> <p>Typing &lt;CTRL-D&gt; (hold down the Control or Ctrl key on your keyboard and press the "d" key) will finish your message, showing you a "CC:" field, in which you can type in other email addresses if you want to test sending to multiple addresses. When you then hit &lt;ENTER&gt;, it will attempt to send this email. It might take a few minutes to work its way through to the receiving email system (having to run the gauntlet of spam and virus filters on the way).</p> <p>You can also always check the postfix system logs to see what postfix thinks about it using the command above. Hit &lt;SHIFT-F&gt; to have the log update in real time.</p> <h2>Done</h2> <p>Now you've got working outgoing email from your server. That means many higher-level web applications you might install on your infrastructure will work out-of-the-box, because what you've set up, for example, enables the default PHP email service and that used by other stacks.</p> <h2>Sending from Docker Containers</h2> <p>You can configure your server so you can reference it from services you run from Docker containers on your host. You do this by referencing the host, like via an ad hoc SMTP server on your container like <a href="https://marlam.de/msmtp/">msmtp</a>, and you can just reference it as 172.17.0.1, which is the default base IP for Docker hosts from the perspective of Docker containers. You might find it's different on your particular install. In that case, you have to make your Postfix SmartHost accept email for sending from the Docker containers on that server. There're quite a few examples of that among <a href="https://git.oeru.org/explore/projects?utf8=%E2%9C%93&amp;name=docker&amp;sort=latest_activity_desc">my Docker recipes on the OERu's git repository</a>.</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=28&amp;2=field_blog_comments&amp;3=comment" token="lLRkGAi5P6j9iM99_jKG1YxvJLITyB02GoT7oM-A7oA"></drupal-render-placeholder> </div> </section> Mon, 02 Aug 2021 02:08:28 +0000 dave 28 at http://tech.oeru.org Working from home http://tech.oeru.org/working-home <span class="field field--name-title field--type-string field--label-hidden">Working from home</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--covid-19"> <span class="field__item-wrapper"><a href="/taxonomy/term/73" hreflang="en">covid-19</a></span> </div> <div class="field__item field__item--working-from-home"> <span class="field__item-wrapper"><a href="/taxonomy/term/74" hreflang="en">working from home</a></span> </div> <div class="field__item field__item--foss"> <span class="field__item-wrapper"><a href="/taxonomy/term/10" hreflang="en">foss</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Wed 18/03/2020 - 09:11</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>In light of the current global pandemic, there's lots of interest out there on "working remotely" or "working from home". We've been doing a for years, and it has a lot of advantages... here's how we do it....</p> <p>The OER Foundation (we're the charitable foundation that coordinates the <a href="https://oeru.org">OERu</a>) is a "remote organisation". There is no central office - after all only two of us work for the Foundation full time. We both work from our homes, in <a href="https://osm.org/go/r9ZYCbT?layers=N">Mosgiel</a> and <a href="https://osm.org/go/uoyJ~MZ-?layers=N">Christchurch</a>, two centres about 5 hours drive apart in the South Island of New Zealand. We also have colleagues, who dedicate part of their time to our projects, based at <a href="https://op.ac.nz">Otago Polytechnic</a> in Dunedin. Despite our small (but plucky) team, we collaborate with educators at<em> <a href="https://oeru.org/oeru-partners">40+ tertiary institution partners</a> (universities and polytechnics) in 5 continents around the world who amplify our efforts</em>...  Luckily for us, we both have the benefit of high speed internet access and a knowledge of Free and Open Source Software (FOSS) and how to make it work for our requirements - and scale for a <em>global</em> audience of educators and learners!</p> <h2>How we work</h2> <p>At the OER Foundation, we're task oriented (not time oriented - no timesheets). Every week, I have a set of standard tasks I undertake, as do my colleagues, and we have to get them done. For example, I "go around the (virtual) traps" at the start of each week to ensure all our remote systems are up-to-date and secure as possible. When those are done, we have a bunch of longer term (important-but-not-urgent) projects that are under way, scheduled in a kanban board (we use either our NextCloud's "Desk" plug-in or the very fine <a href="https;//kanboard.net">Kanboard</a> FOSS app) or by agreement.</p> <p>During the day we have frequent communication and collaboration among our small team, as well as with the broader group of OERu partner representatives - our Rocket.Chat system is our virtual meeting room and water cooler for both meeting and informal discussion.</p> <p>A few of us meet online for a video chat each day to catch up on the days work, talk strategy, and socialise a bit. We usually have our chats using Jitsi Meet, which is triggered by Rocket.Chat. Lately, we've been using Big Blue Button a lot more for our catchups, just to mix it up a little - they're both great for small groups (Big Blue Button also supports much larger groups).</p> <p>Here's a table laying out our "work from home" stack:</p> <table style="width: 90%; padding: 4px 8px; border: 0px black solid" summary="The FOSS tools we use to work remotely, all the time."><tbody><tr><th>Technology category</th> <th>Specific FOSS technology</th> <th>Replaces</th> </tr><tr><th>Desktop/Laptop Computers</th> <td>Lenovo Thinkpads</td> <td> </td> </tr><tr><th>Operating System</th> <td><a href="https://ubuntu.com/download/desktop/thank-you?version=18.04.4&amp;architecture=amd64">Ubuntu Linux</a> or <a href="https://linuxmint.com/download.php">Linux Mint</a></td> <td>Microsoft Windows, Apple MacOS</td> </tr><tr><th>Email service</th> <td><a href="https://mailcow.email/">Mailcow</a> (self-hosted - <a href="https://github.com/mailcow/mailcow-dockerized">how to do it</a>)(<a href="https://www.servercow.de/mailcow?lang=en#managed">commercial service</a>)</td> <td>Google Mail, Microsoft Outlook, Fastmail</td> </tr><tr><th>Email client (reading/composing)</th> <td><a href="https://www.thunderbird.net/en-US/">Thunderbird</a> (desktop), <a href="https://sogo.nu/about.html">SOGo</a> (webmail - built into Mailcow)</td> <td>Microsoft Outlook, Apple MailApp, Gmail, Hotmail, etc.</td> </tr><tr><th>Web browser</th> <td><a href="https://www.chromium.org/Home">Chromium</a> or <a href="https://www.mozilla.org/en-US/firefox/new/">Firefox</a> or <a href="https://brave.com/">Brave</a></td> <td>Google Chrome, Microsoft IE or Edge, Apple Safari</td> </tr><tr><th>Chat service, video conf</th> <td><a href="https://rocket.chat">Rocket.Chat</a> (self-hosted - <a href="/upgrading-rocketchat-10x-and-mongodb-40">how to do it</a>)(<a href="https://rocket.chat/pricing#cloud">commercial service</a>) - (access via browser, or <a href="https://rocket.chat/install">desktop and mobile</a> interfaces)</td> <td>Slack, Discord, Gitter, Microsoft Teams</td> </tr><tr><th>Document management and sharing</th> <td><a href="https://nextcloud.com">NextCloud Hub</a> (self-hosted - <a href="/installing-nextcloud-hub-onlyoffice-ubuntu-1804">how to do it</a>)(<a href="https://nextcloud.com/pricing/">commercial service</a> - gov't and education discounts available)</td> <td>Dropbox, Google Drive, Microsoft OneDrive</td> </tr><tr><th>Collaborative documents</th> <td>NextCloud's <a href="https://onlyoffice.com">OnlyOffice</a> integration (self-hosted - <a href="/installing-nextcloud-hub-onlyoffice-ubuntu-1804">how to do it</a>)(<a href="https://nextcloud.com/pricing/">commercial service</a> - gov't and education discounts available)</td> <td>Google Docs, MS Office 365</td> </tr><tr><th>Kanban project management</th> <td>NextCloud's <a href="https://apps.nextcloud.com/apps/deck">Deck app</a> or <a href="https://kanboard.org">Kanboard</a> (commercial service)</td> <td> <p> </p> <p>Trello, MS Project</p> </td> </tr><tr><th>Shared calendaring and contacts</th> <td>NextCloud's <a href="https://apps.nextcloud.com/apps/calendar">Calendaring</a> and <a href="https://apps.nextcloud.com/apps/contacts">Contact</a> apps</td> <td>MS Outlook, iCal, Google Calendars</td> </tr><tr><th>Scheduling (timezone aware)</th> <td>NextCloud's Poll app (part of NextCloud)</td> <td>Doodle Polls</td> </tr><tr><th>Password manager (share and personal)</th> <td><a href="https://bitwarden.com/">BitWarden</a> (self-hosted - <a href="/setting-your-own-bitwarden-password-keeper-and-sync-server">how to do it</a>)</td> <td>LastPass, 1Password, Keepassx, etc.</td> </tr><tr><th>Video Conferencing (&lt; 8 participants)</th> <td><a href="https://meet.jit.si">Jitsi Meet</a> (externally hosted) browser-based or mobile apps (all FOSS)</td> <td>Google Hangouts, Apple Facetime, Microsoft Skype</td> </tr><tr><th>Video Conferencing (webinars, hundreds of participants, breakout rooms, etc.)</th> <td><a href="https://bigbluebutton.org">Big Blue Button</a> (self-hosted)(commercial service <a href="https://moodlecloud.com/app/en/">one</a>, <a href="https://blindsidenetworks.com/services/">two</a>) browser-based, mobile apps available - if you want to try it yourself (gratis) go to the NZOSS' "<a href="https://nzoss.nz/workfromhome">Work from home</a>" page.</td> <td>Zoom</td> </tr><tr><th>Social Media</th> <td><a href="https://joinmastodon.org">Mastodon</a> (self-hosted)</td> <td>Twitter</td> </tr></tbody></table><p>Outside of that stack of software applications, we use lots of other FOSS apps on the desktop, like <a href="https://inkscape.org">Inkscape</a> (for vector drawings, replacing Adobe Illustrator), <a href="https://krita.org">Krita</a> and <a href="https://www.gimp.org">Gimp</a> (to modify images, replacing Adobe Photoshop), <a href="https://kdenlive.org">Kdenlive</a> (to edit video), <a href="https://libreoffice.org">LibreOffice</a> (to edit PDFs, documents, spreadsheets, drawings, presentations, replacing MS Office), <a href="https://www.audacityteam.org/">Audacity</a> (editing audio like voice-overs and podcasts), and many others, and many dozens of development tools to produce digital artefacts. If you're looking for a FOSS tool to do a job, <a href="/contact">let us know</a> - perhaps we can help you find one that's suitable!</p> <p>If you want to learn more about our cloud infrastructure - which involves a whole additional constellation of FOSS technologies - here's a <a href="/2018-update-oeru-technology-stack">review of our 2018 technology stack</a>, which is still current, with which we are providing a full online university...</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=31&amp;2=field_blog_comments&amp;3=comment" token="sLz-SwBuWdXbxVlQ0hryx85rnXi263GuAbWsaTp-zmk"></drupal-render-placeholder> </div> </section> Tue, 17 Mar 2020 20:11:13 +0000 dave 31 at http://tech.oeru.org