OERu Technology Blog http://tech.oeru.org/rss.xml en Introducing the OERu Tech Blog http://tech.oeru.org/intro <span class="field field--name-title field--type-string field--label-hidden">Introducing the OERu Tech Blog</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--free--open-source"> <span class="field__item-wrapper"><a href="/taxonomy/term/6" hreflang="en">free &amp; open source</a></span> </div> <div class="field__item field__item--technology"> <span class="field__item-wrapper"><a href="/taxonomy/term/7" hreflang="en">technology</a></span> </div> <div class="field__item field__item--kanban"> <span class="field__item-wrapper"><a href="/taxonomy/term/8" hreflang="en">kanban</a></span> </div> <div class="field__item field__item--devops"> <span class="field__item-wrapper"><a href="/taxonomy/term/9" hreflang="en">devops</a></span> </div> <div class="field__item field__item--foss"> <span class="field__item-wrapper"><a href="/taxonomy/term/10" hreflang="en">foss</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Thu 08/09/2016 - 13:00</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>The <a href="http://oeru.org">Open Education Resource universitas</a> (OERu) is an open organisation from top to bottom. Our entire technological infrastructure (with a <a href="/node/1" target="_blank">couple exceptions</a>) is built with and on Free and Open Source Software to which I normally refer as "FOSS".</p> <p>We are committed to using FOSS for our infrastructure because:</p> <ul><li>it is consistent with our "open" philosophy for education - if our educational materials and all our OERu planning processes are "open on principle", then delivering them on "closed" (or, more precisely, "proprietary") technologies would be a clanging bit of <a href="https://en.wikipedia.org/wiki/Cognitive_dissonance">cognitive dissonance</a>. There's even a word for it: "prefigurative" - use means that are consistent with your desired end.</li> <li>it allows us to use "best-of-breed" technologies (in the web-space, most innovation happens as FOSS first), combining many single-purpose solutions together in a modular way - via the open standards, open data formats, and open APIs they all support - to form a robust, feature-rich and secure communications and collaboration infrastructure. If a better solution to one of our requirements emerges, we can rapidly adapt our systems to adopt it, ensuring we are responsive and offer a fresh, state-of-the-art, well-supported and technically excellent platform.</li> <li>our partner institutions collaborators become familiar with these tools, and if they find any of them valuable, they can champion their adoption by their home institutions, helped by the instructions provided by this blog or our FOSS code repositories. The money saved and flexibility gained by our partner institutions by doing so are likely to be of substantially greater value than their annual OERu membership fees!</li> <li>Perhaps most importantly, none of our collaborators or learners are <em>ever</em> compelled to accept dubious (and usually exploitative) proprietary software Terms and Conditions to be part of our community. We respect our community's right to privacy and personal data sovereignty from the ground up.</li> </ul><h2>Physical Infrastructure</h2> <p>Our infrastructure is currently hosted on FOSS <a href="https://ubuntu.com">Ubuntu Linux</a> systems provides by three IaaS (Infrastructure as a Service) providers:</p> <ol><li>our <a href="https://www.mediawiki.org">MediaWiki</a> infrastructure, the star of which is <a href="https://wikieducator.org">Wikieducator.org</a>, is hosted on Amazon's "Elastic Compute" EC technology (located in the US, in our case), running Ubuntu Linux 14.04.</li> <li>the rest of our self-hosted infrastructure, hosted on a high-specification virtual machine, "Hetzner", also running Ubuntu Linux 14.04 (based in Germany). More on the many services provided by Hetzner below.</li> <li>more recently, we have created hosting for some of our new services on <a href="http://azure.com" title="Microsoft's Infrastructure-as-a-service Platform.">Azure</a>, taking advantage of an annual grant given to qualifying non-profit organisations by Microsoft. We run Ubuntu Linux 16.04 hosts there. We ensure we do not make use of any proprietary Azure capabilities in any of our automated scripts so that we can shift hosting providers with minimal cost and inconvenience if/when Microsoft changes their free-hosting policy.</li> </ol><p>Incidentally, the FOSS operating system, Linux, <a href="https://w3techs.com/technologies/details/os-unix/all/all">is the most widely used hosting platform on the Internet </a>today, and <a href="https://w3techs.com/blog/entry/ubuntu_became_the_most_popular_linux_distribution_for_web_servers">Ubuntu is the most widely used</a> "distribution" of Linux.</p> <h2>Externally hosted</h2> <p>We also make use of some externally hosted commercial FOSS services (we pay them for their services) to provide all the functionality we require:</p> <ul><li><a href="http://onlinegroups.net">OnlineGroups.Net</a> for <a href="http://groups.oeru.org">our family of mailing lists</a>.</li> <li><a href="https://mautic.net">Mautic</a> for our newsletter and user-engagement needs (Update 2017-06-16: due to substantial price increases for the hosted Mautic service, we are moving to a self-hosted version <a href="/installing-mautic-php7-fpm-docker-nginx-and-mariadb-ubuntu-1604">set up like this</a>).</li> <li>Update 2017-06-16: we have adopted a new open source Kanban planning tool, <a href="http://kanboard.net">Kanboard</a>, and we're supporting the developer by paying for the hosted service.</li> </ul><h2>Web Applications</h2> <p>We host and maintain a number of websites "that do stuff", otherwise known as web applications. These include:</p> <ul><li>Our <a href="https://course.oeru.org">Course website</a> which acts as a platform for per-course and per-cohort course websites, generated automatically via an OERu innovation: our <a href="/oeru-mediawiki-wordpress-snapshot-toolchain">course "snapshot" process</a> from learning materials formulated on Wikieducator that are transformed into fully-formed, partner-institution-branded websites. Built on the <a href="https://wordpress.org">WordPress</a> blog platform, running in "multisite" mode.</li> <li>Our main <a href="https://oeru.org">OERu Website</a> - which provides information about the organisation relevant to both learners and partners. It is built on the <a href="https://silverstripe.org">Silverstripe</a> Content Management System (CMS).</li> <li>This Technology Blog... which is built on the <a href="https://drupal.org">Drupal</a> CMS (version 8).</li> </ul><h2>Web Services</h2> <p>To maintain control and flexibility, we self-host a myriad of useful web-based resources and services. These include:</p> <ul><li>our <a href="https://cloud.oeru.org">data sharing/digital artefact-storage site</a>, comparble to having our own "Dropbox", is <a href="https://owncloud.org">ownCloud</a>.</li> <li>our <a href="https://etherpad.oerfoundation.org">collaborative document editing</a> platform is <a href="https://github.com/ether/etherpad-lite">Etherpad-Lite</a>.</li> <li>our two "next generation" online forums, <a href="https://community.oeru.org">Community</a> (for educators and OERu collaborators) and <a href="https://forums.oeru.org">Forums</a> (for learners), built on <a href="http://www.discourse.org/">Discourse</a>.</li> <li>our <a href="https://chat.oeru.org">chat system</a>, a <a href="https://rocket.chat">Rocket.Chat</a> instance, similar to the proprietary Slack platform, replaces our venerable geeks-only platform, IRC (Internet Relay Chat).</li> <li>our <a href="https://plan.oeru.org">planning system</a>, an instance of the <a href="http://wekan.org">Wekan</a> "virtual <a href="https://en.wikipedia.org/wiki/Kanban_board">Kanban board</a>" (Update 2017-06-16: although we still use this a bit, we have found we prefer <a href="http://kanboard.net">Kanboard</a>)</li> <li>our <a href="https://oer.nz/admin">link shortening</a> service is an instance of <a href="http://yourls.org/">YourLS</a>.</li> <li>our <a href="https://mantis.oeru.org">issue tracking</a> service is an instance of <a href="https://www.mantisbt.org">Mantis Bug Tracker</a> (Update 2017-06-16: we've retired this as it wasn't quite the right fit for our users)</li> <li>our integrated <a href="https://links.oeru.org">link sharing "course resource bank"</a> is <a href="https://semanticscuttle.sourceforge.net">Semantic Scuttle</a>.</li> <li>our <a href="https://stats.oeru.org">website usage tracking</a> system built with <a href="http://piwik.org">Piwik</a>.</li> <li>Update 2017-06-16: we have recently set up a <a href="https://github.com/tootsuite/mastodon" title="The Mastodon Federated Social Network">Mastodon</a> instance to facilitate training our learners in the use of social networking without having to resort to a proprietary freedom-compromised platform. Here's <a href="/installing-mastodon-docker-compose-ubuntu-1604">how we did it</a>.</li> <li>Update 2017-06-16: we have a <a href="https://nextcloud.com" title="Open source, self-hosted web-based multi-tenented file store similar to Dropbox, but with more freedom.">NextCloud</a> instance, linked to a <a href="https://www.collaboraoffice.com/">Collabora Online office suite</a> instance, a concurrent editing application similar to Google Docs/Sheets. Howto coming soon!</li> <li>Update 2017-09-22: we have a <a href="https://limesurvey.org" title="Comprehensive open source online survey and polling tool.">Lime Survey</a> instance, replacing our use of Google Forms for conducting web-based surveys and polls.</li> </ul><h2>Code Repositories</h2> <p>We have a convention of documenting (including instructions, source code, and configuration examples) all of our individual implementations FOSS implementations. We have multiple projects for public reference stored at both Bitbucket (we have repositories in both the <a href="https://bitbucket.org/oerf/">Wikieducator</a> and the <a href="https://bitbucket.org/oerf/">OER Foundation</a> projects - Bitbucket is a source code "<a href="https://en.wikipedia.org/wiki/Forge_(software)">forge</a>" run by Atlassian in Australia) and<a href="github.com/oeru"> </a><a href="https://bitbucket.org/wikieducator/">Github</a><a href="github.com/oeru"> </a>(a forge run by Github in the US).</p> <p>Over time, a few of us will be writing up some blog posts on specific technologies to introduce them in a gentle way to those for whom terms like "git" and "pull request" do not yet have a respectable technology-related connotation.</p> <h2>Our Toolbox</h2> <p>To build and maintain our infrastructure, we use a cornucopia of additional FOSS tools. Among these are text editors, monitoring systems, backup tools, debugging environments, "<a href="https://en.wikipedia.org/wiki/DevOps">devops</a>" platforms, container technologies, and many others. We'll no doubt cover some of these in future blog posts.</p> <p>A couple noteworthy tools that all of our institutional partners should know about include:</p> <ul><li>we get all of our SSL (Secure Sockets Layer) certificates, that help keep our users' information secure and private by providing browser-to-server end-to-end encryption from <a href="https://letsencrypt.org/">Let's Encrypt</a>, at no cost. We encourage everyone else to do likewise! Here's our <a href="/protecting-your-users-lets-encrypt-ssl-certs">Let's Encrypt howto</a>.</li> <li><a href="https://en.wikipedia.org/wiki/OpenSSH">OpenSSH</a> (Secure SHell) - which ships with all Linux systems of which we're aware - it's the way we access all of our remote systems securely from anywhere.</li> </ul></div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=2&amp;2=field_blog_comments&amp;3=comment" token="CdJY_SzH-b9OZT5uqHtipOPQseJjEJj5aRpmwEa4k8M"></drupal-render-placeholder> </div> </section> Thu, 08 Sep 2016 01:00:54 +0000 dave 2 at http://tech.oeru.org Creating Simple, Semantic HTML Markup from a Google Doc http://tech.oeru.org/creating-simple-semantic-html-markup-google-doc <span class="field field--name-title field--type-string field--label-hidden">Creating Simple, Semantic HTML Markup from a Google Doc</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Wed 23/05/2018 - 12:55</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>Google Docs are great for allowing people to collaboratively build a document, with the ability for people to suggest (and discuss) changes and view revisions and a variety of other useful behaviours. At present, I'm sad to admit, <a href="/installing-nextcloud-and-collabora-office-online-docker-ubuntu-1604">CollaboraOffice</a> isn't quite in the same level (although it's catching up quickly!).</p> <h2>The Problem</h2> <p>A big problem with Google Docs (shared, incidentally, with Microsoft Word and even LibreOffice/CollaboraOffice) is that it's almost impossible to convert a collaboratively created document into a form suitable for publishing on the web.</p> <p>Yes, you can go to File-&gt;Download as...-&gt;Web Page (html, zipped)... but that results in an HTML document that is chock full of badness. Google Docs (and the other word processing packages) all assume you want your document to render on the web <em>exactly</em> like it does in their app. In my experience, is<strong> impossible to get clean, semantic HTML out of any word processing platform</strong> without all the nasty in-line styling and formatting kruft that these applications assume you want.</p> <p>Anyone (like me) who's had to work with non-technical people wanting to manage their own websites, via a content management system (CMS) like Wordpress or Drupal or any one of hundreds of others, can attest to the fact that the average person's first tool for creating content is always their word processor. This, of course, is probably the least appropriate tool, because it</p> <ol><li>creates an expectation of  control and formatting that is invalid on the web, and</li> <li>tries to control the content that is copied and pasted into the CMS' editing interface, which is entirely counter-productive.</li> </ol><p>All you really want is very simple, semantically marked-up content, <em>without any styling whatsoever</em>. The styling for CMS content is typically applied by the theme, rather than the editor. The purpose behind that is to ensure that all the site's content is <em>consisten</em>t in its look-and-feel and <em>the whole lot can be changed with simple tweaks to that theme. </em>Problem is, most people neither understand that nor care - in fact, few people will even know what I mean by "semantically marked-up": simply put, it's the idea that the content is simply marked up to show what type of content it is, e.g. a title, a paragraph, a list, a table, something requiring emphasis (usually achieved by <em>italics</em>) or strong presentation (usually achieved by <strong>bold</strong>).</p> <h2>An Illustrated Example</h2> <p>Here's an example of a document</p> <figure role="group" class="caption caption-img align-center"><img alt="A screenshot of a Google Document used in this example." data-entity-type="file" data-entity-uuid="0a945b14-ffa1-45ae-aea7-b5685e3d6a36" src="/sites/default/files/inline-images/ScreenshotGoogleDocsSample.png" /><figcaption>Screenshot of a Google Doc containing some text with basic formatting.</figcaption></figure><p>Here's the simple, clean semantic mark-up we <em>want</em> to represent this content:</p> <p><blockcode style="html"><code>&lt;h1&gt;Privacy notice&lt;/h1&gt;<br /> &lt;p&gt;The Open Education Resource universitas (OERu) privacy notice provides a simple and concise summary to explain our treatment of personal information from OERu learners and users of the websites hosted by the OER Foundation (OERF).&lt;/p&gt;<br /> &lt;p&gt;This privacy notice complies with our Privacy Policy and Terms of Service Policy and applies to you when you decide to use our services.&lt;/p&gt;<br /> &lt;p&gt;We collect information when users visit websites hosted by the OERF. This includes IP addresses which are unique numbers that identify specific internet connections, and sometimes specific computers and devices that visit our sites.&lt;/p&gt;<br /> &lt;h2&gt;Definitions&lt;/h2&gt;<br /> &lt;p&gt;In the context of this policy&lt;/p&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;&lt;p&gt;Personal data refers to information related to an identified or identifiable natural person used for data processing by the OERF.&lt;/p&gt;&lt;/li&gt;<br /> &lt;/ul&gt; </code></blockcode></p> <p>Here's the mark-up we <em>get</em> if we copy and paste from Google Docs (you'll get similar results copy-and-pasting from any word processor):</p> <p><blockcode style="html"><code>&lt;p&gt;&amp;nbsp;&lt;/p&gt;<br /><br /> &lt;h1 dir="ltr" style="line-height:1.38;margin-top:20pt;margin-bottom:6pt;"&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:20pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;Privacy notice&lt;/span&gt;&lt;/b&gt;&lt;/h1&gt;<br /><br /> &lt;p&gt;&amp;nbsp;&lt;/p&gt;<br /><br /> &lt;p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;The Open Education Resource universitas (OERu) privacy notice provides a simple and concise summary to explain our treatment of personal information from OERu learners and users of the websites hosted by the OER Foundation (OERF).&lt;/span&gt;&lt;/b&gt;&lt;/p&gt;<br /><br /> &lt;p&gt;&amp;nbsp;&lt;/p&gt;<br /><br /> &lt;p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;This privacy notice complies with our Privacy Policy and Terms of Service Policy and applies to you when you decide to use our services. &lt;/span&gt;&lt;/b&gt;&lt;/p&gt;<br /><br /> &lt;p&gt;&amp;nbsp;&lt;/p&gt;<br /><br /> &lt;p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;We collect information when users visit websites hosted by the OERF. This includes IP addresses which are unique numbers that identify specific internet connections, and sometimes specific computers and devices that visit our sites.&lt;/span&gt;&lt;/b&gt;&lt;/p&gt;<br /><br /> &lt;h2 dir="ltr" style="line-height:1.38;margin-top:18pt;margin-bottom:6pt;"&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:16pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;Definitions&lt;/span&gt;&lt;/b&gt;&lt;/h2&gt;<br /><br /> &lt;p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;In the context of this policy &lt;/span&gt;&lt;/b&gt;&lt;/p&gt;<br /><br /> &lt;p&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;Personal data&lt;/span&gt;&lt;span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt; refers to information related to an identified or identifiable natural person used for data processing by the OERF. &lt;/span&gt;&lt;/b&gt;&lt;/p&gt;<br /><br /> &lt;p&gt;&amp;nbsp;&lt;/p&gt;<br /><br /> &lt;p&gt;&amp;nbsp;&lt;/p&gt;<br /><br /> &lt;p&gt;&amp;nbsp;&lt;/p&gt;</code></blockcode></p> <p>As you can see, it's fairly impenetrable (it's not written for human's eyes :) ). It doesn't even use proper semantic tags to represent the content. Instead it uses "ad hoc" styling to achieve a superficial resemblance to what we want to see, and in doing so, it also generates huge amounts of totally awful mark-up.</p> <h2>The (Free and Open Source) Solution</h2> <p>Luckily, after a lot of dives down rabbit holes, I have emerged with a free and open source software (FOSS) solution. It involves an incredible software tool called "<a href="https://pandoc.org" title="The Pandoc Free and Open Source tool">Pandoc</a>" that can be installed on any Windows, MacOS, or Linux computer. Because I use FOSS where ever possible, I'll describe how I achieved a solution on Linux. </p> <p>In addition to providing clean semantic HTML, preserving links and structure (including tables), Pandoc can be asked to build a "Table of Contents" for the HTML, too, based on titles (H1, H2, etc.) to the level desired.</p> <p>The steps:</p> <ol><li>download a "Web Page (html, zipped)" version of the Google Document and unzip it in to a file, in my case,<code> PrivacyNotice.html</code></li> <li>install Pandoc on my computer:<code> <code>sudo apt install pandoc</code></code></li> <li>in the directory in which I have<code> PrivacyNotice.html,</code> I ran<code> <code>pandoc -r html -w docbook --email-obfuscation=none -S -s PrivacyNotice.html | pandoc -r docbook -w html --toc --toc-depth=2 --email-obfuscation=none -S -s -o PrivacyNotice-semantic.html -</code> </code>producing a semantically marked up version of the file PrivacyNotice-semantic.html, and, in this case, with a linked Table of Contents automatically created. If you prefer not to have the Table of Contents, run this instead:<code> <code>pandoc -r html -w docbook --email-obfuscation=none -S -s PrivacyNotice.html | pandoc -r docbook -w html --email-obfuscation=none -S -s -o PrivacyNotice-semantic.html -</code> </code>Note: the trailing "-" is important!</li> </ol><p>What this Pandoc process does, is to convert the nasty HTML provided by the Google Doc "Web Page" export into a different (and very semantically structured) format called DocBook, which strips out all the unnecessary styling. We then convert it back to HTML, preserving the semantic structure, and removing the stuff we don't want.</p> <p>To simplify matters, you can also write a script or shell alias which achieves the same thing in a single command. For example, here's the alias I created in my ~/.bashrc (the hidden bash shell configuration file in my home directory, "~"):</p> <p><blockcode><code><code>alias tosemtoc='pandoc -r html -w docbook --email-obfuscation=none -S -s $1 | pandoc -r docbook -w html --toc --toc-depth=2 --email-obfuscation=none -S -s -o $1 -'<br /> alias tosem='pandoc -r html -w docbook --email-obfuscation=none -S -s $1 | pandoc -r docbook -w html --email-obfuscation=none -S -s -o $1 -'</code></code></blockcode></p> <p>(if you add this to your .bashrc, you can make your bash session aware of it either by logging out or in again, or by running <code>. ~/.bashrc</code> at your command prompt)</p> <p>With these two lines I can achieve the same thing as the above complex command like this (the semantic HTML output will be put into the same file I started with in this case, rather than a separate file - no big loss on that Google Doc-produced HTML travesty!):</p> <p>With Table of Contents: <code>tosemtoc PrivacyNotice.html</code></p> <p>Without the Table of Contents: <code>tosem PrivacyNotice.html</code></p> <p>Pretty straightforward. And it takes less than a second to run. Note that the result is a "well formed" complete HTML document, so you can view it happily in your browser, e.g. type <code>file:///path/to/your/PrivacyNotice.html</code> file into your browser's location bar as a URL.</p> <p>If you want to see the final result of this, have a look at our new GDPR-aware <a href="https://oeru.org/privacy-notice" title="The OERu Privacy Notice">Privacy Notice</a>!</p> <p>Hope this saves someone a lot of frustration and unnecessary remedial mark-up editing!</p> <p><code> </code></p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=20&amp;2=field_blog_comments&amp;3=comment" token="Tk3zaP2726_ZI8MqlLbML0HqaI7MDjD7WmCCTtyS-RM"></drupal-render-placeholder> </div> </section> Wed, 23 May 2018 00:55:19 +0000 dave 20 at http://tech.oeru.org Insight: what does the Open Source Technologist at the OER Foundation do? http://tech.oeru.org/insight-what-does-open-source-technologist-oer-foundation-do <span class="field field--name-title field--type-string field--label-hidden">Insight: what does the Open Source Technologist at the OER Foundation do?</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Fri 18/05/2018 - 13:17</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>A few weeks ago, I got an unexpected (but very flattering!) request from a friend and colleague's son, Felix. He wanted to interview me about my work (at 11, he thinks he might want to become a software developer, too, when he's older) as material he could present at an up-coming <a href="https://en.wikipedia.org/wiki/PechaKucha">PechaKucha</a> session at his school... Wow, we didn't do PechaKucha when I was a kid :)</p> <p>I agreed to respond, and, due to logistical challenges, that response took the form of an email answer his questions. After sending it to him, it occurred to me that it might be of interest to some folks in the OERu community and beyond, so I asked if he'd be ok with me publishing the results under a CC-By license. He was agreed, and so here it is (note, I've made a few minor edits and additions to my original response).</p> <dl><dt>1. What do you do in an average day at work?</dt> <dd> <p>My days are generally very diverse, but I can give you an idea of a typical "business as usual" day... (as opposed to, say, when I'm overseas at one of our very enjoyable international OERu partner meetings).</p> <p>Of late, my usual routine Usually, I make myself a coffee and go up to the garage loft in my house. I sit down at my computer and log in, immediately checking my email and various chat channels to see if there're any urgent problems to look into (my work is global, so there's always someone using the systems I manage).</p> <p>If there something urgent, like a system not running the way it's meant to, I get right onto making it right - our systems are meant to serve hundreds (and eventually tens of thousands or more!) learners daily, so they need to be reliable and responsive.</p> <p>On Mondays, I "go around the traps", applying updates to servers that I'm responsible for in Germany, the US, NZ, Australia, and Singapore - they support a bunch of websites and services that are part of our "Digital Learning Environment". I also maintain some web systems for the NZ Open Source Society (which I sometimes include in "the traps" - my job title with the OER Foundation who employ me, is "Open Source Technologist").</p> <p>In amongst the updates, I sometimes do some research on new technologies I need to deploy as part of my "continual improvement" process (I can do this without needing to concentrate intensely, so I can do it in short bursts well suited to the start and stop nature of the server update process) - finding ways to make it easier for me, as one half of a 2-person team trying to build a global service, to manage all the many systems I'm responsible for. Today, for instance, I'm looking at setting up an open source service monitoring system which will alert me if any of the systems I'm interested in have problems, go do, or even if they're slow... I'm also interested in monitoring whether any of them are running short of hard disk space, as recovering from a full hard disk is often very painful and messy, and can result in both long downtimes and lost user data (something we work <em>very</em> hard to avoid).</p> <p>Most days I also do quite a bit of software development. My role includes the responsibility for building strategic software capabilities. Today that will include working on a plugin for the <a href="https://wordpress.org">WordPress</a> blog platform (like <a href="https://drupal.org">Drupal</a>, but somewhat simpler, but also open source) that we use to provide our course materials to learners. When a learner logs in, and asks to take part in a particular course, the plugin I've written, automatically registers their details with a separate system we use, <a href="/installing-mautic-php7-fpm-docker-nginx-and-mariadb-ubuntu-1604">Mautic</a>, which is a "marketing automation" platform.</p> <p>Many people use Mautic to "spam" people or send out newsletters to people who subscribe to them via a website, for instance. We use it differently: we use it to automate the process of sending out hundreds of carefully timed emails to people taking each of our courses, so that they know when to start working on things, what they need to do, when their projects and assignments are due, and how to go about getting assessed after they finish a course (they need to be assessed to get formal academic credit for our courses).</p> <p>Using Mautic means that we can set up the rules and email templates for each course once, and then their available any time we run a course with a cohort (a group of learners are going through the course together, who could be anywhere in the world - our last course had 310 learners from 58 countries!), or even for people who decide to take our courses via "independent study" in which case they can set their own schedule.</p> <p>The other thing I do quite a lot is to work out better ways to host the various services we offer - because the open source technologies we build our services on are constantly improving, there's a steady stream of new approaches to doing things. I have to use my judgment and experience to decide which improvements are worth making (because they all take my time, which is limited to 8 hrs a day), and which probably aren't worthwhile. I recently changed the way we host our main WordPress "multisite", <a href="https://course.oeru.org">https://course.oeru.org</a>, so that it's built as a collection of Docker containers, and I was able to improve the number of simultaneous learners we could support by 10 times without increasing our costs...</p> <p>Then, just about every afternoon, I have a video chat with my colleague, Wayne Mackintosh, who's the only other employee of the OER Foundation (he's also the founder of it, and the main brains behind the operation). He's a keen supporter of open stuff - particularly open source software and open educational resources (the learning materials we make available to our learners as courses), which are the educational analogue to open source software. Wayne's main focus is on academic things - like coordinating educators at our many partner institutions globally to help contribute towards our open educational resources, to get their academic boards to push through accreditation processes for our courses (so that their institutions formally recognise our courses as being of a suitable quality for them to accept as if they were their own). He's got the really hard job, because it's more to do with people than technology. And, despite claiming he's not at every turn, he's remarkably switched on with the technology, too.</p> <p>We report on the things we've done that day, catch up on other news and developments, talk through problems we're working on, and make plans and prioritise things for the coming days and weeks.</p> <p>From time to time, I also write up "how tos" and explanations of how we use open source technologies at the OERu (the OERu - <a href="https://oeru.org">https://oeru.org</a> is the "network" made up of the OER Foundation and our higher education partners) - you can see it at <a href="https://tech.oeru.org">https://tech.oeru.org</a> - the introduction provides an overview of the mix of technologies we use if you're curious.</p> </dd> <dt>2. What is your favourite thing about being a software developer?</dt> <dd> <p>I love that I have a steady stream of interesting problems to solve, and that, when I solve them, I'm potentially making life better for hundreds or thousands of people who, in this case, might not have had the opportunity to reach their educational potential before.</p> <p>Plus, I really appreciate the fact that my current role allows me to learn new things constantly, and to actively engage with the communities of people who write the software that we use to make our learning materials available. For example, I frequently end up making minor improvements to software, like Mautic, or WordPress, which I then make available to other people (at places like this: <a href="https://github.com/kiwilightweight">https://github.com/kiwilightweight</a> or <a href="https://github.com/oeru">https://github.com/oeru</a>) as a way of contributing back.</p> <p>The fact that I can use my skills and experience do what I love, to benefit the world - people I've never even met - is very satisfying and fulfilling, and I consider myself very lucky.</p> </dd> <dt>3. What is the hardest part about being a software developer?</dt> <dd> <p>There're plenty of stressful things in managing systems that people use - for example, if there's a bug that means parts of the system (or all of it) don't work, I have to really scramble to make things right. I also have to to my very best to protect people's data! Very rarely (thankfully) something goes wrong, and people lose data (sometimes things they've invested a lot of time or energy in) and that's really tough.</p> <p>There're also "good-hard" things about developing software - making complex things easier for others to understand... understanding them myself... solving problems by doing a lot of detective work on the Internet - lots of times it requires hours of focused concentration, and really thinking hard... but if you never drop the ball, eventually you solve the problem, and that feels <em>great</em>.</p> </dd> <dt>4. What sort of training and education do you need to do before becoming a software developer?</dt> <dd> <p>Well, ultimately, as a software developer, you're a people interpreter: you're someone who learns what others want to do, how they want to do it, and then you work out how to "codify" that in software. This takes a LOT of time to do it properly. If you don't, you're likely to end up writing software that solves a problem... but one that's often quite different from what people want solved.  So learning about how people do things is very useful... not sure what formal education you can get here - good software developers are keen observers of human behaviour.</p> <p>Once you know roughly where you're going, software development boils down to a lot of problem solving and "micro-decision-making". You have to be the sort of person who, once you commit to solving a problem, you don't stop. Not sure you can learn that - it's just part of some people's nature...</p> <p>As far as formal education goes, I generally recommend this: <strong>get all the education you possibly can</strong> - in maths, science, and computing, but <em>also in other fields</em>. I think one of the best things I ever did was to do a "liberal arts" degree at university - I was privileged that my family and I were able to afford to do it (it was <em>very</em> expensive). I got to study many other fields besides, especially English. Being able to express myself in writing (be it code or English :) ) confidently is perhaps the best thing I learned. I'd also recommend learning other (human) languages!</p> </dd> <dt>5. What do you find least enjoyable about being a software developer?</dt> <dd> <p>There's always a bit of drudgery in every job. There're routine things that are a bit boring (the great thing about being a software developer is that you have the power to <em>automate</em> the boring stuff if it becomes a real drag - here's a <a href="/creating-simple-semantic-html-markup-google-doc">recent example</a>... ) - and there's also the frustration of having to work with (and keep happy) people who really <em>don't</em> understand technology at all, and see it as a threat.</p> <p>I think people who have a deep understanding of both how people work, <em>and</em> how software works have the best prospects in the world today, and I think that'll continue to be the case into the future.</p> </dd> </dl></div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <h2 class="comment-field__title">Blog comments</h2> <a id="comment-7"></a> <article data-comment-user-id="0" about="/comment/7" typeof="schema:Comment" class="comment js-comment by-anonymous has-title clearfix"> <div class="comment__container"> <h3 property="schema:name" datatype="" class="comment__title"> <a href="/comment/7#comment-7" class="permalink" rel="bookmark" hreflang="en">Why GH not GitLab?</a> <span class="comment__new marker marker--success hidden" data-comment-timestamp="1528242423"></span> </h3> <div class="comment__meta"> <div class="comment__submitted"> <span class="comment__author"><span rel="schema:author"><span lang="" typeof="schema:Person" property="schema:name" datatype="">Danyl Strype (not verified)</span></span> </span> <span class="comment__pubdate">Thu 31/05/2018 - 01:18 <span property="schema:dateCreated" content="2018-05-30T13:18:12+00:00" class="rdf-meta hidden"></span> </span> </div> </div> <div class="comment__content"> <div property="schema:text" class="clearfix text-formatted field field-comment--comment-body field-name-comment-body field-type-text-long field-label-hidden"> <div class="field__items"> <div property="schema:text" class="field__item"><p>Hi Dave, this is a great write-up, and hearing that even a pro like you sometimes loses people&#039;s data makes me feel much less existentially terrified about the idea of going pro as a sysadmin or developer, rather than just paddling around the edges of it. One question, why are you using GH for your code, rather than a self-hosted, free code replacement like GitLab? I know that you run a GL instance for NZOSS, so I&#039;m curious about why you the links you&#039;ve given are to GH.</p> </div> </div> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=7&amp;1=default&amp;2=en&amp;3=" token="gSITky6KmlYh8k3gzKegeQw5Ab0eMLcMkqN0j1ewtsc"></drupal-render-placeholder> </div> </div> </article> <div class="indented"><a id="comment-8"></a> <article data-comment-user-id="1" about="/comment/8" typeof="schema:Comment" class="comment js-comment by-node-author has-title clearfix"> <div class="comment__container"> <h3 property="schema:name" datatype="" class="comment__title"> <a href="/comment/8#comment-8" class="permalink" rel="bookmark" hreflang="en">Network Effect... for now</a> <span class="comment__new marker marker--success hidden" data-comment-timestamp="1528242760"></span> </h3> <div class="comment__meta comment__meta--has-user-picture"> <a href="/blog/1">View recent blog entries</a> <div class="comment__submitted"> <span class="comment__author"><span rel="schema:author"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> </span> <span class="comment__pubdate">Wed 06/06/2018 - 11:52 <span property="schema:dateCreated" content="2018-06-05T23:52:40+00:00" class="rdf-meta hidden"></span> </span> </div> </div> <div class="comment__content"> <p class="comment__parent visually-hidden">In reply to <a href="/comment/7#comment-7" class="permalink" rel="bookmark" hreflang="en">Why GH not GitLab?</a> by <span lang="" typeof="schema:Person" property="schema:name" datatype="">Danyl Strype (not verified)</span></p> <div property="schema:text" class="clearfix text-formatted field field-comment--comment-body field-name-comment-body field-type-text-long field-label-hidden"> <div class="field__items"> <div property="schema:text" class="field__item"><p>Very good question, Danyl. My predecessor used GitHub prior to my involvement. I recognise that GitHub enjoys a network effect, and, given our goal is to maximise the value of our code for "The Commons", I (with misgivings, to be sure) decided to stick with GitHub (and a few projects on BitBucket, too) for the short term. </p> <p>Of course, everything's changed a couple days ago with the revelations that GitHub is being purchased by one of the least trustworthy corporations in the tech world, Microsoft. We will be setting up our own Gitlab instance (or perhaps Gogs, not sure yet) and shifting all our repos away from GitHub and Bitbucket (which is very similar to GitHub). Watch this space.</p> </div> </div> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=8&amp;1=default&amp;2=en&amp;3=" token="roEwVhPI8x3guUlKO4gZlMfjatgWmh9ZcaE9ZvBUEwo"></drupal-render-placeholder> </div> </div> </article> </div> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=19&amp;2=field_blog_comments&amp;3=comment" token="RWzhW49w2Ranf8cyf73ptbSAI2rTzCPypSc-MBKegxE"></drupal-render-placeholder> </div> </section> Fri, 18 May 2018 01:17:23 +0000 dave 19 at http://tech.oeru.org Installing NextCloud and Collabora Office Online with Docker on Ubuntu 16.04 http://tech.oeru.org/installing-nextcloud-and-collabora-office-online-docker-ubuntu-1604 <span class="field field--name-title field--type-string field--label-hidden">Installing NextCloud and Collabora Office Online with Docker on Ubuntu 16.04</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--ubuntu-linux"> <span class="field__item-wrapper"><a href="/taxonomy/term/12" hreflang="en">ubuntu linux</a></span> </div> <div class="field__item field__item--mariadb"> <span class="field__item-wrapper"><a href="/taxonomy/term/48" hreflang="en">mariadb</a></span> </div> <div class="field__item field__item--docker"> <span class="field__item-wrapper"><a href="/taxonomy/term/16" hreflang="en">docker</a></span> </div> <div class="field__item field__item--docker-compose"> <span class="field__item-wrapper"><a href="/taxonomy/term/49" hreflang="en">docker-compose</a></span> </div> <div class="field__item field__item--php"> <span class="field__item-wrapper"><a href="/taxonomy/term/40" hreflang="en">php</a></span> </div> <div class="field__item field__item--collabora-office"> <span class="field__item-wrapper"><a href="/taxonomy/term/50" hreflang="en">collabora office</a></span> </div> <div class="field__item field__item--nextcloud"> <span class="field__item-wrapper"><a href="/taxonomy/term/51" hreflang="en">nextcloud</a></span> </div> <div class="field__item field__item--lets-encrypt"> <span class="field__item-wrapper"><a href="/taxonomy/term/17" hreflang="en">let&#039;s encrypt</a></span> </div> <div class="field__item field__item--redis"> <span class="field__item-wrapper"><a href="/taxonomy/term/21" hreflang="en">redis</a></span> </div> <div class="field__item field__item--productivity"> <span class="field__item-wrapper"><a href="/taxonomy/term/52" hreflang="en">productivity</a></span> </div> <div class="field__item field__item--nginx"> <span class="field__item-wrapper"><a href="/taxonomy/term/30" hreflang="en">nginx</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Mon 29/01/2018 - 17:29</span> <div class="float-none field field-node--field-image field-name-field-image field-type-image field-label-hidden has-multiple"> <figure class="field-type-image__figure image-count-1 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/Files%20-%20OERu%20NextCloud.png?itok=xQHlcyml" title="The NextCloud web interface for browsing your files" data-colorbox-gallery="gallery-field_image-5XSxTYYHdzg" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;The NextCloud web interface for browsing your files&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/Files%20-%20OERu%20NextCloud.png?itok=6v2Kuyct" width="220" height="122" alt="The NextCloud web interface for browsing your files" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-2 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/NextCloud-AppStore.png?itok=DPeCx5Rd" title="The central AppStore (note, almost all apps have no cost and are open source). You get a similar view within your own NextCloud instance." data-colorbox-gallery="gallery-field_image-5XSxTYYHdzg" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;The central AppStore (note, almost all apps have no cost and are open source). You get a similar view within your own NextCloud instance.&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/NextCloud-AppStore.png?itok=WqCJJdGj" width="220" height="175" alt="The central AppStore (note, almost all apps have no cost and are open source). You get a similar view within your own NextCloud instance." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-3 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/NextCloud-Calendar.png?itok=-j0Dq2rG" title="The NextCloud shared calendar plugin works with all major calendaring applications alongside your existing digital calendars." data-colorbox-gallery="gallery-field_image-5XSxTYYHdzg" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;The NextCloud shared calendar plugin works with all major calendaring applications alongside your existing digital calendars.&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/NextCloud-Calendar.png?itok=bP23WxDf" width="220" height="175" alt="The NextCloud shared calendar plugin works with all major calendaring applications alongside your existing digital calendars." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-4 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/Nextcloud-CollaboraSpreadsheet.png?itok=Ovp0KryQ" title="An example of a fairly complex spreadsheet in the Collabora spreadsheet interface." data-colorbox-gallery="gallery-field_image-5XSxTYYHdzg" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;An example of a fairly complex spreadsheet in the Collabora spreadsheet interface.&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/Nextcloud-CollaboraSpreadsheet.png?itok=CNhDR2y-" width="220" height="157" alt="An example of a fairly complex spreadsheet in the Collabora spreadsheet interface." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-5 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/Nextcloud-CollaboraWordprocessor.png?itok=IOyfA_M4" title="A fairly complex document, with variables, shown in the Collabora wordprocessor interface." data-colorbox-gallery="gallery-field_image-5XSxTYYHdzg" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;A fairly complex document, with variables, shown in the Collabora wordprocessor interface.&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/Nextcloud-CollaboraWordprocessor.png?itok=HPawBI-o" width="220" height="157" alt="A fairly complex document, with variables, shown in the Collabora wordprocessor interface." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-6 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/DavInFilemanager.png?itok=rCbwaUUY" title="This is what your NextCloud would look like in your desktop filemanager (this is the Nemo filemanager on a Linux desktop)" data-colorbox-gallery="gallery-field_image-5XSxTYYHdzg" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;This is what your NextCloud would look like in your desktop filemanager (this is the Nemo filemanager on a Linux desktop)&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/DavInFilemanager.png?itok=g2dNm33H" width="220" height="122" alt="This is what your NextCloud would look like in your desktop filemanager (this is the Nemo filemanager on a Linux desktop)" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-7 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-02/CollaboraAdminConsole.png?itok=1tNI9ZdJ" title="Collabora Office admin console" data-colorbox-gallery="gallery-field_image-5XSxTYYHdzg" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Collabora Office admin console&quot;}"><img src="/sites/default/files/styles/medium/public/2018-02/CollaboraAdminConsole.png?itok=iijjMrBK" width="220" height="149" alt="Collabora Office admin console" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> </div> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>Dropbox is the best known of the end-user "cloud storage" services for documents, backups, and synchronising data among multiple devices, although now Google's Drive and Microsoft's OneDrive are functionally similar and are being heavily promoted and tied into all sorts of services.</p> <p>Similarly the collaborative editing of documents, spreadsheets, and presentations in the browser, pioneered by Etherpad, but then adopted in a big way by Google Docs (and more recently, Microsoft Office 365), has revolutionised collective note taking, document preparation, and ease of access to these powerful tools by the mainstream of computer users. Only a browser is required, and no other software needs to be installed.</p> <p>But what about people who don't want to entrust all of their data to foreign corporations, holding their data in foreign jurisdiction, in formats that may or may not be retrievable in the event that the supplier fails or changes "strategic direction"? And many of these services involve "mining" their data to extract useful information that vendors sell to others to <em>help them advertise to us in a more targeted way. </em>Yeah, that's creepy.</p> <p>More-over, often if you want to <em>share</em> your data with others, <em>they</em> have to log into the same service, and accept the service's terms and conditions (usually substantially constraining the user's normal rights and freedoms, although who<em> actually</em> reads those, eh?!) in order to do so... so ones use of those services has a magnifying effect on the loss of privacy and control.</p> <p>Some people sensibly prefer to manage their own, or institution-specific, solutions on the infrastructure of their choosing, in a way that doesn't tie anyone into paying ever increasing amounts for data storage as the volumes increase perpetually, month on month.</p> <p>Some of us simply prefer to have control of our own destiny, without a dependence on, for example, file or data storage formats and practices that are completely opaque to them. Our data reflects our creativity energy, and it seems much more comfortable for many of us to be in charge of our own fates rather than entrusting it to a third party who simply sees us a profit centre.</p> <p>Thankfully, the open source world has created an array of possible equivalent systems, and this post describes how you, too, can set up your own equivalent to Dropbox + Google Docs using entirely open source software on any commodity virtual machine hosting system you want to use by adopting NextCloud and Collabora Office.</p> <h2>NextCloud</h2> <p><a href="https://nextcloud.com">NextCloud</a> is <a href="https://nextcloud.com/files/">functionally similar</a> to Dropbox, however, with its active development community and plug-in architecture, it can provide quite a lot more as well, like shared calendaring, email, video conferencing, contact syncing, image/sound/video galleries, <a href="https://nextcloud.com/files/">among many other services</a>.</p> <p>If you prefer not to organise and run your own server, you can purchase a supported server via their website for a cost similar to Dropbox (although, realise that NextCloud is relatively small by comparison and doesn't have the massive economies of scale enjoyed by the bigger players).</p> <p>For those with an interest in history: NextCloud is a fork created by the founder of OwnCloud, after he decided that the company which formed around his OwnCloud project was moving in a direction that was philosophical unpalatable for him. The beauty of open source is that developers can follow their consciences without requiring anyone's permission. The resulting "forks" in code bases and communities then thrive or die based on the strengths of the communities they can build and sustain. This fork is remarkably similar to that which occurred in the OpenOffice community which resulted in the founding of LibreOffice. LibreOffice has thrived and OpenOffice has faded into irrelevance. More on that below.</p> <p>For those with a technological interest, NextCloud is a mature PHP application (but with a modern architecture, including a command line interface, occ) which stores its data in an RDBMS like MySQL, MariaDB, PostgreSQL, or (usually for development purposes) the lightweight SQLite database. Here are <a href="https://docs.nextcloud.com/server/12/admin_manual/installation/index.html">details for would-be administrators</a>.</p> <h2>Collabora Office</h2> <p>Given how much companies like Google and Microsoft invest on Docs and Office 365 respectively, how is it possible for an open source community to create a credible competitor? Turns out it's not as hard as you might think if they leverage the power of open source.</p> <p>A small software company with headquarters in the UK (although their team appears to be from all over), Collabora Office, has taken on the ambitious mission of creating a "collaborative web interface" allowing users to collaborate using <a href="https://libreoffice.org">LibreOffice</a>, one of the most powerful and widely used office package available anywhere. We're currently at Collabora Office 3.0, and the front end is quite nice and functional, but still pretty simple - that can be a good thing for many users. Collabora is progressively re-imagining the user interface of LibreOffice as a collaborative web interface. This isn't easy, but it's <em>much</em> easier than it otherwise would be because the difficult job of creating the heavy-lifting application back-end is already done - LibreOffice is a mature widely used application (albeit with a desktop interface, not a web-based collaborative interface). So we can expect progress will be rapid, and large sets of new capabilities will be "unlocked" as they progress their efforts.</p> <h2>NextCloud and Collabora - better together!</h2> <p>The beauty of the open source software model is that we can connect NextCloud and Collabora office - completely separate and unrelated communities - thanks to a new integration standard, WOPI (Web-application Open Platform Interface) they form a well integrated component model - with the <em>major </em>added benefit of being able to swap in a better file management platform, or a better collaborative productivity package if one or the other emerges, without having to start from scratch.</p> <h2>Setting up your own NextCloud Collabora Server</h2> <p>If you're game to run your own (and, in my experience, it's a surprisingly well behaved system) here's how you do it.</p> <p>In preparation, you'll want to have the following ready:</p> <ul><li>a Linux virtual machine or "VM" (I recommend running the current Ubuntu LTS version, or current Debian) with a user with Sudo privileges...,</li> <li>your domain name for the NextCloud instance, pointing to the IP address of your VM,</li> <li>your domain name for the Collabora instance, also pointing to the IP of your VM, and</li> <li>credentials for an email address capable of sending from a remote server (usually termed an "authenticating SMTP email account")</li> </ul><h3>Secure access with SSH</h3> <p>First things first, make sure you're logged into your host (probably via SSH) as a user who has "sudo" capabilities! You need to log into the host from your local machine. We recommend setting up <a href="https://www.digitalocean.com/community/tutorials/how-to-configure-ssh-key-based-authentication-on-a-linux-server">key-based authentication</a>.</p> <h3>Firewall with UFW</h3> <p>No computer system is ever full secure - there're always exploits waiting to be found, so security is a process of maintaining vigilance. Part of that is reducing exposure - minimising your "attack surface". Use a firewall - "<a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-ubuntu-16-04" title="Uncomplicated FireWall">ufw</a>" is installed on Ubuntu by default. Make sure you've got exceptions for SSH (without them, you could lock yourself out of your machine! Doh!).</p> <p>Run the following commands to allow your Docker containers to talk to other services on your host.</p> <p><code>sudo ufw allow in on docker0<br /> sudo ufw allow from 172.17.0.0/24 to any<br /> sudo ufw allow from 172.18.0.0/24 to any<br /> sudo ufw allow from 172.19.0.0/24 to any<br /> sudo ufw allow from 172.20.0.0/24 to any</code></p> <p>Specifically for Docker's benefit, you need to tweak the default Forwarding rule (I use "vim" as my editor. If you don't know how to/want to use it, replace <strong>vim</strong> with <strong>nano</strong> everywhere you see it in the following - nano's easier to use for simple edits like this):</p> <p><code>sudo vim /etc/defaults/ufw</code></p> <p>and copy the line <code>DEFAULT_FORWARD_POLICY="DROP"</code> tweak it to look like this (commenting out the default, but leaving it there for future reference!):</p> <p><code>#DEFAULT_FORWARD_POLICY="DROP"<br /> DEFAULT_FORWARD_POLICY="ACCEPT"</code></p> <p>You also have to edit <code>/etc/ufw/sysctl.conf</code> and remove the "#" at the start of the following lines, so they look like this:</p> <p><code>sudo vim /etc/ufw/sysctl.conf</code></p> <p><code># Uncomment this to allow this host to route packets between interfaces<br /> net/ipv4/ip_forward=1<br /> net/ipv6/conf/default/forwarding=1<br /> net/ipv6/conf/all/forwarding=1</code></p> <p>and finally restart the network stack and ufw on your server<code> </code></p> <p><code>sudo service networking restart<br /> sudo service ufw restart</code></p> <h3>Installing the Nginx webserver</h3> <p>In the configuration I'm describing here, you'll need a webserver running on the server - it'll be acting as a "proxy" for the Docker-based Nginx instance described below. I like the efficiency of Nginx and clarity of Nginx configurations over those of Apache and other open source web servers. Here's how you install it.</p> <p><code>sudo apt-get install nginx-full</code></p> <p>To allow nginx to be visible via ports 80 and 443, run</p> <p><code>sudo ufw allow "Nginx Full"</code></p> <p><strong>Note</strong>: make sure your hosting service is not blocking these ports at some outer layer (depending on who's providing that hosting service you may have to set up port forwarding).</p> <h3>Installing MariaDB</h3> <p>MariaDB is effectively a drop-in alternative to MySQL and we prefer it because it's not controlled by Oracle and has a more active developer community. On Ubuntu, MariaDB pretends to be MySQL for compatibility purposes, so don't be weirded out by the interchangeable names below. Install the server and the client like this.</p> <p><code>sudo apt-get install mariadb-server-10.0 mariadb-client-10.0</code></p> <p>You need to set a root (admin) user password - you might want to create a /root/.my.cnf file containing the following (replacing YOURPASSWORD) to let you access MariaDB without a password from the commandline<code>:</code></p> <p><code>[client]<br /> user=root<br /> password=YOURPASSWORD</code></p> <p>You should now be able to type "mysql" at the command prompt</p> <p>Tweak the configuration so that it's listening on</p> <p><code>sudo vim /etc/mysql/mariadb.conf.d/50-server.cnf </code></p> <p>and copy the bind-address line and adjust so it looks like this - we want MariaDB to be listening on all interfaces, not just localhost (127.0.0.1)...</p> <p><code># Instead of skip-networking the default is now to listen only on<br /> # localhost which is more compatible and is not less secure.<br /> #bind-address           = 127.0.0.1<br /> bind-address            = 0.0.0.0</code></p> <p>Then restart MariaDB:</p> <p><code>sudo service mysql restart</code></p> <p>It should now be listening on port 3306 on all interfaces, i.e. 0.0.0.0.</p> <p>Now set up the database which will hold NextCloud's data. Log into the MySQL client on the host (if you've created a .my.cnf file in your home directory as describe above, you won't need to enter your username and password):</p> <p><code>mysql -u root -p</code></p> <p>Enter your root password when prompted. It's also a good idea to gin up a password for your "nextcloud" database user. I usually use pwgen (<code>sudo apt-get install pwgen</code>) - for example running this command will give you a single 12 character password without special characters (just numbers and letters):</p> <p><code>pwgen -s 12 1<br /> T7KR2osrMkyC</code></p> <p>At the prompt (which will look something like MariaDB [(none)]&gt;) enter the following lines (putting your password in place of [passwd]):</p> <p><code>CREATE DATABASE nextcloud CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;<br /> CREATE USER "nextcloud"@"%" IDENTIFIED BY "[passwd]";<br /> GRANT ALL ON nextcloud.* to "nextcloud"@"%";<br /> FLUSH PRIVILEGES;</code></p> <p>Then enter \q to exit.</p> <h2>NextCloud and Collabora Office with Docker</h2> <p>We make use of the NextCloud community's <a href="https://hub.docker.com/_/nextcloud/" title="Documentation for the reference NextCloud Docker container.">stable Docker container</a> which they keep up to date. Similarly, the Collabora community has created a <a href="https://hub.docker.com/collabora/code">reference Docker container</a>.</p> <p>The over all architecture consists of five Docker containers (note, done properly, you aim to ensure that each container runs only one service!):</p> <ol><li>the main NextCloud container (running the PHP-FPM service)</li> <li>an identical container to the PHP one which runs the cron service (which does periodic administrative tasks relevant to NextCloud)</li> <li>the self-contained Collabora Office container (running PHP with an Apache web server instance and a full instance of LibreOffice running in headless server mode (never fear, no servers were harmed in the building of this server!) - yes this server doesn't really adhere to the "one-service per container" convention, but I'm ok with that. It's just a convention after all.)</li> <li>a Redis container (which provides performance improving caching for NextCloud), and</li> <li>an Nginx webserver container which makes it easier to manage the configuration and paths of the NextCloud and Collabora servers via WOPI. It means that on the hosting server, we only need to run a proxying web server, which is easy.</li> </ol><p>The way I prefer to implement this set of containers is to use <a href="https://docs.docker.com/compose/">Docker Compose</a> (after first setting up <a href="https://docs.docker.com/install/linux/docker-ce/ubuntu/">Docker support</a> on your server - I'll assume you've followed the complete instructions including <a href="https://docs.docker.com/install/linux/linux-postinstall/">setting up Docker for your non-root user</a>). I suggest using the latest <a href="https://docs.docker.com/compose/install/#install-compose">installation instructions</a> provided by the Docker community. To be honest, I usually use the alternative instructions, <a href="https://docs.docker.com/compose/install/#install-using-pip">employing the "pip" approach</a>. You can upgrade an existing install by issuing (on your Linux VM's command line):</p> <p><code>sudo pip install -U docker-compose </code></p> <p>To set up your server, I recommend setting up a place for your Docker containers (replace "me" with your non-root username on the server) and the associated persistent data (your Docker containers should hold <em>no</em> important data - you should be able to delete and recreate them entirely without losing any important data or configuration):</p> <p><code>sudo mkdir /home/data</code><br /><code>sudo mkdir /home/data/nextcloud</code><br /><code>sudo mkdir /home/data/nextcloud/apps<br /> sudo mkdir /home/data/nextcloud/config<br /> sudo mkdir /home/data/nextcloud/data<br /> sudo mkdir /home/data/nextcloud/redis<br /> sudo mkdir /home/data/nextcloud/resources<br /> sudo mkdir /home/docker<br /> sudo mkdir /home/docker/nextcloud-collabora<br /> sudo chown -R me:me /home/docker<br /> cd /home/docker/nextcloud-collabora</code></p> <p>Here's an example of the required docker-compose.yml file (you can create this via a text editor like "nano" which should be pre-installed on any VM these days, or use my preferred, but less intuitive, editor, vim via <code>vim docker-compose.yml</code> in the /home/docker/nextcloud-collabora directory):</p> <p><code>version: '2'<br /> networks:<br />   back:<br />     driver: bridge<br /> services:<br />   web:<br />     image: nginx<br />     ports:<br />       - 127.0.0.1:8082:80<br />     volumes:<br />       - ./nginx.conf:/etc/nginx/nginx.conf:ro<br />     links:<br />       - app<br />     volumes_from:<br />       - app<br />     environment:<br />       - VIRTUAL_HOST<br />     networks:<br />     - back<br />     restart: unless-stopped      <br />   app:<br />     image: nextcloud:12-fpm<br />     links:<br />       - redis<br />     volumes:<br />       - /home/data/nextcloud/apps:/var/www/html/apps<br />       - /home/data/nextcloud/config:/var/www/html/config<br />       - /home/data/nextcloud/resources:/var/www/html/resources<br />       - /home/data/nextcloud/data:/var/www/html/data<br />     networks:<br />     - back<br />     restart: unless-stopped      <br />   cron:<br />     image: nextcloud:12-fpm<br />     volumes_from:<br />       - app<br />     user: www-data<br />     entrypoint: |<br />       bash -c 'bash -s &lt;&lt;EOF<br />       trap "break;exit" SIGHUP SIGINT SIGTERM<br />       while /bin/true; do<br />         /usr/local/bin/php /var/www/html/cron.php<br />         sleep 900<br />       done<br />       EOF'<br />     networks:<br />       - back<br />     restart: unless-stopped      <br />   redis:<br />     image: redis:alpine<br />     volumes:<br />       - /home/data/nextcloud/redis:/data<br />     networks:<br />       - back<br />     restart: unless-stopped<br />   collab:<br />     image: collabora/code<br />     environment:</code><br /><code>      # put the domain name you select for your NextCloud instance<br />       # here! Escape any . in your domain name by preceding them with \\<br />       domain: your\\.domain\\.tld<br />       username: admin</code><br /><code>      # put your own strong password in here!<br />       password: some-good-password<br />     cap_add:<br />       - MKNOD<br />     networks:<br />       - back<br />     volumes_from:<br />       - app<br />     ports:<br />       - 127.0.0.1:9980:9980<br />     links:<br />       - app<br />     restart: unless-stopped</code></p> <p>You'll need to substitute the domain name you pick for your NextCloud instance - Collabora's container requires that you specify it so that it doesn't accept connections from other (potentially nefarious) containers elsewhere on the Internet!</p> <p>Also note, the "ports" specified above, 8082 for <code>nginx</code> and 9980 for <code>collab</code> are arbitrary - I picked these to ensure they don't conflict with ports being used by other containers on my server - you can use these if you want, or use <code>sudo netstat -punta</code> to see what ports are currently claimed by other services on your server (if there are any) and pick ones that don't clash! If it scroll past too fast, you can pipe it into less to allow you to scroll and search: <code>sudo netstat -punta | less</code> - hit "q" to exit or "/" to initiate a text search.</p> <p>You will also need to provide the "nginx.conf" file referenced in the nginx section of the file. Do that by using your editor, e.g. <code>vim nginx.conf</code>, and enter this content:</p> <p><code>user www-data;</code></p> <p><code>events {<br />   worker_connections 768;<br /> }</code></p> <p><code>http {<br />   upstream backend {</code><br /><code>      # if you don't call your NextCloud server "app" in your<br />       # docker-compose.yml, you'll need to change app below to </code><br /><code>      # whatever you end up calling it.<br />       server app:9000;<br />   }<br />   include /etc/nginx/mime.types;<br />   default_type application/octet-stream;</code></p> <p><code>  server {<br />     listen 80;<br />     <br />     # Add headers to serve security related headers<br />     add_header X-Content-Type-Options nosniff;<br />     add_header X-Frame-Options "SAMEORIGIN";<br />     add_header X-XSS-Protection "1; mode=block";<br />     add_header X-Robots-Tag none;<br />     add_header X-Download-Options noopen;<br />     add_header X-Permitted-Cross-Domain-Policies none;</code></p> <p><code>    root /var/www/html;</code></p> <p><code>    location = /robots.txt {<br />       allow all;<br />       log_not_found off;<br />       access_log off;<br />     }</code></p> <p><code>    location = /.well-known/carddav {<br />       return 301 $scheme://$host/remote.php/dav;<br />     }<br />     location = /.well-known/caldav {<br />       return 301 $scheme://$host/remote.php/dav;<br />     }</code></p> <p><code>    client_max_body_size 1G;<br />     fastcgi_buffers 64 4K;</code></p> <p><code>    gzip off;</code></p> <p><code>    index index.php;<br />     error_page 403 /core/templates/403.php;<br />     error_page 404 /core/templates/404.php;<br />  <br />     location / {<br />         rewrite ^ /index.php$uri;<br />     }</code></p> <p><code>    location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {<br />         deny all;<br />     }<br />     location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) {<br />         deny all;<br />     }</code></p> <p><code>    location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+|core/templates/40[34])\.php(?:$|/) {<br />         include fastcgi_params;<br />         fastcgi_split_path_info ^(.+\.php)(/.*)$;<br />         fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;<br />         fastcgi_param PATH_INFO $fastcgi_path_info;<br />         fastcgi_param HTTPS on;<br />         #Avoid sending the security headers twice<br />         fastcgi_param modHeadersAvailable true;<br />         fastcgi_param front_controller_active true;<br />         fastcgi_pass backend;<br />         fastcgi_intercept_errors on;<br />         fastcgi_request_buffering off;<br />     }</code></p> <p><code>    location ~ ^/(?:updater|ocs-provider)(?:$|/) {<br />         try_files $uri/ =404;<br />         index index.php;<br />     }</code></p> <p><code>    # Adding the cache control header for js and css files<br />     # Make sure it is BELOW the PHP block<br />     location ~* \.(?:css|js)$ {<br />         try_files $uri /index.php$uri$is_args$args;<br />         add_header Cache-Control "public, max-age=7200";<br />         # Add headers to serve security related headers (It is intended to<br />         # have those duplicated to the ones above)<br />         # Before enabling Strict-Transport-Security headers please read into<br />         # this topic first.<br />         # add_header Strict-Transport-Security "max-age=15768000;<br />         #  includeSubDomains; preload;";<br />         add_header X-Content-Type-Options nosniff;<br />         add_header X-Frame-Options "SAMEORIGIN";<br />         add_header X-XSS-Protection "1; mode=block";<br />         add_header X-Robots-Tag none;<br />         add_header X-Download-Options noopen;<br />         add_header X-Permitted-Cross-Domain-Policies none;<br />         # Optional: Don't log access to assets<br />         access_log off;<br />     }</code></p> <p><code>    location ~* \.(?:svg|gif|png|html|ttf|woff|ico|jpg|jpeg)$ {<br />         try_files $uri /index.php$uri$is_args$args;<br />         # Optional: Don't log access to other assets<br />         access_log off;<br />     }<br />   }<br /> }</code></p> <p>That should be all the configuration you need to make the Docker containers go.</p> <h2>Configuring Nginx to proxy NextCloud and Collabora</h2> <p>The next step is configuring the local nginx proxy servers for NextCloud and Collabora using the nginx instance you installed earlier. That's what responds to the domain name you choose for this service. In our case, the name is <a href="https://docs.oeru.org">https://docs.oeru.org</a> - you can have a look at it to see what you should be seeing when you first start things up! We use <a href="https://letsencrypt.org" title="This is an incredible free and open source service, that is single-handedly making the web a much safer place.">Let's Encrypt</a> to provide secure hosting - <a href="/protecting-your-users-lets-encrypt-ssl-certs">here're my Let's Encrypt instructions</a> on setting it up. The key thing to realise is that your "certificates" need to exist for Nginx to restart with the new configurations below - use the "commenting out the intervening lines" trick mentioned in my instructions to bootstrap the creation of your secure certificates!</p> <p>To configure the proxies, you need to create two configuration files in your /etc/nginx/sites-available/ directory.</p> <h3>NextCloud Proxy Configuration</h3> <p>Create a file with a meaningful name for your NextCloud Proxy, perhaps based on the domain name you've chosen (our file for docs.oeru.org is called "docs") using the same editing approach as the last few (although this is in a different directory) for example <code>sudo vim /etc/nginx/sites-available/docs</code> with the following contents, replacing "nextcloud.domain" with your selected domain name (and the port number 8082 if you've opted to change to a different one!):</p> <p><code>server {<br />     listen 80;<br />     server_name nextcloud.domain;</code></p> <p><code>    include /etc/nginx/includes/letsencrypt.conf;</code></p> <p><code>    # redirect all HTTP traffic to HTTPS.<br />     location / {<br />         return  302 https://nextcloud.domain$request_uri;<br />     }<br /> }</code></p> <p><code># This configuration assumes that there's an nginx container talking to the mautic PHP-fpm container,<br /> # and this is a reverse proxy for that Mautic instance.<br /> server {<br />     listen 443 ssl;<br />     server_name your.domain;</code></p> <p><code>    ssl_certificate /etc/letsencrypt/live/nextcloud.domain/fullchain.pem;<br />     ssl_certificate_key /etc/letsencrypt/live/nextcloud.domain/privkey.pem;<br />     ssl_protocols TLSv1 TLSv1.1 TLSv1.2;<br />     # to create this, see https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html<br />     ssl_dhparam /etc/ssl/certs/dhparam.pem;<br />     keepalive_timeout 20s;</code></p> <p><code>    include /etc/nginx/includes/letsencrypt.conf;<br />    <br />     location ^~ / {<br />         proxy_pass http://localhost:8082;<br />         proxy_set_header Upgrade $http_upgrade;<br />         proxy_set_header Connection "Upgrade";<br />         proxy_set_header Host $http_host;<br />         proxy_read_timeout 36000s;<br />     }<br />     client_max_body_size 1G;<br />     fastcgi_buffers 64 4K;</code></p> <p><code>    add_header X-Frame-Options "SAMEORIGIN";<br />     add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";<br /> }</code></p> <h3>Collab Proxy Configuration</h3> <p>Now create a collabora proxy configuration.</p> <p>Note: This will probably never by used by any user directly (there is a resource analysis service on the collabora system that might be of interest) - instead it'll be referenced by the NextCloud instance transparently to your users. </p> <p>In our case, we chose the domain collab.oeru.org and the file is called "collab", created via <code>sudo vim /etc/nginx/sites-available/collab</code> and containing (replace collab.domain with the one you've selected - similarly replace the port number 9980 with whatever you've selected if you've opted for a different one!):</p> <p><code>server {<br />     listen 80;<br />     server_name collab.domain;</code></p> <p><code>    # for let's encrypt renewals!<br />     include /etc/nginx/includes/letsencrypt.conf;</code></p> <p><code>    # redirect all HTTP traffic to HTTPS.<br />     location / {<br />         return  302 https://collab.domain$request_uri;<br />     }<br /> }</code></p> <p><code>server {<br />     listen 443 ssl;<br />     server_name collab.domain;</code></p> <p><code>    ssl_certificate /etc/letsencrypt/live/collab.domain/fullchain.pem;<br />     ssl_certificate_key /etc/letsencrypt/live/collab.domain/privkey.pem;<br />     ssl_protocols TLSv1 TLSv1.1 TLSv1.2;<br />     # to create this, see https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html<br />     ssl_dhparam /etc/ssl/certs/dhparam.pem;<br />     keepalive_timeout 20s;</code></p> <p><code>    # for let's encrypt renewals!<br />     include /etc/nginx/includes/letsencrypt.conf;</code></p> <p><code>    proxy_http_version 1.1;<br />     proxy_buffering off;</code></p> <p><code>    # static files<br />     location ^~ /loleaflet {<br />         proxy_pass https://localhost:9980;<br />         proxy_set_header Host $http_host;<br />     }</code></p> <p><code>    # WOPI discovery URL<br />     location ^~ /hosting/discovery {<br />         proxy_pass https://localhost:9980;<br />         proxy_set_header Host $http_host;<br />     }</code><br /><br /><code>    # download, presentation and image upload<br />     location ^~ /lool {<br />         proxy_pass https://localhost:9980;<br />         proxy_set_header Upgrade $http_upgrade;<br />         proxy_set_header Conection "upgrade";<br />         proxy_set_header Host $http_host;<br />     }<br /> }</code></p> <p>Once those are created, you have to make sure that they're "enabled" (replacing with your file names, of course):</p> <p><code>sudo cd /etc/nginx/sites-enabled<br /> sudo ln -sf ../sites-available/docs .<br /> sudo ln -sf ../sites-available/collab .</code></p> <p>To confirm that there aren't any typos or issues that might make nginx unhappy, run</p> <p><code>sudo nginx -t</code></p> <p>If all's well, get nginx to reread its configuration with the new files:</p> <p><code>sudo service nginx reload</code></p> <h2>Firing it all up!</h2> <p>Phew - congratulations on getting here! We've reached the moment of truth where we need to see if this whole thing will work!</p> <p>We need to make sure we're back in the Docker directory we set up:</p> <p><code>cd /home/docker/nextcloud-collabora</code></p> <p>and then we need to try running our docker-compose script to "pull" in the pre-built Docker containers we've specified in our docker-compose.yml file:</p> <p><code>docker-compose pull</code></p> <p>All going well, after a few minutes (longer or shorter depending on the speed of your server's connection) you should have download the Nginx, Redis, NextCloud and Collabora-CODE Docker images. Then you can run:</p> <p><code>docker-compose up -d &amp;&amp; docker-compose logs -f</code></p> <p>This will attempt to start up the containers (bringing them "up" in daemon mode, thus the -d) and then show you a stream of log messages from the containers, preceded by the container name. This should help you debug any problems that occur during the process (ideally, none).</p> <p>Once you see log messages streaming past, and no obvious "container exited" or other error messages (which will usually contain the word "error" a lot), you should be able to point your browser at your selected domain name and bring it up in your browser!</p> <h3>Setting up the database</h3> <p>On doing so, if all is well, you should be directed through the database set up process for your NextCloud instance. Your details should be:</p> <p>database IP: 172.17.0.1 - this is the default IP of the Docker host server.<br /> database name: nextcloud<br /> database user: nextcloud<br /> database password: (the one you came up with above)</p> <h3>Setting the Admin user</h3> <p>Once that's set and working, NextCloud will install all the relevant database tables and initial data. You'll be asked to set up an <em>admin user</em> account, which can be "admin" (you could make it something different to help stymie nefarious probes that assume you've got a user called "admin" - but don't forget what you've called it!) and some strong password you create (you can use the pwgen utility you used earlier) - I'd recommend recording it somewhere. I would <em>not</em> recommend making your own account, in your name, the main admin account. I recommend creating a second account, <em>with administrator privileges</em> for yourself, but leave the admin account purely for administrative activities.</p> <h3>Configuring Outgoing Email</h3> <p>To allow your NextCloud instance to send outgoing email, so that your site can alert you to security updates that need to be applied, or so that users can request a replacement password if they've forgot theirs, you'll need an <em>authenticating SMTP account</em> somewhere. Most of you already have one. You'll probably want to set up a dedicated email address for this server somewhere, perhaps something like "<a href="mailto:nextcloud@your.domain">nextcloud@your.domain</a>" or similar, with a username (often just the email address) and a password. You'll need the following details:</p> <p>SMTP server : an IP address or a domain name<br /> SMTP username: a username or an email address<br /> SMTP password: a strong password already configured for the username on that server<br /> SMTP login security: whether login is via TLS, SSL, or unsecure (!!), and<br /> SMTP login method: plain, encrypted, "login" or some other value.</p> <p>You should be able to text your email settings to make sure the details you've entered are valid. If you need to adjust these settings later, you can go to the admin menu (top right of the web browser interface) and go to Admin-&gt;Additional Settings  - should have a path of <a href="https://your.domain/settings/admin/additional">https://your.domain/settings/admin/additional</a></p> <h3>Configuring Collabora Office Integration</h3> <p>Once you're logged in as your own user, looking at your own default folders, you can start having a look around. You should have an "admin" menu (assuming you've created your user with Administrator privileges) at the top right of the web interface. If you go to Apps, you can use the search box to search for "Collabora" or go to the "Office &amp; text" App category. You'll need to "enable" the Collabora Online "official" app, at which point it will download the latest version of the connector app and install it (it should appear in your /home/data/nextcloud/apps directory)</p> <p>Once you've done that, go to your top right menu again, selecting Admin, and you should see "Collabora Online" as an option in the left column (which starts with "Basic settings"). Selecting that, you'll need to enter  "<a href="https://collab.domain">https://collab.domain</a>" (replacing with your domain, of course). I don't have any of the other options ticked.</p> <p>If it works, you should have the ability to go back to the home of your NextCloud install, which should show you your top-level folders. If you click the "+" next to the home icon (top left of the folder pane) you should now have the option to create (in addition to "Upload file", "New folder", "New text file") a "New Document", "New Spreadsheet", and "New Presentation". Clicking those should give you the Collabora Office interface for the designated content type.</p> <p>Similarly, you can use the "Upload file" to upload a document in a format that is supported by Collabora Office, once uploaded clicking on the filename should open it for editing in the appropriate Collabora Office interface.</p> <p>It is saved as it is change, you shouldn't need to save it explicitly.</p> <h2>Upgrading it</h2> <p>So, as you're no doubt aware, both NextCloud and Collabora Office are always being improved and updated. I certainly encourage you to keep your installation up-to-date.</p> <p>While you'll periodically see that NextCloud apps have available updates (these can be upgraded through the browser interface) updates to the NextCloud and Collabora Office systems themselves need to be undertaken by upgrading the containers. Luckily it's easy to do (although I strongly urge you to ensure you have a very recent backup of both database and uploaded files - they're the files in /home/data/nextcloud/data:</p> <p>Updating the container should be as easy as either doing another</p> <p><code>docker pull oeru/mautic</code></p> <p>and then shutting down Docker container via a</p> <p><code>docker-compose stop</code></p> <p>removing the old containers (this won't remove any data you want to save if you followed the directions above! But remember to do it in the right directory!) via</p> <p><code>docker-compose rm -v</code></p> <p>and then restarting it via</p> <p><code>docker-compose up -d</code></p> <p>Use <code>docker-compose logs -f</code> to watch the logs - you'll likely see debugging information in the unlikely event that something goes wrong in the upgrade process.</p> <h2>Backing it up</h2> <p>To back up your instance on your server, you need two things: a file system backup of your /home/data/nextcloud directory, and database dumps of your database.</p> <p>There're lots of ways to back up your files (I personally use a bash script that I wrote in a past role, which uses <a href="http://www.nongnu.org/rdiff-backup/">rdiff-backup</a> to create versioned backups either locally or on a remote server, although there're <a href="https://www.howtoforge.com/linux_rdiff_backup">other documented approaches</a> - leave a comment below if you'd like to learn more about my approach!).</p> <p>Backing up your database is as easy installing automysqlbackups:</p> <p><code>sudo apt install automysqlbackups</code></p> <p>You'll find daily versioned dumps of your MariaDB database(s) in /var/lib/automysqlbackups. To run an ad hoc backup (which will replace the previous backup from that day, if there is one) just run</p> <p><code>sudo automysqlbackups</code></p> <h2>Collabora Admin Console</h2> <p>Once you've got everything set up, you can access the admin console of the Collabora Office instance at the collab.domain you specified above - it'll have the path <code>https://collab.domain/loleaflet/dist/admin/admin.html</code> (of course replacing collab.domain with your domain) which gives you useful info about the system resources being used, number of documents being edited and by whom, and some other interesting details. I've included a screen shot.</p> <p>When prompted for login details, use the collab username - "admin" if you used the default I provided, and the password you set in your docker-compose.yml file above.</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=17&amp;2=field_blog_comments&amp;3=comment" token="aF_f2aYW3SGRfmrsffb5fOJlN5wEXJibvMUiQKm2VjE"></drupal-render-placeholder> </div> </section> Mon, 29 Jan 2018 04:29:13 +0000 dave 17 at http://tech.oeru.org Installing LimeSurvey with Docker on Ubuntu 16.04 with Nginx and Mariadb http://tech.oeru.org/installing-limesurvey-docker-ubuntu-1604-nginx-and-mariadb <span class="field field--name-title field--type-string field--label-hidden">Installing LimeSurvey with Docker on Ubuntu 16.04 with Nginx and Mariadb</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Fri 19/01/2018 - 10:27</span> <div class="float-none field field-node--field-image field-name-field-image field-type-image field-label-hidden has-multiple"> <figure class="field-type-image__figure image-count-1 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-06/OERuSurvey-mainpage.png?itok=U9bb22aE" title="Limesurvey main page and menu" data-colorbox-gallery="gallery-field_image--L25Bhz3pro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Limesurvey main page and menu&quot;}"><img src="/sites/default/files/styles/medium/public/2018-06/OERuSurvey-mainpage.png?itok=DUz2wzqw" width="220" height="165" alt="Limesurvey main page and menu" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-2 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-06/OERuSurvey-config.png?itok=0yRPUASD" title="Limesurvey administrator configuration" data-colorbox-gallery="gallery-field_image--L25Bhz3pro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Limesurvey administrator configuration&quot;}"><img src="/sites/default/files/styles/medium/public/2018-06/OERuSurvey-config.png?itok=jx88srCu" width="220" height="168" alt="Limesurvey administrator configuration" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-3 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-06/OERuSurvey-question-edit.png?itok=8BB0VF1J" title="Limesurvey question editing interface" data-colorbox-gallery="gallery-field_image--L25Bhz3pro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Limesurvey question editing interface&quot;}"><img src="/sites/default/files/styles/medium/public/2018-06/OERuSurvey-question-edit.png?itok=ERuWcUOS" width="209" height="220" alt="Limesurvey question editing interface" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-4 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-06/OERuSurvey-sample-survey_0.png?itok=OO_sL2D4" title="Limesurvey sample survey" data-colorbox-gallery="gallery-field_image--L25Bhz3pro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Limesurvey sample survey&quot;}"><img src="/sites/default/files/styles/medium/public/2018-06/OERuSurvey-sample-survey_0.png?itok=jP-g_vyQ" width="220" height="219" alt="Limesurvey sample survey" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> </div> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p><a href="https://www.limesurvey.org/">LimeSurvey</a> is an open source survery tool, functionally similar to far more heavily marketed proprietary tools like SurveyMonkey and Google Forms. It's a very mature, fully-featured system. You can either use the reasonably priced hosted service available on the LimeSurvey site, or you can host your own, holding your own data and collecting your own analytics if you prefer. The only cost of hosting your own is your time to set it up and any costs associated with your hosting environment (which, today, can be negligible). Our instance is on <a href="https://survey.oeru.org">https://survey.oeru.org</a>...</p> <h2>Setup</h2> <p>It's as simple as having a server online somewhere (perhaps provided by your organisation or institution or if you're not sure where to go, I can recommend <a href="https://digitalocean.com" title="Their low-end Virtual Server would be sufficent, at USD5/month... ">Digital Ocean</a> - note: I have no affiliation with them other than being a happy customer)  and installing <a href="https://docs.docker.com/install/linux/docker-ce/ubuntu/">Docker-CE</a> (including <a href="https://docs.docker.com/install/linux/linux-postinstall/">use by non-root users</a>) and <a href="https://docs.docker.com/compose/install/#install-using-pip">Docker Compose</a> (I prefer to do this with 'pip' as described in the link).</p> <p>First create Docker and Data directories on your server:</p> <p><code>sudo mkdir /home/docker home/data<br /> sudo chown ${USER}:${USER} /home/docker home/data</code></p> <p>And we'll create a directory for Limesurvey's docker compose configuration and data respectively:</p> <p><code>mkdir /home/docker/limesurvey /home/data/limesurvey</code></p> <p>Then you simply have to create a <code>docker-compose.yml</code> file in /home/docker/limesurvey with your preferred editor (I use vim. If you don't know vim or vi, use nano which is easy to use and self-explanatory, e.g. <code>nano /home/docker/limesurvey/docker-compose.yml</code> ) containing the following:</p> <p><blockcode><code>version: '3'<br /> services:<br />   lime:<br />     restart: unless-stopped<br />     image: oeru/limesurvey<br />     ports:<br />       - '127.0.0.1:8080:80'<br />     volumes:<br />       - '/home/data/limesurvey/upload:/var/www/html/upload'<br />       - '/home/data/limesurvey/plugins:/var/www/html/plugins'<br />       - '/home/data/limesurvey/config:/var/www/html/application/config'<br />       - '/home/data/limesurvey/reference:/var/www/reference'</code></blockcode></p> <p>Last steps, you need your domain name you want to use, and to install Nginx on your server and configure it</p> <p><code>apt update &amp;&amp; apt install nginx-full</code></p> <p>and you then need to add a configuration file in /etc/nginx/sites-available/limesurvey that looks like this (let's say your domain is "survey.oeru.org" - obviously, replace with your own domain :) ):</p> <p><blockcode><code>server {<br />   listen 80;<br />   root /var/www/html;<br />   index index.html index.htm;<br />   server_name survey.oeru.org;</code></blockcode><br /><blockcode><code>  ## Access and error logs.<br />   access_log /var/log/nginx/survey.oeru.org_access.log;<br />   error_log /var/log/nginx/survey.oeru.org_error.log;</code></blockcode><blockcode><br /><code>#  include /etc/nginx/includes/letsencrypt.conf;</code></blockcode><blockcode><br /><code>#  location / {<br /> #     return 302 https://survey.oeru.org$request_uri;<br /> #  }<br /> #}</code></blockcode><blockcode><br /><code># server {<br /> #  listen 443 ssl;<br /> #  ssl on;<br /> #  ssl_certificate /etc/letsencrypt/live/survey.oeru.org/fullchain.pem;<br /> #  ssl_certificate_key /etc/letsencrypt/live/survey.oeru.org/privkey.pem;<br /> #  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;<br /> #  ssl_dhparam /etc/ssl/certs/dhparam.pem;</code></blockcode><blockcode><br /><code>#  keepalive_timeout 20s;</code></blockcode><blockcode><br /><code>#  root /var/www/html;<br /> #  index index.html index.htm;</code></blockcode><blockcode><br /><code>#  server_name survey.oeru.org;</code></blockcode><blockcode><br /><code>#  ## Access and error logs.<br /> #  access_log /var/log/nginx/survey.oeru.org_access.log;<br /> #  error_log /var/log/nginx/survey.oeru.org_error.log;</code></blockcode><blockcode><br /><code>  location / {<br />     proxy_set_header Host $host;<br />     proxy_set_header X-Forwarded-Host $host;<br />     proxy_set_header X-Real-IP $remote_addr;<br />     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br />     proxy_set_header X-Forwarded-Proto $scheme;<br />     add_header Front-End-Https  on;<br />     proxy_pass http://127.0.0.1:8080;<br />   }<br /> }</code></blockcode></p> <p>Note the port number 8080 - this is set in your <code>docker-compose.yml</code> file - it should be any free high port on your server (i.e. not already in use by another application - <code>netstat -punta</code> will help you work out what's in use). Change the port number to whatever you've set in the docker-compose.yml file.</p> <p>You'll note that many of the lines start with a # - these are commented out, particularly the info about SSL (secure sockets layer - encrypted transmission between users' computers and your server). Use our <a href="/protecting-your-users-lets-encrypt-ssl-certs">Lets Encrypt howto</a> to set that up - it's worth it for your users' security! Once you do that, you can remove the extra layer of # to create two server configurations - one for HTTP which redirects users to the other which implements HTTPS.</p> <p>You need to "enable" your nginx configuration like this:</p> <p><code>sudo ln -sf /etc/nginx/sites-available/limewire /etc/nginx/sites-enabled</code></p> <p>Then you can confirm that it's set up properly by running</p> <p><code>sudo nginx -t </code></p> <p>and if it returns without errors, you can implement your settings with (this should work even if you've never started nginx before!):</p> <p><code>sudo service nginx restart</code></p> <p>You should then be able to go to your site on <a href="http://survey.oeru.org">http://survey.oeru.org</a> (replaced, of course, with your own domain).</p> <h2>Upgrades</h2> <p>No recipe is very useful unless it gives you an idea how to provide ongoing maintenance for your application. Limesurvey is a wee bit tricky, but not too hard. This is how we do it when you see that there's an upgrade available (check <a href="https://hub.docker.com/r/oeru/limesurvey/">https://hub.docker.com/r/oeru/limesurvey/</a> to see!)</p> <ol><li>make a copy of your existing plugins and configuration: <code>sudo mkdir /home/data/limesurvey-backup &amp;&amp; cp -a /home/data/limesurvey/config /home/data/limesurvey/plugins /home/data/limesurvey-backup</code></li> <li>go to /home/docker/limesurvey (<code>cd /home/docker/limesurvey)</code> and run <code>docker-compose pull</code> to get the most recent Docker container I've produced.</li> <li>prepare your users for brief downtime.</li> <li>run docker-compose stop to stop the current Limesurvey site</li> <li>move the old config and plugin directories <code>sudo mv /home/data/limesurvey/config /home/data/limesurvey/config.old &amp;&amp; sudo mv /home/data/limesurvey/plugins /home/data/limesurvey/plugins.old</code> (this might overwrite previous old updates, which is hopefully ok! If not, choose something other than .old as a file suffix - perhaps the date)</li> <li>run <code>docker-compose up -d</code> This will start the new Limesurvey docker container.</li> <li>You can then run<code> docker-compose exec lime cp -a /var/www/html/application/config.orig /var/www/html/plugins.orig /var/www/reference/ </code>which is a sneaky way of running commands on the container itself. When I build the docker container you've "pulled" down, I copy the default config and plugins directories to config.orig and plugins.orig to ensure they're still there for you - this copies them somewhere on your local filesystem where you can usefully access them...</li> <li>You need to copy the reference configuration and plugins to the appropriate directories with <code>sudo cp -a /home/data/limesurvey/reference/config.orig/* /home/data/limesurvey/config/ &amp;&amp; sudo cp -a /home/data/limesurvey/reference/plugins.orig/* /home/data/limesurvey/plugins/</code></li> <li>Then you need to copy your previous config.php file to where it belongs: <code>sudo cp /home/data/limesurvey/config.old/config.php /home/data/limesurvey/config/</code> and the same with any plugins you have added to your instance. The latter will require you go copy them each individually from <code>/home/data/limesurvey/plugins.old/_____ to /home/data/limesurvey/plugins</code> - Just be sure not to overwrite plugins provided by the updated Limesurvey!</li> <li>make sure all the permissions are ok: <code>sudo chown -R www-data:${USER} /home/data/limesurvey/* &amp;&amp; sudo chmod -R g+w /home/data/limesurvey/* </code>which will allow both you to read and write all files and the webserver user (www-data) to read (and potentially write) the files it needs to.</li> <li>Finally, you need to run the database update in your browser (again, replacing with your domain name, and https if applicable): <code>http://survey.oeru.org/admin/upgrade</code> which should walk you through the process of upgrading. After that, you're good to go!</li> </ol><p> </p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=18&amp;2=field_blog_comments&amp;3=comment" token="Hm8NpVysFuvAd5MqABltr4y4y5jisiYytp23JzS6M-s"></drupal-render-placeholder> </div> </section> Thu, 18 Jan 2018 21:27:34 +0000 dave 18 at http://tech.oeru.org WikiEducator Notes: OERu's course feed aggregation and messaging system http://tech.oeru.org/wikieducator-notes-oerus-course-feed-aggregation-and-messaging-system <span class="field field--name-title field--type-string field--label-hidden">WikiEducator Notes: OERu&#039;s course feed aggregation and messaging system</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--wenotes"> <span class="field__item-wrapper"><a href="/taxonomy/term/41" hreflang="en">wenotes</a></span> </div> <div class="field__item field__item--couchdb"> <span class="field__item-wrapper"><a href="/taxonomy/term/42" hreflang="en">couchdb</a></span> </div> <div class="field__item field__item--faye"> <span class="field__item-wrapper"><a href="/taxonomy/term/43" hreflang="en">faye</a></span> </div> <div class="field__item field__item--javascript"> <span class="field__item-wrapper"><a href="/taxonomy/term/44" hreflang="en">javascript</a></span> </div> <div class="field__item field__item--nodejs"> <span class="field__item-wrapper"><a href="/taxonomy/term/39" hreflang="en">node.js</a></span> </div> <div class="field__item field__item--docker"> <span class="field__item-wrapper"><a href="/taxonomy/term/16" hreflang="en">docker</a></span> </div> <div class="field__item field__item--docker-compose"> <span class="field__item-wrapper"><a href="/taxonomy/term/25" hreflang="en">docker compose</a></span> </div> <div class="field__item field__item--mastodon"> <span class="field__item-wrapper"><a href="/taxonomy/term/31" hreflang="en">mastodon</a></span> </div> <div class="field__item field__item--hypothesis"> <span class="field__item-wrapper"><a href="/taxonomy/term/45" hreflang="en">hypothesis</a></span> </div> <div class="field__item field__item--oeru"> <span class="field__item-wrapper"><a href="/taxonomy/term/46" hreflang="en">oeru</a></span> </div> <div class="field__item field__item--lida101"> <span class="field__item-wrapper"><a href="/taxonomy/term/47" hreflang="en">lida101</a></span> </div> <div class="field__item field__item--mediawiki"> <span class="field__item-wrapper"><a href="/taxonomy/term/38" hreflang="en">mediawiki</a></span> </div> <div class="field__item field__item--wikieducator"> <span class="field__item-wrapper"><a href="/taxonomy/term/34" hreflang="en">wikieducator</a></span> </div> <div class="field__item field__item--wordpress"> <span class="field__item-wrapper"><a href="/taxonomy/term/35" hreflang="en">wordpress</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Thu 24/08/2017 - 09:21</span> <div class="float-none field field-node--field-image field-name-field-image field-type-image field-label-hidden has-multiple"> <figure class="field-type-image__figure image-count-1 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-08/WEnotes_diagram.png?itok=_aB7Lt-Y" title="A diagram describing the function of WikiEducator Notes (WEnotes)" data-colorbox-gallery="gallery-field_image-fqhif-1jxa4" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;A diagram describing the function of WikiEducator Notes (WEnotes)&quot;}"><img src="/sites/default/files/styles/medium/public/2017-08/WEnotes_diagram.png?itok=EncLVnTc" width="174" height="220" alt="A diagram describing the function of WikiEducator Notes (WEnotes)" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-2 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-09/WENotesPostMW.png?itok=sAdOfXNT" title="Example of a WEnotes feed and post widget on WikiEducator - MediaWiki" data-colorbox-gallery="gallery-field_image-fqhif-1jxa4" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Example of a WEnotes feed and post widget on WikiEducator - MediaWiki&quot;}"><img src="/sites/default/files/styles/medium/public/2017-09/WENotesPostMW.png?itok=0QqAZWQf" width="220" height="182" alt="Example of a WEnotes feed and post widget on WikiEducator - MediaWiki" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-3 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-09/WENotesPostWP.png?itok=_58SHpuz" title="Example of a WEnotes feed and post widget on Course - WordPress Multisite" data-colorbox-gallery="gallery-field_image-fqhif-1jxa4" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Example of a WEnotes feed and post widget on Course - WordPress Multisite&quot;}"><img src="/sites/default/files/styles/medium/public/2017-09/WENotesPostWP.png?itok=QQl6B5hF" width="220" height="182" alt="Example of a WEnotes feed and post widget on Course - WordPress Multisite" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> </div> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>Here at the OERu, we provide a service, attached to all of our online courses (and available to all of our partners - or anyone else for that matter) which allows anyone involved in those courses to communicate with one another from any one of a dazzling array of online "places" with WikiEducator Notes (aka WEnotes). The entire system is free and open source software (FOSS). </p> <p>The magic glue is "tags" - most social media and online systems support "tagging" in one way or another. Tools like Twitter, <a href="https://github.com/tootsuite/mastodon" title="A FOSS alternative to Twitter, but distributed rather than centralised. Ruby-on-Rails and React.js are the underlying technology.">Mastodon</a>, and G+ support "hashtags" like "#OERu" as shorthand. Other tools like blog engines (<a href="https://medium.com" title="A proprietary blogging cloud service and aggregator">Medium</a>, <a href="https://wordpress.org" title="The worlds most widely used web platform. FOSS blogging engine used by 25% of websites.">WordPress</a>, and others), forums (like <a href="https://discourse.org" title="A next generation forum engine, FOSS, built on Ruby on Rails and Ember.js for the front end.">Discourse</a>), content annotation services (like <a href="https://web.hypothes.is/" title="FOSS website annotation system - review any web page, sentence by sentence. ">Hypothesis</a>), social bookmarking services (like <a href="http://semanticscuttle.sourceforge.net/" title="The software underlying our https://bookmarks.oeru.org service">Semantic Scuttle</a>), and instant messaging services (like <a href="https://rocket.chat" title="A FOSS private messaging system built on Meteor and React.JS, with MongoDB">Rocket.Chat</a>, <a href="https://about.riot.im/" title="A FOSS React.js-based private messaging client system built on top of the Matrix (matrix.org) messaging standard and the &quot;Synapse&quot; server (Python + Twisted).">Riot</a> + <a href="https://matrix.org/docs/projects/server/synapse.html" title="The Matrix server used by the Riot messaging client. Written in Python + Twisted.">Synapse</a>, <a href="https://zulip.org" title="A FOSS private messaging service originally created by Dropbox for internal use. Written in Python with the Django framework and Electron clients.">Zulip</a>, and <a href="https://about.mattermost.com" title="A FOSS private messaging service built on Go (aka Golang)">Mattermost</a>, among others) support adding  "tags", "labels" or "categories" to any given post. WEnotes is sensitive to those tags, and thanks to the modern trend towards services providing feeds, it can happily "harvest" relevant content from all over the web, in real time (or nearly, depending on the technology).</p> <h2>How WEnotes components fit together</h2> <p>WEnotes is a collection of a significant number of separate FOSS components. See the graphical diagram of WEnotes alongside this article for an overview of the WEnotes component architecture. To provide further explanation, WEnotes is made up of several functional components:</p> <ol><li>A set of feed harvesting scripts that are part of the <a href="https://bitbucket.org/wikieducator/wenotes-tools">WEnotes-tools project</a> (written in Javascript and Python, for those interested in that sort of detail, the former ideal for those sites producing JSON feeds and real time updates, the latter our choice for periodic scans - every 10 minutes by default at the moment - of RSS and ATOM feeds)</li> <li>A central <a href="http://couchdb.apache.org/">CouchDB</a> instance, which stores all posts harvested, along with metadata like their source, a unique ID (to ensure we don't re-harvest it) and some useful info to ensure we can push the messages to subscribers effectively.</li> <li>The WEnotes client for MediaWiki or WordPress, part of the <a href="https://bitbucket.org/wikieducator/wenotes">WEnotes project</a>, which provides a real-time feed of posts, either a full feed (for example <a href="https://wikieducator.org/WENotesCompleteFeed">this one</a>) or specific to a tag (for example a specific <a href="https://course.oeru.org/lida101/interactions/course-feed/">OERu Course</a>). The client also offers a posting interface, allowing logged in users to post new posts (up to 300 characters) automatically tagged in the relevant context. See two screenshots attached.</li> <li>A "publish-subscribe" or "Pub-Sub" service, <a href="https://bitbucket.org/wikieducator/wenotes-server">WEnotes-server project</a>, a Node.JS based <a href="https://faye.jcoglan.com/">Faye</a> implementation of the <a href="https://docs.cometd.org/current/reference/index.html#_bayeux">Bayeux protocol</a>. It uses <a href="https://en.wikipedia.org/wiki/WebSocket">websockets</a> to push out real-time updates to subscribed WEnotes clients.</li> <li> <p>The "couchwatch.js" script, also part of the WEnotes-tools project, watches the CouchDB and alerts Faye whenever a new post is received, which in turn pushes it out to subscribed WENote clients.</p> </li> </ol><h2>Whence WEnotes posts come</h2> <p>Here are three scenarios which outline the process by which a WEnote post is created.</p> <h3>Handy WEnotes post form</h3> <p>The most obvious way to create a WEnotes post is to go to a WEnote feed page - for <a href="https://course.oeru.org/lida101/interactions/course-feed/">instance</a> - and log into the system (assuming you have authentication credentials or can create some). If you're logged in, you should see a simple form as illustrated in the two attached screen shots. The system then knows who you are so that your post can be properly attributed, and it knows the "context" for the feed you're looking at (namely the associated tag). You can enter a message of up to 300 characters, and hit the "Post a WEnote" button, and this will send - direct from your browser - a suitably formatted post (a simple JSON object - example below) into the WEnotes CouchDB where it is then available for inclusion in relevant feed displays.</p> <h3>Personal blog post, appropriately tagged</h3> <p>If you register for our <a href="https://course.oeru.org" title="The main OERu course delivery platform for OERu itself and many partner institutions.">Course site</a>, or any course on that site, you have the option of adding a "Blog URL" - that's the web address of your blog (the address of <em>this </em>blog is <code>https://tech.oeru.org</code>, for instance). Because most widely used blogging platforms, like the open source <a href="https://wordpress.org">WordPress</a> platform (which is the most widely use web platform in the world, the basis for <a href="https://w3techs.com/blog/entry/wordpress-powers-25-percent-of-all-websites">25% of <em>all</em> websites</a>!! For the record, this website uses <a href="https://drupal.org">Drupal</a>, which is the 3rd biggest web platform, and is also open source), and proprietary cloud blogging platforms like Blogger and Medium, provide both "tagging" of posts and a built-in <a href="http://www.whatisrss.com/">RSS feed</a>. When you know what to look for, you'll start to see them all over the place (this blog has <a href="/blog/feed">one</a> - the logo is the little grey icon on the bottom left corner of <a href="/blog" title="The Tech Blog index page.">this page</a>).</p> <p>When you create a blog post on your registered blog and tag it with a course-specific tag (each course we run has one, it's usually the short code immediately after the main course website URL: <code>https://course.oeru.org/[course code here]/...</code>) the WEnotes feed harvesters will periodically visit your blog's RSS feed and look for new posts tagged with a course tag. If it finds one, it will create a new post in CouchDB with all the necessary info to provide an informative post in the WEnotes feed, proper author attribution (and an "avatar" picture of you where available!), and a way for interested readers to get to your original post.</p> <h3>Tagged/Hashtagged social media reference</h3> <p>Because the attention of many people is heavily focused on social media - dominated by "network effect" companies like Twitter and Facebook, for better or worse, but also by an array of more open and egalitarian services (which we, on principle, prefer, but recognise that they're far less well marketed due mostly to not taking anyone's money) - we also provide a variety of social media specific "feed harvesters".  Some of these periodically scan specific social media technologies for new relevant posts, others perpetually monitor their sources providing instant updates.</p> <p>If you</p> <ul><li><a href="https://mastodon.oeru.org" title="OERu's Instance of the open source Mastodon social media platform.">Toot</a> including a relevant hashtag (e.g. #oeru or #lida101 for the "Learning in a Digital Age 101" course),</li> <li>use <a href="https://hypothes.is">Hypothesis</a> to annotate a web page, tagging it appropriately (e.g. oeru or lida101),</li> <li>add a <a href="http://semanticscuttle.sourceforge.net/">Semantic Scuttle</a> "<a href="https://bookmarks.oeru.org" title="Our Semantic Scuttle instance.">Bookmark</a>" to a web reference of interest, tagging it appropriately,</li> <li>add a topic or reply on our <a href="https://community.oeru.org" title="Our Discourse forum for educators and OER developers">Community</a> or <a href="https://forums.oeru.org" title="The OERu's Discourse forum for learners">Forums</a> Discourse instances also suitably tagged, or</li> <li>create a <a href="https://moodle.org/" title="Market leading learning management system (it's also open source).">Moodle</a> post on a Moodle site running a suitable "student bot",</li> </ul><p>the WEnotes feed harvesters will find it and create a suitable post in the CouchDB.</p> <h2>How WEnotes feeds are made</h2> <p>Once you have a bunch of WEnotes posts stored in CouchDB, how do we create a feed? At present, we have "WEnotes client" scripts that run either on MediaWiki instances, like Wikieducator, or on WordPress sites, like our Course multisite, for which content is usually provisioned per-course using our "<a href="/oeru-mediawiki-wordpress-snapshot-toolchain">Snapshot</a>" toolchain. In both cases, the client is invoked via the inclusion of a "Widget" in the MediaWiki content that will either be viewed on the <a href="https://wikieducator.org/WENotesCompleteFeed" title="An example of the WENotes Widget, in this case a full feed, not restricted to one or more tags.">MediaWiki instance itself</a>, or on a target <a href="https://course.oeru.org/lida101/interactions/course-feed/" title="This is an example of the widget being transferred to a WordPress course subsite. Note the 'Content' link at the bottom of the page, which points to the Wikieducator page which contains the actual Widget.">WordPress course subsite</a>.</p> <p>After you log into the Wikieducator site, and create or find a page on which you want a feed, you "Edit source" and you can add both a <a href="https://wikieducator.org/Widget:WEnotesPost" title="The WENotes Post Widget how-to documentation.">WEnotes Post template</a> and <a href="https://wikieducator.org/Widget:WEnotes" title="The WENotes Feed Widget documentation">WEnotes Feed template</a> two simple one line recipes like this:</p> <p><code>{{#widget:WEnotesPost|tag=lida101}}<br /> {{#widget:WEnotes|tag=lida101|count=20}} </code></p> <p>This invocation will create a page showing a WEnotes Post Widget, where any submitted post will automatically be tagged by "lida101", and directly below that, a feed showing up to 20 posts from the CouchDB tagged with "lida101". Note, any <em>new</em> suitably tagged posts will pop into the feed in realtime as they're added to CouchDB thanks to couchwatch.js and Faye - invoking a WEnotes Feed Widget automatically subscribes you to realtime updates for that tag.</p> <p>You can also create a cross-tag feed using the magic <code>"_"</code> designator. WEnotes Posts require a tag (the default, if none is specified, is "wikieducator"):</p> <p><code>{{#widget:WEnotes|tag=_|count=20}} </code></p> <h2>WEnotes technical details</h2> <p>This is a tech-blog, so I'm not going to shy away from the technical details. Here're some useful bits and pieces for folks wanting to get a better understanding of this system. If you're scared of technical details, you can stop reading here.</p> <h3>WEnotes technology infrastructure</h3> <p>We implement the services making up the WEnotes stack using a set of three Docker containers as well as our main MediaWiki implementation, WikiEducator, and our main WordPress multisite implementation, the OERu Course multisite (we also maintain development instance of each of these). The Docker containers are managed with <a href="/docker-compose-better-way-deploy-rocketchat-wekan-and-mongodb">Docker Compose</a> and the roles are divided  We've created a separate <a href="https://github.com/oeru/wenotes-docker">code repository on GitHub </a>for our evolving WEnotes stack to facilitate others making use of some or all of it! The three containers currently being deployed do the following:</p> <ol><li>run CouchDB version 2.0 or later, as well as its "Fauxton" interactive web front-end.</li> <li>run the Faye Pub-Sub websocket destination for realtime feed updates, implemented as a Node.JS service running under <a href="http://pm2.keymetrics.io/">pm2</a>.</li> <li>run the Javascript (using pm2) and Python-based (using cron to run periodically) feed harvesters and the Javascript "couchwatch.js" script which alerts Faye to the addition of new posts.</li> </ol><p>Once configured (we're working on making that a straight forward "out-of-the-box" experience, although it's not quite there yet), the three containers can be deployed together with a single invocation of Docker Compose - at the moment, though, this system is quite dependent on a properly configured "options.json" file, the format of which is still in a state of flux.</p> <h3>WENotes JSON object</h3> <p>A WEnotes post is stored in our CouchDB as a JSON object (JSON = <a href="json.org" title="JSON is rapidly superseding XML as a way of storing and propogating machine-readable structured data. ">JavaScript Object Notation</a>). This is an example of a Post as it's stored in our CouchDB:</p> <p>{<br />   "_id": "3e1c16eb1c99ae59899761a680006f4a",<br />   "_rev": "1-7462f14a0515d98b5985b9f4b8c39dd5",<br />   "media_attachments": [<br />     {<br />       "url": "<a href="https://mastodon.oeru.org/system/media_attachments/files/000/000/017/original/c3e85e56142c5c4d.png?1505016473">https://mastodon.oeru.org/system/media_attachments/files/000/000/017/or…</a>",<br />       "text_url": "<a href="https://mastodon.oeru.org/media/jvUoIpj6OAt-e5t-H8k">https://mastodon.oeru.org/media/jvUoIpj6OAt-e5t-H8k</a>",<br />       "preview_url": "<a href="https://mastodon.oeru.org/system/media_attachments/files/000/000/017/small/c3e85e56142c5c4d.png?1505016473">https://mastodon.oeru.org/system/media_attachments/files/000/000/017/sm…</a>",<br />       "meta": {<br />         "small": {<br />           "width": 400,<br />           "size": "400x321",<br />           "aspect": 1.2461059190031152,<br />           "height": 321<br />         },<br />         "original": {<br />           "width": 1148,<br />           "size": "1148x921",<br />           "aspect": 1.246471226927253,<br />           "height": 921<br />         }<br />       },<br />       "type": "image",<br />       "id": 17,<br />       "remote_url": ""<br />     }<br />   ],<br />   "truncated": true,<br />   "reblog": null,<br />   "id": 21,<br />   "in_reply_to_id": null,<br />   "content": "&lt;p&gt;Productive weekend fixing hashtags and instructions for &lt;a href=\"<a href="https://mastodon.oeru.org/tags/lida101">https://mastodon.oeru.org/tags/lida101</a>\" class=\"mention hashtag\" rel=\"tag\"&gt;#&lt;span&gt;LiDA101&lt;/span&gt;&lt;/a&gt; photo challenges . Super cool to see mastodon.oeru,org toots in the course feed. &lt;a href=\"<a href="https://mastodon.oeru.org/tags/lida101photo">https://mastodon.oeru.org/tags/lida101photo</a>\" class=\"mention hashtag\" rel=\"tag\"&gt;#&lt;span&gt;lida101photo&lt;/span&gt;&lt;/a&gt; &lt;a href=\"<a href="https://mastodon.oeru.org/media/jvUoIpj6OAt-e5t-H8k">https://mastodon.oeru.org/media/jvUoIpj6OAt-e5t-H8k</a>\" rel=\"nofollow noopener\" target=\"_blank\"&gt;&lt;span class=\"invisible\"&gt;https://&lt;/span&gt;&lt;span class=\"ellipsis\"&gt;mastodon.oeru.org/media/jvUoIp&lt;/span&gt;&lt;span class=\"invisible\"&gt;j6OAt-e5t-H8k&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;",<br />   "application": null,<br />   "text": "Productive weekend fixing hashtags and instructions for #LiDA101 photo challenges . Super cool to see mastodon.oeru,org toots in the course feed. #lida101photo...",<br />   "profile_url": "<a href="https://mastodon.oeru.org/@mackiwg">https://mastodon.oeru.org/@mackiwg</a>",<br />   "tags": [<br />     {<br />       "url": "<a href="https://mastodon.oeru.org/tags/lida101">https://mastodon.oeru.org/tags/lida101</a>",<br />       "name": "lida101"<br />     },<br />     {<br />       "url": "<a href="https://mastodon.oeru.org/tags/lida101photo">https://mastodon.oeru.org/tags/lida101photo</a>",<br />       "name": "lida101photo"<br />     }<br />   ],<br />   "visibility": "public",<br />   "we_tags": [<br />     "lida101"<br />   ],<br />   "user": {<br />     "screen_name": "mackiwg",<br />     "profile_image_url": "<a href="https://mastodon.oeru.org/system/accounts/avatars/000/000/002/original/1575afda498ac49c.jpg?1494995233">https://mastodon.oeru.org/system/accounts/avatars/000/000/002/original/…</a>",<br />     "name": "Wayne Mackintosh"<br />   },<br />   "spoiler_text": "",<br />   "we_timestamp": "2017-09-10T04:09:05.000Z",<br />   "account": {<br />     "username": "mackiwg",<br />     "display_name": "Wayne Mackintosh",<br />     "statuses_count": 10,<br />     "following_count": 1,<br />     "url": "<a href="https://mastodon.oeru.org/@mackiwg">https://mastodon.oeru.org/@mackiwg</a>",<br />     "locked": false,<br />     "created_at": "2017-04-30T01:51:59.778Z",<br />     "avatar_static": "<a href="https://mastodon.oeru.org/system/accounts/avatars/000/000/002/original/1575afda498ac49c.jpg?1494995233">https://mastodon.oeru.org/system/accounts/avatars/000/000/002/original/…</a>",<br />     "note": "&lt;p&gt;Open sourcing education at OERu&lt;/p&gt;",<br />     "header": "<a href="https://mastodon.oeru.org/headers/original/missing.png">https://mastodon.oeru.org/headers/original/missing.png</a>",<br />     "followers_count": 1,<br />     "avatar": "<a href="https://mastodon.oeru.org/system/accounts/avatars/000/000/002/original/1575afda498ac49c.jpg?1494995233">https://mastodon.oeru.org/system/accounts/avatars/000/000/002/original/…</a>",<br />     "header_static": "<a href="https://mastodon.oeru.org/headers/original/missing.png">https://mastodon.oeru.org/headers/original/missing.png</a>",<br />     "acct": "mackiwg",<br />     "id": 2<br />   },<br />   "favourites_count": 0,<br />   "language": "en",<br />   "url": "<a href="https://mastodon.oeru.org/@mackiwg/21">https://mastodon.oeru.org/@mackiwg/21</a>",<br />   "we_source": "mastodon",<br />   "in_reply_to_account_id": null,<br />   "uri": "tag:mastodon.oeru.org,2017-09-10:objectId=21:objectType=Status",<br />   "reblogs_count": 0,<br />   "sensitive": false,<br />   "mentions": [],<br />   "created_at": "2017-09-10T04:09:58.197Z"<br /> }</p> <p>The most crucial things for WEnotes are the value that start with <code>we_</code> They are used by the system which filters and displays WEnotes posts. </p> <h3>The OERu Tag List</h3> <p>The list of tags which OERu's WEnotes looks for changes from time-to-time, usually growing as we add new course tags or other tags of business. Currently, the list is being updated manually, but it is <a href="http://wenotes.oeru.org/resources/feed-sources.json" title="WENotes Tag List">available as a web feed</a> in JSON format. We intend to provide both a simple editing interface for this list and to provide an API to automate adding new tags. Eventually, when we create a new course on our Course multisite, we expect to be able to automatically registered the new course "tag" with the tag list.</p> <h2>Credits</h2> <p>The entire underlying mechanics and most of the cleverness of WikiEducator Notes are thanks to the ingenuity and hard work of <a href="https://wikieducator.org/User:JimTittsler">Jim Tittsler</a>, my predecessor as OER Foundation Technical Lead.</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=16&amp;2=field_blog_comments&amp;3=comment" token="PxBEEjMNnN6bxJwA5uTvkIiXXSU76SK-WRBHv4dgcXc"></drupal-render-placeholder> </div> </section> Wed, 23 Aug 2017 21:21:04 +0000 dave 16 at http://tech.oeru.org http://tech.oeru.org/wikieducator-notes-oerus-course-feed-aggregation-and-messaging-system#comments OERu MediaWiki to WordPress Snapshot Toolchain http://tech.oeru.org/oeru-mediawiki-wordpress-snapshot-toolchain <span class="field field--name-title field--type-string field--label-hidden">OERu MediaWiki to WordPress Snapshot Toolchain</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--wikieducator"> <span class="field__item-wrapper"><a href="/taxonomy/term/34" hreflang="en">wikieducator</a></span> </div> <div class="field__item field__item--wordpress"> <span class="field__item-wrapper"><a href="/taxonomy/term/35" hreflang="en">wordpress</a></span> </div> <div class="field__item field__item--multisite"> <span class="field__item-wrapper"><a href="/taxonomy/term/36" hreflang="en">multisite</a></span> </div> <div class="field__item field__item--snapshot"> <span class="field__item-wrapper"><a href="/taxonomy/term/37" hreflang="en">snapshot</a></span> </div> <div class="field__item field__item--mediawiki"> <span class="field__item-wrapper"><a href="/taxonomy/term/38" hreflang="en">mediawiki</a></span> </div> <div class="field__item field__item--nodejs"> <span class="field__item-wrapper"><a href="/taxonomy/term/39" hreflang="en">node.js</a></span> </div> <div class="field__item field__item--php"> <span class="field__item-wrapper"><a href="/taxonomy/term/40" hreflang="en">php</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Fri 16/06/2017 - 11:19</span> <div class="float-none field field-node--field-image field-name-field-image field-type-image field-label-hidden has-multiple"> <figure class="field-type-image__figure image-count-1 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-06/WikiEducatorSnapshotExample.png?itok=RUIJFzD9" title="Example of a WikiEducator Outline Page with the Snapshot button" data-colorbox-gallery="gallery-field_image--L25Bhz3pro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Example of a WikiEducator Outline Page with the Snapshot button&quot;}"><img src="/sites/default/files/styles/medium/public/2017-06/WikiEducatorSnapshotExample.png?itok=4cDqr9-n" width="220" height="125" alt="Example of a WikiEducator Outline Page with the Snapshot button" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-2 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-06/WikiEducatorSnapshotAuthenticationExample.png?itok=y_-fZCbJ" title="Example of Snapshot authentication for remote WordPress site" data-colorbox-gallery="gallery-field_image--L25Bhz3pro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Example of Snapshot authentication for remote WordPress site&quot;}"><img src="/sites/default/files/styles/medium/public/2017-06/WikiEducatorSnapshotAuthenticationExample.png?itok=iCXP2eXj" width="220" height="125" alt="Example of Snapshot authentication for remote WordPress site" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-3 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-06/WikiEducatorSnapshotSampleCoursePage.png?itok=4dq_F2B7" title="A sample course page showing the adaptive layout (mobile-friendly) interface and navigation." data-colorbox-gallery="gallery-field_image--L25Bhz3pro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;A sample course page showing the adaptive layout (mobile-friendly) interface and navigation.&quot;}"><img src="/sites/default/files/styles/medium/public/2017-06/WikiEducatorSnapshotSampleCoursePage.png?itok=DOT2vvTM" width="220" height="145" alt="A sample course page showing the adaptive layout (mobile-friendly) interface and navigation." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-4 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-06/WikiEducatorSnapshotSampleCoursePageContentLink.png?itok=Mx5-yzwM" title="Sample of the link (#1) back to the WikiEducator page from which this WordPress Course page is derived" data-colorbox-gallery="gallery-field_image--L25Bhz3pro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Sample of the link (#1) back to the WikiEducator page from which this WordPress Course page is derived&quot;}"><img src="/sites/default/files/styles/medium/public/2017-06/WikiEducatorSnapshotSampleCoursePageContentLink.png?itok=PTtiGfSy" width="220" height="145" alt="Sample of the link (#1) back to the WikiEducator page from which this WordPress Course page is derived" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-5 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-06/WikiEducatorSnapshotSampleCoursePageNarrowLayout.png?itok=RUWJTmk0" title="The same Course Page example viewed on a narrower screen, demonstrating the adaptive &quot;mobile-friendly&quot; layout." data-colorbox-gallery="gallery-field_image--L25Bhz3pro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;The same Course Page example viewed on a narrower screen, demonstrating the adaptive &quot;mobile-friendly&quot; layout.&quot;}"><img src="/sites/default/files/styles/medium/public/2017-06/WikiEducatorSnapshotSampleCoursePageNarrowLayout.png?itok=m5NnfsfY" width="65" height="220" alt="The same Course Page example viewed on a narrower screen, demonstrating the adaptive &quot;mobile-friendly&quot; layout." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> </div> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>One of the OERu's most compelling technological capabilities is the set of internally developed open source tools that allow us to automatically transform and inject a collection of MediaWiki content on <a href="https://wikieducator.org">WikiEducator</a> (usually in form of micro-courses) making up a course into a mobile-friendly, easy-to-navigate WordPress site, usually situated on OERu's <a href="https://course.oeru.org">Course WordPress Multisite</a> implementation.</p> <h2>How it works</h2> <p>Educators, usually working with one of our <a href="https://oeru.org/oeru-partners/">OERu partner institutions</a>, can assemble the materials (either written by them or recruited from existing OERs on Wikieducator) into an "outline" which references all the content in one place. If they want to make the course available (either a cohort-based group or a "rolling" enrolment, either affiliated with their institution or under the OERu banner) they can do so either on a subsite on OERu's WordPress instance or some other WordPress, for example one hosted by their institution.</p> <h3>MediaWiki Widget</h3> <p>The process of linking an outline to a WordPress site and pushing a "snapshot" of the course content outlined to the WordPress site is remarkably simple. First, the author of the outline page includes a one line MediaWiki template or "widget" - which is managed by the open source <a href="https://bitbucket.org/wikieducator/wesnapshot" title="Open Source MediaWiki plugin component of the OERu Snapshotting process">OERu MediaWiki plugin</a> (written in <a href="http://php.net">PHP</a>, like the rest of <a href="https://www.mediawiki.org/wiki/MediaWiki">MediaWiki</a>) developed by <a href="http://wikieducator.org/User:JimTittsler">Jim Tittsler</a> - that will look something like this (usually in one line, expanded here for clarity):</p> <p><code>{{ #widget:Snapshot |<br /> url=http://course.oeru.org/shortcode/ |<br /> logo=https://wikieducator.org/images/f/f1/MyInstitutionLogo.png |<br /> link=http://oeru.org/oeru-partners/my-institution-profile/}}</code></p> <p>This defines</p> <ul><li>the URL of the WordPress site, using the "shortcode" the educator has defined for that course (like the institution's course code),</li> <li>a logo image to display at the bottom of each page of the WordPress course site to identify the relevant institution, and</li> <li>the link to which a user will be sent if they click on the logo.</li> </ul><p>There is a <a href="https://wikieducator.org/Widget:Snapshot">handy instruction page</a> for the widget if you want a bit more detail.</p> <h3>Course Snapshot Script</h3> <p>When the page containing this template is saved, the resulting page will have a "Request snapshot" button at the top of it (see attached screenshot). Clicking this will popup a modal dialog box into which the user will be asked to enter their login and password <em>for an admin user on the listed WordPress site</em> (either multisite or stand-alone implementation).</p> <p>If the WordPress site in question accepts these credentials, a new Snapshot task is added to WikiEducator's task list, handing over control to Jim's rather clever open source <a href="https://bitbucket.org/wikieducator/course">Course Snapshot</a> script which runs in the background, and checks the task list ever minute or two. When it identifies as snapshot request, it uses the outline page and WikiEducator's API to grab all the relevant content automatically, and it builds a structured data archive (in a variant of XML) which, when completed, is imported directly into a WordPress site via WordPress' XMLRPC capabilities, replacing any existing content.</p> <p>Note: the default export format used by the Course Snapshot script is a WordPress-importable data object, however for historical reasons, it also has the ability to generate (more manually at this stage) a static, internally linked HTML version of the content.</p> <h3>WordPress Mobile-Friendly Course Site</h3> <p>The resulting WordPress course site uses the OERu-created open source mobile-friendly <a href="https://github.com/oeru/oeru_course">Course WordPress theme</a> - see the sample page screen shot. Each page includes a link back to the original page in WikiEducator from which it was generated - see sample page screen shot with the link identified. Also see the sample image of the same page as it would look on a mobile device with a narrower (lower pixel resolution) screen width.</p> <p>Note: anyone can copy an outline page written by someone else and use it as the starting point for assembling their own version of the course - they will, however, have to have admin rights on the target WordPress site (and log in to it to initiate the process) protecting any existing snapshot sites from being overwritten by someone else's course content.</p> <h3>More examples</h3> <p><a href="http://wikieducator.org/OERu/OERu_authoring_model" title="Examples of courses and course &quot;Outline pages&quot; on WikEducator">Here's a list</a> of some of the currently available course outlines on WikiEducator with corresponding links to the "rendered" WordPress courses so you can see what it looks like.</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=15&amp;2=field_blog_comments&amp;3=comment" token="Su39uEtmGb5TGwGIb3Eyr4NoVO4Ofw4MyngXZefYMqA"></drupal-render-placeholder> </div> </section> Thu, 15 Jun 2017 23:19:06 +0000 dave 15 at http://tech.oeru.org Installing Mastodon with Docker-Compose on Ubuntu 16.04 http://tech.oeru.org/installing-mastodon-docker-compose-ubuntu-1604 <span class="field field--name-title field--type-string field--label-hidden">Installing Mastodon with Docker-Compose on Ubuntu 16.04</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--mastodon"> <span class="field__item-wrapper"><a href="/taxonomy/term/31" hreflang="en">mastodon</a></span> </div> <div class="field__item field__item--ubuntu-linux"> <span class="field__item-wrapper"><a href="/taxonomy/term/12" hreflang="en">ubuntu linux</a></span> </div> <div class="field__item field__item--_604"> <span class="field__item-wrapper"><a href="/taxonomy/term/27" hreflang="en">16.04</a></span> </div> <div class="field__item field__item--nginx"> <span class="field__item-wrapper"><a href="/taxonomy/term/30" hreflang="en">nginx</a></span> </div> <div class="field__item field__item--lets-encrypt"> <span class="field__item-wrapper"><a href="/taxonomy/term/17" hreflang="en">let&#039;s encrypt</a></span> </div> <div class="field__item field__item--ruby-on-rails"> <span class="field__item-wrapper"><a href="/taxonomy/term/22" hreflang="en">ruby on rails</a></span> </div> <div class="field__item field__item--docker"> <span class="field__item-wrapper"><a href="/taxonomy/term/16" hreflang="en">docker</a></span> </div> <div class="field__item field__item--docker-compose"> <span class="field__item-wrapper"><a href="/taxonomy/term/25" hreflang="en">docker compose</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Fri 02/06/2017 - 15:02</span> <div class="float-none field field-node--field-image field-name-field-image field-type-image field-label-hidden has-multiple"> <figure class="field-type-image__figure image-count-1 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-06/mastodon_userscreen2.png?itok=pyxlNcbL" title="The Mastodon webapp, showing a federated timeline (right panel - and yes, Mastodon is popular in Japan)" data-colorbox-gallery="gallery-field_image--L25Bhz3pro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;The Mastodon webapp, showing a federated timeline (right panel - and yes, Mastodon is popular in Japan)&quot;}"><img src="/sites/default/files/styles/medium/public/2017-06/mastodon_userscreen2.png?itok=P5YMy5Yf" width="220" height="141" alt="The Mastodon webapp, showing a federated timeline (right panel - and yes, Mastodon is popular in Japan)" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-2 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-06/mastodon_user_settings.png?itok=1CcpPZP0" title="Mastodon user settings" data-colorbox-gallery="gallery-field_image--L25Bhz3pro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Mastodon user settings&quot;}"><img src="/sites/default/files/styles/medium/public/2017-06/mastodon_user_settings.png?itok=w2QahwB4" width="220" height="141" alt="Mastodon user settings" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-3 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-06/mastodon_admin_settings.png?itok=cRSdDIKu" title="Mastodon administrator settings" data-colorbox-gallery="gallery-field_image--L25Bhz3pro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Mastodon administrator settings&quot;}"><img src="/sites/default/files/styles/medium/public/2017-06/mastodon_admin_settings.png?itok=jrYKu26i" width="220" height="141" alt="Mastodon administrator settings" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-4 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-06/mastodon_social_info.png?itok=_LbvxOiq" title="Info on a heavily used Mastodon node (Mastodon.Social, the reference node)" data-colorbox-gallery="gallery-field_image--L25Bhz3pro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Info on a heavily used Mastodon node (Mastodon.Social, the reference node)&quot;}"><img src="/sites/default/files/styles/medium/public/2017-06/mastodon_social_info.png?itok=N2x5gmY2" width="220" height="143" alt="Info on a heavily used Mastodon node (Mastodon.Social, the reference node)" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-5 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-06/mastodon_nzoss_info.png?itok=mT-9glYy" title="Info on a more humble node (the NZ Open Source Society node I run)" data-colorbox-gallery="gallery-field_image--L25Bhz3pro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Info on a more humble node (the NZ Open Source Society node I run)&quot;}"><img src="/sites/default/files/styles/medium/public/2017-06/mastodon_nzoss_info.png?itok=93rckq5L" width="220" height="141" alt="Info on a more humble node (the NZ Open Source Society node I run)" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> </div> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>Not long ago, <a href="https://github.com/tootsuite/mastodon" title="Scroll down to see the write up - the source code is front and centre.">Mastodon</a>, an open source, <em>federated</em> alternative to the proprietary network-effect wunderkind, Twitter, came out of no where. Actually, it came out of an insane amount of work done by free and open source powerhouse Eugen Rochko aka <a href="https://github.com/Gargron">Gargron</a> and a small elite developer community, and many predecessors who are part of the <a href="https://www.coactivate.org/projects/disintermedia/blog/2017/04/01/a-brief-history-of-the-gnu-social-fediverse-and-the-federation/">GNU Social Fediverse</a> (kudos to <a href="http://www.coactivate.org/people/strypey/profile">Danyl Strype</a> for compiling that excellent history).</p> <p>Mastodon, unlike Twitter, is entirely community driven - there are no ads, there are no privacy threats, there are no corporate Terms and Conditions to blindly "I Accept". And your Mastodon "persona" can be on a server you control (or that is controlled by someone you trust). Despite being distributed, you're still part of a global network, but one made resilient by its federated, decoupled nature.</p> <p>Instead of "Tweeting" in 140 characters like on Twitter, your "Toots" are limited to 500 characters (a lot more information can usefully be passed). You can follow people (and they you) by learning their handle - which looks like an email. I've got a couple Mastodon accounts, but my main one right now is <a href="mailto:lightweight@mastodon.social">lightweight@mastodon.social</a> (I set it up quite a while back, before I set up my first couple Mastodon servers). Actually, Mastodon's biggest problem (in my opinion) right now is that you can't easily migrate your "main" persona from one server to another without losing a lot of its value (historical toots, followers, those you follow, etc.). You can migrate some things, like those you're following, and any users you've "blocked" but it's still fairly rudimentary.</p> <p>Mastodon includes a nice web interface which will look somewhat familiar to anyone who's used Twitter's "Tweetdeck" web application. Similarly, the GNU Social community has rallied to provide at least 2 separate open source mobile apps (I run <a href="https://play.google.com/store/apps/details?id=com.keylesspalace.tusky&amp;hl=en">Tusky</a> on my <a href="https://lineageos.org" title="Open Source Android - the way it was supposed to be before the OEMs messed it up.">LineageOS</a> powered phone at the moment) - I think there're some for iOS, too, although Apple's not as amenable to open source apps. </p> <p>There's a useful <a href="https://github.com/tootsuite/documentation/blob/master/Using-Mastodon/FAQ.md">Mastodon FAQ</a>.</p> <h2>Running with the Mastodon Herd</h2> <p>The way I implement a complex Ruby on Rails app like Mastodon is to do as much as possible to keep it at arms length (and stop it from getting anything gooey on my virtual machine). To achieve that comforting isolation, I employ Docker Compose on Ubuntu Linux 16.04. See our <a href="/docker-compose-better-way-deploy-rocketchat-wekan-and-mongodb">Docker Compose article</a> on how to install it (and its dependencies, like Docker itself).</p> <p>Once you've got Docker Compose running, you can do what I did. </p> <p>A couple notes:</p> <ul><li>I have an unprivileged user on my server, "ubuntu". You can use any unprivileged users - I'd encourage you to use sudo rather than login as root.</li> <li>I use "vim" as my terminal-based text editor below. I think it's a great tool, but it does have a learning curve. If you're daunted (no shame in that), I recommend using "nano" instead - it'll probably installed on most Ubuntu 16.04 instances. If someone suggests you use "emacs" instead, they're jerkin' yer chain (I used emacs for over a decade, I know what I'm talking about).</li> <li>make sure you have the "git" VCS system installed... <code>sudo apt install git</code> should do it.</li> <li>you'll need nginx installed, too... <code>sudo apt install nginx-full</code> will do that for you.</li> </ul><p>After logging into my server (via SSH remotely) as the ubuntu user (you might have different non-privileged user name, that's ok), I did the following (to avoid permissions problems later on, we'll create a "mastodon" group and user with the id 991, used by the Mastodon app by default, on the hosting platform):</p> <p><code>groupadd -g 991 mastodon<br /> useradd -u 991 -g 991 -c "Mastodon User" -s /usr/bin/nologin -d /home/data/mastodon mastodon</code><br /><code>sudo mkdir -p /home/docker /home/data/mastodon<br /> sudo chown -R ubuntu:ubuntu /home/docker<br /> sudo chown -R mastodon:mastodon /home/data/mastodon</code><br /><code>cd /home/docker<br /> git clone https://github.com/tootsuite/mastodon.git docker-mastodon<br /> cd docker-mastodon</code></p> <p>What you then need to do is ensure you're using the current "tagged" release (it'll make your life easiest). You can find out what tags are available:</p> <p><code>git tag -l </code></p> <p>At present, the latest tag is "v1.4.7" - to use it do this:</p> <p><code>git checkout tags/v1.4.7</code></p> <p>Obviously, replace this with the most recent tag (note, you might have to look through the whole list to find it!). Then you're using the specific collection of files corresponding to the v1.4.7 tagged release. We can carry on...</p> <p><code>cp .env.production.sample .env.production<br /> vim .env.production</code></p> <p>Edit this file to look like the .env.production sample below, but replacing the [tokens] with your values. Then run this:</p> <p><code>vim docker-compose.yml</code></p> <p>Edit this file to look like docker-compose.yml below.</p> <p><code>docker-compose run --rm web rake secret </code></p> <p>Run this last command 3 times - to get 3 secrets - long random strings - for .env.production! Copy and paste your 3 secrets into your .env.production file with your preferred editor as shown below.</p> <p><code>docker-compose build<br /> docker-compose up</code></p> <p>That should download the required Docker images (might take quite a while depending on how fast your server's network connection is) and result in starting 5 different Docker containers, and you'll be able to watch them put out status (and error) messages as they boot and find their various dependencies. If there're no obvious errors, you can hit CTRL-C to shut them down again and restart them in a mode that keeps them running after you log out</p> <p><code>docker-compose up -d</code></p> <p>Note, you can always stop the containers by running docker-compose stop in that directory. You can check their status by running</p> <p><code>docker ps</code></p> <p>which should show you something like this:</p> <p><code>CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                NAMES<br /> c6be9f3eef1e        gargron/mastodon         "bundle exec rails s "   13 days ago         Up 13 days          127.0.0.1:3000-&gt;3000/tcp, 4000/tcp   dockermastodon_web_1<br /> 6a123d9b1843        gargron/mastodon         "bundle exec sidekiq "   13 days ago         Up 13 days          3000/tcp, 4000/tcp                   dockermastodon_sidekiq_1<br /> f06c4a9bc479        gargron/mastodon         "npm run start"          13 days ago         Up 13 days          3000/tcp, 127.0.0.1:4000-&gt;4000/tcp   dockermastodon_streaming_1<br /> 6dbfad0669f8        postgres:alpine          "docker-entrypoint.sh"   2 weeks ago         Up 13 days          5432/tcp                             dockermastodon_postgres_1<br /> 8026b79e976d        redis:alpine             "docker-entrypoint.sh"   4 weeks ago         Up 13 days          6379/tcp                             dockermastodon_redis_1</code></p> <p>You can use the 12 digit IDs to run other Docker commands, like <code>docker inspect [ID]</code> or <code>docker exec -it [ID] bash</code> to log into the container itself and get a bash prompt. After all that's running, you can do some final housekeeping:</p> <p><code>docker-compose run --rm web rails db:migrate<br /> docker-compose run --rm web rails assets:precompile<br /> sudo vim /etc/nginx/sites-available/mastodon</code></p> <p>Edit this to look like the mastodon nginx config file below.</p> <p><code>sudo cd /etc/nginx/sites-enabled<br /> sudo ln -sf ../sites-available/mastodon .</code></p> <p>to enable the new configuration...</p> <p><code>sudo nginx -t</code></p> <p>To check for typos in you file. If you get no errors, you can restart nginx:</p> <p><code>sudo service nginx restart</code></p> <p>When that's done,  to [your domain] in your browser, which should take you to https://[your domain] and create a new user. If your email is set up properly, you'll get an email confirmation, and this will allow you to log in. If that works, I'd encourage you to modify your configuration to use a Let's Encrypt SSL certificate to protect your users' (and your server's) security. <a href="/protecting-your-users-lets-encrypt-ssl-certs">We provide this dedicated howto</a>! The .env.production template below <em>assumes you've done this</em>, so if your Mastodon isn't working, that might be why (you can try turning <code>LOCAL_HTTPS=false</code> temporarily if that's helpful).</p> <p>You will want to create an admin user - create the user first through the web interface, and then on the command line run (replacing <code>[admin username] </code>with the username you set up:</p> <p><code>cd /home/docker</code><code>/docker-mastodon<br /> docker-compose run --rm web rails mastodon:make_admin USERNAME=[admin username]</code></p> <p>Then go to that user's Mastodon preferences and define the relevant info for your instance (see the administration options).</p> <h2>Debugging</h2> <p>If you run in to problems, a very useful Docker Compose option to use (from within the docker-mastodon directory) is </p> <p><code>docker-compose logs -f</code></p> <p>It will provide you with the automatically updating integrated logs of all the containers you've unleashed!</p> <h3>Sample .env.production</h3> <p>Here's a sample with (hopefully obviously named) [placeholders]</p> <p><code># Service dependencies<br /> REDIS_HOST=redis<br /> REDIS_PORT=6379<br /> DB_HOST=postgres<br /> DB_USER=postgres<br /> DB_NAME=postgres<br /> DB_PASS=<br /> DB_PORT=5432</code></p> <p><code># Federation<br /> LOCAL_DOMAIN=[your domain]<br /> LOCAL_HTTPS=true</code></p> <p><code># Application secrets<br /> # Generate each with the `rake secret` task (`docker-compose run --rm web rake secret` if you use docker compose)<br /> PAPERCLIP_SECRET=[first secret]<br /> SECRET_KEY_BASE=[second secret]<br /> OTP_SECRET=</code>[third secret]</p> <p><code># Registrations<br /> # Single user mode will disable registrations and redirect frontpage to the first profile<br /> # SINGLE_USER_MODE=true<br /> # Prevent registrations with following e-mail domains<br /> # EMAIL_DOMAIN_BLACKLIST=example1.com|example2.de|etc</code></p> <p><code># E-mail configuration<br /> SMTP_SERVER=[smtp server domain name]<br /> SMTP_PORT=587<br /> SMTP_LOGIN=[smtp user name]<br /> SMTP_PASSWORD=[smtp user password]<br /> SMTP_FROM_ADDRESS=[sender address for outgoing mastodon emails]<br /> SMTP_DOMAIN=[your site's base domain]<br /> SMTP_OPENSSL_VERIFY_MODE=none</code></p> <p><code># Optional asset host for multi-server setups<br /> # CDN_HOST=assets.example.com</code></p> <p><code># S3 (optional)<br /> # S3_ENABLED=true<br /> # S3_BUCKET=<br /> # AWS_ACCESS_KEY_ID=<br /> # AWS_SECRET_ACCESS_KEY=<br /> # S3_REGION=<br /> # S3_PROTOCOL=http<br /> # S3_HOSTNAME=192.168.1.123:9000</code></p> <p><code># Optional alias for S3 if you want to use Cloudfront or Cloudflare in front<br /> # S3_CLOUDFRONT_HOST=</code></p> <p><code># Streaming API integration<br /> # STREAMING_API_BASE_URL=</code></p> <h3>Sample docker-compose.yml</h3> <p>Here's a sample with [placeholders]. Note - this generates <strong>five Docker containers. </strong>Yeah, like I said, this is a serious, complex app.</p> <p><code>version: '2'<br /> services:<br />   postgres:<br />     restart: unless-stopped<br />     image: postgres:alpine<br />     volumes:<br />      - /home/data/mastodon/postgres:/var/lib/postgresql/data<br />   redis:<br />     restart: unless-stopped<br />     image: redis:alpine<br />     volumes:<br />      - /home/data/mastodon/redis:/data<br />   web:<br />     restart: unless-stopped<br />     build: .<br />     image: gargron/mastodon<br />     env_file: .env.production<br />     command: bundle exec rails s -p 3000 -b '0.0.0.0'<br />     ports:<br />       - "127.0.0.1:3000:3000"<br />     depends_on:<br />       - postgres<br />       - redis<br />     volumes:<br />       - /home/data/mastodon/packs:/mastodon/public/packs</code><br /><code>      - /home/data/mastodon/assets:/mastodon/public/assets</code><br /><code>      - /home/data/mastodon/system:/mastodon/public/system<br />   streaming:<br />     restart: unless-stopped<br />     build: .<br />     image: gargron/mastodon<br />     env_file: .env.production<br />     command: npm run start<br />     ports:<br />       - "127.0.0.1:4000:4000"<br />     depends_on:<br />       - postgres<br />       - redis<br />   sidekiq:<br />     restart: unless-stopped<br />     build: .<br />     image: gargron/mastodon<br />     env_file: .env.production<br />     command: bundle exec sidekiq -q default -q mailers -q pull -q push<br />     depends_on:<br />       - postgres<br />       - redis<br />     volumes:<br />       - /home/data/mastodon/system:/mastodon/public/system</code></p> <h3>Sample nginx mastodon config file</h3> <p>Here's a copy of the nginx configuration file I use (with [placeholders], obviously) - it's the result of quite a lot of tweaking. Have fun!</p> <p><code>map $http_upgrade $connection_upgrade {<br />     default upgrade;<br />     ''      close;<br /> }</code></p> <p><code>server {<br />     listen 80;<br /> #    listen [::]:80;<br />     server_name [your domain];<br />     root /var/www/html;</code></p> <p><code>    # for let's encrypt renewals!<br />     location /.well-known/acme-challenge/ {<br />         default_type text/plain;<br />         root /var/www/html;<br />    }</code></p> <p><code>    # redirect all HTTP traffic to HTTPS.<br />     location / {<br />         return 302 https://[your domain]$request_uri;<br />     }<br /> }</code></p> <p><code>server {<br />     listen 443 ssl;<br /> #    listen [::]:443 ssl;<br />     server_name [your domain];</code></p> <p><code>    ssl_certificate /etc/letsencrypt/live/[your domain]/fullchain.pem;<br />     ssl_certificate_key /etc/letsencrypt/live/[your domain]/privkey.pem;<br />     ssl_protocols TLSv1 TLSv1.1 TLSv1.2;<br />     # from https://0x39b.fr/post/nginx_security/<br />     ssl_session_timeout 1d;<br />     ssl_session_cache shared:SSL:50m;<br />     #ssl_session_tickets off;<br />     ssl_prefer_server_ciphers on;<br />     ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';<br />     # OCSP Stapling ---<br />     # fetch OCSP records from URL in ssl_certificate and cache them<br />     ssl_stapling on;<br />     ssl_stapling_verify on;<br />     # to create this, see https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html<br />     ssl_dhparam /etc/ssl/certs/dhparam.pem;</code></p> <p><code>    # for let's encrypt renewals!<br />     location /.well-known/acme-challenge/ {<br />         default_type text/plain;<br />         root /var/www/html;<br />     }</code></p> <p><code>    keepalive_timeout    70;<br />     sendfile             on;<br />     client_max_body_size 0;<br />    </code>  <code># update from https://github.com/tootsuite/documentation/blob/master/Running-Mastodon/Production-guide.md<br />     gzip on;<br />     gzip_vary on;<br />     gzip_proxied any;<br />     gzip_comp_level 6;<br />     gzip_buffers 16 8k;<br />     gzip_http_version 1.1;<br />     gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;</code></p> <p><code>    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";</code></p> <p><code>    location / {<br />         try_files $uri @proxy;<br />     }</code></p> <p><code>    location ~ ^/(packs|system/media_attachments/files|system/accounts/avatars) {<br />         add_header Cache-Control "public, max-age=31536000, immutable";<br />         try_files $uri @proxy;<br />     }</code></p> <p> </p> <p><code>    location @proxy {<br />         proxy_set_header Host $host;<br />         proxy_set_header X-Real-IP $remote_addr;<br />         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br />         proxy_set_header X-Forwarded-Proto https;</code><br /><code>        proxy_set_header Proxy "";</code><br /><code>        proxy_pass_header Server;<br />         proxy_pass http://localhost:3000;<br />         proxy_buffering off;<br />         proxy_redirect off;<br />         proxy_http_version 1.1;<br />         proxy_set_header Upgrade $http_upgrade;<br />         proxy_set_header Connection $connection_upgrade;<br />         tcp_nodelay on;<br />     }</code></p> <p><code>    location /api/v1/streaming {<br />         proxy_set_header Host $host;<br />         proxy_set_header X-Real-IP $remote_addr;<br />         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br />         proxy_set_header X-Forwarded-Proto https;<br />         proxy_set_header Proxy "";<br />         proxy_pass http://localhost:4000;<br />         proxy_buffering off;<br />         proxy_redirect off;<br />         proxy_http_version 1.1;<br />         proxy_set_header Upgrade $http_upgrade;<br />         proxy_set_header Connection $connection_upgrade;<br />         tcp_nodelay on;<br />     }</code></p> <p><code>    error_page 500 501 502 503 504 /500.html;<br />     # this should give you an A+ rating on https://instances.mastodon.xyz/<br />     add_header X-XSS-Protection "1; mode=block";<br />     add_header Content-Security-Policy "default-src 'none'; font-src 'self'; media-src 'self'; style-src 'self' 'unsafe-inline'; script-src 'self'; img-src 'self' data: blob:; connect-src 'self' wss://[your domain]";<br /> }</code></p> <p>Enjoy!</p> <h2>Keeping Mastodon up to date</h2> <p>To ensure your Mastodon doesn't become a run down abandoned trailerpark node bit rotting quietly in the ether, I recommend you keep it up to date! There is a useful <a href="https://github.com/tootsuite/documentation/blob/master/Running-Mastodon/Docker-Guide.md">Mastodon Administrator's guide for Docker instances</a> that I consult every time I want to update. Note, if the "git stash" part of it is too hard, I recommend that any time you change your docker-compose.yml file, you copy it to</p> <p><code>cp docker-compose.yml docker-compose.yml-backup</code></p> <p>That way, you can simply remove docker-compose.yml (double check your docker-compose.yml-backup is up-to-date first!), do the <code>git checkout TAG_NAME</code>, and then</p> <p><code>cp docker-compose.yml-backup docker-compose.yml </code></p> <p>and you're done. Welcome the Fediverse!</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=14&amp;2=field_blog_comments&amp;3=comment" token="MWkKHP2AmYlJPPoRbiZwT_Yf5xOfApOMw_Kb2WBZEpo"></drupal-render-placeholder> </div> </section> Fri, 02 Jun 2017 03:02:31 +0000 dave 14 at http://tech.oeru.org Docker Compose: A better way to deploy Rocketchat, Wekan, and MongoDB http://tech.oeru.org/docker-compose-better-way-deploy-rocketchat-wekan-and-mongodb <span class="field field--name-title field--type-string field--label-hidden">Docker Compose: A better way to deploy Rocketchat, Wekan, and MongoDB</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--docker-compose"> <span class="field__item-wrapper"><a href="/taxonomy/term/25" hreflang="en">docker compose</a></span> </div> <div class="field__item field__item--rocketchat"> <span class="field__item-wrapper"><a href="/taxonomy/term/18" hreflang="en">rocket.chat</a></span> </div> <div class="field__item field__item--wekan"> <span class="field__item-wrapper"><a href="/taxonomy/term/15" hreflang="en">wekan</a></span> </div> <div class="field__item field__item--ubuntu-linux"> <span class="field__item-wrapper"><a href="/taxonomy/term/12" hreflang="en">ubuntu linux</a></span> </div> <div class="field__item field__item--_604"> <span class="field__item-wrapper"><a href="/taxonomy/term/27" hreflang="en">16.04</a></span> </div> <div class="field__item field__item--docker"> <span class="field__item-wrapper"><a href="/taxonomy/term/16" hreflang="en">docker</a></span> </div> <div class="field__item field__item--mongodb"> <span class="field__item-wrapper"><a href="/taxonomy/term/14" hreflang="en">mongodb</a></span> </div> <div class="field__item field__item--nginx"> <span class="field__item-wrapper"><a href="/taxonomy/term/30" hreflang="en">nginx</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Tue 23/05/2017 - 11:03</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>A few months back, I posted instructions on deploying <a href="/installing-rocketchat-docker-ubuntu-linux-1404">Rocket.Chat</a> and <a href="/installing-wekan-docker-ubuntu-linux-1404">Wekan</a> instances (and their mutual dependency, <a href="/installing-mongodb-docker-ubuntu-linux-1404">MongoDB</a>) individually. Since then, I've spent some time with Docker Compose, a set of scripts which help you to define, build, and manage a set of Docker containers. Docker Compose is a thing of beauty. This is the way I now deploy Rocket.Chat, Wekan, and MongoDB together.</p> <h2>Install Docker and Docker Compose</h2> <p>Install <a href="https://docs.docker.com/engine/installation/linux/ubuntu/">Docker</a> (including the "<a href="https://docs.docker.com/engine/installation/linux/linux-postinstall/">post-installation</a>" steps to allow non-root users to run Docker) and <a href="https://docs.docker.com/compose/install/#alternative-install-options" title="We recommend the &quot;pip&quot; install method">Docker Compose</a> on your server (we recommend Ubuntu 16.04 or the older 14.04). We recommend using the "pip" (Python package manager) to do the install.</p> <h2>Create your Docker Compose recipe</h2> <p>We recommend creating a directory with an obvious name - in my case, it's <code>/home/www/docker-rocketchat-wekan-mongo</code></p> <p>In that directory, I create a file called <code>docker-compose.yml</code> containing (I've removed implementation specific details and replaced them with [placeholders]):</p> <p><code>version: '2'<br /> services:<br />   mongo:<br />     restart: unless-stopped<br />     image: mongo<br />     volumes:<br />       - [data directory path]:/data/db<br />       - [backup directory path]:/backups<br />     command: --smallfiles<br />   rocketchat-oeru:<br />     restart: unless-stopped<br />     image: rocketchat/rocket.chat<br />     ports:<br />       - "127.0.0.1:[port number]:3000" # should be a free port above 1024<br />     depends_on:<br />       - mongo<br />     environment:<br />       - MONGO_URL=mongodb://mongo/rocket<br />       - ROOT_URL=[domain name (including schema, e.g. http://)]<br />     volumes:<br />       - [upload directory path]:/var/www/rocket.chat/uploads<br />   wekan:<br />     restart: unless-stopped<br />     image: mquandalle/wekan<br />     ports:<br />       - "127.0.0.1:[port number]:80" # should be a free port above 1024<br />     depends_on:<br />       - mongo<br />     environment:<br />       - VIRTUAL_HOST=[domain name (don't include schema, e.g. https://)]<br />       - MONGO_URL=mongodb://mongo/plan<br />       - ROOT_URL=[domain name (include schema, e.g. https://)]<br />       - MAIL_URL=smtp://[smtp username]:[smtp password]@[server name or IP]:[port: 25, 465, or 587]/<br />     volumes:<br />       - [path to public content]:/built_app/programs/web.browser/app</code></p> <p>Note, you can include multiple instances of either Rocket.Chat or Wekan simply by providing a new name (e.g. rocketchat2 or wekan2 or similar) and a new set of properties - just make sure you're using a unique (and otherwise unused) port number! You can check what's on your server's ports using <code>netstat -punta | less </code>to make sure you're not doubling up. </p> <p>In case it's not obvious, you can leave out either the rocketchat or wekan definitions if you don't want to run those services!</p> <h2>Creating and Running the Docker Containers</h2> <p>It's easy to create the containers: simply run</p> <p><code>docker pull mongo<br /> docker pull rocket.chat<br /> docker pull mquandalle/wekan</code></p> <p>and when it's finished, run</p> <p><code>docker-compose up </code></p> <p>which should start all your containers, but leave you with a running log - this is great for testing, but when you're happy it's all running you hit CTRL-C (to shut down the current set of containers) and then run</p> <p><code>docker-compose up -d </code></p> <p>which runs the containers in daemon mode, without the running log. You can then log out of your server, and your containers will continue running!</p> <p>Based on the configuration above (the "unless-stopped" directive), your containers will restart automatically if your server is rebooted. If you <em>do</em> want to stop them for some reason, you can via</p> <p><code>docker-compose stop</code></p> <p>Easy.</p> <h2>Serving them to the Web</h2> <p>Once you've got your containers running, you need to make sure you've got a web server running on your host to act as the reverse proxy so that external requests get sent to them reliably! We use <a href="nginx.org">Nginx</a>.</p> <h3>RocketChat Nginx</h3> <p>Here's our configuration (with appropriate [substitutions]) - you can create it as <code>/etc/nginx/sites-available/[domain name]</code>:</p> <p><code>server {<br />     listen 80;<br />     server_name [domain name];</code></p> <p><code>  ## Access and error logs.<br />   access_log /var/log/nginx/[domain name]_access.log;<br />   error_log /var/log/nginx/[domain name]_error.log;</code></p> <p><code>  # see https://tech.oeru.org/protecting-your-users-lets-encrypt-ssl-certs<br />   include /etc/nginx/includes/letsencrypt.conf</code></p> <p><code>  # we use a 302 temporary redirect rather than a 301 permanent redir</code><br /><code>  location / {<br />     return 302 https://[domain name]$request_uri;<br />   }<br /> }<br /><br /> server {<br />     listen 443 ssl;<br />     ssl on;</code><br />      <code>  # see https://tech.oeru.org/protecting-your-users-lets-encrypt-ssl-certs<br />     ssl_certificate /etc/letsencrypt/live/[domain name]/fullchain.pem;<br />     ssl_certificate_key /etc/letsencrypt/live/[domain name]/privkey.pem;<br />     ssl_protocols TLSv1 TLSv1.1 TLSv1.2;<br />     ssl_dhparam /etc/ssl/certs/dhparam.pem;</code></p> <p><code>    keepalive_timeout 20s;</code></p> <p><code>    root /var/www/html;<br />     index index.html index.htm;</code></p> <p><code>    server_name [domain name];</code></p> <p><code>    ## Access and error logs.<br />     access_log /var/log/nginx/[domain name]_access.log;<br />     error_log /var/log/nginx/[domain name]_error.log;</code></p> <p><code>    location / {<br />         proxy_pass http://127.0.0.1:[your rocketchat port];<br />         proxy_http_version 1.1;<br />         proxy_set_header Upgrade $http_upgrade;<br />         proxy_set_header Connection "upgrade";<br />         proxy_set_header Host $http_host;<br />         proxy_set_header X-Forwarded-Host $host;<br />         proxy_set_header X-Real-IP $remote_addr;<br />         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br />         proxy_set_header X-Forward-Proto http;<br />         proxy_set_header X-Nginx-Proxy true;<br />         proxy_redirect off;<br />     }<br /> }</code></p> <p> </p> <h3>Wekan Nginx</h3> <p>Here's our configuration (with appropriate [substitutions]) - you can create it as <code>/etc/nginx/sites-available/[domain name]</code>: </p> <p><code># from https://github.com/wekan/wekan/wiki/Install-Wekan-Docker-in-production<br /> upstream websocket {<br />     server 127.0.0.1:[wekan port];<br /> }</code></p> <p><code>map $http_upgrade $connection_upgrade {<br />     default upgrade;<br />     '' close;<br /> }</code><br /><br /><code>server {<br />     listen    80;<br /><br />     root /var/www/html;<br />     index index.html index.htm;</code></p> <p><code>    # Make site accessible from http://localhost/<br />     server_name [domain name];</code></p> <p><code>    access_log /var/log/nginx/[domain name]_access.log;<br />     error_log /var/log/nginx/[domain name]_error.log;</code></p> <p><code>    # see https://tech.oeru.org/protecting-your-users-lets-encrypt-ssl-certs<br />     include /etc/nginx/includes/letsencrypt.conf</code></p> <p><code>    location / {<br />         return 302 https://[domain name]$request_uri; <br />     }<br /> }</code></p> <p><code>server {<br />     listen 443 ssl;<br />     ssl on;</code><br />      <code>  # see https://tech.oeru.org/protecting-your-users-lets-encrypt-ssl-certs<br />     ssl_certificate /etc/letsencrypt/live/[domain name]/fullchain.pem;<br />     ssl_certificate_key /etc/letsencrypt/live/[domain name]/privkey.pem;<br />     ssl_protocols TLSv1 TLSv1.1 TLSv1.2;<br />     ssl_dhparam /etc/ssl/certs/dhparam.pem;</code></p> <p><code>    keepalive_timeout 20s;</code></p> <p><code>    access_log /var/log/nginx/[domain name]_access.log;<br />     error_log /var/log/nginx/[domain name]_error.log;</code></p> <p><code>    root /var/www/html;<br />     index index.html index.htm;</code></p> <p><code>    server_name [domain name];</code></p> <p><code>    location / {<br />         proxy_read_timeout 300;<br />         proxy_connect_timeout 300;<br />         proxy_redirect off;<br />         proxy_set_header Host $http_host;<br />         proxy_set_header X-Real-IP $remote_addr;<br />         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br />         proxy_set_header X-Forwarded-Proto scheme;<br />         proxy_pass http://127.0.0.1:[your wekan port];<br />         proxy_set_header Host $host;<br />     }</code></p> <p><code>    location ~ websocket$ {<br />         proxy_pass http://websocket;<br />         proxy_http_version 1.1;<br />         proxy_set_header Upgrade $http_upgrade;<br />         proxy_set_header Connection $connection_upgrade;<br />     }</code><br /><code>}</code></p> <h3>Enable Nginx Configuration</h3> <p>To make your configurations active, do the following for each of your Nginx configurations:</p> <p><code>cd /etc/nginx/sites-enabled</code></p> <p>Do this for each file:<br /><code>ln -sf ../sites-available/[filename] .</code></p> <p>To check if there are any errors in the files, run</p> <p><code>nginx -t</code></p> <p>If not, you can restart Nginx to incorporate the new configuration files:</p> <p><code>sudo service nginx reload</code></p> <p>You can check for errors in the relevant log files specified in your nginx configurations above in <code>/var/log/nginx/*_error.log</code> or <code>/var/log/nginx/*_access.log</code>.</p> <h2>Protecting your users and reputation with encryption</h2> <p>We encourage you to ensure that these services are made available with full encryption to protect your users' privacy. It's <a href="/protecting-your-users-lets-encrypt-ssl-certs">easy (and no cost) to set up</a>!  The "include" directive in the Nginx configuration files above are examples of this approach.</p> <h2>Upgrades and Backups</h2> <p>We also encourage you to keep your services upgraded. It's easy to do and you'll experience little if any perceptible down time!</p> <p>Simply re-pull the containers and restart them - the updated containers will be launched without loss of data!</p> <p><code>docker pull mongo<br /> docker pull rocket.chat<br /> docker pull mquandalle/wekan</code></p> <p><code>docker-compose up -d</code></p> <p>If you want to back up your data - you need to do normal file backups of the directories on your local server that you've configured in the <code>docker-compose.yml</code> file, and you can do MongoDB backups based on <a href="/installing-mongodb-docker-ubuntu-linux-1404">our previous article</a> on the topic!</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=13&amp;2=field_blog_comments&amp;3=comment" token="IKqqNGzna0isfyEknk_oRE2sfWThSeFDbglHdf4AmjA"></drupal-render-placeholder> </div> </section> Mon, 22 May 2017 23:03:29 +0000 dave 13 at http://tech.oeru.org Many simple tools, loosely coupled http://tech.oeru.org/many-simple-tools-loosely-coupled <span class="field field--name-title field--type-string field--label-hidden">Many simple tools, loosely coupled</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--philosophy"> <span class="field__item-wrapper"><a href="/taxonomy/term/28" hreflang="en">philosophy</a></span> </div> <div class="field__item field__item--unix"> <span class="field__item-wrapper"><a href="/taxonomy/term/29" hreflang="en">unix</a></span> </div> <div class="field__item field__item--ubuntu-linux"> <span class="field__item-wrapper"><a href="/taxonomy/term/12" hreflang="en">ubuntu linux</a></span> </div> <div class="field__item field__item--docker"> <span class="field__item-wrapper"><a href="/taxonomy/term/16" hreflang="en">docker</a></span> </div> <div class="field__item field__item--lets-encrypt"> <span class="field__item-wrapper"><a href="/taxonomy/term/17" hreflang="en">let&#039;s encrypt</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Mon 08/05/2017 - 15:10</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>Our approach to technology here at the <a href="https://oeru.org" title="The Open Education Resource universitas">OERu</a> is inspired by the <a href="http://www.faqs.org/docs/artu/ch01s06.html">UNIX tool philosophy</a> which can be summarised as follows:</p> <blockquote> <p>"create simple tools that do one job well, and make it easy to combine them to work together"</p> </blockquote> <p>In the UNIX (and, somewhat more recently, the Linux) computing environment, this originally meant a lot of small commandline applications like "ls" for listing the contents of file directories, and "grep" for searching directories of files for words and other snippets of content, and "diff" for showing the difference between two files, and many many more. These all output text, and they also accept text as an input - you can <em>chain</em> all of these simple little applications together to create, on the fly, remarkably complex capabilities. This is one of the things that makes Linux and the commandline so powerful for those who have learned its lore (and so intimidating for those who haven't yet done so). </p> <p>This idea of "loosely coupled" tools, working together is also a good way to describe both the OERu technology and documentation philosophy.</p> <p>On this website, the way it manifests is interesting - each time I write a howto article, there're certain common tasks - things like setting up Docker, or creating secure SSL certificates for encrypting user interactions with a web service.</p> <p>Initially, I wrote howtos with all of those details contained in one document, however the instructions fairly quickly become outdated, for example, the install process for Docker or Let's Encrypt is changed by its community (usually to make it faster and more convenient) or to reflect the release of new software dependencies. It doesn't sit well with me to be leaving outdated or inaccurate resources on the web - I feel a responsibility for curating them to improve the signal-to-noise ratio of the 'net. Also, it rapidly becomes an intractable problem to go through old howtos to update all the slight variations on the same instructions to something new (the problem grows exponentially as more howtos are added).</p> <p>So, taking the UNIX approach, I use my experience writing a few howtos to provide insight into parts of each that are repeated. Any section repeated (more or less unchanged) in each howto is a candidate for replacement with a stand-alone howto.</p> <p>As it turns out, the community that's building the Docker container technology has done a good job of keeping their <a href="https://docs.docker.com/engine/installation/linux/ubuntu/">installation documentation</a> up-to-date and making it easy to find the relevant info for our target platforms, Ubuntu Linux 14.04 and 16.04. As a result, there's no point in my repeating their instructions. Instead, I just point my readers there when it's time to install Docker.</p> <p>My first candidate for documentation "<a href="https://en.wikipedia.org/wiki/Code_refactoring">refactoring</a>" was the "<a href="/protecting-your-users-lets-encrypt-ssl-certs">Let's Encrypt</a>" SSL Certificate process (it's a feature of almost all my howtos to date). Having now created that stand-alone howto, I can replace the largely repeated sections of several howtos with a single link to the same place.</p> <p>Yes, this change adds the overhead of the reader of a howto needing to go to a different article on this site, but I think this is greatly outweighed by the benefits: if I need to update or improve the description of this operation, I can simply update one document and ensure it's got the "best of" tips across all the different howtos. Also, if people leave questions or comments, all relevant ones will be in the same place, making it easier for other site visitors to find them.</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=12&amp;2=field_blog_comments&amp;3=comment" token="OgsykCUI1TUoj8ce8ZZ9L5Ax7Aq8mwkhmaCmxsLyqQI"></drupal-render-placeholder> </div> </section> Mon, 08 May 2017 03:10:35 +0000 dave 12 at http://tech.oeru.org