Drupal blog posts http://tech.oeru.org/blog en Hourly versioned MongoDB backup http://tech.oeru.org/hourly-versioned-mongodb-backup <span class="field field--name-title field--type-string field--label-hidden">Hourly versioned MongoDB backup</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--mongodb"> <span class="field__item-wrapper"><a href="/taxonomy/term/14" hreflang="en">mongodb</a></span> </div> <div class="field__item field__item--backup"> <span class="field__item-wrapper"><a href="/taxonomy/term/57" hreflang="en">backup</a></span> </div> <div class="field__item field__item--docker"> <span class="field__item-wrapper"><a href="/taxonomy/term/16" hreflang="en">docker</a></span> </div> <div class="field__item field__item--docker-compose"> <span class="field__item-wrapper"><a href="/taxonomy/term/49" hreflang="en">docker-compose</a></span> </div> <div class="field__item field__item--bash"> <span class="field__item-wrapper"><a href="/taxonomy/term/58" hreflang="en">bash</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Mon 29/04/2019 - 16:28</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>Because the collaboration of an open community is its real history, I place a high value on backing up the Rocket.Chat servers I'm responsible for, and <em>especially</em> the data they generate, held in MongoDB files (on the host) managed by a MongoDB container.</p> <p>To do that reliably, I have set up <a href="https://git.oeru.org/oeru/mongobackup">a bash script</a> which does an hourly backup of all MongoDB "databases" and automatically maintains 24 hourly, 7 daily, 4 weekly, 12 monthly, and 7 yearly snapshots of the databases.</p> <p>Once you've configured your backup script properly, you should be able to run this command to do a backup...</p> <p><code>/etc/mongobackup/dbbackup-mongo --hourly</code></p> <p>easy-peasy.</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=27&amp;2=field_blog_comments&amp;3=comment" token="a3hwmZDJbiAJ569rEIXJYUfTF3BaDQbUaOh1gRHfc5g"></drupal-render-placeholder> </div> </section> Mon, 29 Apr 2019 04:28:43 +0000 dave 27 at http://tech.oeru.org Upgrading RocketChat to 1.0.x and MongoDB to 4.0 http://tech.oeru.org/upgrading-rocketchat-10x-and-mongodb-40 <span class="field field--name-title field--type-string field--label-hidden">Upgrading RocketChat to 1.0.x and MongoDB to 4.0</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--rocketchat"> <span class="field__item-wrapper"><a href="/taxonomy/term/18" hreflang="en">rocket.chat</a></span> </div> <div class="field__item field__item--mongodb"> <span class="field__item-wrapper"><a href="/taxonomy/term/14" hreflang="en">mongodb</a></span> </div> <div class="field__item field__item--docker"> <span class="field__item-wrapper"><a href="/taxonomy/term/16" hreflang="en">docker</a></span> </div> <div class="field__item field__item--docker-compose"> <span class="field__item-wrapper"><a href="/taxonomy/term/49" hreflang="en">docker-compose</a></span> </div> <div class="field__item field__item--ubuntu-linux"> <span class="field__item-wrapper"><a href="/taxonomy/term/12" hreflang="en">ubuntu linux</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Mon 29/04/2019 - 14:39</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>With the recent release of Rocket.Chat 1.0.x (after a couple years undergoing development at a fairly blistering pace), it's time for many of us to upgrade!</p> <p>Previously, I showed how to <a href="/docker-compose-better-way-deploy-rocketchat-wekan-and-mongodb">install Rocket.Chat via Docker Compose</a> but that was a much earlier version of Rocket.Chat and version 3.4 of MongoDB, which is now quite old (by FOSS standards at least). And it turns out upgrading everything has a few gotchas, so here's how I managed to do it.</p> <p>Before you do <em>anything</em> <a href="/hourly-versioned-mongodb-backup">do a backup of your MongoDB</a>!</p> <p>The first thing you need to do is upgrade <em>the way</em> in which you're running MongoDB. You have to enable a capability called "Local Replication".</p> <h2>Update your Docker Compose configuration</h2> <p>My first step, after logging into my virtual machine via SSH as the <em>unprivileged user</em> that I created to run docker commands, was to update my <code>docker-compose.yml</code> file (if you followed my previous instructions, you'll find it in <code>/home/www/docker-rocketchat-wekan-mongo</code>). </p> <p>First, make a backup of it nearby...</p> <p>cd <code>/home/www/docker-rocketchat-wekan-mongo</code><br /> cp docker-compose.yml docker-compose.yml-mongo3.4</p> <p>and then edit the file to say this:</p> <p><code>version: '2'<br /> services:<br />   mongo:<br />     restart: unless-stopped<br />     image: mongo<strong>:3.4</strong><br />     volumes:<br />       - [data directory path]:/data/db<br />       - [backup directory path]:/backups<br />     command: --smallfiles <strong>--oplogSize 128 --replSet rs0</strong><br /><strong>  # this container's job is just run the command to initialize the replica set.<br />   # it will run the command and remove himself (it will not stay running)<br />   mongo-init-replica:<br />     image: mongo:3.4<br />     command: 'bash -c "for i in `seq 1 30`; do mongo mongo/rocketchat --eval \"rs.initiate({ _id: ''rs0'', members: [ { _id: 0, host: ''localhost:27017'' } ]})\" &amp;&amp; s=$$? &amp;&amp; break || s=$$?; echo \"Tried $$i times. Waiting 5 secs...\"; sleep 5; done; (exit $$s)"'<br />     depends_on:<br />       - mongo</strong><br />   rocketchat:<br />     restart: unless-stopped<br />     image: rocketchat/rocket.chat<strong>:latest</strong></code><br /><code><strong>    command: bash -c 'for i in `seq 1 30`; do node main.js &amp;&amp; s=$$? &amp;&amp; break || s=$$?; echo "Tried $$i times. Waiting 5 secs..."; sleep 5; done; (exit $$s)'</strong><br />     ports:<br />       - "127.0.0.1:[port number]:3000" # should be a free port above 1024<br />     depends_on:<br />       - mongo<br />     environment:<br />       - MONGO_URL=mongodb://mongo/rocket<br /><strong>      - MONGO_OPLOG_URL=mongodb://mongo/local</strong><br />       - ROOT_URL=[domain name (including schema, e.g. http://)]<br />     volumes:<br />       - [upload directory path]:/var/www/rocket.chat/uploads</code><br /><strong><code>    labels:<br />       - "traefik.backend=rocketchat"<br />       - "traefik.frontend.rule=Host: [your domain name (<em>not </em>including schema)]"</code></strong></p> <p>Now, having updated your docker-compose.yml file, you have to do a couple other things. To do the upgrade from MongoDB 3.4 to 4.0, you have to do the interim upgrade to 3.6 first.</p> <h2>Enabling Local Replication</h2> <p>First you need to check what version of MongoDB you're <em>currently</em> using - both the version you're running <em>and</em> the "Feature Compatibility Version" (you can run a newer version of MongoDB, but configure it to only run features from some previous version to avoid breaking older software that depends on old features)... Do this as follows.</p> <p>Access your MongoDB instance:</p> <p><code>docker-compose exec mongo bash</code></p> <p>That should give you a command prompt that looks like this:</p> <p><code>root@a56eefe9f352: # </code></p> <p>but the container identifier (after the @) will be different (but the same length). At that prompt, you can run this command:</p> <p><code>mongo --eval "db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )"</code></p> <p>Tip: if you're on a Linux desktop, you can copy this command (via CTRL-C) from this document and past it into your SSH terminal window (via CTRL+SHIFT-V).</p> <p>It should tell you you're either running "featureCompatibilityVersion" 3.2 or 3.4. If it's the latter, skip this next step. If not, run this next:</p> <p><code>mongo --eval "db.adminCommand( { setFeatureCompatibilityVersion: '3.4' } )"</code></p> <p>to set the version to 3.4. If the command succeeds you'll likely see something like</p> <p><code>{ "ok" : 1 }</code></p> <p>as the response.</p> <p>Now you can upgrade your Mongo 3.4 is the latest version (should be 3.4.20 at the time of this writing). Get out of the container (back to your Docker host) via CTRL-D (or "exit" - they're synonymous for logging out of a terminal session). Then you can run:</p> <p><code>docker-compose pull mongo</code></p> <p>That should update both your Mongo docker container to the latest version in the 3.4 series.</p> <p>The final step to enabling local replication is to run</p> <p><code>docker-compose up -d mongo mongo-init-replica &amp;&amp; docker-compose logs -f</code></p> <p>That will restart MongoDB and drop you into the stream of logging from all the containers (including the rocket.chat container). It'll also start the "mongo-init-replica" container.  That container should run briefly <em>and then exit cleanly</em> having set up the local replication that you'll need for subsequent upgrades to MongoDB!</p> <p>Check for any errors in the output... there might be a couple if it takes your MongoDB a bit of time to accept connections... as long as it eventually stops showing errors, you should be ok!</p> <p> </p> <h2>Upgrading Rocket.Chat to 1.0.x</h2> <p>Now that you're fully on version 3.4, running in local replica mode, you can update your Rocket.Chat instance.   Rocket.Chat still supports Mongo 3.4 (it won't for long, thus this tutorial!), so you can now upgrade the Rocket.Chat container as well as make sure your Mongo 3.4 is the latest version (should be 3.4.20 at the time of this writing).</p> <p>Note that the latest version of the Rocket.Chat docker container could be quite a lot higher when you read this... if it's beyond, say, 1.1 it might be unsafe to use the approach I'm describing. You can <a href="https://github.com/RocketChat/Rocket.Chat/releases">check the current version release status</a>. To protect yourself, you can alter the rocket.chat image line in your docker-compose.yml file to explicitly tell it to use the 1.0.x series for which these instructions should continue to apply... pick the highest 1.0.x version you can find in the releases and alter the line in docker-compose.yml to specify that version:</p> <p><code>    image: rocketchat/rocket.chat<strong>:1.0.1</strong></code></p> <p>or whatever the latest 1.0.x version is. Then you can run:</p> <p><code>docker-compose pull</code> rocketchat</p> <p>which will update your Rocket.Chat from the current version to the one specified.</p> <p>Then restart your Rocket.Chat instance:</p> <p><code>docker-compose up -d &amp;&amp; docker-compose logs -f</code></p> <p>That should restart both MongoDB and Rocket.Chat, and drop you into the stream of logging from both containers. It'll also start the "mongo-init-replica" container again, but having done its job it should exit happily again.</p> <p>Check for any errors in the output... there might be a couple if it takes your MongoDB a bit of time to accept connections... as long as it eventually stops showing errors, you should be ok! Eventually, you should see something similar to (with version details updated appropriately):</p> <p> </p> <p><code>+----------------------------------------------+<br /> |                SERVER RUNNING                |<br /> +----------------------------------------------+<br /> |                                              |<br /> |  Rocket.Chat Version: 1.0.1                  |<br /> |       NodeJS Version: 8.11.4 - x64           |<br /> |      MongoDB Version: 3.4.20                 |<br /> |       MongoDB Engine: wiredTiger             |<br /> |             Platform: linux                  |<br /> |         Process Port: 3000                   |<br /> |             Site URL: https://chat.oeru.org  |<br /> |     ReplicaSet OpLog: Enabled                |<br /> |          Commit Hash: 60f1a4afd6             |<br /> |        Commit Branch: HEAD                   |<br /> |                                              |<br /> +----------------------------------------------+</code></p> <p>Your instance is now running the right version! Time to tidy things up by upgrading Mongo the rest of the way to 4.0!</p> <p> </p> <h2>Upgrading to MongoDB 3.6</h2> <p>Now you can upgrade Mongo to 3.6. First, adjust your docker-compose.yml file.  Update both occurances of this line:</p> <p><code>image: mongo:3.4</code></p> <p>to</p> <p><code>image: mongo:3.6</code></p> <p>Then you can do another</p> <p><code>docker-compose pull mongo</code></p> <p>which will download the newer Mongo 3.6 docker container. Then you can again run</p> <p><code>docker-compose up -d &amp;&amp; docker-compose logs -f</code></p> <p>Again check for errors. If there are none (other than perhaps a brief set of "mongo is not accepting connections" errors), you should be fine to update the "compatibility version" from 3.4 to 3.6... Get a session on your Mongo container via</p> <p><code>docker-compose exec mongo bash</code></p> <p>and then (as above) run this:</p> <p><code>mongo --eval "db.adminCommand( { setFeatureCompatibilityVersion: '3.6' } )"</code></p> <p>which should give you a more complicated response than that for the 3.4 transition, but it should still more or less say "ok"... To make sure everything's happy with the change, it's wise to run</p> <p><code>docker-compose up -d &amp;&amp; docker-compose logs -f</code></p> <p>again and make sure there're no obvious errors. If not (after making another database backup for safety!!) we can proceed to Mongo 4.0!</p> <h2>Final push to MongoDB 4.0</h2> <p>Finally, you can again edit your docker-compose.yml and change both occurrences of</p> <p> </p> <p><code>image: mongo:3.6</code></p> <p>to</p> <p><code>image: mongo:4.0</code></p> <p>Then you can do a final</p> <p><code>docker-compose pull mongo</code></p> <p>which will download the newer Mongo 4.0 docker container. Then you can again run</p> <p><code>docker-compose up -d &amp;&amp; docker-compose logs -f</code></p> <p>And, assuming you don't see any errors, you can push the feature compatibility to 4.0:</p> <p><code>docker-compose exec mongo bash</code></p> <p>and then run:</p> <p><code>mongo --eval "db.adminCommand( { setFeatureCompatibilityVersion: '4.0' } )"</code></p> <p>followed by a final</p> <p><code>docker-compose up -d &amp;&amp; docker-compose logs -f</code></p> <p>And, again, if you don't see any errors... you should get something a bit like this:</p> <p><code>+----------------------------------------------+<br /> |                SERVER RUNNING                |<br /> +----------------------------------------------+<br /> |                                              |<br /> |  Rocket.Chat Version: 1.0.1                  |<br /> |       NodeJS Version: 8.11.4 - x64           |<br /> |      MongoDB Version: 4.0.9                  |<br /> |       MongoDB Engine: wiredTiger             |<br /> |             Platform: linux                  |<br /> |         Process Port: 3000                   |<br /> |             Site URL: https://chat.oeru.org  |<br /> |     ReplicaSet OpLog: Enabled                |<br /> |          Commit Hash: 60f1a4afd6             |<br /> |        Commit Branch: HEAD                   |<br /> |                                              |<br /> +----------------------------------------------+</code></p> <p>you're done and future proofed!</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=26&amp;2=field_blog_comments&amp;3=comment" token="VK6tqiyjVB7HYLx9xMKgF1KeXWiFxiQ-XNY71DIaaF0"></drupal-render-placeholder> </div> </section> Mon, 29 Apr 2019 02:39:33 +0000 dave 26 at http://tech.oeru.org Setting up your own BitWarden password keeper and sync server http://tech.oeru.org/setting-your-own-bitwarden-password-keeper-and-sync-server <span class="field field--name-title field--type-string field--label-hidden">Setting up your own BitWarden password keeper and sync server</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--docker-compose"> <span class="field__item-wrapper"><a href="/taxonomy/term/49" hreflang="en">docker-compose</a></span> </div> <div class="field__item field__item--docker"> <span class="field__item-wrapper"><a href="/taxonomy/term/16" hreflang="en">docker</a></span> </div> <div class="field__item field__item--rust"> <span class="field__item-wrapper"><a href="/taxonomy/term/59" hreflang="en">rust</a></span> </div> <div class="field__item field__item--bitwarden"> <span class="field__item-wrapper"><a href="/taxonomy/term/60" hreflang="en">BitWarden</a></span> </div> <div class="field__item field__item--password-keeper"> <span class="field__item-wrapper"><a href="/taxonomy/term/61" hreflang="en">password keeper</a></span> </div> <div class="field__item field__item--privacy"> <span class="field__item-wrapper"><a href="/taxonomy/term/62" hreflang="en">privacy</a></span> </div> <div class="field__item field__item--security"> <span class="field__item-wrapper"><a href="/taxonomy/term/63" hreflang="en">security</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Wed 23/01/2019 - 11:30</span> <div class="float-none field field-node--field-image field-name-field-image field-type-image field-label-hidden has-multiple"> <figure class="field-type-image__figure image-count-1 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2019-04/bitwarden_desktop_app.png?itok=9ZE2TxZh" title="BitWarden desktop app (Electron-based)" data-colorbox-gallery="gallery-field_image-0Nhui6ShQUs" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;BitWarden desktop app (Electron-based)&quot;}"><img src="/sites/default/files/styles/medium/public/2019-04/bitwarden_desktop_app.png?itok=hekOd__9" width="220" height="169" alt="BitWarden desktop app (Electron-based)" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-2 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2019-04/bitwarden_web_interface_0.png?itok=QShOqyOM" title="BitWarden website interface" data-colorbox-gallery="gallery-field_image-0Nhui6ShQUs" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;BitWarden website interface&quot;}"><img src="/sites/default/files/styles/medium/public/2019-04/bitwarden_web_interface_0.png?itok=QitHloR0" width="220" height="158" alt="BitWarden website interface" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-3 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2019-04/bitwarden_desktop_app_logged_in_0.png?itok=FAbPSjim" title="Desktop BitWarden app, with user logged in. " data-colorbox-gallery="gallery-field_image-0Nhui6ShQUs" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Desktop BitWarden app, with user logged in. &quot;}"><img src="/sites/default/files/styles/medium/public/2019-04/bitwarden_desktop_app_logged_in_0.png?itok=p9Oarp8h" width="220" height="156" alt="Desktop BitWarden app, with user logged in. " typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-4 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2019-04/bitwarden_adding_new_item_via_web_interface_0.png?itok=_PTdZqjB" title="Web interface for BitWarden with user logged in." data-colorbox-gallery="gallery-field_image-0Nhui6ShQUs" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Web interface for BitWarden with user logged in.&quot;}"><img src="/sites/default/files/styles/medium/public/2019-04/bitwarden_adding_new_item_via_web_interface_0.png?itok=YQftj3vf" width="220" height="209" alt="Web interface for BitWarden with user logged in." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> </div> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>One of the key requirements of pursuing Good Digital Hygiene is <em>using strong passwords</em>, and a <em>different strong password for every application</em>. This is relatively easy to do in theory, <em>with the aid of clever software</em>, but it's something desperately few people do well in practice. I'm going to explain how I've addressed this issue of digital hygiene for myself, and how you can do it for yourself, <em>and your entire family, social circle, or community</em>.</p> <p>Password Keepers (or managers or safes) have emerged as that "clever software". A good password keeper has to do a bunch of things to be really useful:</p> <ol><li>It needs to store your passwords somewhere in an encrypted form (so if someone gets your password database, they can't work out your entire collection of passwords). You only need to remember <em>one </em><em>really strong password/phrase to unlock all of them. </em></li> <li>It needs to work in whatever context you need a password. Like <ol><li>your desktop/laptop, where you need to remember logins for a variety of apps and services,</li> <li>in your browser (for web apps that require authentication), and</li> <li>on your mobile platforms (because most services you use via apps or browsers require authentication)</li> </ol></li> <li>It needs to be cross platform <ol><li>must support Windows, MacOS, and Linux OSs,</li> <li>must support extensions for many browsers like Firefox, Chrome/<a href="https://chromium.org" title="The open source project that underlies Chrome and is 99% identical.">Chromium</a>, Safari, and others, and</li> <li>must support mobile OSs like iOS and Android.</li> </ol></li> <li>It needs to sync data in a timely manner among all the different contexts in which a given user needs it.</li> </ol><p>That's a lot of requirements. There're quite a few efforts that have had a crack at solving this. The <a href="https://www.keepassx.org/">KeePassX</a> community has been addressing this for ages and has created a comprehensive (if variable) ecosystem of apps which work across all of the required platforms, but only with a lot of work.</p> <p>In the proprietary world, there're many options, with a few front runners like 1Password and <a href="https://lastpass.com">LastPass</a>. The former doesn't work on Linux, so it only gets a passing reference and no link :) (update 2019-05-31 - <a href="https://1password.com">1Password</a> has added Linux support). The latter, which I used (grudgingly, mostly because I couldn't get KeePassX to work for me) for a few years, works across all the platforms relevant to me, but it was becoming progressively more invasive and annoying to use. Also, because it has a <em>lot</em> of users, and stores everything (albeit, encrypted) in a centralised cloud repository, it's a <em>big target</em>. Also, with its largely proprietary code, I wasn't happy trusting it. </p> <p>Then I heard about <a href="https://bitwarden.com">BitWarden</a>. They offered a commercial service (with a free tier) that I could quickly try... they supported all the OSs, mobile and desktop, and browsers that I use... <em>and they release their entire codebase </em>(server and clients) <em><strong>under open source licenses.</strong></em> I tried it, it worked for me, I was sold!</p> <p>Then I decided I wanted to run my own BitWarden server, rather than use their commercial centralised cloud platform (because, as with LastPass, it's a tempting target). That's when I found out the server of BitWarden was written using Microsoft technologies, C# (yeah, it's mostly open source, but it's dirty to me due to its Microsoft legacy), and MS SQL Server, which is a nasty proprietary dependency (especially given how basic the database requirements for this sort of application are).</p> <p>So I was devastated that I couldn't set up my own server... until another Free and Open Source Software aficionado pointed me at Daniel Garcia's work! Daniel has implemented a <a href="https://github.com/dani-garcia/bitwarden_rs">full (unofficial) BitWarden work-alike using a fully FOSS stack</a>: the Rust language, storing data in SQLite, and (quite thoughtfully) re-using other open source licensed components of the BitWarden system that don't have proprietary dependencies, including the website code and layout (which is part of the server).</p> <p>Daniel's server implementation also unlocks all the "premium" services that BitWarden offers through their hosted service, too... so that's a nice bonus.</p> <p>Another open source developer, <a href="https://github.com/mprasil">mpasil</a>, has created <a href="https://github.com/mprasil/bitwarden_rs">a "fork" of Daniel's project</a> from which he maintains an up-to-date Docker container on hub.docker.com. <strong>Thanks to both Daniel Garcia and mpasil's efforts</strong>, it turns out to be quite straightforward to set up your own Docker-based BitWarden-compatible service! Here's how...</p> <h2>Creating your own BitWarden Service</h2> <h3>Set up a Virtual Server</h3> <p>The first step is to get yourself an entry-level virtual server or compute instance somewhere. I generally use DigitalOcean (I have no affiliation with the company), but there are many other commodity hosting services around the world which offer comparably (or better ) spec'd servers for <strong>USD5.00/month</strong>, or <strong>USD60.00/year</strong> - I encourage you to do a bit of research. For that you get a Gigabyte (GB) of RAM, a processor, and 40GB of SSD (Static Storage Device = faster) storage. </p> <p>I suggest you create an account for yourself (and I encourage you to use Two Factor Authentication, aka 2FA) and create an Ubuntu 18.04 (or the most recent LTS version - the next will be 20.04, in April 2020 :) ) in the zone nearest to you. You'll need to note the server's IP address (it'll be a series of 4 numbers, 0-254, separated by full stops, e.g. 103.99.72.244). With that, you can <a href="https://www.digitalocean.com/community/tutorials/how-to-use-ssh-to-connect-to-a-remote-server-in-ubuntu">log into it via SSH</a>.</p> <h3>Get your Domain lined up</h3> <p>You will want to have a domain to point at your server, so you don't have to remember the IP number. There're are thousands of domain "registrars" in the world who'll help you do that... You just need to "register" a name, and you pay yearly fee (usually between USD10-30 depending on the country and the "TLD" (Top Level Domain. There're national ones like .nz, .au, .uk, .tv, .sa, .za, etc., or international domains (mostly associated with the US) like .com, .org, .net, and a myriad of others. Countries decide on how much their domains wholesale for and registrars add a margin for the registration service).</p> <p>Here in NZ, I use the services of <a href="https://metaname.net">Metaname</a> (they're local to me in Christchurch, and I know them personally and trust their technical capabilities). If you're not sure who to use, ask your friends. Someone's bound to have recommendations (either positive or negative).</p> <p>Once you have selected and registered your domain, you can set up (usually through a web interface provided by the registrar) an "A Record" which associates your website's name to the IP address of your server. So you should just be able to enter your server's IP address, the domain name (or sub-domain) you want to use for your BitWarden service, and that's it. For a password safe, I tend to use the subdomain "safe", so, for example, safe.mydomain.nz or similar.</p> <p>You might be asked to set a "Time-to-live" (which has to do with the length of time Domain Name Servers are asked to "cache" the association that the A Record specifies) in which case you can put in 3600 seconds or an hour depending on the time units your interface requests... but in most cases that'll be set to a default of an hour automatically.</p> <p>You should be able to test that your A Record has been set correctly by SSHing to your domain name rather than the IP address. It should (after you accept the SSH warning that the server's name has changed) work the same way your original SSH login did.</p> <h3>Set up a Docker Server</h3> <p>Once I've first logged into it as the "root" (full admin) user, here's what I usually do:</p> <ol><li>I create an "unprivileged user", either with my name "dave" or sometimes an "ubuntu" user (some hosting providers create a default unprivileged user of "ubuntu" when you create an Ubuntu-based virtual machine. Some create a "debian" user for Debian-based VMs, etc.) via<br /><code>adduser ubuntu</code></li> <li>I install a few core applications: my preferred editor <a href="https://en.wikipedia.org/wiki/Vim_(text_editor)">vim</a> (<a href="https://en.wikipedia.org/wiki/GNU_nano">nano</a> is another easy option and comes pre-installed on Ubuntu), version control system, <a href="https://en.wikipedia.org/wiki/Git">git</a>, and a very handy configuration tracker, <a href="http://joeyh.name/code/etckeeper/">etckeeper</a>:<br /><code>apt-get update &amp;&amp; apt-get install vim git etckeeper</code></li> <li>I do some basic configuration of git (replace the [tokens] with the real values for you, minus the []):<br /><code>git config --global user.email "[your email]"<br /> git config --global user.name "[your full name, e.g. Jane Doe]"</code></li> <li>Initialise etckeeper - it will track configuration changes you make to your system which can be invaluable in replicating a server or working out what's changed if something breaks.<br /><code>etckeeper init<br /> etckeeper commit -m "initial commit of BitWarden host"</code></li> <li>Install Docker dependencies:<br /><code>apt-get install apt-transport-https ca-certificates curl software-properties-common pwgen</code><br /> Install secure key needed to add the docker.com package repository to your system<br /><code>curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -</code><br /> Confirm the key is valid<br /><code>apt-key fingerprint 0EBFCD88</code><br /> (you should see something like "<code>uid [ unknown] Docker Release (CE deb) &lt;docker@docker.com&gt;</code>" among the 4 lines)</li> <li>Add the repository for your Ubuntu version (this will pick it automatically)<br /><code>add-apt-repository    "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"</code></li> <li>Update the package repository to include the packages from docker.com<br /><code>apt-get update</code></li> <li>Install the Community Edition of the Docker service<br /><code>apt-get install docker-ce</code></li> <li>Add your unprivileged user ("ubuntu" in this case - substitute the unprivileged user you created!) to a new "docker" group and add that user to other useful groups:<br /><code>groupadd docker<br /> adduser ubuntu<br /> adduser ubuntu sudoers<br /> adduser ubuntu admin</code><br /><code>adduser ubuntu docker</code></li> <li>Create an SSH key for your unprivileged user and allow logins for that user from external connection:<br /><code>sudo -Hu ubuntu ssh-keygen -t rsa<br /> cp /root/.ssh/authorized_keys /home/ubuntu/.ssh/<br /> chown ubuntu:ubuntu /home/ubuntu/.ssh/<br /> adduser ubuntu ssh</code></li> <li>Install the Python packaging system, "pip" to allow you to install and maintain the Docker Compose framework for managing collections of Docker containers:<br /><code>apt install python-pip<br /> pip install -U pip<br /> pip install docker-compose</code></li> <li>Set a convenience variable for [your domain] here (note: it'll only be recognised for this session, i.e. until you log out):<br /><code>DOMAIN=[your domain]<br /> USER=[unprivileged user, e.g. ubuntu]</code><br /> Below, anytime you see $DOMAIN in a command, it'll be replaced by whatever you put in for [your domain] and similarly $USER...</li> <li>Create directories to hold both the Docker Compose configurations and the persistent data you don't want to lose if you remove your Docker containers (namely your password database and configuration information):<br /><code>mkdir -p /home/docker/$DOMAIN &amp;&amp; mkdir -p /home/data/$DOMAIN<br /> chown -R ${USER}:${USER} /home/data /home/docker/</code></li> <li>Install the NGINX (pronounced "Engine X") webserver which will act as a reverse proxy for the BitWarden service and terminate the encryption via HTTPS:<br /><code>apt-get install nginx-full</code></li> <li>Configure the server's firewill and make an exception for SSH and NGINX services<br /><code>ufw allow OpenSSH<br /> ufw allow "Nginx Full"<br /> ufw enable</code><br /> Check that its running via<br /><code>ufw status</code></li> <li>Create a directory for including files for NGINX<br /><code>cd /etc/nginx</code><br /><code>mkdir includes</code><br /> Choose your text editor for editing files. Here're options for Vim or Nano - you can install and select others. Setting the EDIT shall variable allows you to copy and paste these commands regardless of which editor you prefer as it'll replace the value of $EDIT with the full path to your preferred editor.<br /><code>EDIT=`which nano`</code> or <code>EDIT=`which vim`</code></li> <li>To support encrypted data transfer between external devices and your server using HTTPS,  you need a valid SSL certificate. Until recently, these were costly and hard to get. With <a href="/protecting-your-users-lets-encrypt-ssl-certs">Let's Encrypt</a>, they've become a straightforward and essential part of any good (user-respecting) web site or service. To facilitate getting and periodically renewing your SSL certificate, you need to create the file letsencrypt.conf:<br /><code>$EDIT includes/letsencrypt.conf</code><br /> and enter the following content: <p><blockcode><code>#############################################################################<br /> # Configuration file for Let's Encrypt ACME Challenge location<br /> # This file is already included in listen_xxx.conf files.<br /> # Do NOT include it separately!<br /> #############################################################################<br /> #<br /> # This config enables to access /.well-known/acme-challenge/xxxxxxxxxxx<br /> # on all our sites (HTTP), including all subdomains.<br /> # This is required by ACME Challenge (webroot authentication).<br /> # You can check that this location is working by placing ping.txt here:<br /> # /var/www/letsencrypt/.well-known/acme-challenge/ping.txt<br /> # And pointing your browser to:<br /> # http://xxx.domain.tld/.well-known/acme-challenge/ping.txt<br /> #<br /> # Sources:<br /> # https://community.letsencrypt.org/t/howto-easy-cert-generation-and-renewal-with-nginx/3491<br /> #<br /> # Rule for legitimate ACME Challenge requests<br /> location ^~ /.well-known/acme-challenge/ {<br />     default_type "text/plain";<br />     # this can be any directory, but this name keeps it clear<br />     root /var/www/letsencrypt;<br /> }<br /> # Hide /acme-challenge subdirectory and return 404 on all requests.<br /> # It is somewhat more secure than letting Nginx return 403.<br /> # Ending slash is important!<br /> location = /.well-known/acme-challenge/ {<br />     return 404;<br /> }</code></blockcode></p> </li> <li> <p>Now you need to create the directory described in the letsencrypt.conf file:<br /><code>mkdir /var/www/letsencrypt</code></p> </li> <li> <p>Create "<a href="https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html#Forward_Secrecy_&amp;_Diffie_Hellman_Ephemeral_Parameters">forward secrecy &amp; Diffie Hellman ephemeral parameters</a>" to make your server more secure... The result will be a secure signing key stored in <code>/etc/ssl/certs/dhparam.pem</code> (note, getting enough "entropy" to generate sufficient randomness to calculate this will take a few minutes!<code>):</code><br /><code>openssl dhparam -out /etc/ssl/certs/dhparam.pem 4096</code></p> </li> <li> <p>and then you need to create the reverse proxy configuration file as follows:<br /><code>cd ../sites-available</code><br /><br /> and fill it with this content, replacing all [tokens] with your relevant values:<br /><blockcode><code>#<br /> # HTTP does *soft* redirect to HTTPS<br /> #<br /> server {<br />     # add [IP-Address:]80 in the next line if you want to limit this to a single interface<br />     listen 0.0.0.0:80;<br />     </code></blockcode><blockcode><code>server_name [your domain];<br />     root /home/data/[your domain];<br />     index index.php;<br /><br />     # change the file name of these logs to include your server name<br />     # if hosting many services...<br />     access_log /var/log/nginx/[your domain]_access.log;<br />     error_log /var/log/nginx/[your domain]_error.log;  <br />     include includes/letsencrypt.conf;</code></blockcode><blockcode><br /><br /><code>    # redirect all HTTP traffic to HTTPS.<br />     location / {<br />         return  302 https://[your domain]$request_uri;<br />     }<br /> }</code></blockcode><br /> and make the configuration available to NGINX by linking the file from sites-available into sites-enabled (you can disable the site by removing the link and reloading NGINX)<br /><code>ln -sf sites-available/bitwarden sites-enabled/bitwarden</code><br /> Check to make sure NGINX is happy with the configuration<br /><code>nginx -t </code><br /> If you don't get any errors, you can restart NGINX<br /><code>service nginx restart</code><br /> and it should be configured properly to respond to requests at <code>http://[your domain]/.well-known/acme-challenge/ </code>which is required for creating a Let's Encrypt certificate.<br /><code>$EDIT sites-available/bitwarden</code></p> </li> <li> <p>So now we can create the certificate. You'll need to install the letscencrypt scripts:<br /><code>apt-get install letsencrypt</code><br /> You will be asked to enter some information about yourself, including an email address - this is necessary so that the letsencrypt service can email you if any of your certificates are not successfully updated (they need to be renewed every few weeks - normally this happens automatically!) so that you site and users aren't affected by an expired SSL certificate (a bad look!). Trust me, these folks are the good guys.<br /> You create a certificate for [your domain] with the following command (with relevant substitutions):<br /><code>letsencrypt certonly --webroot -w /var/www/letsencrypt -d $DOMAIN</code><br /> If the process works, you should see a "Congratulations!" message.</p> </li> <li> <p>Edit the nginx configuration file for the BitWarden service again<br /><code>$EDIT sites-available/bitwarden</code><br /> and add the following to the bottom of <code>file (starting the line below the final "}")<br /><blockcode>#<br /> # HTTPS<br /> #<br /> # This assumes you're using Let's Encrypt for your SSL certs (and why wouldn't<br /> # you!?)... https://letsencrypt.org<br /> server {<br />     # add [IP-Address:]443 ssl in the next line if you want to limit this to a single interface<br />     listen 0.0.0.0:443 ssl;<br />     ssl on;<br />     ssl_certificate /etc/letsencrypt/live/[your domain]/fullchain.pem;<br />     ssl_certificate_key /etc/letsencrypt/live/[your domain]/privkey.pem;<br />     ssl_protocols TLSv1 TLSv1.1 TLSv1.2;<br />     # to create this, see https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html<br />     ssl_dhparam /etc/ssl/certs/dhparam.pem;<br />     keepalive_timeout 20s;</blockcode></code><blockcode></blockcode><blockcode><br /><code>    server_name [your domain];<br />     root /home/data/[your domain];<br />     index index.php;</code></blockcode><blockcode><br /><code>    # change the file name of these logs to include your server name<br />     # if hosting many services...<br />     access_log /var/log/nginx/[your domain]_access.log;<br />     error_log /var/log/nginx/[your domain]_error.log;</code></blockcode><blockcode><br /><br /><code>    location /notifications/hub/negotiate {<br />         proxy_pass http://127.0.0.1:8080;<br />         proxy_set_header Upgrade $http_upgrade;<br />         proxy_set_header Connection "upgrade";<br />         proxy_set_header Host $http_host;<br />         proxy_set_header X-Real-IP $remote_addr;<br />         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br />         proxy_set_header X-Forwarded-Host $server_name;<br />         proxy_set_header X-Forwarded-Proto https;<br />         proxy_connect_timeout 2400;<br />         proxy_read_timeout 2400;<br />         proxy_send_timeout 2400;<br />     }</code></blockcode><blockcode><br /><code>    location / {<br />         proxy_pass http://127.0.0.1:8080;<br />         proxy_set_header Upgrade $http_upgrade;<br />         proxy_set_header Connection "upgrade";<br />         proxy_set_header Host $http_host;<br />         proxy_set_header X-Real-IP $remote_addr;<br />         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br />         proxy_set_header X-Forwarded-Host $server_name;<br />         proxy_set_header X-Forwarded-Proto https;<br />         proxy_connect_timeout 2400;<br />         proxy_read_timeout 2400;<br />         proxy_send_timeout 2400;<br />     }</code></blockcode><blockcode><br /><code>    location /notifications/hub {<br />         proxy_pass http://127.0.01:3012;<br />         proxy_set_header Upgrade $http_upgrade;<br />         proxy_set_header Connection "upgrade";<br />     }<br />     #<br />     # These "harden" your security<br />     add_header 'Access-Control-Allow-Origin' "*";<br /> }</code></blockcode></p> </li> <li>You should now be able to run<br /><code>nginx -t </code><br /> again, and it you haven't got an accidental errors in the files, it should return no errors. You can restart nginx to make sure it picks up your SSL certificates...<br /><code>service nginx restart</code></li> </ol><p>Now everything is read to set up your BitWarden Docker containers!</p> <h3>Setting up your BitWarden "rust" service</h3> <p>Before we start this part, you'll need a few bits of information. First, you'll need a 64 character random string to be your "admin token"... you can create that like this:<br /><code>pwgen -y 64 1</code></p> <p>copy the result (highlight the text and hit CTRL+SHIFT+C) and paste it somewhere so you can copy-and-paste it into the file below later.</p> <p>Also, if you want your BitWarden server to be able to send out emails, like for password recovery, you'll need to have an "authenticating SMTP email account"... I would recommend setting one up specifically for this purpose. You can use a random gmail account or any other email account that lets you send mail by logging into an SMTP (Simple Mail Transfer Protocol) server, i.e. most mail servers. You'll need to know the SMTP [host name], the [port] (usually 465 or 587), the [login security] (usually "true" or "TLS"), and your authenticating [username] (possibly this is also the email address) and [password]. You'll also need a "[from email] like bitwarden@[your domain] or similar, which will be the sender of email from your server.</p> <p>You're going to be setting up your configuration in the directory we created earlier, so run<br /><code>cd /home/docker/$DOMAIN</code></p> <p>and there<br /><code>$EDIT docker-compose.yml</code></p> <p>copy-and-pasting in the following, replacing the [tokens] appropriately:</p> <p><blockcode><code>version: "3"<br /> services:<br />     app:<br />         image: mprasil/bitwarden<br />         environment:<br />             - DOMAIN=https://[your domain]<br />             - WEBSOCKET_ENABLED=true<br />             - SIGNUPS_ALLOWED=false<br />             - LOG_FILE="/data/bitwarden.log"<br />             - INVITATIONS_ALLOWED=true<br />             - ADMIN_TOKEN=[admin token]<br />             - SMTP_HOST=[host name]<br />             - SMTP_FROM=[from email]<br />             - SMTP_PORT=[port]<br />             - SMTP_SSL=[login security]<br />             - SMTP_USERNAME=[username]<br />             - SMTP_PASSWORD=[password]<br />         volumes:<br />             - /home/data/[your domain]/data/:/data/<br />         ports:<br />             - "127.0.0.1:8080:80"<br />             - "127.0.0.1:3012:3012"<br />         restart:<br />             unless-stopped</code></blockcode></p> <p><em>Note that the indentation has to be exact in this file - Docker Compose will complain otherwise.</em></p> <p>With the docker-compose file completed, you're ready to "pull" your package!</p> <p><code>docker-compose pull</code></p> <p>This will download the BitWarden Docker container from hub.docker.com. Then all you need to do is start it:</p> <p><code>docker-compose up -d &amp;&amp; docker-compose logs -f</code></p> <p>the "up -d" option actually starts the container called "app" which is actually your BitWarden rust server in "daemon" mode, which means it'll keep running unless you tell it to stop. If that's successful, it automatically then shows you the logs of that container. You can exit at any time with CTRL-C which will put you back on the command prompt. If you <em>do</em> want the container to stop, just run</p> <p><code>docker-compose stop</code></p> <p>If your start up was successful, you should see a message like this (albeit your version number could be higher - 1.9.0 is the current version of the Rust implementation at the time of writing):</p> <p><blockcode><code>/--------------------------------------------------------------------\<br /> |                       Starting Bitwarden_RS                        |<br /> |                           Version 1.9.0                            |<br /> |--------------------------------------------------------------------|<br /> | This is an *unofficial* Bitwarden implementation, DO NOT use the   |<br /> | official channels to report bugs/features, regardless of client.   |<br /> | Report URL: https://github.com/dani-garcia/bitwarden_rs/issues/new |<br /> \--------------------------------------------------------------------/</code></blockcode></p> <p>You should now be able to point your browser at <code>http://[your domain]</code> which, in turn, should automatically redirect you to <code><strong>https://</strong>[your domain]</code> and you should see the BitWarden web front end similar to that shown in the attached screen shot!</p> <p>To do your initial login, I believe (I'll test this and update this howto!) you'll be asked to provide your "admin token" to create a first user with administration privileges.</p> <p>For additional info on setting up these services - and new options as Daniel and his co-developers add them in - consult the <a href="https://github.com/dani-garcia/bitwarden_rs">repository pages</a> and <a href="https://github.com/dani-garcia/bitwarden_rs/issues">issues</a> and for Docker-specific questions, look at <a href="https://github.com/mprasil/bitwarden_rs">mpasil's pages</a>.</p> <h3>Sending Emails</h3> <p>It'll be worth testing if your email services work, like by requesting a password hint! You should be able to see what the server's doing via the</p> <p><code>docker-compose logs -f</code></p> <h2>Tips</h2> <p>I recommend <em>not </em>including your login credentials to your BitWarden instance in your BitWarden database ;) that's the one thing you need to remember. If you need to write it down somewhere, then do so (but make sure you don't include <em>all</em> the info needed to log in on the same piece of paper, that's just asking for trouble).</p> <p>Also, you can easily configure all the BitWarden clients - browser plugins, mobile apps, or the desktop app -  to use your server rather than BitWarden's default hosted service. Just click the "gear" settings icon on each app's interface, and set the "Self-Hosted Environment" Server URL to be your server, i.e. https://[your domain]</p> <h3>Backing it all up</h3> <p>I'll add information on my SQLite backup scripts (which maintain automatic versioned hourly, daily, weekly, monthly, and yearly database dumps, the content in which is encrypted)...</p> <h3>Two Factor Authentication</h3> <p>This configuration should allow you to simply turn on Two Factor Authentication for any given BitWarden user.</p> <h3>Keeping it up-to-date</h3> <p>One of the best things about this Docker configuration is that it's very straightforward to upgrade your installation to Daniel's (via mpasil's Docker work) latest server version. Just log into the server as your unprivileged user,</p> <p><code>cd /home/docker/[your domain]<br /> docker-compose  pull<br /> docker-compose up -d &amp;&amp; docker-compose logs -f</code></p> <p>The whole process shouldn't take much more than a minute, with a few seconds downtime only as your new Docker BitWarden container is being created...</p> <p>Hope this helps a few folks! If you find any of the above doesn't work, please let me know in the comments. I'll do my best to make sure this how-to is accurate and up-to-date, and I'll do my best to assist people if they're having trouble.</p> <p>Have (secure and private) fun!t</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=25&amp;2=field_blog_comments&amp;3=comment" token="hKQtrXBA7ErwoC_B3-AO9Vd7GHqkXf0X-QAroc-dhuE"></drupal-render-placeholder> </div> </section> Tue, 22 Jan 2019 22:30:13 +0000 dave 25 at http://tech.oeru.org Democratising Higher Education with OERs & FOSS http://tech.oeru.org/democratising-higher-education-oers-foss <span class="field field--name-title field--type-string field--label-hidden">Democratising Higher Education with OERs &amp; FOSS</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--oeru18"> <span class="field__item-wrapper"><a href="/taxonomy/term/65" hreflang="en">oeru18</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Mon 21/01/2019 - 14:02</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>Rather than write a slide-based presentation for the Open Education miniconf at <a href="https://linux.conf.au" title="Linux.Conf.AU, the premier FOSS annual conference in the southern hemisphere... ">Linux.Conf.AU 2019</a>, I thought I'd try an approach that seemed to work quite well <a href="/2018-update-oeru-technology-stack">recently</a>: write my presentation as a blog post and make it available for posterity, before and after. You can find this post quickly by going to <strong>oer.nz/lca</strong> if you want to look at it on your own device.</p> <p>Update: If you'd like to see me presenting this talk, <a href="https://www.youtube.com/watch?v=o9KaDiHj-H4">the video is now available</a>. There're <a href="https://www.youtube.com/user/linuxconfau2019/videos">lots of other talks</a> from the conference, too - I encourage you to have a browse!</p> <p>So why am I here and what am I talking about?</p> <p>I'm here because I love building stuff. I'm also a fan of sharing, and of the Commons (both analogue and digital). Unsurprisingly, I'm a committed adherent to and advocate for the principles underlying Free and Open Source Software (FOSS) and Creative Commons licensing.</p> <p>As a person who has a lot of time working with technology and has had a great education thanks to a combination of luck, hard work, and privilege, I have realised that I have an opportunity to parley my good fortune to make the world a better place.</p> <p>After quite a bit of thinking about it, I've really only identified one thing that I feel confident is an unambiguously "good thing" to do: increasing opportunities in education, particularly for girls and women. So I<em> </em>focus my energies on<em> removing as many barriers as I can that hinder people all over the world from reaching their educational potentials</em>.</p> <h2>Living the Dream</h2> <p>After following a relatively <a href="https://davelane.nz">convoluted path</a> for many years, I consider myself very fortunate to be working in my dream role today: I'm the <a href="/insight-what-does-open-source-technologist-oer-foundation-do">Open Source Technologist</a> for the Open Educational Resource Foundation (OERF). I say it's my dream role because it allows me to support my family, work on something I find personally fulfilling because it's completely aligned with my personal interests, skills, and values, and gives me the huge satisfaction of knowing I'm doing something that will create opportunities for millions of people I've never even met.</p> <p>The OERF is a charitable foundation, hosted by the <a href="https://op.ac.nz">Otago Polytech</a> in Dunedin (South Island, New Zealand). I think it's fair to say we're a lean operation... with two full-time staff: the founder, <a href="https://wikieducator.org/User:Mackiwg" title="Formal bio for Wayne Mackintosh"><strong>Wayne Mackintosh</strong></a>, who's a committed FOSS-using academic and a genius at getting people to do the right thing (as well being the UNESCO chair in OER), and me - so the Open Source Technologist is the <em>only technologist</em>. You can start to see why I love this role.</p> <h2>Our Aims</h2> <p>The Foundation has three main aims:</p> <ol><li>to create the curriculum materials for an entire university's worth of academical "courses" that are entirely licensed as "<a href="https://en.wikipedia.org/wiki/Definition_of_Free_Cultural_Works" title="The definition of Free Cultural Works">Free Cultural Works</a>" (CC-By or CC-By-SA) - this is the curriculum analogue to FOSS.</li> <li>to create and lead a partnership with higher education institutions around the world to create a new type of higher education model. These partners recognise the value of collaborating on learning materials and are committed to expanding global access to high education, the articulation of academic credit, and see an opportunity in extending their assessment services to learners who aren't formally enrolled in their institutions.</li> <li>to create a platform for delivering these collaboratively assembled and accredited courses to <em>anyone who has access to the internet</em>, allow them to study those materials at no cost, and with low cost<em> formal assessment services</em> provided by our partners. This allows learners to gain <em>academic credit from accredited institutions,</em> potentially leading to a formal qualification. We encourage others (especially our partners) to copy our approach where it appeals to them.</li> </ol><h2>Radical Openness</h2> <p>Prior to my involvement, Wayne decided that the organisation would be "radically open". And he means it. How it works:</p> <ul><li>our constantly evolving strategic direction is openly published as a Free Cultural Work,</li> <li>all decision making is done collectively with our partners and the results (and thinking behind them) is proactively published,</li> <li>all our meetings are streamed, videoed, and made available for posterity, along with post meeting reports summarising the discussion and any decisions reached, and</li> <li><a href="/proprietary-software-holdouts">wherever possible</a>, the OERF uses FOSS (we all use Linux internally), contribute to upstream projects, and release any internally developed software as FOSS.</li> </ul><p>Here's an example of what it looks like: <a href="https://wikieducator.org/OERu/OERu_18.11_Meeting/Report#OERu_milestones">our 2018 partner meeting in Port Macqurie, NSW, Australia.</a></p> <h2>Associated Policies</h2> <p>In addition to its commitment to openness, the OERF undertakes its activities in accordance with a few general principles:</p> <ol><li>All our curriculum materials should be available for any educator to incorporate into any course an educator chooses, in a way that respects their authorship and the provenance of the materials.</li> <li>Most higher education institutions choose a monolithic learning management system (LMS) to frame learner interactions, allowing only authenticated learners to participate, and on matriculation they lose access to their learning materials and artefacts. Instead, we prefer a more flexible, open, and persistent environment that is also more relevant to learners: the Internet. </li> <li>No learners studying, nor educators contributing to, our course materials should be disadvantaged because they cannot afford proprietary software to do so, nor should they need to entrust any of their creative efforts, or personal privacy to any third party with a commercial interest.</li> <li> <p>This entire process should be fiscally sustainable, not based on pure charity, but due to enlightened self-interest from educational institutions (who realise they are in imminent danger of having their business models permanently disrupted).</p> </li> </ol><h2>Open, Reusable, Editable, Distributable content</h2> <p>All our curriculum materials should be available for any educator to incorporate into any course an educator chooses. We use the powerful (but relatively transparent to non-technical users) versioning capabilities of MediaWiki to version all content and provide an educators' analogue to a git repository. Our curriculum material site is <a href="https://wikieducator.org">WikiEducator.org</a> and it is usually among the top 100k sites on the web.</p> <h2>Loosely Coupled, no Monoliths</h2> <p>Rather than an LMS, we adhere to the "<a href="https://en.wikipedia.org/wiki/Unix_philosophy">Unix philosophy</a>" - all of our learning materials are delivered by a <a href="/many-simple-tools-loosely-coupled">loosely coupled</a> selection of excellent FOSS applications via the open Internet, and can be viewed by anyone, anonymously.</p> <p>Collaborating with other learners does require authentication to manage behaviour and demonstrate participation, however this is almost entirely done using FOSS web services we host.</p> <p>We select the software components for our "<a href="/2018-update-oeru-technology-stack">Next Generation Digital Learning Environment</a>" based on being FOSS licensed, web-based, open standards compliant, fit for purpose, mobile-friendly, and flexible (e.g. with respect to authentication methods).</p> <p>We also consider scalability - although our current learner numbers are low, in the hundreds or thousands, we expect to scale to hundreds of thousands in the coming year or two, and eventually to millions of learners.</p> <p>Thanks to our component model, if we identify a better FOSS component to fill a particular niche, we simply replace what's there already. Our solutions can evolve quickly (especially compared to most of academia) and relatively painlessly from our learners' perspective.</p> <h2>Respect People</h2> <p>To ensure we do not open our learners or collaborators to exploitation by corporate interests outside of our control... we don't use any in our teaching process. We see them as largely incompatible with the <em>sharing</em> necessary for real teaching and learning. Our entire software stack is comprised of FOSS, including collaboration tools for both learners and educators.</p> <h2>Sustainable Model</h2> <p>Of course, as with any FOSS project, for the OERF's approach to succeed, it must appeal to the <em>enlightened self-interest</em> of its participants. Our aim, as a charitable foundation is not to profit, but to achieve sustainability while maximising our impact...</p> <p>Part of that is consistently achieving sufficient income and the other part is minimising costs. We have done the latter to the extent possible (our small staff overhead and an annual IT budget for software and infrastructure is $5000!). Other than a small but much appreciated grant from the Hewlett Foundation (we're a long time grantee), we are already self-sustaining.</p> <p>The major of our income comes from our partners' membership fees (which are about USD4000 per year, although some opt to pay somewhat more), which they choose to pay for a variety of different reasons.</p> <p>For some it's a matter of fulfilling their "greater good" mission. For others, it's a philosophical direction which appeals - improving access to education to those who currently have no realistic opportunities to pursue it due to their location, cultural restrictions, and personal circumstances.</p> <p>Many of our partners realise that their current situations are precarious - that they are in imminent danger of having their business models permanently disrupted by technologically savvy initiatives like ours  - it's in their best interest to be part of them.</p> <p>Many also recognise an opportunity in becoming familiar with the FOSS technologies that we're implementing and making available to our partners - if they choose to implement any one of our chosen technologies at their own institutions rather than a proprietary alternative, the cost savings alone (as well as increase capabilities) would easily compensate of many years worth of our membership fee.</p> <h2>The FOSS Stack</h2> <p>At a conference like this, people like to get specific about the technologies we use, so <a href="/2018-update-oeru-technology-stack">let's look into that</a> for the time we have left! Happy to talk to anyone further about all the details!</p> <h2> </h2> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <h2 class="comment-field__title">Blog comments</h2> <article data-comment-user-id="0" id="comment-49" about="/comment/49" typeof="schema:Comment" class="comment js-comment by-anonymous has-title clearfix"> <div class="comment__container"> <h3 property="schema:name" datatype="" class="comment__title"> <a href="/comment/49#comment-49" class="permalink" rel="bookmark" hreflang="en">contribute</a> <span class="comment__new marker marker--success hidden" data-comment-timestamp="1560387860"></span> </h3> <div class="comment__meta"> <div class="comment__submitted"> <span class="comment__author"><span rel="schema:author"><span lang="" typeof="schema:Person" property="schema:name" datatype="">Stephen (not verified)</span></span> </span> <span class="comment__pubdate">Tue 28/05/2019 - 19:57 <span property="schema:dateCreated" content="2019-05-28T07:57:31+00:00" class="rdf-meta hidden"></span> </span> </div> </div> <div class="comment__content"> <div property="schema:text" class="clearfix text-formatted field field-comment--comment-body field-name-comment-body field-type-text-long field-label-hidden"> <div class="field__items"> <div property="schema:text" class="field__item"><p>This is very inspiring!</p> <p>Looking forward to chasing those links and learning more. I would love to contribute to these changes in education.</p> </div> </div> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=49&amp;1=default&amp;2=en&amp;3=" token="2bW4gH3KUsfISmgVb-IEas6wHmfqlnC3dv3_lvUWzdE"></drupal-render-placeholder> </div> </div> </article> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=24&amp;2=field_blog_comments&amp;3=comment" token="mjUav4-YhA77NHR_0jlXW-3giZNhdMcjCjhc8ozAcyM"></drupal-render-placeholder> </div> </section> Mon, 21 Jan 2019 01:02:01 +0000 dave 24 at http://tech.oeru.org 2018 update on the OERu Technology Stack http://tech.oeru.org/2018-update-oeru-technology-stack <span class="field field--name-title field--type-string field--label-hidden">2018 update on the OERu Technology Stack</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Wed 31/10/2018 - 10:24</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>As I prepare for the <a href="https://course.oeru.org/2018-meetings/">2018 OERu Partners' Meeting</a>, I'm working how to convey the depth and breadth of the OERu open source technology stack - our infrastructure and applications - to our partners and other attendees.</p> <p>Over the past year, we've had our first live OERu courses allowing learners to work towards a formal exit qualification (our "1st year of study"), and our infrastructure has worked as intended throughout, so I'm chalking that up as a win.</p> <h2>OERu Infrastructure - NGDLE</h2> <p>At the OERu, in following our open principles, we have ended up creating a "Next Generation Digital Learning Environment" (NGDLE) to meet the needs of our learners, partners, and OER collaborators. It's a distributed, <a href="/many-simple-tools-loosely-coupled">loosely coupled component</a> model. It's also entirely made up of Free and Open Source software, from top to bottom.</p> <figure role="group" class="caption caption-img"><img alt="OERu Technology Service wheel - the basis for our Next Generation Digital Learning Environment" data-entity-type="file" data-entity-uuid="19dbe4e0-3256-42a3-a34a-575c518dc532" height="484" src="/sites/default/files/inline-images/OERu_technology_wheel.png" width="701" /><figcaption>Services making up parts of the OERu's NGDLE. These are either hosted by the community or on the OERu's fully open source technology infrastructure.</figcaption></figure><p>We believe our approach has a lot of advantages and, if emulated by our partners and other academic institutions, could revolutionise both the quality of digital services used in education, as well as vastly reducing costs and increasing the autonomy and resilience of technical solutions while providing unprecedented technology-related learning opportunities and agency for learners and educators alike. The purpose of this post is to describe our technology infrastructure, explain some of its advantages, and highlight the challenges it presents.</p> <p>A key takeaway: <em>if any of our partners adopted even one of the technologies we have incorporated into our NGDLE, they could easily save many times the value of their annual OERu subscription fees  in the first year, and in every subsequent year</em>. This blog exists to provide handy howtos to make it easy for our partners to trial and adopt these technologies in a maintainable way.</p> <h3>Geographic Diversity</h3> <p>In my role of Open Source Technologist, I maintain our global computing infrastructure, currently made up of <a href="https://www.mapcustomizer.com/map/OERu%20Global%20Infrastructure" title="Map of OERu's global hosting infrastructure.">four separate nodes</a>, on behalf of the <a href="https://oeru.org">OERu</a>:</p> <ol><li><a href="https://osm.org/go/0DvZ0tbZ--" title="The likely hosting facility for u.oerfoundation.org">u.oerfoundation.org</a> (5.9.142.102) - a dedicated server hosted in an Hetzner Online facility in Gunzenhausen, Bavaria, Germany.</li> <li><a href=" https://osm.org/go/uG4HYrE-?m=">open.oerfoundation.org</a> (40.127.81.149) - a compute instance hosted in a Microsoft Azure facility in Melbourne, Victoria, Australia (charitable organisation grant from TechSoup).</li> <li><a href="https://osm.org/go/urEwlGNt-?m=">containers.oerfoundation.org</a> (202.49.241.195) - a compute instance hosted in a Catalyst Cloud facility in Wellington, New Zealand.</li> <li><a href="https://osm.org/go/WIDlXmu-?m=">we6.wikieducator.org</a> (54.218.47.90) - a compute instance hosted in an Amazon Web Services facility in Portland, Oregon, USA.</li> </ol><figure role="group" class="caption caption-img"><a href="https://www.mapcustomizer.com/map/OERu%20Global%20Infrastructure" title="Link to a live map (using the open source OpenStreetMap platform!) - you can drill down to each hosting facility."><img alt="A map view of the OERu's open source tech infrastructure." data-entity-type="file" data-entity-uuid="225adadd-5811-47e3-8c03-a86d81f5ae78" src="/sites/default/files/inline-images/OERu_global_infrastructure_map.png" /></a> <figcaption>This image is linked to a live <a href="https://www.openstreetmap.org">OpenStreetMap</a> view (open source alternative to Google Maps) - there you can zoom into each facility in which our hosting infrastructure is held. This interactive map was created using <a href="https://www.mapcustomizer.com">MapCustomiser</a>.</figcaption></figure><h3>Functional Diversity and Containers</h3> <p>All of these servers run the open source operating system, <a href="https://en.wikipedia.org/wiki/Linux">Linux</a>, and with the exception of we6.wikieducator.org, run multiple services in <a href="https://www.docker.com/docker-community">Docker</a> containers. We6 is the original OERu server and it runs the technology on which our first initiative is built: a <a href="https://www.mediawiki.org/wiki/MediaWiki">MediaWiki</a> implementation answering to <a href="https://wikieducator.org">https://wikieducator.org</a> - it's where all our curriculum materials are assembled and maintained.</p> <p>The story on our other hosts is more complicated. Here's a run down of the <em>fully open source</em> services each provides:</p> <ol><li>u.oerfoundation.org is currently running a number of services, including 26 Docker containers, which collectively provide the following services: <ol><li>Our <a href="https://www.silverstripe.org/">SilverStripe</a> OERu website, <a href="https://oeru.org">https://oeru.org</a>, which is our main contact with prospective partners and our course advertising platform.</li> <li>Our <a href="https://wordpress.org">WordPress</a> <a href="https://codex.wordpress.org/Create_A_Network">Multisite</a>, <a href="https://course.oeru.org">https://course.oeru.org</a>, our primary course delivery platform (each course instance is a separate "sub-site" but a user on our course site can enrol in any course sub-site), to which we can automatically deploy fully formed courses from <a href="https://wikieducator.org">WikiEducator</a> via our Snapshotting system.</li> <li>Our <a href="https://www.discourse.org/">Discourse Forums</a>: <ol><li><a href="https://community.oeru.org">https://community.oeru.org</a> - for our educator and partner community</li> <li><a href="https://forums.oeru.org">https://forums.oeru.org</a> - for our learners</li> </ol></li> <li>Our <a href="https://rocket.chat">Rocket.Chat</a> instance, <a href="https://chat.oeru.org">https://chat.oeru.org</a>, for real-time communication - text &amp; media messages, audio and video calls - within our team, and with the broader partner and learner community. We also host a couple other Rocket.Chat instances for well aligned groups and partners trialling it. Or you could <a href="/docker-compose-better-way-deploy-rocketchat-wekan-and-mongodb">run your own Rocket.Chat</a>...</li> <li>Our <a href="/wikieducator-notes-oerus-course-feed-aggregation-and-messaging-system">WEnotes Course Feed</a> stack made up of several containers that monitor various social media and blogs, and then scan, store, and serve up our dynamic course feeds - <a href="https://course.oeru.org/lida101/interactions/course-feed/">for example</a>.</li> <li>Our <a href="https://matomo.org">Matomo</a> (formerly Piwik) instance, <a href="https://stats.oeru.org">https://stats.oeru.org</a>, allows us to track the use of our websites ourselves. It's functionally similar to Google Analytics, but without giving information on our web visitors to Google.</li> <li>Additionally this server hosts a number of native services (not in containers) including <ol><li><a href="https://yourls.org">YourLS</a>, our link shortener on <a href="https://oer.nz">https://oer.nz</a>, and</li> <li><a href="http://semanticscuttle.sourceforge.net/">Semantic Scuttle</a>, on <a href="https://bookmarks.oeru.org">https://bookmarks.oeru.org</a>.</li> </ol></li> <li>Finally, we run a number of test systems including replicas of WikiEducator and our Course site.</li> </ol></li> <li>open.oerfoundation.org runs 27 Docker containers which collectively provide <ol><li>Our <a href="https://joinmastodon.org/">Mastodon</a> instance, <a href="https://mastodon.oeru.org">https://mastodon.oeru.org</a>, our open education corner of the "<a href="https://en.wikipedia.org/wiki/Fediverse">Fediverse</a>" - it's a micro-blogging platform similar to Twitter, but distributed - without the centralised corporate control and surveillance capitalism business model. Here's how to <a href="/installing-mastodon-docker-compose-ubuntu-1604">run your own Mastodon</a>...</li> <li>Our <a href="https://www.collaboraoffice.com/">Collabora Office</a> + <a href="https://nextcloud.com/">NextCloud</a> instance, <a href="https://doc.oeru.org">https://doc.oeru.org</a>, which together are functionally similar to Google Drive or Dropbox + Google Docs, offering web based document storage, combined with the ability to collaboratively edit documents, spreadsheets, and presentations. Here's how to <a href="/installing-nextcloud-and-collabora-office-online-docker-ubuntu-1604">make your own</a>...</li> <li>Our <a href="http://etherpad.org/">Etherpad-Lite</a> instance, <a href="https://etherpad.oeru.org">https://etherpad.oeru.org</a>, for collaborative note taking.</li> <li>Our <a href="https://mautic.com/">Mautic</a> instance, <a href="https://mautic.oeru.org">https://mautic.oeru.org</a>, which manages and automates our email out to learners and partners, as well as  prospective partners and learners. It's an invaluable tool for magnifying the responsiveness and effectiveness of our small OER Foundation team. Replicate our approach to <a href="/installing-mautic-php7-fpm-docker-nginx-and-mariadb-ubuntu-1604">run your own</a>.</li> <li>Our <a href="https://www.limesurvey.org/">Lime Survey</a> instance, <a href="https://survey.oeru.org">https://survey.oeru.org</a>, where we manage our learner and partner surveys.</li> <li>Our <a href="https://moodle.org">Moodle</a> instance, <a href="https://moodle.oeru.org">https://moodle.oeru.org</a>, for various assessment related testing and other activities.</li> <li>Our <a href="http://wallabag.org/">Wallabag</a> instance, <a href="https://wallabag.oeru.org">https://wallabag.oeru.org</a>, a very slick shared bookmarking tool, which we're evaluating as a replacement for Semantic Scuttle.</li> <li>Our <a href="https://drupal.org">Drupal 8 CMS</a> instance, that powers this very Tech Blog!</li> <li>Our own <a href="https://bitwarden.com">BitWarden</a> instance, <a href="https://safe.oeru.org">https://safe.oeru.org</a>, as our password keeper for managing our passwords and ensuring we have <strong>strong</strong> passwords, that we can optionally share in specific organisation-level groups.</li> <li>We also run a couple of test websites built on a blogging/CMS platform called "<a href="https://getgrav.org">Grav</a>". Watch this space.</li> </ol></li> <li>containers.oerfoundation.org is our newest computing resource, based in New Zealand (made available through generous sponsorship from <a href="https://catalystcloud.nz">Catalyst Cloud</a>). It is intended to run containers hosting services that are local to us. Its current set of Docker containers runs our <a href="https://gitlab.org">Gitlab</a> instance, at <a href="https://git.oeru.org">https://git.oeru.org</a> - you can view <a href="https://git.oeru.org/explore/projects">our existing projects</a> you're welcome to use, learn from, and improve - to which we are moving all of our remote software code repositories. Any software developers are encouraged to have a look around and make use of our code! Contributions are always welcome.</li> </ol><h2>Agile and flexible</h2> <p>One of the best things about our "loosely coupled component" model is that we have the ability to quickly incorporate useful new capabilities, reinforce and update components, and readily replace weaker components with stronger ones without sentimentality as we identify them - our open source policy allows rapid evolution without the sentimentality that the <a href="https://en.wikipedia.org/wiki/Sunk_cost">sunk-cost fallacy</a> tends to create.</p> <p>In recent months, we have tested a number of new open source technologies, and introduced a few of those into our component mix, including Lime Survey, Gitlab, and BitWarden, as well as enjoyed steadily improving capabilities - features, security, privacy, and performance - by tracking community-driven updates to most of our existing components.</p> <p>On the back end, there have been even more improvements, like our move from normal Docker to <a href="https://docs.docker.com/compose/">Docker Compose</a> which has revolutionised our ability to rapidly deploy and <em>move around</em> services between hosting infrastructure.</p> <figure role="group" class="caption caption-img"><img alt="Side by side comparison of Semantic Scuttle and Wallabag." data-entity-type="file" data-entity-uuid="5a1a508e-bf3f-4932-9ec8-9437497f5d1f" height="384" src="/sites/default/files/inline-images/wallabag_and_bookmarks.png" width="714" /><figcaption>This is a side-by-side comparison of Semantic Scuttle (left) and Wallabag (right).</figcaption></figure><p>We have also been able to identify and test potentially better components for certain uses - for example,  our "Bookmarking" tool, provided by an instance of Semantic Scuttle, is not as modern as many of our other components, and its interface is decidedly dowdy by comparison. We have since found another social bookmarking tool, Wallabag, which promises to provide a more usable service, with high quality open source mobile apps available, as well as having in-built support for a variety of Single-Sign-On technologies which Semantic Scuttle lacks.</p> <h2>Capacity and scalability</h2> <p>The OERu is starting small - with tens or hundreds of learners participating in our courses. This places minimal load on our infrastructure, but allows us to validate that it is working as intended. But for many software implementations, those sorts of numbers might already be challenging the ability of normally available computing infrastructure to supply a usable service.</p> <p>One of the major advantages of our loosely coupled component model, making use of best-of-breed open source applications working together, is that each of those components is in active use in other contexts. They have all already had their "trial-by-fire", confronting the challenges of efficiently supporting large numbers of users (by large, I mean not tens or hundreds, but millions or tens of millions) and they have <em>already</em> evolved to meet those challenges.</p> <p>Although the applications we choose as components are all the products of different communities, different developers, and different technologies, all adhere to a set of well tested robust and <em>scalable</em> internet software service patterns. All have separate data stores (mostly databases, including MariaDB,  PostgreSQL, MongoDB, CouchDB, and SQLite) themselves decoupled from the containers doing the computing, usually running scripting engines (we use PHP, Ruby on Rails, Python, and Node.JS) that manipulate that data in a "stateless" way pushing and pulling data from users' browsers which employ open standards compliant HTML5 (comprising HTML markup, CSS for styling, and Javascript for in-browser client application functionality).</p> <p>That model makes it possible to scale up <em>all</em> of these services simply by adding more computing containers (which is facilitated by the use of Docker).</p> <h2>Costs<a id="costs" name="costs"></a></h2> <p>One way that the OERu maintains its capabilities with such a small infrastructure and development budget is that we adhere to a few key principles:</p> <ol><li>Use commodity open source hosting (we only run Linux), allowing for rapid movement between hosting providers with minimal trouble or disruption to our services.</li> <li>If using a Software-as-a-Service (SaaS) solution, strongly prefer open source options which gives us a safety valve if the pricing model/service doesn't suit our needs. (At this point, Zoom is the only proprietary software in our stack). This largely removes vendor lock-in.</li> <li>Ensure any external purchased service is fixed price and does not increase with number of users.</li> <li>Include internal maintenance time in cost of ownership.</li> </ol><p>As described above, we use four main hosting providers across the globe. All of our computing instances and dedicated machines are commodity system without proprietary features. We never exceed the (usually very generous) in-built data and storage allotments, so our prices are fixed and predictable.</p> <p>The OERu's entire annual infrastructure/IT costs can be summarised as follows (values approximate, in USD):</p> <ul><li>Four Servers: AWS ($4000) + Hetzner ($440) + Azure ($0) + Catalyst Cloud ($0) = $4440.</li> <li>SaaS: Zoom ($180) + <a href="https://kanboard.org" title="A SaaS offering built around an open source web-based Kanban application.">Kanboard</a> ($360)</li> </ul><p>Total <strong>annual</strong> software + infrastructure budget: <strong>$4800</strong></p> <p>Some of the OER Foundation's hosting infrastructure costs are covered by a "Charitable Organisation Grant" which includes $5000/year worth of Microsoft Azure hosting services. We also gratefully receive $500/month sponsored hosting services from the NZ-based hosting provider, Catalyst Cloud, who provide a fully open source cloud hosting infrastructure, and like the work the OERu is doing!</p> <h3>Commodity</h3> <p>We run Linux on all of them (Ubuntu or Debian) which is available at no cost. We can run as many or as few systems as we want at a fixed cost. Only the sunk cost of my time and the relative computing resource requirements are variable - but these vary in far less than a linear fashion with user numbers, e.g. for 10 times more users, we might require only 10% more of my time, and maybe 50% more computing resources.</p> <h3>Case Study: SaaS and the value of open source<a id="case-study" name="case-study"></a></h3> <p>At last year's Partners' meeting, we described Mautic, an impressive new open source "marketing automation" tool we were using to automate much of our email communication with learners and partners as well as prospective learners and partners.</p> <p>Initially, to test it, we opted to use the $30/month SaaS "entry-level" offering from Mautic.com which allowed us a single login to get started right away and test the Mautic system's fit to our requirements. It allowed us up to 2000 contacts, with an increased cost beyond that.</p> <p>Shortly after last partner meeting, however, the folks at Mautic.com contacted us to say that their pricing model was changing and that our new costs would go up by more than 10 times - $500/month, and with a much steeper increase for new contacts. For example, 10k contacts would cost us more like $1000/month.</p> <figure role="group" class="caption caption-img"><img alt="Mautic SaaS vs. Self-Hosted" data-entity-type="file" data-entity-uuid="e63e203a-1b37-40d9-bb70-804f7bdfc6c0" height="273" src="/sites/default/files/inline-images/MauticCostPlots.png" width="622" /><figcaption>Relative cost models of the hosted (SaaS) Mautic service vs. the self-hosted OERu Mautic instance.</figcaption></figure><p>Given the substantial price rise and unpleasant "scaling factor" for new contacts, combined with the fact that we found Mautic a very useful piece of software, we were in an uncomfortable position. Thankfully the Mautic platform itself is open source software. So <em>unlike all proprietary SaaS offerings </em>we had the option of hosting the service ourselves.</p> <p>In the course of a day or two of my time invested into creating an implementation on our hosting infrastructure (which <a href="/installing-mautic-php7-fpm-docker-nginx-and-mariadb-ubuntu-1604">I have documented</a>), we had <a href="https://mautic.oeru.org">https://mautic.oeru.org</a> up and running.</p> <p>Our new cost profile for Mautic is far more favourable. I spend perhaps an hour or so every month or two to update the Mautic system (it's improving continually thanks to the efforts of its Mautic.com-led developer community - which now includes the OERu!). Mautic places a negligible additional load on our AU-based infrastructure, but I have assigned it a small proportion of the (donated) hosting cost, a bit of my time, and a negligible amount for outgoing email costs (the 20k or so emails our Mautic has sent so far using the AWS "Simple Email Service" commodity SaaS have cost us a whopping $0.50 so far).</p> <p>Overall, the per-month cost-comparison for 10000 contacts looks like its about $1000 vs. $80, or an annual savings of approximately $11,040 off a total of $12,000. Massive 92% savings, <em>amounting to twice our total infrastructure budget</em>.</p> <p>And the savings only increases as we grow as we full intend to do! If, for example, we ended up with 100k contacts in the next year, the SaaS cost would be more like $4000/month. For 100k contacts, our self hosted system's costs might increase to $120/month (due to increased infrastructure resources being used)... The resulting savings would then be $46,560 off a total of $48,000 - a non-trivial savings of 97% or <em>almost 10 times our total infrastructure budget</em>!</p> <p>More importantly, the absolute cost of our environment is a tiny fraction, perhaps 5-10%, of that represented by a managed, outsourced SaaS offering. The open source self-hosted approach we've been able to validate represents a huge opportunity, particularly for higher education in emerging economies.</p> <h2>Progress since last year</h2> <p>In addition to building and maintaining the above open source software stack, our small team on occasionally identifies opportunities act decisively and strategically, to write our own code that bridges gaps or improves the usability of components in our NGDLE. Since the <a href="https://course.oeru.org/2017-meetings/">last OERu partner meeting</a>, we've rolled up our sleeves to create the <a href="/oeru-blog-feed-finder">Blog Feed Finder</a>, and we're about to release our new Registration and Enrolment plugin for WordPress MultiSite implementations (still a <a href="https://git.oeru.org/oeru/register-enrol">work-in-progress</a>).</p> <p>We have lots of other strategic initiatives stacked up and waiting to receive our full attention. The main constraint on that progress is our capacity, namely me (as the OER Foundation's <a href="/insight-what-does-open-source-technologist-oer-foundation-do">Open Source Technologist</a>). Mine is a very gratifying role, but a bit daunting at times - so much to learn and do!</p> <h2>What this means for OERu Partners</h2> <p>So, what should our OERu partners take from this description of the OERu's NGDLE? I think the most important thing is this: the status quo for higher learning institution IT conventions is neither the only way to do things, nor is it the best way.</p> <p>Because we are unbound to convention or historical decisions, we at the OERu are able to pioneer new approaches, from a "technology expert" perspective. We are driven by open principles and very tight resource constraints, but we also need fulfil our vision: to build a rich, fit-for-purpose infrastructure for learners and OER collaborators alike, which has the potential to scale to unprecedented global learner volumes.</p> <p>Implementing an open source end-to-end service gives the OERu a unique perspective and experience compared to organisations who only implement the occasional open source component in the midst of IT infrastructure dominated by proprietary software that is costly and extremely restrictive by comparison.</p> <p>We are also building (anonymous) monitoring systems into these services to ensure we can measure our success (without impinging on the privacy of our learners or collaborators). The insight we gain may well be of value to our partners.</p> <h3>Return on (open source technology) Investment</h3> <p>Return on Investment (ROI) can be achieved in a number of ways: investment can increase productivity and therefore value created (usually measured as <em>profit</em>) <em>or</em> it can be achieved by reducing costs. The best ROI achieves a combination of both.</p> <p>Our OERu partners can safely appraise our NGDLE and pick and choose from among the many (always improving) best-of-breed open source components that might fill an emerging need within their own institutional context. We can provide assistance and demonstration instances to help them initiate pilot implementations and work with local advocates and experts to provide ongoing support.</p> <p>Any partner interested in the capabilities of technologies we have included in our NGDLE has the potential to save their institutions many times the cost of their OERu partner membership if they adopt any one of them - along the lines of the Mautic case study above... And, of course, that savings increases with the number of technologies adopted. </p> <p> </p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <h2 class="comment-field__title">Blog comments</h2> <article data-comment-user-id="0" id="comment-27" about="/comment/27" typeof="schema:Comment" class="comment js-comment by-anonymous has-title clearfix"> <div class="comment__container"> <h3 property="schema:name" datatype="" class="comment__title"> <a href="/comment/27#comment-27" class="permalink" rel="bookmark" hreflang="en">Swapping out the last proprietary layer of the stack</a> <span class="comment__new marker marker--success hidden" data-comment-timestamp="1546981578"></span> </h3> <div class="comment__meta"> <div class="comment__submitted"> <span class="comment__author"><span rel="schema:author"><span lang="" typeof="schema:Person" property="schema:name" datatype="">Danyl Strype (not verified)</span></span> </span> <span class="comment__pubdate">Fri 21/12/2018 - 07:34 <span property="schema:dateCreated" content="2018-12-20T18:34:52+00:00" class="rdf-meta hidden"></span> </span> </div> </div> <div class="comment__content"> <div property="schema:text" class="clearfix text-formatted field field-comment--comment-body field-name-comment-body field-type-text-long field-label-hidden"> <div class="field__items"> <div property="schema:text" class="field__item"><p>This is a very impressive accomplishment Dave. The only thing that wrinkles my brow is that a significant chunk of OERU spending is going to the very corporate datafarms we&#039;re trying to replace (Microsoft and Amazon), by developing software and services that respects their users&#039; software freedom. If we are swapping from gratis services managed by the datafarms, to taking on management work ourselves while starting to pay them for hosting, is this actually progress? They still make all the money, but now we pay them for doing some of their work for them ;-P</p> <p>I&#039;m much happier to see OERU money going to Catalyst. As you know, having worked for them, they are not and have never been datafarmers. They have been involved in free code development and deployment in both the commercial and governmental sectors in Aotearoa, and huge supporters of the libre commons communities in the country (including sponsoring and organizing the NZ Open Source Awards). It would be great to read your next annual report, and see that all the OERU infrastructure hosted on the datafarms has been moved to companies like Catalyst. It&#039;s the final piece of the puzzle :)</p> </div> </div> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=27&amp;1=default&amp;2=en&amp;3=" token="otqyAq8w5Pd0NDlWnsqJTdnPUTbkZ3076qwkZ7AXGc8"></drupal-render-placeholder> </div> </div> </article> <div class="indented"><article data-comment-user-id="1" id="comment-34" about="/comment/34" typeof="schema:Comment" class="comment js-comment by-node-author has-title clearfix"> <div class="comment__container"> <h3 property="schema:name" datatype="" class="comment__title"> <a href="/comment/34#comment-34" class="permalink" rel="bookmark" hreflang="en">Fair comment, Danyl. Yes, we…</a> <span class="comment__new marker marker--success hidden" data-comment-timestamp="1546981964"></span> </h3> <div class="comment__meta comment__meta--has-user-picture"> <a href="/blog/1">View recent blog entries</a> <div class="comment__submitted"> <span class="comment__author"><span rel="schema:author"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> </span> <span class="comment__pubdate">Wed 09/01/2019 - 10:12 <span property="schema:dateCreated" content="2019-01-08T21:12:44+00:00" class="rdf-meta hidden"></span> </span> </div> </div> <div class="comment__content"> <p class="comment__parent visually-hidden">In reply to <a href="/comment/27#comment-27" class="permalink" rel="bookmark" hreflang="en">Swapping out the last proprietary layer of the stack</a> by <span lang="" typeof="schema:Person" property="schema:name" datatype="">Danyl Strype (not verified)</span></p> <div property="schema:text" class="clearfix text-formatted field field-comment--comment-body field-name-comment-body field-type-text-long field-label-hidden"> <div class="field__items"> <div property="schema:text" class="field__item"><p>Fair comment, Danyl. Yes, we agonised over the use of Microsoft's Azure cloud quite a bit... we decided in the end that, given our use of their service is within their USD5000 annual charitable organisation grant (so we're taxing their services rather than funding them) and our policy of actively avoiding the use of any service-specific proprietary tools (i.e. we treat all our hosting services as commodity services), we're effectively exploiting their efforts to lock organsations into their offering (and convert them to paying customers) and using them to promote their antithesis, namely fully open, community focused services... </p> <p>The use of Amazon is somewhat less contentious (given that Amazon hasn't historically been actively hostile to FOSS like Microsoft has been for most of its history), and although we're currently paying for their service (albeit not very much) we're similarly treating their services as commodity, so that <em>the instant</em> we're able to identify an alternative service that's fit for our purpose but is better aligned with our principle, we can move there toot-sweet! :)</p> <p>Keep on keeping us honest!</p> </div> </div> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=34&amp;1=default&amp;2=en&amp;3=" token="5Ndj6-NY2Rp4CIlcOJ9Py-2-MLvBNF34yZ1q6Vjwifg"></drupal-render-placeholder> </div> </div> </article> <div class="indented"><article data-comment-user-id="0" id="comment-48" about="/comment/48" typeof="schema:Comment" class="comment js-comment by-anonymous has-title clearfix"> <div class="comment__container"> <h3 property="schema:name" datatype="" class="comment__title"> <a href="/comment/48#comment-48" class="permalink" rel="bookmark" hreflang="en">Thanks for acknowledging the…</a> <span class="comment__new marker marker--success hidden" data-comment-timestamp="1557185946"></span> </h3> <div class="comment__meta"> <div class="comment__submitted"> <span class="comment__author"><span rel="schema:author"><span lang="" typeof="schema:Person" property="schema:name" datatype="">Danyl Strype (not verified)</span></span> </span> <span class="comment__pubdate">Thu 14/03/2019 - 19:16 <span property="schema:dateCreated" content="2019-03-14T06:16:27+00:00" class="rdf-meta hidden"></span> </span> </div> </div> <div class="comment__content"> <p class="comment__parent visually-hidden">In reply to <a href="/comment/34#comment-34" class="permalink" rel="bookmark" hreflang="en">Fair comment, Danyl. Yes, we…</a> by <span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></p> <div property="schema:text" class="clearfix text-formatted field field-comment--comment-body field-name-comment-body field-type-text-long field-label-hidden"> <div class="field__items"> <div property="schema:text" class="field__item"><p>Thanks for acknowledging the issues Dave, and great to be able to see the details of your reasoning. Can I suggest you include these details in future reports where these hosting choices are mentioned? Pleased to hear the OERu is not actually giving Microsoft any money :)</p> <p>&gt; Amazon hasn&#039;t historically been actively hostile to FOSS like Microsoft has been for most of its history</p> <p>True, but many of their business activities are bad for software freedom, and human freedom in general. The one that always sticks in my head is the centralized censorship power that allowed them to delete copies of George Orwell&#039;s &#039;1984&#039; from thousands of Amazon Swindle devices. A long list of their other bad behaviours can be found here, with links to sources:<br /> <a href="https://stallman.org/amazon.html">https://stallman.org/amazon.html</a></p> <p>&gt; the instant we&#039;re able to identify an alternative service that&#039;s fit for our purpose but is better aligned with our principle, we can move there toot-sweet! :)</p> <p>Good to hear. Could you perhaps write more about exactly what these hosts provide that make them difficult to find a replacement for? Such a description could make an excellent topic for a future blog post. It may be that someone who reads it is inspired to set up a suitable service, or can point you to one that already exists.</p> <p>Keep up the great work, both on the open tech stack itself and on the open documentation practice, which is equally important IMHO.</p> </div> </div> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=48&amp;1=default&amp;2=en&amp;3=" token="OrvEVqV-UezZv7CqEUFdTN7wkGEbOXOU7VDTkUNmtKE"></drupal-render-placeholder> </div> </div> </article> </div></div> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=23&amp;2=field_blog_comments&amp;3=comment" token="2zE34of-pPoUy4YnQR_I3RaAyQnePTxGQyL0a8q2djg"></drupal-render-placeholder> </div> </section> Tue, 30 Oct 2018 21:24:12 +0000 dave 23 at http://tech.oeru.org The OERu Blog Feed Finder http://tech.oeru.org/oeru-blog-feed-finder <span class="field field--name-title field--type-string field--label-hidden">The OERu Blog Feed Finder</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--lida101"> <span class="field__item-wrapper"><a href="/taxonomy/term/47" hreflang="en">lida101</a></span> </div> <div class="field__item field__item--wenotes"> <span class="field__item-wrapper"><a href="/taxonomy/term/41" hreflang="en">wenotes</a></span> </div> <div class="field__item field__item--blog-feed-finder"> <span class="field__item-wrapper"><a href="/taxonomy/term/53" hreflang="en">blog feed finder</a></span> </div> <div class="field__item field__item--free--open-source"> <span class="field__item-wrapper"><a href="/taxonomy/term/6" hreflang="en">free &amp; open source</a></span> </div> <div class="field__item field__item--rss"> <span class="field__item-wrapper"><a href="/taxonomy/term/54" hreflang="en">rss</a></span> </div> <div class="field__item field__item--atom"> <span class="field__item-wrapper"><a href="/taxonomy/term/55" hreflang="en">atom</a></span> </div> <div class="field__item field__item--json"> <span class="field__item-wrapper"><a href="/taxonomy/term/56" hreflang="en">json</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Mon 15/10/2018 - 11:40</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>One of the <a href="/many-simple-tools-loosely-coupled">distributed tools</a> that the <a href="https://oeru.org">OERu</a> makes available as part of its open source distributed digital learning environment is WEnotes. I've <a href="/wikieducator-notes-oerus-course-feed-aggregation-and-messaging-system">described what it is and how it works previously</a>. One of the most powerful distributed tools has also been the most complicated to achieve: the ability for learners to complete assignments and reflect on their learning journeys on their own blogs, tagging posts with relevant OERu course ids (e.g. <em>lida101</em> - see the <a href="https://course.oeru.org/lida101/interactions/course-feed/">Learning in a Digital Age 101 course feed</a> for example) and then having our blog feed scanner:</p> <ol><li>know to look at the learner's blog in a timely manner,</li> <li>identify the newly published appropriate blog post (i.e. with the relevant tag), and</li> <li>create a WEnotes message with a reference to the blog post</li> </ol><p>for the other learners participating in the course to see.</p> <p>There are a lot of moving parts in achieving this ambition, but the most unexpectedly difficult (from my perspective as a technologist) was solving the first problem: knowing how to find the learner blog feeds to monitor!</p> <h2>The mysterious blog feed</h2> <p>It turns out few people who use the web, even those who blog, are familiar with "feeds". A "<a href="https://en.wikipedia.org/wiki/Web_feed">feed</a>" is a web accessible file, adhering to a well-known and publicly defined "open standard" <em>machine readable </em>format, which summarises <em>and references</em> (links to), the content on a website, in this case a blog. Most blog platforms enable them right out-of-the-box, by default. This fact isn't widely appreciated, even among active bloggers!</p> <p>If you haven't seen a feed, here're some examples and an explanation of the two most common types: <a href="https://www.w3schools.com/XML/xml_rss.asp">RSS</a> (which stands for "Really Simple Syndication") and <a href="https://en.wikipedia.org/wiki/Atom_(Web_standard)">Atom</a>, both of which support "tagging" content.</p> <p>When we (here at the OER Foundation) first looked into offering the monitoring of learner (or educator) blogs for suitably tagged posts to include in course interaction feeds, I thought it would be a simple process of allowing learners to nominate a "Blog Feed URL" when they registered an account on our <a href="https://course.oeru.org" title="The OERu Course Site">Course Site</a>. I was quite mistaken.</p> <p>Over the course of six or so months after we made a "Blog Feed URL" field available in learner profiles, entered during the registration process, we had perhaps a hundred or so learners nominate a URL. (N.B. our model allows a learner to nominate a different blog feed URL for each course in which they're participating, allowing for the possibility that they might create a new blog site for each course. Naturally, they can nominate the <em>same</em> blog URL for multiple courses) Of those, perhaps one in 50 was an actual valid blog <em>feed</em> URL. Most were not even valid blog addresses. The most common entries were either facebook.com, google.com, a wikieducator.org user profile, or the course's own url - none of which are blogs, much less blog feeds. So this was useful market research: the market does not, in general, know what a blog is or what its feed might be, if it has one.</p> <p>Among other problems this creates is the fact that our automated feed monitoring code could not effectively sift the wheat from the chaff to find <em>valid</em> blog feeds to monitor. Clearly, a different approach was required.</p> <h2>Back to the drawing board</h2> <p>Turns out that if a learner<em> has a blog</em>, they generally know its web address. If provided clear instructions, they can fairly reliably go to it, and copy and paste the URL into a text field. So I decided to use this as a starting point for creating our "Blog Feed Finder" - a simple tool that helps us side step the issue of requiring a learner to find their blog feed URL - by finding it for them. Our approach was to provide a clear, well documented interface that embraces "<a href="http://www.instructionaldesign.org/theories/elaboration-theory/" title="A summary of elaboration theory">elaboration theory</a>", providing enough information at a glance to guide a learner, but offers further information to those who want to understand more.</p> <p>Here's what it looks like:</p> <figure role="group" class="caption caption-img"><img alt="The BFF with no blog URL entered yet. " data-entity-type="file" data-entity-uuid="b6e1d29c-ed8b-44bb-8842-e37d432f81e1" src="/sites/default/files/inline-images/bff_default_0.png" /><figcaption>Here's the default view of the BFF - you can enter your blog URL as per the instructions, or seek further instruction if you're not sure how to do that.</figcaption></figure><p>I'll copy in the URL of this very blog (and I won't even bother with the "https://" that appears at the start of it):</p> <figure role="group" class="caption caption-img"><img alt="BFF with block URL entered... " data-entity-type="file" data-entity-uuid="52be95bc-c609-4e8f-b966-04bbd4d8a2a1" src="/sites/default/files/inline-images/bff_with_blog_url_0.png" /><figcaption>The URL is pasted in. Now push the "Submit" button...</figcaption></figure><p>After a few seconds of behind-the-scenes whirring and grinding...</p> <figure role="group" class="caption caption-img"><img alt="BFF with an initial result." data-entity-type="file" data-entity-uuid="57caa975-4761-4369-bfba-71b2bde08802" src="/sites/default/files/inline-images/bff_with_result.png" /><figcaption>We have a result from the BFF. Note that the BFF also determines the type of feed, typically RSS or Atom. Some blog platforms offer both, in which case the BFF lets you choose which one to use.</figcaption></figure><p>If we want to know more about what the BFF had to do to find that blog feed, we can review its output, and even ask for more information by hovering over the info icons, fo example:</p> <figure role="group" class="caption caption-img"><img alt="Showing further elaboration of the blog feed finding process." data-entity-type="file" data-entity-uuid="da32b127-3e68-418a-8e1a-3ad8360fa218" src="/sites/default/files/inline-images/bff_with_result_and_elaboration_0.png" /><figcaption>Showing an elaborating pop-up, triggered by hovering over (or clicking on) an information icon.</figcaption></figure><p>And having found a feed, we can now update any of the courses for which we're registered to use that feed as the designated "feed to scan periodically" for blog posts tagged with the relevant course code (in brackets next to each course title, e.g. analyticsdev, lida101, mun).</p> <p>Here's what it looks like after I've replaced <em>all</em> of my blog feed associations with the new feed URL:</p> <figure role="group" class="caption caption-img"><img alt="After 'replacing' the feed URLs previously associated with my courses." data-entity-type="file" data-entity-uuid="a66ddf80-ec3b-46cf-b550-9dd4b4e1f83e" src="/sites/default/files/inline-images/bff_with_result_after_updating_all.png" /><figcaption>This is what the interface looks like after I've elected to "Replace" each of my blog feed URLs with the newly found URL.</figcaption></figure><p>And, what if, instead of tech.oeru.org, I'd mistakenly entered a URL like one of those I mentioned above? Here's how the BFF responds if someone enters, say, Facebook.com as their blog URL:</p> <figure role="group" class="caption caption-img"><img alt="BFF with the 'common accidental' blog URL message... " data-entity-type="file" data-entity-uuid="e699e2b2-9a83-4c58-bddf-a6c81ce0b1d3" src="/sites/default/files/inline-images/bff_with_common_accident_message_0.png" /><figcaption>This is what the BFF says to a learner if they mistakenly enter one of a handful of frequently entered non-blog URLs like facebook.com, google.com, or even course.oeru.org.</figcaption></figure><h2>Try it now, or adapt it! It's free (and open source)</h2> <p>Of course, you can <a href="https://course.oeru.org/blog-feed-finder" title="The OERu's Block Feed Finder. It's all open source.">try it yourself right now</a> without even needing to log in, although you get additional functionality if you're logged in an enrolled in one or more courses, like the ability to assign blog feed URLs to any courses in which you're enrolled.</p> <p>Even though we're only on our first iteration of it (any software developer will tell you: no software application is ever <em>finished</em>) our Blog Feed Finder (BFF for short) has also been <a href="http://cogdogblog.com/2018/06/better-magic-box/">assessed by others in the online learning realm</a>, who have given it a good going over and their seal of approval.</p> <p>If anyone is curious about how it works, the <a href="https://github.com/oeru/blog-feed-finder">entire source code is available</a> under the terms of the <a href="https://www.gnu.org/licenses/agpl-3.0.en.html">GNU Affero General Public License</a> (same as what's used by the Linux kernel, WordPress, Drupal, MediaWiki, and thousands of other well know free and open source projects) to make its review and reuse convenient and reliable!</p> <h2>OERu Blog Scanner is Go!</h2> <p>As of a few weeks ago, following the introduction of the BFF (and a bit of manual tidy-up of previously entered blog feed URL where OERu administrators used the BFF to find legitimate feed URLs where-ever possible) we now have a working scan which periodically checks all registered learner blog feeds and, if a suitably tagged post is found on any of them, incorporates a link to the post in the relevant course feed!</p> <p>The <a href="https://course.oeru.org/lida101/interactions/course-feed/">course interaction feed</a> (also linked above) includes some posts with a "blog" designator, which indicates they are references to blog posts which have been scanned by our WEnotes blog scanner (which is a different piece of software).</p> <h2>Loose Ends</h2> <p>As with any first iteration software release, there are quite a few known issues (a whole itemised, prioritised list, actually) we'd like to address for the next release. The first among them in the BFF's case is making the interface properly "responsive" (i.e. friendly for mobile device users)... At present, it doesn't do very well at that.</p> <p>If you have a go with the BFF and run into trouble related to the software itself, we encourage you to let us know in <a href="https://github.com/oeru/blog-feed-finder/issues">the project's Issue Queue</a>. </p> <p>Final note: we will be moving our source code repositories to a properly open source code repository (our own <a href="https://gitlab.com" title="Gitlab is an open source alternative to Github, which is not open source.">Gitlab</a> instance) in the coming weeks, so you may find you're being redirected from Github to a different site. Please don't be alarmed! It's by design.</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=22&amp;2=field_blog_comments&amp;3=comment" token="DjJy08iCX70uHh392eLxPjDOG3f9SoByCElo9m3CGxs"></drupal-render-placeholder> </div> </section> Sun, 14 Oct 2018 22:40:06 +0000 dave 22 at http://tech.oeru.org What is the OERu? - Pizza Thursday talk for Catalyst Christchurch http://tech.oeru.org/what-oeru-pizza-thursday-talk-catalyst-christchurch <span class="field field--name-title field--type-string field--label-hidden">What is the OERu? - Pizza Thursday talk for Catalyst Christchurch</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--lida101"> <span class="field__item-wrapper"><a href="/taxonomy/term/47" hreflang="en">lida101</a></span> </div> <div class="field__item field__item--oeru"> <span class="field__item-wrapper"><a href="/taxonomy/term/46" hreflang="en">oeru</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Thu 23/08/2018 - 09:31</span> <div class="float-none field field-node--field-image field-name-field-image field-type-image field-label-hidden has-single"> <figure class="field-type-image__figure image-count-1 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-08/OERu_technology_wheel.png?itok=SujshCxP" title="The wheel of interactive technologies OERu learners can work with that will be recruited into their course feed." data-colorbox-gallery="gallery-field_image-0Nhui6ShQUs" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;The wheel of interactive technologies OERu learners can work with that will be recruited into their course feed.&quot;}"><img src="/sites/default/files/styles/medium/public/2018-08/OERu_technology_wheel.png?itok=c-VCQ8aT" width="220" height="152" alt="The wheel of interactive technologies OERu learners can work with that will be recruited into their course feed." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> </div> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>I've been asked to speak about the OERu to my former colleagues at <a href="https://catalyst.net.nz" title="NZ's premier open source development and cloud hosting company">Catalyst IT</a>'s Christchurch office as part of their monthly "Pizza Thursday" talks. Amusingly, this talk will be on a Friday, and lunch will be curries rather than pizza :)</p> <h2>What is the OERu?</h2> <p>Last week, an <a href="https://opensource.com/article/18/8/oeru-courses">article and interview discussing exactly that</a> was published on <a href="https://opensource.com">OpenSource.com</a> which examined both the OERu and its relationship to free and open source software, which is core to the organisation's ethos. We're a "radically open" organisation... our terms of reference state that we</p> <ul><li>use exclusively openly licensed educational resources (OERs), i.e. written materials and digital artefacts that conform with the "<a href="https://freedomdefined.org/Definition">free cultural works</a>" criteria.</li> <li>we only use free and open source software for <em>everything</em> where-ever possible</li> <li>all our planning processes are conducted openly and with external participation encouraged (where-ever possible, they're streamed live and <a href="https://www.youtube.com/channel/UCCMLj7wLhiIYq_CH84D_TPg">recorded for posterity</a>, so others can see how we arrived at strategic decisions)</li> </ul><p>I work in the role of "<a href="/insight-what-does-open-source-technologist-oer-foundation-do">Open Source Technologist</a>" for the OER Foundation (a Dunedin-based charitable foundation, which in turn, is owned and formally hosted by the Otago Polytech, my official employer), who facilitate and enable the OER universitas (OERu), the global network of <a href="https://oeru.org/oeru-partners/">higher education institutions and related organsations</a> (like donors and sponsors), who are working together to meet the OERu's strategic goals. </p> <p>This is how we explain the OERu to prospective partners:</p> <p><iframe allow="autoplay; encrypted-media" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/Ia3g2qh-qjQ" width="560"></iframe></p> <p>Of course, as an educational provider, the OERu cannot make a lot of progress without "learners" (they're not "students" because that implies a contractual relationship with an institution - people choosing to learn with us can do so anonymously, without any formal relationship, so we've adopted the term "learners"). This is what the OERu looks like from a learner's perspective:<br /><iframe allow="autoplay; encrypted-media" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/F4q3CjOQARI" width="560"></iframe></p> <p>Earlier this year, we launched our "MVP", a "1st year of study" - a sequence of courses a learner can take (equivalent to a first year of undergraduate study) which, if they choose to be assessed (and meet the assessment criteria) can lead to a formal exit qualification.</p> <p>Our current "flagship" course is <a href="https://oeru.org/learning-in-a-digital-age/">Learning in a Digital Age</a>, a series of four "microcourses" which can be undertaking in a sequence or individually:</p> <ol><li><a href="https://oeru.org/oeru-partners/otago-polytechnic/digital-literacies-for-online-learning/">Digital literacies for online learning</a> (LiDA101)</li> <li><a href="https://oeru.org/oeru-partners/otago-polytechnic/digital-citizenship/">Digital citizenship</a> (LiDA102)</li> <li><a href="https://oeru.org/oeru-partners/otago-polytechnic/open-education-copyright-and-open-licensing-in-a-digital-world/">Open education, copyright and open licensing in a digital world</a> (LiDA103)</li> <li><a href="https://oeru.org/oeru-partners/otago-polytechnic/critical-media-literacies-and-associated-digital-skills/">Critical media literacies and associated digital skills</a> (LiDA104)</li> </ol><p>So, as a learner, what does a course look like? Let's have a look at <a href="https://course.oeru.org/lida101" title="Digital literacies for online learning">LiDA101</a>...</p> <p>What you see is it's a WordPress site holding all the course materials. These materials are not edited or altered by course designers within WordPress. Instead, they're created as a course outline on our collaborative OER assembly platform, <a href="https://wikieducator.org">WikiEducator</a>, an instance of the venerable MediaWiki platform (yes, it's high time for a facelift - never fear, in the works). Here's the <a href="https://wikieducator.org/Learning_in_a_digital_age/_Outline_LiDA101">course outline</a> used to build the LiDA101 course.</p> <p>You can see that the course outline is just a selection of content from elsewhere on the site, including reference materials, media objects (videos, images, audio), quizzes and assignments, and assessment rubrics. We use MediaWiki as the collaboration platform thanks to the (relatively) user-friendly and flexible version control of content it provides. (n.b. the WikiEducator site, one of the OER Foundation's main initiatives, the other being the OERu, is very heavily used by educators around the world, and <a href="https://www.alexa.com/siteinfo/wikieducator.org">hovers near the 100k sites on the web</a>).</p> <p>You'll notice the "Snapshot" button at the top of the outline - behind that button is a one line specification for a target WordPress site - clicking that button allows someone with "administrator" permissions on that site (or "sub site") to automatically push the course materials contained in that outline to the chosen WordPress site - it will be formatted as you see, with the selected branding (e.g. for our partner institutions), <em>particularly designed for use with mobile devices</em> (although that could use some refinement) which is our largest potential audience.</p> <p>Learners can access all course materials anonymously, however, if they want to <a href="https://wikieducator.org/File:OERu_technology_wheel.svg">interact with one another, and have their participation recorded</a>, they must create an account on the site.</p> <h2>OERu's FOSS Tech Stack</h2> <p>The technology stack on which the OERu is built has a few parts. We have created what we refer to as a "next generation digital learning environment" made up of a number of carefully selected free and open source tools all "<a href="/many-simple-tools-loosely-coupled">loosely coupled</a>" together (the "UNIX philosophy"), with some strategic custom glue...</p> <ul><li>educator collaboration tools (Discourse, Rocket.Chat, OnlineGroups, Jitsi Meet)</li> <li>educator collaborative course assembly tools (MediaWiki, NextCloud/CollaboraOffice, Kanboard)</li> <li>market-automation, learner instruction email tools, market research (Mautic, LimeSurvey)</li> <li>learner course sites (WordPress)</li> <li>learner assessment tools (Moodle)</li> <li>learner interaction (Mastodon, Discourse, Semantic Scuttle, Hypothes.is, other social media, learner blogs)</li> <li>learner <a href="https://course.oeru.org/lida101/interactions/course-feed/" title="An example of a course feed">interaction feed</a> (<a href="/wikieducator-notes-oerus-course-feed-aggregation-and-messaging-system">WEnotes</a>: Node.Js + Python + CouchDB + MediaWiki extensions + WordPress plugins)</li> <li>organisational info platforms (MediaWiki, Silverstripe, Drupal, Grav)</li> <li>reporting/promotional presentations (Reveal.JS)</li> <li>analytics (Matomo)</li> <li>infrastructure (Ubuntu/Debian Linux VMs/Instances, Git, Docker, Docker-Compose)</li> </ul><p>We use this website (tech.oeru.org) as a place to explain how we do what we do, and why. We include how-tos for many of the technologies we use in hopes that more organisations will start to use them inspired by our example - our underlying "enlightened self-interest" culture sees that we all fare better when we have an interest in making these tools the best they can be...</p> <h2>Why I do this</h2> <p>In my career as a technologist, I've been very aware of (and involved in activism for improving) the many problems faced by our society (and others around the world). There's one compelling, effective, and unequivocally <em>good</em> thing I've identified as a key component to all the solutions to our social ills:<strong> education</strong>. By that, I mean education (which depends on aptitude, motivation, and opportunity) from literacy to higher learning, across all of social strata, particularly for girls and women.</p> <p>As we in the OERu network build a platform for low cost higher education, we hope to create something that <em>anyone</em> can build upon, and scale up to meet the latent demand for education (globally, fewer than 10% of people ever have the opportunity to complete a tertiary qualification), providing people the <em>realistic</em> opportunity to reach their individual educational potentials. That opportunity depends only on people having access to suitable technology and Internet connectivity, both of which are spreading across the world faster than anyone could ever have imagined.</p> <p>The fact that I can do something I enjoy, have real passion for, and derive satisfaction from because it's the "right thing to do for our world" - also known as "enlightened self-interest" - with people I respect and admire (and get on with really well), means that I have found my calling and consider myself incredibly fortunate.</p> <h2>Demos</h2> <p>Time permitting, I'd be keen to show you a few of the tech things:</p> <ol><li>Demo of the WEnotes <a href="https://course.oeru.org/lida101/interactions/course-feed/">Course Feed </a>(git repos: <a href="https://bitbucket.org/wikieducator/wenotes-tools/src" title="The tools">tools</a>, <a href="https://github.com/oeru/wenotes" title="WordPress plugin and MediaWiki extension">plugins</a>, <a href="https://github.com/oeru/wenotes-docker">deployment</a>) <ol><li>The <a href="https://wikieducator.org/WENotesCompleteFeed">full noise</a></li> <li>A look at Hypothesis annotations</li> <li>A look at Mastodon</li> </ol></li> <li>A look at our <a href="https://wpms.oeru.org/blog-feed-finder">Blog Feed Finder</a> (git <a href="https://github.com/oeru/blog-feed-finder">repo</a>) which helps learners who want to submit their personal blog for scanning (to include suitably "tagged" posts in their course feed) actually find the feed and associate it with their course so our WEnotes scanner periodically checks it for new posts of interest.</li> <li>A tour of our <a href="https://mautic.oeru.org">Mautic</a> and our approach to keeping learners and partners in the loop!</li> </ol><p>I reckon many of these tools are the sort of thing Catalyst could offer as managed services commercially.</p> <p>N.B. we're not happy with Microsoft acquiring Github, and will be moving our repos from Github (and Bitbucket, too, while we're at it, because we're not overly excited to be using proprietary repos in general) to our own free and open source Git hosting as soon as possible. We will leave pointers to the new locations for these (and our many other) repos when they are created.</p> <p> </p> <p> </p> <p> </p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=21&amp;2=field_blog_comments&amp;3=comment" token="99KWcafILHKzzimelPVVtuDa5XZ3jjZyJ2-reyAbBAI"></drupal-render-placeholder> </div> </section> Wed, 22 Aug 2018 21:31:44 +0000 dave 21 at http://tech.oeru.org Creating Simple, Semantic HTML Markup from a Google Doc http://tech.oeru.org/creating-simple-semantic-html-markup-google-doc <span class="field field--name-title field--type-string field--label-hidden">Creating Simple, Semantic HTML Markup from a Google Doc</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Wed 23/05/2018 - 12:55</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>Google Docs are great for allowing people to collaboratively build a document, with the ability for people to suggest (and discuss) changes and view revisions and a variety of other useful behaviours. At present, I'm sad to admit, <a href="/installing-nextcloud-and-collabora-office-online-docker-ubuntu-1604">CollaboraOffice</a> isn't quite in the same level (although it's catching up quickly!).</p> <h2>The Problem</h2> <p>A big problem with Google Docs (shared, incidentally, with Microsoft Word and even LibreOffice/CollaboraOffice) is that it's almost impossible to convert a collaboratively created document into a form suitable for publishing on the web.</p> <p>Yes, you can go to File-&gt;Download as...-&gt;Web Page (html, zipped)... but that results in an HTML document that is chock full of badness. Google Docs (and the other word processing packages) all assume you want your document to render on the web <em>exactly</em> like it does in their app. In my experience, is<strong> impossible to get clean, semantic HTML out of any word processing platform</strong> without all the nasty in-line styling and formatting kruft that these applications assume you want.</p> <p>Anyone (like me) who's had to work with non-technical people wanting to manage their own websites, via a content management system (CMS) like Wordpress or Drupal or any one of hundreds of others, can attest to the fact that the average person's first tool for creating content is always their word processor. This, of course, is probably the least appropriate tool, because it</p> <ol><li>creates an expectation of  control and formatting that is invalid on the web, and</li> <li>tries to control the content that is copied and pasted into the CMS' editing interface, which is entirely counter-productive.</li> </ol><p>All you really want is very simple, semantically marked-up content, <em>without any styling whatsoever</em>. The styling for CMS content is typically applied by the theme, rather than the editor. The purpose behind that is to ensure that all the site's content is <em>consisten</em>t in its look-and-feel and <em>the whole lot can be changed with simple tweaks to that theme. </em>Problem is, most people neither understand that nor care - in fact, few people will even know what I mean by "semantically marked-up": simply put, it's the idea that the content is simply marked up to show what type of content it is, e.g. a title, a paragraph, a list, a table, something requiring emphasis (usually achieved by <em>italics</em>) or strong presentation (usually achieved by <strong>bold</strong>).</p> <h2>An Illustrated Example</h2> <p>Here's an example of a document</p> <figure role="group" class="caption caption-img align-center"><img alt="A screenshot of a Google Document used in this example." data-entity-type="file" data-entity-uuid="0a945b14-ffa1-45ae-aea7-b5685e3d6a36" src="/sites/default/files/inline-images/ScreenshotGoogleDocsSample.png" /><figcaption>Screenshot of a Google Doc containing some text with basic formatting.</figcaption></figure><p>Here's the simple, clean semantic mark-up we <em>want</em> to represent this content:</p> <p><blockcode style="html"><code>&lt;h1&gt;Privacy notice&lt;/h1&gt;<br /> &lt;p&gt;The Open Education Resource universitas (OERu) privacy notice provides a simple and concise summary to explain our treatment of personal information from OERu learners and users of the websites hosted by the OER Foundation (OERF).&lt;/p&gt;<br /> &lt;p&gt;This privacy notice complies with our Privacy Policy and Terms of Service Policy and applies to you when you decide to use our services.&lt;/p&gt;<br /> &lt;p&gt;We collect information when users visit websites hosted by the OERF. This includes IP addresses which are unique numbers that identify specific internet connections, and sometimes specific computers and devices that visit our sites.&lt;/p&gt;<br /> &lt;h2&gt;Definitions&lt;/h2&gt;<br /> &lt;p&gt;In the context of this policy&lt;/p&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;&lt;p&gt;Personal data refers to information related to an identified or identifiable natural person used for data processing by the OERF.&lt;/p&gt;&lt;/li&gt;<br /> &lt;/ul&gt; </code></blockcode></p> <p>Here's the mark-up we <em>get</em> if we copy and paste from Google Docs (you'll get similar results copy-and-pasting from any word processor):</p> <p><blockcode style="html"><code>&lt;p&gt;&amp;nbsp;&lt;/p&gt;<br /><br /> &lt;h1 dir="ltr" style="line-height:1.38;margin-top:20pt;margin-bottom:6pt;"&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:20pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;Privacy notice&lt;/span&gt;&lt;/b&gt;&lt;/h1&gt;<br /><br /> &lt;p&gt;&amp;nbsp;&lt;/p&gt;<br /><br /> &lt;p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;The Open Education Resource universitas (OERu) privacy notice provides a simple and concise summary to explain our treatment of personal information from OERu learners and users of the websites hosted by the OER Foundation (OERF).&lt;/span&gt;&lt;/b&gt;&lt;/p&gt;<br /><br /> &lt;p&gt;&amp;nbsp;&lt;/p&gt;<br /><br /> &lt;p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;This privacy notice complies with our Privacy Policy and Terms of Service Policy and applies to you when you decide to use our services. &lt;/span&gt;&lt;/b&gt;&lt;/p&gt;<br /><br /> &lt;p&gt;&amp;nbsp;&lt;/p&gt;<br /><br /> &lt;p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;We collect information when users visit websites hosted by the OERF. This includes IP addresses which are unique numbers that identify specific internet connections, and sometimes specific computers and devices that visit our sites.&lt;/span&gt;&lt;/b&gt;&lt;/p&gt;<br /><br /> &lt;h2 dir="ltr" style="line-height:1.38;margin-top:18pt;margin-bottom:6pt;"&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:16pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;Definitions&lt;/span&gt;&lt;/b&gt;&lt;/h2&gt;<br /><br /> &lt;p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;In the context of this policy &lt;/span&gt;&lt;/b&gt;&lt;/p&gt;<br /><br /> &lt;p&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;Personal data&lt;/span&gt;&lt;span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt; refers to information related to an identified or identifiable natural person used for data processing by the OERF. &lt;/span&gt;&lt;/b&gt;&lt;/p&gt;<br /><br /> &lt;p&gt;&amp;nbsp;&lt;/p&gt;<br /><br /> &lt;p&gt;&amp;nbsp;&lt;/p&gt;<br /><br /> &lt;p&gt;&amp;nbsp;&lt;/p&gt;</code></blockcode></p> <p>As you can see, it's fairly impenetrable (it's not written for human's eyes :) ). It doesn't even use proper semantic tags to represent the content. Instead it uses "ad hoc" styling to achieve a superficial resemblance to what we want to see, and in doing so, it also generates huge amounts of totally awful mark-up.</p> <h2>The (Free and Open Source) Solution</h2> <p>Luckily, after a lot of dives down rabbit holes, I have emerged with a free and open source software (FOSS) solution. It involves an incredible software tool called "<a href="https://pandoc.org" title="The Pandoc Free and Open Source tool">Pandoc</a>" that can be installed on any Windows, MacOS, or Linux computer. Because I use FOSS where ever possible, I'll describe how I achieved a solution on Linux. </p> <p>In addition to providing clean semantic HTML, preserving links and structure (including tables), Pandoc can be asked to build a "Table of Contents" for the HTML, too, based on titles (H1, H2, etc.) to the level desired.</p> <p>The steps:</p> <ol><li>download a "Web Page (html, zipped)" version of the Google Document and unzip it in to a file, in my case,<code> PrivacyNotice.html</code></li> <li>install Pandoc on my computer:<code> <code>sudo apt install pandoc</code></code></li> <li>in the directory in which I have<code> PrivacyNotice.html,</code> I ran<code><br /><br /><code>pandoc -r html -w docbook --email-obfuscation=none -S -s PrivacyNotice.html | pandoc -r docbook -w html --toc --toc-depth=2 --email-obfuscation=none -S -s -o PrivacyNotice-semantic.html -</code></code><br /><br /> Which produces a semantically marked up version of the file PrivacyNotice-semantic.html, and, in this case, with a linked Table of Contents automatically created, using headings up to a depth of "2" (i.e. H1 and H2) as the headings.<br /><br /> If you prefer not to have the Table of Contents, run this instead:<code><br /><br /><code>pandoc -r html -w docbook --email-obfuscation=none -S -s PrivacyNotice.html | pandoc -r docbook -w html --email-obfuscation=none -S -s -o PrivacyNotice-semantic.html -</code> </code><br /><br /> Note: the trailing "-" is important (it's a way of telling the script to take as input the output of the script described prior to the "pipe" character, "|")!</li> </ol><p>What this Pandoc process does, is to convert the nasty HTML provided by the Google Doc "Web Page" export into a different (and very semantically structured) format called DocBook, which strips out all the unnecessary styling. We then convert it back to HTML, preserving the semantic structure, and removing the stuff we don't want.</p> <p>To simplify matters, you can also write a script or shell alias which achieves the same thing in a single command. For example, here's the alias I created in my ~/.bashrc (the hidden bash shell configuration file in my home directory, "~"):</p> <p><blockcode><code><code>alias tosemtoc='pandoc -r html -w docbook --email-obfuscation=none -S -s $1 | pandoc -r docbook -w html --toc --toc-depth=2 --email-obfuscation=none -S -s -o $1 -'</code></code></blockcode></p> <p><blockcode><br /><code><code>alias tosem='pandoc -r html -w docbook --email-obfuscation=none -S -s $1 | pandoc -r docbook -w html --email-obfuscation=none -S -s -o $1 -'</code></code></blockcode></p> <p>(if you add this to your .bashrc, you can make your bash session aware of it either by logging out or in again, or by running <code>. ~/.bashrc</code> at your command prompt)</p> <p>With these two lines I can achieve the same thing as the above complex command like this (the semantic HTML output will be put into the same file I started with in this case, rather than a separate file - no big loss on that Google Doc-produced HTML travesty!):</p> <p>With Table of Contents (down to H2): <code>tosemtoc PrivacyNotice.html</code></p> <p>Without the Table of Contents: <code>tosem PrivacyNotice.html</code></p> <p>Pretty straightforward. And it takes less than a second to run. Note that the result is a "well formed" complete HTML document, so you can view it happily in your browser, e.g. type <code>file:///path/to/your/PrivacyNotice.html</code> file into your browser's location bar as a URL.</p> <p>If you want to see the final result of this, have a look at our new GDPR-aware <a href="https://oeru.org/privacy-notice" title="The OERu Privacy Notice">Privacy Notice</a>!</p> <p>Hope this saves someone a lot of frustration and unnecessary remedial mark-up editing!</p> <p><code> </code></p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=20&amp;2=field_blog_comments&amp;3=comment" token="Tk3zaP2726_ZI8MqlLbML0HqaI7MDjD7WmCCTtyS-RM"></drupal-render-placeholder> </div> </section> Wed, 23 May 2018 00:55:19 +0000 dave 20 at http://tech.oeru.org Insight: what does the Open Source Technologist at the OER Foundation do? http://tech.oeru.org/insight-what-does-open-source-technologist-oer-foundation-do <span class="field field--name-title field--type-string field--label-hidden">Insight: what does the Open Source Technologist at the OER Foundation do?</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Fri 18/05/2018 - 13:17</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>A few weeks ago, I got an unexpected (but very flattering!) request from a friend and colleague's son, Felix. He wanted to interview me about my work (at 11, he thinks he might want to become a software developer, too, when he's older) as material he could present at an up-coming <a href="https://en.wikipedia.org/wiki/PechaKucha">PechaKucha</a> session at his school... Wow, we didn't do PechaKucha when I was a kid :)</p> <p>I agreed to respond, and, due to logistical challenges, that response took the form of an email answer his questions. After sending it to him, it occurred to me that it might be of interest to some folks in the OERu community and beyond, so I asked if he'd be ok with me publishing the results under a CC-By license. He was agreed, and so here it is (note, I've made a few minor edits and additions to my original response).</p> <dl><dt>1. What do you do in an average day at work?</dt> <dd> <p>My days are generally very diverse, but I can give you an idea of a typical "business as usual" day... (as opposed to, say, when I'm overseas at one of our very enjoyable international OERu partner meetings).</p> <p>Of late, my usual routine Usually, I make myself a coffee and go up to the garage loft in my house. I sit down at my computer and log in, immediately checking my email and various chat channels to see if there're any urgent problems to look into (my work is global, so there's always someone using the systems I manage).</p> <p>If there something urgent, like a system not running the way it's meant to, I get right onto making it right - our systems are meant to serve hundreds (and eventually tens of thousands or more!) learners daily, so they need to be reliable and responsive.</p> <p>On Mondays, I "go around the traps", applying updates to servers that I'm responsible for in Germany, the US, NZ, Australia, and Singapore - they support a bunch of websites and services that are part of our "Digital Learning Environment". I also maintain some web systems for the NZ Open Source Society (which I sometimes include in "the traps" - my job title with the OER Foundation who employ me, is "Open Source Technologist").</p> <p>In amongst the updates, I sometimes do some research on new technologies I need to deploy as part of my "continual improvement" process (I can do this without needing to concentrate intensely, so I can do it in short bursts well suited to the start and stop nature of the server update process) - finding ways to make it easier for me, as one half of a 2-person team trying to build a global service, to manage all the many systems I'm responsible for. Today, for instance, I'm looking at setting up an open source service monitoring system which will alert me if any of the systems I'm interested in have problems, go do, or even if they're slow... I'm also interested in monitoring whether any of them are running short of hard disk space, as recovering from a full hard disk is often very painful and messy, and can result in both long downtimes and lost user data (something we work <em>very</em> hard to avoid).</p> <p>Most days I also do quite a bit of software development. My role includes the responsibility for building strategic software capabilities. Today that will include working on a plugin for the <a href="https://wordpress.org">WordPress</a> blog platform (like <a href="https://drupal.org">Drupal</a>, but somewhat simpler, but also open source) that we use to provide our course materials to learners. When a learner logs in, and asks to take part in a particular course, the plugin I've written, automatically registers their details with a separate system we use, <a href="/installing-mautic-php7-fpm-docker-nginx-and-mariadb-ubuntu-1604">Mautic</a>, which is a "marketing automation" platform.</p> <p>Many people use Mautic to "spam" people or send out newsletters to people who subscribe to them via a website, for instance. We use it differently: we use it to automate the process of sending out hundreds of carefully timed emails to people taking each of our courses, so that they know when to start working on things, what they need to do, when their projects and assignments are due, and how to go about getting assessed after they finish a course (they need to be assessed to get formal academic credit for our courses).</p> <p>Using Mautic means that we can set up the rules and email templates for each course once, and then they're available any time we run a course with a cohort (a group of learners are going through the course together, who could be anywhere in the world - our last course had 310 learners from 58 countries!), or even for people who decide to take our courses via "independent study" in which case they can set their own schedule.</p> <p>The other thing I do quite a lot is to work out better ways to host the various services we offer - because the open source technologies we build our services on are constantly improving, there's a steady stream of new approaches to doing things. I have to use my judgment and experience to decide which improvements are worth making (because they all take my time, which is limited to 8 hrs a day), and which probably aren't worthwhile. I recently changed the way we host our main WordPress "multisite", <a href="https://course.oeru.org">https://course.oeru.org</a>, so that it's built as a collection of Docker containers, and I was able to improve the number of simultaneous learners we could support by 10 times without increasing our costs...</p> <p>Then, just about every afternoon, I have a video chat with my colleague, Wayne Mackintosh, who's based in Mosgiel (a town south of Dunedin), and is the only other employee of the OER Foundation (he's also the founder of it, and the main brains behind the operation). He's a keen supporter of open stuff - particularly open source software and open educational resources (the learning materials we make available to our learners as courses), which are the educational analogue to open source software. Wayne's main focus is on academic things - like coordinating educators at our many partner institutions globally to help contribute towards our open educational resources, to get their academic boards to push through accreditation processes for our courses (so that their institutions formally recognise our courses as being of a suitable quality for them to accept as if they were their own). He's got the really hard job, because it's more to do with people than technology. And, despite claiming he's not at every turn, he's remarkably switched on with the technology, too.</p> <p>We report on the things we've done that day, catch up on other news and developments, talk through problems we're working on, and make plans and prioritise things for the coming days and weeks.</p> <p>From time to time, I also write up "how tos" and explanations of how we use open source technologies at the OERu (the OERu - <a href="https://oeru.org">https://oeru.org</a> is the "network" made up of the OER Foundation and our higher education partners) - you can see it at <a href="https://tech.oeru.org">https://tech.oeru.org</a> - the introduction provides an overview of the mix of technologies we use if you're curious.</p> </dd> <dt>2. What is your favourite thing about being a software developer?</dt> <dd> <p>I love that I have a steady stream of interesting problems to solve, and that, when I solve them, I'm potentially making life better for hundreds or thousands of people who, in this case, might not have had the opportunity to reach their educational potential before.</p> <p>Plus, I really appreciate the fact that my current role allows me to learn new things constantly, and to actively engage with the communities of people who write the software that we use to make our learning materials available. For example, I frequently end up making minor improvements to software, like Mautic, or WordPress, which I then make available to other people (at places like this: <a href="https://git.oeru.org/explore/projects">https://git.oeru.org/explore/projects</a> or <a href="https://git.nzoss.org.nz/lightweight">https://git.nzoss.org.nz/lightweight</a>) as a way of contributing back.</p> <p>The fact that I can use my skills and experience do what I love, to benefit the world - people I've never even met - is very satisfying and fulfilling, and I consider myself very lucky.</p> </dd> <dt>3. What is the hardest part about being a software developer?</dt> <dd> <p>There're plenty of stressful things in managing systems that people use - for example, if there's a bug that means parts of the system (or all of it) don't work, I have to really scramble to make things right. I also have to to my very best to protect people's data! Very rarely (thankfully) something goes wrong, and people lose data (sometimes things they've invested a lot of time or energy in) and that's really tough.</p> <p>There're also "good-hard" things about developing software - making complex things easier for others to understand... understanding them myself... solving problems by doing a lot of detective work on the Internet - lots of times it requires hours of focused concentration, and really thinking hard... but if you never drop the ball, eventually you solve the problem, and that feels <em>great</em>.</p> </dd> <dt>4. What sort of training and education do you need to do before becoming a software developer?</dt> <dd> <p>Well, ultimately, as a software developer, you're a people interpreter: you're someone who learns what others want to do, how they want to do it, and then you work out how to "codify" that in software. This takes a LOT of time to do it properly. If you don't, you're likely to end up writing software that solves a problem... but one that's often quite different from what people want solved.  So learning about how people do things is very useful... not sure what formal education you can get here - good software developers are keen observers of human behaviour.</p> <p>Once you know roughly where you're going, software development boils down to a lot of problem solving and "micro-decision-making". You have to be the sort of person who, once you commit to solving a problem, you don't stop. Not sure you can learn that - it's just part of some people's nature...</p> <p>As far as formal education goes, I generally recommend this: <strong>get all the education you possibly can</strong> - in maths, science, and computing, but <em>also in other fields</em>. I think one of the best things I ever did was to do a "liberal arts" degree at university - I was privileged that my family and I were able to afford to do it (it was <em>very</em> expensive). I got to study many other fields besides, especially English. Being able to express myself in writing (be it code or English :) ) confidently is perhaps the best thing I learned. I'd also recommend learning other (human) languages!</p> </dd> <dt>5. What do you find least enjoyable about being a software developer?</dt> <dd> <p>There's always a bit of drudgery in every job. There're routine things that are a bit boring (the great thing about being a software developer is that you have the power to <em>automate</em> the boring stuff if it becomes a real drag - here's a <a href="/creating-simple-semantic-html-markup-google-doc">recent example</a>... ) - and there's also the frustration of having to work with (and keep happy) people who really <em>don't</em> understand technology at all, and see it as a threat.</p> <p>I think people who have a deep understanding of both how people work, <em>and</em> how software works have the best prospects in the world today, and I think that'll continue to be the case into the future.</p> </dd> </dl></div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <h2 class="comment-field__title">Blog comments</h2> <article data-comment-user-id="0" id="comment-7" about="/comment/7" typeof="schema:Comment" class="comment js-comment by-anonymous has-title clearfix"> <div class="comment__container"> <h3 property="schema:name" datatype="" class="comment__title"> <a href="/comment/7#comment-7" class="permalink" rel="bookmark" hreflang="en">Why GH not GitLab?</a> <span class="comment__new marker marker--success hidden" data-comment-timestamp="1528242423"></span> </h3> <div class="comment__meta"> <div class="comment__submitted"> <span class="comment__author"><span rel="schema:author"><span lang="" typeof="schema:Person" property="schema:name" datatype="">Danyl Strype (not verified)</span></span> </span> <span class="comment__pubdate">Thu 31/05/2018 - 01:18 <span property="schema:dateCreated" content="2018-05-30T13:18:12+00:00" class="rdf-meta hidden"></span> </span> </div> </div> <div class="comment__content"> <div property="schema:text" class="clearfix text-formatted field field-comment--comment-body field-name-comment-body field-type-text-long field-label-hidden"> <div class="field__items"> <div property="schema:text" class="field__item"><p>Hi Dave, this is a great write-up, and hearing that even a pro like you sometimes loses people&#039;s data makes me feel much less existentially terrified about the idea of going pro as a sysadmin or developer, rather than just paddling around the edges of it. One question, why are you using GH for your code, rather than a self-hosted, free code replacement like GitLab? I know that you run a GL instance for NZOSS, so I&#039;m curious about why you the links you&#039;ve given are to GH.</p> </div> </div> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=7&amp;1=default&amp;2=en&amp;3=" token="gSITky6KmlYh8k3gzKegeQw5Ab0eMLcMkqN0j1ewtsc"></drupal-render-placeholder> </div> </div> </article> <div class="indented"><article data-comment-user-id="1" id="comment-8" about="/comment/8" typeof="schema:Comment" class="comment js-comment by-node-author has-title clearfix"> <div class="comment__container"> <h3 property="schema:name" datatype="" class="comment__title"> <a href="/comment/8#comment-8" class="permalink" rel="bookmark" hreflang="en">Network Effect... for now</a> <span class="comment__new marker marker--success hidden" data-comment-timestamp="1528242760"></span> </h3> <div class="comment__meta comment__meta--has-user-picture"> <a href="/blog/1">View recent blog entries</a> <div class="comment__submitted"> <span class="comment__author"><span rel="schema:author"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> </span> <span class="comment__pubdate">Wed 06/06/2018 - 11:52 <span property="schema:dateCreated" content="2018-06-05T23:52:40+00:00" class="rdf-meta hidden"></span> </span> </div> </div> <div class="comment__content"> <p class="comment__parent visually-hidden">In reply to <a href="/comment/7#comment-7" class="permalink" rel="bookmark" hreflang="en">Why GH not GitLab?</a> by <span lang="" typeof="schema:Person" property="schema:name" datatype="">Danyl Strype (not verified)</span></p> <div property="schema:text" class="clearfix text-formatted field field-comment--comment-body field-name-comment-body field-type-text-long field-label-hidden"> <div class="field__items"> <div property="schema:text" class="field__item"><p>Very good question, Danyl. My predecessor used GitHub prior to my involvement. I recognise that GitHub enjoys a network effect, and, given our goal is to maximise the value of our code for "The Commons", I (with misgivings, to be sure) decided to stick with GitHub (and a few projects on BitBucket, too) for the short term. </p> <p>Of course, everything's changed a couple days ago with the revelations that GitHub is being purchased by one of the least trustworthy corporations in the tech world, Microsoft. We will be setting up our own Gitlab instance (or perhaps Gogs, not sure yet) and shifting all our repos away from GitHub and Bitbucket (which is very similar to GitHub). Watch this space.</p> </div> </div> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=8&amp;1=default&amp;2=en&amp;3=" token="roEwVhPI8x3guUlKO4gZlMfjatgWmh9ZcaE9ZvBUEwo"></drupal-render-placeholder> </div> </div> </article> <article data-comment-user-id="1" id="comment-46" about="/comment/46" typeof="schema:Comment" class="comment js-comment by-node-author has-title clearfix"> <div class="comment__container"> <h3 property="schema:name" datatype="" class="comment__title"> <a href="/comment/46#comment-46" class="permalink" rel="bookmark" hreflang="en">OERu code repos now all on GitLab</a> <span class="comment__new marker marker--success hidden" data-comment-timestamp="1552341237"></span> </h3> <div class="comment__meta comment__meta--has-user-picture"> <a href="/blog/1">View recent blog entries</a> <div class="comment__submitted"> <span class="comment__author"><span rel="schema:author"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> </span> <span class="comment__pubdate">Tue 12/03/2019 - 10:53 <span property="schema:dateCreated" content="2019-03-11T21:53:57+00:00" class="rdf-meta hidden"></span> </span> </div> </div> <div class="comment__content"> <p class="comment__parent visually-hidden">In reply to <a href="/comment/7#comment-7" class="permalink" rel="bookmark" hreflang="en">Why GH not GitLab?</a> by <span lang="" typeof="schema:Person" property="schema:name" datatype="">Danyl Strype (not verified)</span></p> <div property="schema:text" class="clearfix text-formatted field field-comment--comment-body field-name-comment-body field-type-text-long field-label-hidden"> <div class="field__items"> <div property="schema:text" class="field__item"><p>Hi Danyl,</p> <p>Just a follow-up - all of the OERu's source code repositories (which is quite a few!) are now on <a href="https://git.oeru.org">https://git.oeru.org</a>, which is  a GitLab installation I set up and maintain here in New Zealand, on local cloud hosting infrastructure generously provided (sponsored) by <a href="https://catalystcloud.nz">CatalystCloud.nz!</a></p> </div> </div> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=46&amp;1=default&amp;2=en&amp;3=" token="gedjSAkNeSxUx8t2wJaPnNQFELS_DwaigcB_qfBpLRs"></drupal-render-placeholder> </div> </div> </article> </div> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=19&amp;2=field_blog_comments&amp;3=comment" token="RWzhW49w2Ranf8cyf73ptbSAI2rTzCPypSc-MBKegxE"></drupal-render-placeholder> </div> </section> Fri, 18 May 2018 01:17:23 +0000 dave 19 at http://tech.oeru.org Installing NextCloud and Collabora Office Online with Docker on Ubuntu 16.04 http://tech.oeru.org/installing-nextcloud-and-collabora-office-online-docker-ubuntu-1604 <span class="field field--name-title field--type-string field--label-hidden">Installing NextCloud and Collabora Office Online with Docker on Ubuntu 16.04</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--ubuntu-linux"> <span class="field__item-wrapper"><a href="/taxonomy/term/12" hreflang="en">ubuntu linux</a></span> </div> <div class="field__item field__item--mariadb"> <span class="field__item-wrapper"><a href="/taxonomy/term/48" hreflang="en">mariadb</a></span> </div> <div class="field__item field__item--docker"> <span class="field__item-wrapper"><a href="/taxonomy/term/16" hreflang="en">docker</a></span> </div> <div class="field__item field__item--docker-compose"> <span class="field__item-wrapper"><a href="/taxonomy/term/49" hreflang="en">docker-compose</a></span> </div> <div class="field__item field__item--php"> <span class="field__item-wrapper"><a href="/taxonomy/term/40" hreflang="en">php</a></span> </div> <div class="field__item field__item--collabora-office"> <span class="field__item-wrapper"><a href="/taxonomy/term/50" hreflang="en">collabora office</a></span> </div> <div class="field__item field__item--nextcloud"> <span class="field__item-wrapper"><a href="/taxonomy/term/51" hreflang="en">nextcloud</a></span> </div> <div class="field__item field__item--lets-encrypt"> <span class="field__item-wrapper"><a href="/taxonomy/term/17" hreflang="en">let&#039;s encrypt</a></span> </div> <div class="field__item field__item--redis"> <span class="field__item-wrapper"><a href="/taxonomy/term/21" hreflang="en">redis</a></span> </div> <div class="field__item field__item--productivity"> <span class="field__item-wrapper"><a href="/taxonomy/term/52" hreflang="en">productivity</a></span> </div> <div class="field__item field__item--nginx"> <span class="field__item-wrapper"><a href="/taxonomy/term/30" hreflang="en">nginx</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Mon 29/01/2018 - 17:29</span> <div class="float-none field field-node--field-image field-name-field-image field-type-image field-label-hidden has-multiple"> <figure class="field-type-image__figure image-count-1 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/Files%20-%20OERu%20NextCloud.png?itok=xQHlcyml" title="The NextCloud web interface for browsing your files" data-colorbox-gallery="gallery-field_image-epmBg3IHh8o" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;The NextCloud web interface for browsing your files&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/Files%20-%20OERu%20NextCloud.png?itok=6v2Kuyct" width="220" height="122" alt="The NextCloud web interface for browsing your files" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-2 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/NextCloud-AppStore.png?itok=DPeCx5Rd" title="The central AppStore (note, almost all apps have no cost and are open source). You get a similar view within your own NextCloud instance." data-colorbox-gallery="gallery-field_image-epmBg3IHh8o" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;The central AppStore (note, almost all apps have no cost and are open source). You get a similar view within your own NextCloud instance.&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/NextCloud-AppStore.png?itok=WqCJJdGj" width="220" height="175" alt="The central AppStore (note, almost all apps have no cost and are open source). You get a similar view within your own NextCloud instance." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-3 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/NextCloud-Calendar.png?itok=-j0Dq2rG" title="The NextCloud shared calendar plugin works with all major calendaring applications alongside your existing digital calendars." data-colorbox-gallery="gallery-field_image-epmBg3IHh8o" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;The NextCloud shared calendar plugin works with all major calendaring applications alongside your existing digital calendars.&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/NextCloud-Calendar.png?itok=bP23WxDf" width="220" height="175" alt="The NextCloud shared calendar plugin works with all major calendaring applications alongside your existing digital calendars." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-4 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/Nextcloud-CollaboraSpreadsheet.png?itok=Ovp0KryQ" title="An example of a fairly complex spreadsheet in the Collabora spreadsheet interface." data-colorbox-gallery="gallery-field_image-epmBg3IHh8o" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;An example of a fairly complex spreadsheet in the Collabora spreadsheet interface.&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/Nextcloud-CollaboraSpreadsheet.png?itok=CNhDR2y-" width="220" height="157" alt="An example of a fairly complex spreadsheet in the Collabora spreadsheet interface." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-5 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/Nextcloud-CollaboraWordprocessor.png?itok=IOyfA_M4" title="A fairly complex document, with variables, shown in the Collabora wordprocessor interface." data-colorbox-gallery="gallery-field_image-epmBg3IHh8o" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;A fairly complex document, with variables, shown in the Collabora wordprocessor interface.&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/Nextcloud-CollaboraWordprocessor.png?itok=HPawBI-o" width="220" height="157" alt="A fairly complex document, with variables, shown in the Collabora wordprocessor interface." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-6 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/DavInFilemanager.png?itok=rCbwaUUY" title="This is what your NextCloud would look like in your desktop filemanager (this is the Nemo filemanager on a Linux desktop)" data-colorbox-gallery="gallery-field_image-epmBg3IHh8o" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;This is what your NextCloud would look like in your desktop filemanager (this is the Nemo filemanager on a Linux desktop)&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/DavInFilemanager.png?itok=g2dNm33H" width="220" height="122" alt="This is what your NextCloud would look like in your desktop filemanager (this is the Nemo filemanager on a Linux desktop)" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-7 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-02/CollaboraAdminConsole.png?itok=1tNI9ZdJ" title="Collabora Office admin console" data-colorbox-gallery="gallery-field_image-epmBg3IHh8o" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Collabora Office admin console&quot;}"><img src="/sites/default/files/styles/medium/public/2018-02/CollaboraAdminConsole.png?itok=iijjMrBK" width="220" height="149" alt="Collabora Office admin console" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> </div> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>Dropbox is the best known of the end-user "cloud storage" services for documents, backups, and synchronising data among multiple devices, although now Google's Drive and Microsoft's OneDrive are functionally similar and are being heavily promoted and tied into all sorts of services.</p> <p>Similarly the collaborative editing of documents, spreadsheets, and presentations in the browser, pioneered by Etherpad, but then adopted in a big way by Google Docs (and more recently, Microsoft Office 365), has revolutionised collective note taking, document preparation, and ease of access to these powerful tools by the mainstream of computer users. Only a browser is required, and no other software needs to be installed.</p> <p>But what about people who don't want to entrust all of their data to foreign corporations, holding their data in foreign jurisdiction, in formats that may or may not be retrievable in the event that the supplier fails or changes "strategic direction"? And many of these services involve "mining" their data to extract useful information that vendors sell to others to <em>help them advertise to us in a more targeted way. </em>Yeah, that's creepy.</p> <p>More-over, often if you want to <em>share</em> your data with others, <em>they</em> have to log into the same service, and accept the service's terms and conditions (usually substantially constraining the user's normal rights and freedoms, although who<em> actually</em> reads those, eh?!) in order to do so... so ones use of those services has a magnifying effect on the loss of privacy and control.</p> <p>Some people sensibly prefer to manage their own, or institution-specific, solutions on the infrastructure of their choosing, in a way that doesn't tie anyone into paying ever increasing amounts for data storage as the volumes increase perpetually, month on month.</p> <p>Some of us simply prefer to have control of our own destiny, without a dependence on, for example, file or data storage formats and practices that are completely opaque to them. Our data reflects our creativity energy, and it seems much more comfortable for many of us to be in charge of our own fates rather than entrusting it to a third party who simply sees us a profit centre.</p> <p>Thankfully, the open source world has created an array of possible equivalent systems, and this post describes how you, too, can set up your own equivalent to Dropbox + Google Docs using entirely open source software on any commodity virtual machine hosting system you want to use by adopting NextCloud and Collabora Office.</p> <h2>NextCloud</h2> <p><a href="https://nextcloud.com">NextCloud</a> is <a href="https://nextcloud.com/files/">functionally similar</a> to Dropbox, however, with its active development community and plug-in architecture, it can provide quite a lot more as well, like shared calendaring, email, video conferencing, contact syncing, image/sound/video galleries, <a href="https://nextcloud.com/files/">among many other services</a>.</p> <p>If you prefer not to organise and run your own server, you can purchase a supported server via their website for a cost similar to Dropbox (although, realise that NextCloud is relatively small by comparison and doesn't have the massive economies of scale enjoyed by the bigger players).</p> <p>For those with an interest in history: NextCloud is a fork created by the founder of OwnCloud, after he decided that the company which formed around his OwnCloud project was moving in a direction that was philosophical unpalatable for him. The beauty of open source is that developers can follow their consciences without requiring anyone's permission. The resulting "forks" in code bases and communities then thrive or die based on the strengths of the communities they can build and sustain. This fork is remarkably similar to that which occurred in the OpenOffice community which resulted in the founding of LibreOffice. LibreOffice has thrived and OpenOffice has faded into irrelevance. More on that below.</p> <p>For those with a technological interest, NextCloud is a mature PHP application (but with a modern architecture, including a command line interface, occ) which stores its data in an RDBMS like MySQL, MariaDB, PostgreSQL, or (usually for development purposes) the lightweight SQLite database. Here are <a href="https://docs.nextcloud.com/server/12/admin_manual/installation/index.html">details for would-be administrators</a>.</p> <h2>Collabora Office</h2> <p>Given how much companies like Google and Microsoft invest on Docs and Office 365 respectively, how is it possible for an open source community to create a credible competitor? Turns out it's not as hard as you might think if they leverage the power of open source.</p> <p>A small software company with headquarters in the UK (although their team appears to be from all over), Collabora Office, has taken on the ambitious mission of creating a "collaborative web interface" allowing users to collaborate using <a href="https://libreoffice.org">LibreOffice</a>, one of the most powerful and widely used office package available anywhere. We're currently at Collabora Office 3.0, and the front end is quite nice and functional, but still pretty simple - that can be a good thing for many users. Collabora is progressively re-imagining the user interface of LibreOffice as a collaborative web interface. This isn't easy, but it's <em>much</em> easier than it otherwise would be because the difficult job of creating the heavy-lifting application back-end is already done - LibreOffice is a mature widely used application (albeit with a desktop interface, not a web-based collaborative interface). So we can expect progress will be rapid, and large sets of new capabilities will be "unlocked" as they progress their efforts.</p> <h2>NextCloud and Collabora - better together!</h2> <p>The beauty of the open source software model is that we can connect NextCloud and Collabora office - completely separate and unrelated communities - thanks to a new integration standard, WOPI (Web-application Open Platform Interface) they form a well integrated component model - with the <em>major </em>added benefit of being able to swap in a better file management platform, or a better collaborative productivity package if one or the other emerges, without having to start from scratch.</p> <h2>Setting up your own NextCloud Collabora Server</h2> <p>If you're game to run your own (and, in my experience, it's a surprisingly well behaved system) here's how you do it.</p> <p>In preparation, you'll want to have the following ready:</p> <ul><li>a Linux virtual machine or "VM" (I recommend running the current Ubuntu LTS version, or current Debian) with a user with Sudo privileges...,</li> <li>your domain name for the NextCloud instance, pointing to the IP address of your VM,</li> <li>your domain name for the Collabora instance, also pointing to the IP of your VM, and</li> <li>credentials for an email address capable of sending from a remote server (usually termed an "authenticating SMTP email account")</li> </ul><h3>Secure access with SSH</h3> <p>First things first, make sure you're logged into your host (probably via SSH) as a user who has "sudo" capabilities! You need to log into the host from your local machine. We recommend setting up <a href="https://www.digitalocean.com/community/tutorials/how-to-configure-ssh-key-based-authentication-on-a-linux-server">key-based authentication</a>.</p> <h3>Firewall with UFW</h3> <p>No computer system is ever full secure - there're always exploits waiting to be found, so security is a process of maintaining vigilance. Part of that is reducing exposure - minimising your "attack surface". Use a firewall - "<a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-ubuntu-16-04" title="Uncomplicated FireWall">ufw</a>" is installed on Ubuntu by default. Make sure you've got exceptions for SSH (without them, you could lock yourself out of your machine! Doh!).</p> <p>Run the following commands to allow your Docker containers to talk to other services on your host.</p> <p><code>sudo ufw allow in on docker0<br /> sudo ufw allow from 172.17.0.0/24 to any<br /> sudo ufw allow from 172.18.0.0/24 to any<br /> sudo ufw allow from 172.19.0.0/24 to any<br /> sudo ufw allow from 172.20.0.0/24 to any</code></p> <p>Specifically for Docker's benefit, you need to tweak the default Forwarding rule (I use "vim" as my editor. If you don't know how to/want to use it, replace <strong>vim</strong> with <strong>nano</strong> everywhere you see it in the following - nano's easier to use for simple edits like this):</p> <p><code>sudo vim /etc/default/ufw</code></p> <p>and copy the line <code>DEFAULT_FORWARD_POLICY="DROP"</code> tweak it to look like this (commenting out the default, but leaving it there for future reference!):</p> <p><code>#DEFAULT_FORWARD_POLICY="DROP"<br /> DEFAULT_FORWARD_POLICY="ACCEPT"</code></p> <p>You also have to edit <code>/etc/ufw/sysctl.conf</code> and remove the "#" at the start of the following lines, so they look like this:</p> <p><code>sudo vim /etc/ufw/sysctl.conf</code></p> <p><code># Uncomment this to allow this host to route packets between interfaces<br /> net/ipv4/ip_forward=1<br /> net/ipv6/conf/default/forwarding=1<br /> net/ipv6/conf/all/forwarding=1</code></p> <p>and finally restart the network stack and ufw on your server<code> </code></p> <p><code>sudo service networking restart<br /> sudo service ufw restart</code></p> <h3>Installing the Nginx webserver</h3> <p>In the configuration I'm describing here, you'll need a webserver running on the server - it'll be acting as a "proxy" for the Docker-based Nginx instance described below. I like the efficiency of Nginx and clarity of Nginx configurations over those of Apache and other open source web servers. Here's how you install it.</p> <p><code>sudo apt-get install nginx-full</code></p> <p>To allow nginx to be visible via ports 80 and 443, run</p> <p><code>sudo ufw allow "Nginx Full"</code></p> <p><strong>Note</strong>: make sure your hosting service is not blocking these ports at some outer layer (depending on who's providing that hosting service you may have to set up port forwarding).</p> <h3>Installing MariaDB</h3> <p>MariaDB is effectively a drop-in alternative to MySQL and we prefer it because it's not controlled by Oracle and has a more active developer community. On Ubuntu, MariaDB pretends to be MySQL for compatibility purposes, so don't be weirded out by the interchangeable names below. Install the server and the client like this.</p> <p><code>sudo apt-get install mariadb-server-10.0 mariadb-client-10.0</code></p> <p>You need to set a root (admin) user password - you might want to create a /root/.my.cnf file containing the following (replacing YOURPASSWORD) to let you access MariaDB without a password from the commandline<code>:</code></p> <p><code>[client]<br /> user=root<br /> password=YOURPASSWORD</code></p> <p>You should now be able to type "mysql" at the command prompt</p> <p>Tweak the configuration so that it's listening on</p> <p><code>sudo vim /etc/mysql/mariadb.conf.d/50-server.cnf </code></p> <p>and copy the bind-address line and adjust so it looks like this - we want MariaDB to be listening on all interfaces, not just localhost (127.0.0.1)...</p> <p><code># Instead of skip-networking the default is now to listen only on<br /> # localhost which is more compatible and is not less secure.<br /> #bind-address           = 127.0.0.1<br /> bind-address            = 0.0.0.0</code></p> <p>Then restart MariaDB:</p> <p><code>sudo service mysql restart</code></p> <p>It should now be listening on port 3306 on all interfaces, i.e. 0.0.0.0.</p> <p>Now set up the database which will hold NextCloud's data. Log into the MySQL client on the host (if you've created a .my.cnf file in your home directory as describe above, you won't need to enter your username and password):</p> <p><code>mysql -u root -p</code></p> <p>Enter your root password when prompted. It's also a good idea to gin up a password for your "nextcloud" database user. I usually use pwgen (<code>sudo apt-get install pwgen</code>) - for example running this command will give you a single 12 character password without special characters (just numbers and letters):</p> <p><code>pwgen -s 12 1<br /> T7KR2osrMkyC</code></p> <p>At the prompt (which will look something like MariaDB [(none)]&gt;) enter the following lines (putting your password in place of [passwd]):</p> <p><code>CREATE DATABASE nextcloud CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;<br /> CREATE USER "nextcloud"@"%" IDENTIFIED BY "[passwd]";<br /> GRANT ALL ON nextcloud.* to "nextcloud"@"%";<br /> FLUSH PRIVILEGES;</code></p> <p>Then enter \q to exit.</p> <h2>NextCloud and Collabora Office with Docker</h2> <p>We make use of the NextCloud community's <a href="https://hub.docker.com/_/nextcloud/" title="Documentation for the reference NextCloud Docker container.">stable Docker container</a> which they keep up to date. Similarly, the Collabora community has created a <a href="https://hub.docker.com/collabora/code">reference Docker container</a>.</p> <p>The over all architecture consists of five Docker containers (note, done properly, you aim to ensure that each container runs only one service!):</p> <ol><li>the main NextCloud container (running the PHP-FPM service)</li> <li>an identical container to the PHP one which runs the cron service (which does periodic administrative tasks relevant to NextCloud)</li> <li>the self-contained Collabora Office container (running PHP with an Apache web server instance and a full instance of LibreOffice running in headless server mode (never fear, no servers were harmed in the building of this server!) - yes this server doesn't really adhere to the "one-service per container" convention, but I'm ok with that. It's just a convention after all.)</li> <li>a Redis container (which provides performance improving caching for NextCloud), and</li> <li>an Nginx webserver container which makes it easier to manage the configuration and paths of the NextCloud and Collabora servers via WOPI. It means that on the hosting server, we only need to run a proxying web server, which is easy.</li> </ol><p>The way I prefer to implement this set of containers is to use <a href="https://docs.docker.com/compose/">Docker Compose</a> (after first setting up <a href="https://docs.docker.com/install/linux/docker-ce/ubuntu/">Docker support</a> on your server - I'll assume you've followed the complete instructions including <a href="https://docs.docker.com/install/linux/linux-postinstall/">setting up Docker for your non-root user</a>). I suggest using the latest <a href="https://docs.docker.com/compose/install/#install-compose">installation instructions</a> provided by the Docker community. To be honest, I usually use the alternative instructions, <a href="https://docs.docker.com/compose/install/#install-using-pip">employing the "pip" approach</a>. You can upgrade an existing install by issuing (on your Linux VM's command line):</p> <p><code>sudo pip install -U docker-compose </code></p> <p>To set up your server, I recommend setting up a place for your Docker containers (replace "me" with your non-root username on the server) and the associated persistent data (your Docker containers should hold <em>no</em> important data - you should be able to delete and recreate them entirely without losing any important data or configuration):</p> <p><code>sudo mkdir /home/data</code><br /><code>sudo mkdir /home/data/nextcloud</code><br /><code>sudo mkdir /home/data/nextcloud/apps<br /> sudo mkdir /home/data/nextcloud/config<br /> sudo mkdir /home/data/nextcloud/data<br /> sudo mkdir /home/data/nextcloud/redis<br /> sudo mkdir /home/data/nextcloud/resources<br /> sudo mkdir /home/docker<br /> sudo mkdir /home/docker/nextcloud-collabora<br /> sudo chown -R me:me /home/docker<br /> cd /home/docker/nextcloud-collabora</code></p> <p>Here's an example of the required docker-compose.yml file (you can create this via a text editor like "nano" which should be pre-installed on any VM these days, or use my preferred, but less intuitive, editor, vim via <code>vim docker-compose.yml</code> in the /home/docker/nextcloud-collabora directory):</p> <p><code>version: '2'<br /> networks:<br />   back:<br />     driver: bridge<br /> services:<br />   web:<br />     image: nginx<br />     ports:<br />       - 127.0.0.1:8082:80<br />     volumes:<br />       - ./nginx.conf:/etc/nginx/nginx.conf:ro<br />     links:<br />       - app<br />     volumes_from:<br />       - app<br />     environment:<br />       - VIRTUAL_HOST<br />     networks:<br />     - back<br />     restart: unless-stopped      <br />   app:<br />     image: nextcloud:12-fpm<br />     links:<br />       - redis<br />     volumes:<br />       - /home/data/nextcloud/apps:/var/www/html/apps<br />       - /home/data/nextcloud/config:/var/www/html/config<br />       - /home/data/nextcloud/resources:/var/www/html/resources<br />       - /home/data/nextcloud/data:/var/www/html/data<br />     networks:<br />     - back<br />     restart: unless-stopped      <br />   cron:<br />     image: nextcloud:12-fpm<br />     volumes_from:<br />       - app<br />     user: www-data<br />     entrypoint: |<br />       bash -c 'bash -s &lt;&lt;EOF<br />       trap "break;exit" SIGHUP SIGINT SIGTERM<br />       while /bin/true; do<br />         /usr/local/bin/php /var/www/html/cron.php<br />         sleep 900<br />       done<br />       EOF'<br />     networks:<br />       - back<br />     restart: unless-stopped      <br />   redis:<br />     image: redis:alpine<br />     volumes:<br />       - /home/data/nextcloud/redis:/data<br />     networks:<br />       - back<br />     restart: unless-stopped<br />   collab:<br />     image: collabora/code<br />     environment:</code><br /><code>      # put the domain name you select for your NextCloud instance<br />       # here! Escape any . in your domain name by preceding them with \\<br />       domain: your\\.domain\\.tld<br />       username: admin</code><br /><code>      # put your own strong password in here!<br />       password: some-good-password<br />     cap_add:<br />       - MKNOD<br />     networks:<br />       - back<br />     volumes_from:<br />       - app<br />     ports:<br />       - 127.0.0.1:9980:9980<br />     links:<br />       - app<br />     restart: unless-stopped</code></p> <p>You'll need to substitute the domain name you pick for your NextCloud instance - Collabora's container requires that you specify it so that it doesn't accept connections from other (potentially nefarious) containers elsewhere on the Internet!</p> <p>Also note, the "ports" specified above, 8082 for <code>nginx</code> and 9980 for <code>collab</code> are arbitrary - I picked these to ensure they don't conflict with ports being used by other containers on my server - you can use these if you want, or use <code>sudo netstat -punta</code> to see what ports are currently claimed by other services on your server (if there are any) and pick ones that don't clash! If it scroll past too fast, you can pipe it into less to allow you to scroll and search: <code>sudo netstat -punta | less</code> - hit "q" to exit or "/" to initiate a text search.</p> <p>You will also need to provide the "nginx.conf" file referenced in the nginx section of the file. Do that by using your editor, e.g. <code>vim nginx.conf</code>, and enter this content:</p> <p><code>user www-data;</code></p> <p><code>events {<br />   worker_connections 768;<br /> }</code></p> <p><code>http {<br />   upstream backend {</code><br /><code>      # if you don't call your NextCloud server "app" in your<br />       # docker-compose.yml, you'll need to change app below to </code><br /><code>      # whatever you end up calling it.<br />       server app:9000;<br />   }<br />   include /etc/nginx/mime.types;<br />   default_type application/octet-stream;</code></p> <p><code>  server {<br />     listen 80;<br />     <br />     # Add headers to serve security related headers<br />     add_header X-Content-Type-Options nosniff;<br />     add_header X-Frame-Options "SAMEORIGIN";<br />     add_header X-XSS-Protection "1; mode=block";<br />     add_header X-Robots-Tag none;<br />     add_header X-Download-Options noopen;<br />     add_header X-Permitted-Cross-Domain-Policies none;</code></p> <p><code>    root /var/www/html;</code></p> <p><code>    location = /robots.txt {<br />       allow all;<br />       log_not_found off;<br />       access_log off;<br />     }</code></p> <p><code>    location = /.well-known/carddav {<br />       return 301 $scheme://$host/remote.php/dav;<br />     }<br />     location = /.well-known/caldav {<br />       return 301 $scheme://$host/remote.php/dav;<br />     }</code></p> <p><code>    client_max_body_size 1G;<br />     fastcgi_buffers 64 4K;</code></p> <p><code>    gzip off;</code></p> <p><code>    index index.php;<br />     error_page 403 /core/templates/403.php;<br />     error_page 404 /core/templates/404.php;<br />  <br />     location / {<br />         rewrite ^ /index.php$uri;<br />     }</code></p> <p><code>    location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {<br />         deny all;<br />     }<br />     location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) {<br />         deny all;<br />     }</code></p> <p><code>    location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+|core/templates/40[34])\.php(?:$|/) {<br />         include fastcgi_params;<br />         fastcgi_split_path_info ^(.+\.php)(/.*)$;<br />         fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;<br />         fastcgi_param PATH_INFO $fastcgi_path_info;<br />         fastcgi_param HTTPS on;<br />         #Avoid sending the security headers twice<br />         fastcgi_param modHeadersAvailable true;<br />         fastcgi_param front_controller_active true;<br />         fastcgi_pass backend;<br />         fastcgi_intercept_errors on;<br />         fastcgi_request_buffering off;<br />     }</code></p> <p><code>    location ~ ^/(?:updater|ocs-provider)(?:$|/) {<br />         try_files $uri/ =404;<br />         index index.php;<br />     }</code></p> <p><code>    # Adding the cache control header for js and css files<br />     # Make sure it is BELOW the PHP block<br />     location ~* \.(?:css|js)$ {<br />         try_files $uri /index.php$uri$is_args$args;<br />         add_header Cache-Control "public, max-age=7200";<br />         # Add headers to serve security related headers (It is intended to<br />         # have those duplicated to the ones above)<br />         # Before enabling Strict-Transport-Security headers please read into<br />         # this topic first.<br />         # add_header Strict-Transport-Security "max-age=15768000;<br />         #  includeSubDomains; preload;";<br />         add_header X-Content-Type-Options nosniff;<br />         add_header X-Frame-Options "SAMEORIGIN";<br />         add_header X-XSS-Protection "1; mode=block";<br />         add_header X-Robots-Tag none;<br />         add_header X-Download-Options noopen;<br />         add_header X-Permitted-Cross-Domain-Policies none;<br />         # Optional: Don't log access to assets<br />         access_log off;<br />     }</code></p> <p><code>    location ~* \.(?:svg|gif|png|html|ttf|woff|ico|jpg|jpeg)$ {<br />         try_files $uri /index.php$uri$is_args$args;<br />         # Optional: Don't log access to other assets<br />         access_log off;<br />     }<br />   }<br /> }</code></p> <p>That should be all the configuration you need to make the Docker containers go.</p> <h2>Configuring Nginx to proxy NextCloud and Collabora</h2> <p>The next step is configuring the local nginx proxy servers for NextCloud and Collabora using the nginx instance you installed earlier. That's what responds to the domain name you choose for this service. In our case, the name is <a href="https://docs.oeru.org">https://docs.oeru.org</a> - you can have a look at it to see what you should be seeing when you first start things up! We use <a href="https://letsencrypt.org" title="This is an incredible free and open source service, that is single-handedly making the web a much safer place.">Let's Encrypt</a> to provide secure hosting - <a href="/protecting-your-users-lets-encrypt-ssl-certs">here're my Let's Encrypt instructions</a> on setting it up. The key thing to realise is that your "certificates" need to exist for Nginx to restart with the new configurations below - use the "commenting out the intervening lines" trick mentioned in my instructions to bootstrap the creation of your secure certificates!</p> <p>To configure the proxies, you need to create two configuration files in your /etc/nginx/sites-available/ directory.</p> <h3>NextCloud Proxy Configuration</h3> <p>Create a file with a meaningful name for your NextCloud Proxy, perhaps based on the domain name you've chosen (our file for docs.oeru.org is called "docs") using the same editing approach as the last few (although this is in a different directory) for example <code>sudo vim /etc/nginx/sites-available/docs</code> with the following contents, replacing "nextcloud.domain" with your selected domain name (and the port number 8082 if you've opted to change to a different one!):</p> <p><code>server {<br />     listen 80;<br />     server_name nextcloud.domain;</code></p> <p><code>    include /etc/nginx/includes/letsencrypt.conf;</code></p> <p><code>    # redirect all HTTP traffic to HTTPS.<br />     location / {<br />         return  302 https://nextcloud.domain$request_uri;<br />     }<br /> }</code></p> <p><code># This configuration assumes that there's an nginx container talking to the mautic PHP-fpm container,<br /> # and this is a reverse proxy for that Mautic instance.<br /> server {<br />     listen 443 ssl;<br />     server_name nextcloud.domain;</code></p> <p><code>    ssl_certificate /etc/letsencrypt/live/nextcloud.domain/fullchain.pem;<br />     ssl_certificate_key /etc/letsencrypt/live/nextcloud.domain/privkey.pem;<br />     ssl_protocols TLSv1 TLSv1.1 TLSv1.2;<br />     # to create this, see https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html<br />     ssl_dhparam /etc/ssl/certs/dhparam.pem;<br />     keepalive_timeout 20s;</code></p> <p><code>    include /etc/nginx/includes/letsencrypt.conf;<br />    <br />     location ^~ / {<br />         proxy_pass http://localhost:8082;<br />         proxy_set_header Upgrade $http_upgrade;<br />         proxy_set_header Connection "Upgrade";<br />         proxy_set_header Host $http_host;<br />         proxy_read_timeout 36000s;<br />     }<br />     client_max_body_size 1G;<br />     fastcgi_buffers 64 4K;</code></p> <p><code>    add_header X-Frame-Options "SAMEORIGIN";<br />     add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";<br /> }</code></p> <h3>Collab Proxy Configuration</h3> <p>Now create a collabora proxy configuration.</p> <p>Note: This will probably never by used by any user directly (there is a resource analysis service on the collabora system that might be of interest) - instead it'll be referenced by the NextCloud instance transparently to your users. </p> <p>In our case, we chose the domain collab.oeru.org and the file is called "collab", created via <code>sudo vim /etc/nginx/sites-available/collab</code> and containing (replace collab.domain with the one you've selected - similarly replace the port number 9980 with whatever you've selected if you've opted for a different one!):</p> <p><code>server {<br />     listen 80;<br />     server_name collab.domain;</code></p> <p><code>    # for let's encrypt renewals!<br />     include /etc/nginx/includes/letsencrypt.conf;</code></p> <p><code>    # redirect all HTTP traffic to HTTPS.<br />     location / {<br />         return  302 https://collab.domain$request_uri;<br />     }<br /> }</code></p> <p><code>server {<br />     listen 443 ssl;<br />     server_name collab.domain;</code></p> <p><code>    ssl_certificate /etc/letsencrypt/live/collab.domain/fullchain.pem;<br />     ssl_certificate_key /etc/letsencrypt/live/collab.domain/privkey.pem;<br />     ssl_protocols TLSv1 TLSv1.1 TLSv1.2;<br />     # to create this, see https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html<br />     ssl_dhparam /etc/ssl/certs/dhparam.pem;<br />     keepalive_timeout 20s;</code></p> <p><code>    # for let's encrypt renewals!<br />     include /etc/nginx/includes/letsencrypt.conf;</code></p> <p><code>    proxy_http_version 1.1;<br />     proxy_buffering off;</code></p> <p><code>    # static files<br />     location ^~ /loleaflet {<br />         proxy_pass https://localhost:9980;<br />         proxy_set_header Host $http_host;<br />     }</code></p> <p><code>    # WOPI discovery URL<br />     location ^~ /hosting/discovery {<br />         proxy_pass https://localhost:9980;<br />         proxy_set_header Host $http_host;<br />     }</code><br /><br /><code>    # download, presentation and image upload<br />     location ^~ /lool {<br />         proxy_pass https://localhost:9980;<br />         proxy_set_header Upgrade $http_upgrade;<br />         proxy_set_header Conection "upgrade";<br />         proxy_set_header Host $http_host;<br />     }<br /> }</code></p> <p>Once those are created, you have to make sure that they're "enabled" (replacing with your file names, of course):</p> <p><code>sudo cd /etc/nginx/sites-enabled<br /> sudo ln -sf ../sites-available/docs .<br /> sudo ln -sf ../sites-available/collab .</code></p> <p>To confirm that there aren't any typos or issues that might make nginx unhappy, run</p> <p><code>sudo nginx -t</code></p> <p>If all's well, get nginx to reread its configuration with the new files:</p> <p><code>sudo service nginx reload</code></p> <h2>Firing it all up!</h2> <p>Phew - congratulations on getting here! We've reached the moment of truth where we need to see if this whole thing will work!</p> <p>We need to make sure we're back in the Docker directory we set up:</p> <p><code>cd /home/docker/nextcloud-collabora</code></p> <p>and then we need to try running our docker-compose script to "pull" in the pre-built Docker containers we've specified in our docker-compose.yml file:</p> <p><code>docker-compose pull</code></p> <p>All going well, after a few minutes (longer or shorter depending on the speed of your server's connection) you should have download the Nginx, Redis, NextCloud and Collabora-CODE Docker images. Then you can run:</p> <p><code>docker-compose up -d &amp;&amp; docker-compose logs -f</code></p> <p>This will attempt to start up the containers (bringing them "up" in daemon mode, thus the -d) and then show you a stream of log messages from the containers, preceded by the container name. This should help you debug any problems that occur during the process (ideally, none).</p> <p>Once you see log messages streaming past, and no obvious "container exited" or other error messages (which will usually contain the word "error" a lot), you should be able to point your browser at your selected domain name and bring it up in your browser!</p> <h3>Setting up the database</h3> <p>On doing so, if all is well, you should be directed through the database set up process for your NextCloud instance. Your details should be:</p> <p>database IP: 172.17.0.1 - this is the default IP of the Docker host server.<br /> database name: nextcloud<br /> database user: nextcloud<br /> database password: (the one you came up with above)</p> <h3>Setting the Admin user</h3> <p>Once that's set and working, NextCloud will install all the relevant database tables and initial data. You'll be asked to set up an <em>admin user</em> account, which can be "admin" (you could make it something different to help stymie nefarious probes that assume you've got a user called "admin" - but don't forget what you've called it!) and some strong password you create (you can use the pwgen utility you used earlier) - I'd recommend recording it somewhere. I would <em>not</em> recommend making your own account, in your name, the main admin account. I recommend creating a second account, <em>with administrator privileges</em> for yourself, but leave the admin account purely for administrative activities.</p> <h3>Configuring Outgoing Email</h3> <p>To allow your NextCloud instance to send outgoing email, so that your site can alert you to security updates that need to be applied, or so that users can request a replacement password if they've forgot theirs, you'll need an <em>authenticating SMTP account</em> somewhere. Most of you already have one. You'll probably want to set up a dedicated email address for this server somewhere, perhaps something like "<a href="mailto:nextcloud@your.domain">nextcloud@your.domain</a>" or similar, with a username (often just the email address) and a password. You'll need the following details:</p> <p>SMTP server : an IP address or a domain name<br /> SMTP username: a username or an email address<br /> SMTP password: a strong password already configured for the username on that server<br /> SMTP login security: whether login is via TLS, SSL, or unsecure (!!), and<br /> SMTP login method: plain, encrypted, "login" or some other value.</p> <p>You should be able to text your email settings to make sure the details you've entered are valid. If you need to adjust these settings later, you can go to the admin menu (top right of the web browser interface) and go to Admin-&gt;Additional Settings  - should have a path of <a href="https://your.domain/settings/admin/additional">https://your.domain/settings/admin/additional</a></p> <h3>Configuring Collabora Office Integration</h3> <p>Once you're logged in as your own user, looking at your own default folders, you can start having a look around. You should have an "admin" menu (assuming you've created your user with Administrator privileges) at the top right of the web interface. If you go to Apps, you can use the search box to search for "Collabora" or go to the "Office &amp; text" App category. You'll need to "enable" the Collabora Online "official" app, at which point it will download the latest version of the connector app and install it (it should appear in your /home/data/nextcloud/apps directory)</p> <p>Once you've done that, go to your top right menu again, selecting Admin, and you should see "Collabora Online" as an option in the left column (which starts with "Basic settings"). Selecting that, you'll need to enter  "<a href="https://collab.domain">https://collab.domain</a>" (replacing with your domain, of course). I don't have any of the other options ticked.</p> <p>If it works, you should have the ability to go back to the home of your NextCloud install, which should show you your top-level folders. If you click the "+" next to the home icon (top left of the folder pane) you should now have the option to create (in addition to "Upload file", "New folder", "New text file") a "New Document", "New Spreadsheet", and "New Presentation". Clicking those should give you the Collabora Office interface for the designated content type.</p> <p>Similarly, you can use the "Upload file" to upload a document in a format that is supported by Collabora Office, once uploaded clicking on the filename should open it for editing in the appropriate Collabora Office interface.</p> <p>It is saved as it is change, you shouldn't need to save it explicitly.</p> <h2>Upgrading it</h2> <p>So, as you're no doubt aware, both NextCloud and Collabora Office are always being improved and updated. I certainly encourage you to keep your installation up-to-date.</p> <p>While you'll periodically see that NextCloud apps have available updates (these can be upgraded through the browser interface) updates to the NextCloud and Collabora Office systems themselves need to be undertaken by upgrading the containers. Luckily it's easy to do (although I strongly urge you to ensure you have a very recent backup of both database and uploaded files - they're the files in /home/data/nextcloud/data:</p> <p>Updating the container should be as easy as either doing another</p> <p><code>docker pull oeru/mautic</code></p> <p>and then shutting down Docker container via a</p> <p><code>docker-compose stop</code></p> <p>removing the old containers (this won't remove any data you want to save if you followed the directions above! But remember to do it in the right directory!) via</p> <p><code>docker-compose rm -v</code></p> <p>and then restarting it via</p> <p><code>docker-compose up -d</code></p> <p>Use <code>docker-compose logs -f</code> to watch the logs - you'll likely see debugging information in the unlikely event that something goes wrong in the upgrade process.</p> <h2>Backing it up</h2> <p>To back up your instance on your server, you need two things: a file system backup of your /home/data/nextcloud directory, and database dumps of your database.</p> <p>There're lots of ways to back up your files (I personally use a bash script that I wrote in a past role, which uses <a href="http://www.nongnu.org/rdiff-backup/">rdiff-backup</a> to create versioned backups either locally or on a remote server, although there're <a href="https://www.howtoforge.com/linux_rdiff_backup">other documented approaches</a> - leave a comment below if you'd like to learn more about my approach!).</p> <p>Backing up your database is as easy installing automysqlbackups:</p> <p><code>sudo apt install automysqlbackups</code></p> <p>You'll find daily versioned dumps of your MariaDB database(s) in /var/lib/automysqlbackups. To run an ad hoc backup (which will replace the previous backup from that day, if there is one) just run</p> <p><code>sudo automysqlbackups</code></p> <h2>Collabora Admin Console</h2> <p>Once you've got everything set up, you can access the admin console of the Collabora Office instance at the collab.domain you specified above - it'll have the path <code>https://collab.domain/loleaflet/dist/admin/admin.html</code> (of course replacing collab.domain with your domain) which gives you useful info about the system resources being used, number of documents being edited and by whom, and some other interesting details. I've included a screen shot.</p> <p>When prompted for login details, use the collab username - "admin" if you used the default I provided, and the password you set in your docker-compose.yml file above.</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=17&amp;2=field_blog_comments&amp;3=comment" token="aF_f2aYW3SGRfmrsffb5fOJlN5wEXJibvMUiQKm2VjE"></drupal-render-placeholder> </div> </section> Mon, 29 Jan 2018 04:29:13 +0000 dave 17 at http://tech.oeru.org