OERu Technology Blog http://tech.oeru.org/rss.xml en Introducing the OERu Tech Blog http://tech.oeru.org/intro <span class="field field--name-title field--type-string field--label-hidden">Introducing the OERu Tech Blog</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--free--open-source"> <span class="field__item-wrapper"><a href="/taxonomy/term/6" hreflang="en">free &amp; open source</a></span> </div> <div class="field__item field__item--technology"> <span class="field__item-wrapper"><a href="/taxonomy/term/7" hreflang="en">technology</a></span> </div> <div class="field__item field__item--kanban"> <span class="field__item-wrapper"><a href="/taxonomy/term/8" hreflang="en">kanban</a></span> </div> <div class="field__item field__item--devops"> <span class="field__item-wrapper"><a href="/taxonomy/term/9" hreflang="en">devops</a></span> </div> <div class="field__item field__item--foss"> <span class="field__item-wrapper"><a href="/taxonomy/term/10" hreflang="en">foss</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Thu 08/09/2016 - 13:00</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>The <a href="http://oeru.org">Open Education Resource universitas</a> (OERu) is an open organisation from top to bottom. Our entire technological infrastructure (with a <a href="/node/1" target="_blank">couple exceptions</a>) is built with and on Free and Open Source Software to which I normally refer as "FOSS".</p> <p>We are committed to using FOSS for our infrastructure because:</p> <ul><li>it is consistent with our "open" philosophy for education - if our educational materials and all our OERu planning processes are "open on principle", then delivering them on "closed" (or, more precisely, "proprietary") technologies would be a clanging bit of <a href="https://en.wikipedia.org/wiki/Cognitive_dissonance">cognitive dissonance</a>. There's even a word for it: "prefigurative" - use means that are consistent with your desired end.</li> <li>it allows us to use "best-of-breed" technologies (in the web-space, most innovation happens as FOSS first), combining many single-purpose solutions together in a modular way - via the open standards, open data formats, and open APIs they all support - to form a robust, feature-rich and secure communications and collaboration infrastructure. If a better solution to one of our requirements emerges, we can rapidly adapt our systems to adopt it, ensuring we are responsive and offer a fresh, state-of-the-art, well-supported and technically excellent platform.</li> <li>our partner institutions collaborators become familiar with these tools, and if they find any of them valuable, they can champion their adoption by their home institutions, helped by the instructions provided by this blog or our FOSS code repositories. The money saved and flexibility gained by our partner institutions by doing so are likely to be of substantially greater value than their annual OERu membership fees!</li> <li>Perhaps most importantly, none of our collaborators or learners are <em>ever</em> compelled to accept dubious (and usually exploitative) proprietary software Terms and Conditions to be part of our community. We respect our community's right to privacy and personal data sovereignty from the ground up.</li> </ul><h2>Physical Infrastructure</h2> <p>Our infrastructure is currently hosted on FOSS <a href="https://ubuntu.com">Ubuntu</a> and <a href="https://debian.org">Debian</a> Linux systems provides by four IaaS (Infrastructure as a Service) providers:</p> <ol><li>our <a href="https://www.mediawiki.org">MediaWiki</a> infrastructure, the star of which is <a href="https://wikieducator.org">Wikieducator.org</a>, is hosted on Amazon's "Elastic Compute" EC technology (located in the US, in our case), running Ubuntu Linux 14.04.</li> <li>the rest of our self-hosted infrastructure, hosted on a high-specification virtual machine, "Hetzner", also running Ubuntu Linux 14.04 (based in Germany). More on the many services provided by Hetzner below.</li> <li>more recently, we have created hosting for some of our new services on <a href="http://azure.com" title="Microsoft's Infrastructure-as-a-service Platform.">Azure</a>, taking advantage of an annual grant given to qualifying non-profit organisations by Microsoft. We run Ubuntu Linux 16.04 hosts there. We ensure we do not make use of any proprietary Azure capabilities in any of our automated scripts so that we can shift hosting providers with minimal cost and inconvenience if/when Microsoft changes their free-hosting policy.</li> <li>in 2018, we also received sponsored hosting capacity from NZ's own cloud service provider - whose cloud offering is entirely open source software built on open stack - <a href="https://catalystcloud.nz">Catalyst Cloud</a>. We host our <a href="https://git.oeru.org">code repositories</a> on their infrastructure. Many thanks to Catalyst!</li> </ol><p>Incidentally, the FOSS operating system, Linux, <a href="https://w3techs.com/technologies/details/os-unix/all/all">is the most widely used hosting platform on the Internet </a>today, and <a href="https://w3techs.com/blog/entry/ubuntu_became_the_most_popular_linux_distribution_for_web_servers">Ubuntu is the most widely used</a> "distribution" of Linux.</p> <h2>Externally hosted</h2> <p>We also make use of some externally hosted commercial FOSS services (we pay them for their services) to provide all the functionality we require:</p> <ul><li><a href="http://onlinegroups.net">OnlineGroups.Net</a> for <a href="http://groups.oeru.org">our family of mailing lists</a>.</li> <li><a href="https://mautic.net">Mautic</a> for our newsletter and user-engagement needs (Update 2017-06-16: due to substantial price increases for the hosted Mautic service, we are moving to a self-hosted version <a href="/installing-mautic-php7-fpm-docker-nginx-and-mariadb-ubuntu-1604">set up like this</a>).</li> <li>Update 2017-06-16: we have adopted a new open source Kanban planning tool, <a href="http://kanboard.net">Kanboard</a>, and we're supporting the developer by paying for the hosted service.</li> </ul><h2>Web Applications</h2> <p>We host and maintain a number of websites "that do stuff", otherwise known as web applications. These include:</p> <ul><li>Our <a href="https://course.oeru.org">Course website</a> which acts as a platform for per-course and per-cohort course websites, generated automatically via an OERu innovation: our <a href="/oeru-mediawiki-wordpress-snapshot-toolchain">course "snapshot" process</a> from learning materials formulated on Wikieducator that are transformed into fully-formed, partner-institution-branded websites. Built on the <a href="https://wordpress.org">WordPress</a> blog platform, running in "multisite" mode.</li> <li>Our main <a href="https://oeru.org">OERu Website</a> - which provides information about the organisation relevant to both learners and partners. It is built on the <a href="https://silverstripe.org">Silverstripe</a> Content Management System (CMS).</li> <li>This Technology Blog... which is built on the <a href="https://drupal.org">Drupal</a> CMS (version 8).</li> </ul><h2>Web Services</h2> <p>To maintain control and flexibility, we self-host a myriad of useful web-based resources and services. These include:</p> <ul><li>our <a href="https://cloud.oeru.org">data sharing/digital artefact-storage site</a>, comparable to having our own "Dropbox", is <a href="https://nextcloud.org">NextCloud</a>.</li> <li>our <a href="https://etherpad.oerfoundation.org">collaborative document editing</a> platform is <a href="https://github.com/ether/etherpad-lite">Etherpad-Lite</a>.</li> <li>our two "next generation" online forums, <a href="https://community.oeru.org">Community</a> (for educators and OERu collaborators) and <a href="https://forums.oeru.org">Forums</a> (for learners), built on <a href="http://www.discourse.org/">Discourse</a>.</li> <li>our <a href="https://chat.oeru.org">chat system</a>, a <a href="https://rocket.chat">Rocket.Chat</a> instance, similar to the proprietary Slack platform, replaces our venerable geeks-only platform, IRC (Internet Relay Chat).</li> <li>our <a href="https://plan.oeru.org">planning system</a>, an instance of the <a href="http://wekan.org">Wekan</a> "virtual <a href="https://en.wikipedia.org/wiki/Kanban_board">Kanban board</a>" (Update 2017-06-16: although we still use this a bit, we have found we prefer <a href="http://kanboard.net">Kanboard</a>)</li> <li>our <a href="https://oer.nz/admin">link shortening</a> service is an instance of <a href="http://yourls.org/">YourLS</a>.</li> <li>our <a href="https://mantis.oeru.org">issue tracking</a> service is an instance of <a href="https://www.mantisbt.org">Mantis Bug Tracker</a> (Update 2017-06-16: we've retired this as it wasn't quite the right fit for our users)</li> <li>our integrated <a href="https://links.oeru.org">link sharing "course resource bank"</a> is <a href="https://semanticscuttle.sourceforge.net">Semantic Scuttle</a>.</li> <li>our <a href="https://stats.oeru.org">website usage tracking</a> system built with <a href="http://piwik.org">Piwik</a>.</li> <li>Update 2017-06-16: we have recently set up a <a href="https://github.com/tootsuite/mastodon" title="The Mastodon Federated Social Network">Mastodon</a> instance to facilitate training our learners in the use of social networking without having to resort to a proprietary freedom-compromised platform. Here's <a href="/installing-mastodon-docker-compose-ubuntu-1604">how we did it</a>.</li> <li>Update 2017-06-16: we have a <a href="https://nextcloud.com" title="Open source, self-hosted web-based multi-tenented file store similar to Dropbox, but with more freedom.">NextCloud</a> instance, linked to a <a href="https://www.collaboraoffice.com/">Collabora Online office suite</a> instance, a concurrent editing application similar to Google Docs/Sheets. Howto coming soon!</li> <li>Update 2017-09-22: we have a <a href="https://limesurvey.org" title="Comprehensive open source online survey and polling tool.">Lime Survey</a> instance, replacing our use of Google Forms for conducting web-based surveys and polls.</li> </ul><h2>Code Repositories</h2> <p>We have a convention of documenting (including instructions, source code, and configuration examples) all of our individual implementations FOSS implementations. We have multiple projects for public reference stored at both Bitbucket (we have repositories in both the <a href="https://bitbucket.org/oerf/">Wikieducator</a> and the <a href="https://bitbucket.org/oerf/">OER Foundation</a> projects - Bitbucket is a source code "<a href="https://en.wikipedia.org/wiki/Forge_(software)">forge</a>" run by Atlassian in Australia) and<a href="github.com/oeru"> </a><a href="https://bitbucket.org/wikieducator/">Github</a><a href="github.com/oeru"> </a>(a forge run by Github in the US).</p> <p>Over time, a few of us will be writing up some blog posts on specific technologies to introduce them in a gentle way to those for whom terms like "git" and "pull request" do not yet have a respectable technology-related connotation.</p> <h2>Our Toolbox</h2> <p>To build and maintain our infrastructure, we use a cornucopia of additional FOSS tools. Among these are text editors, monitoring systems, backup tools, debugging environments, "<a href="https://en.wikipedia.org/wiki/DevOps">devops</a>" platforms, container technologies, and many others. We'll no doubt cover some of these in future blog posts.</p> <p>A couple noteworthy tools that all of our institutional partners should know about include:</p> <ul><li>we get all of our SSL (Secure Sockets Layer) certificates, that help keep our users' information secure and private by providing browser-to-server end-to-end encryption from <a href="https://letsencrypt.org/">Let's Encrypt</a>, at no cost. We encourage everyone else to do likewise! Here's our <a href="/protecting-your-users-lets-encrypt-ssl-certs">Let's Encrypt howto</a>.</li> <li><a href="https://en.wikipedia.org/wiki/OpenSSH">OpenSSH</a> (Secure SHell) - which ships with all Linux systems of which we're aware - it's the way we access all of our remote systems securely from anywhere.</li> </ul></div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=2&amp;2=field_blog_comments&amp;3=comment" token="CdJY_SzH-b9OZT5uqHtipOPQseJjEJj5aRpmwEa4k8M"></drupal-render-placeholder> </div> </section> Thu, 08 Sep 2016 01:00:54 +0000 dave 2 at http://tech.oeru.org http://tech.oeru.org/intro#comments Democratising Higher Education with OERs & FOSS http://tech.oeru.org/democratising-higher-education-oers-foss <span class="field field--name-title field--type-string field--label-hidden">Democratising Higher Education with OERs &amp; FOSS</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Mon 21/01/2019 - 14:02</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>Rather than write a slide-based presentation for the Open Education miniconf at <a href="https://linux.conf.au" title="Linux.Conf.AU, the premier FOSS annual conference in the southern hemisphere... ">Linux.Conf.AU 2019</a>, I thought I'd try an approach that seemed to work quite well <a href="/2018-update-oeru-technology-stack">recently</a>: write my presentation as a blog post and make it available for posterity, before and after. You can find this post quickly by going to <strong>oer.nz/lca</strong> if you want to look at it on your own device.</p> <p>Update: If you'd like to see me presenting this talk, <a href="https://www.youtube.com/watch?v=o9KaDiHj-H4">the video is now available</a>. There're <a href="https://www.youtube.com/user/linuxconfau2019/videos">lots of other talks</a> from the conference, too - I encourage you to have a browse!</p> <p>So why am I here and what am I talking about?</p> <p>I'm here because I love building stuff. I'm also a fan of sharing, and of the Commons (both analogue and digital). Unsurprisingly, I'm a committed adherent to and advocate for the principles underlying Free and Open Source Software (FOSS) and Creative Commons licensing.</p> <p>As a person who has a lot of time working with technology and has had a great education thanks to a combination of luck, hard work, and privilege, I have realised that I have an opportunity to parley my good fortune to make the world a better place.</p> <p>After quite a bit of thinking about it, I've really only identified one thing that I feel confident is an unambiguously "good thing" to do: increasing opportunities in education, particularly for girls and women. So I<em> </em>focus my energies on<em> removing as many barriers as I can that hinder people all over the world from reaching their educational potentials</em>.</p> <h2>Living the Dream</h2> <p>After following a relatively <a href="https://davelane.nz">convoluted path</a> for many years, I consider myself very fortunate to be working in my dream role today: I'm the <a href="/insight-what-does-open-source-technologist-oer-foundation-do">Open Source Technologist</a> for the Open Educational Resource Foundation (OERF). I say it's my dream role because it allows me to support my family, work on something I find personally fulfilling because completely aligned with my personal interests, skills, and values, and gives me the huge satisfaction of knowing I'm doing something that will create opportunities for millions of people I've never even met.</p> <p>The OERF is a charitable foundation, hosted by the <a href="https://op.ac.nz">Otago Polytech</a> in Dunedin (South Island, New Zealand). I think it's fair to say we're a lean operation... with two full-time staff: the founder, <a href="https://wikieducator.org/User:Mackiwg" title="Formal bio for Wayne Mackintosh"><strong>Wayne Mackintosh</strong></a>, who's a committed FOSS-using academic and a genius at getting people to do the right thing (as well being the UNESCO chair in OER), and me - so the Open Source Technologist is the <em>only technologist</em>. You can start to see why I love this role.</p> <h2>Our Aims</h2> <p>The Foundation has three main aims:</p> <ol><li>to create the curriculum materials for an entire university's worth of academical "courses" that are entirely licensed as "<a href="https://en.wikipedia.org/wiki/Definition_of_Free_Cultural_Works" title="The definition of Free Cultural Works">Free Cultural Works</a>" (CC-By or CC-By-SA) - this is the curriculum analogue to FOSS.</li> <li>to create and lead a partnership with higher education institutions around the world to create a new type of higher education model. These partners recognise the value of collaborating on learning materials and are committed to expanding global access to high education, the articulation of academic credit, and see an opportunity in extending their assessment services to learners who aren't formally enrolled in their institutions.</li> <li>to create a platform for delivering these collaboratively assembled and accredited courses to <em>anyone who has access to the internet</em>, allow them to study those materials at no cost, and with low cost<em> formal assessment services</em> provided by our partners. This allows learners to gain <em>academic credit from accredited institutions,</em> potentially leading to a formal qualification. We encourage others (especially our partners) to copy our approach where it appeals to them.</li> </ol><h2>Radical Openness</h2> <p>Prior to my involvement, Wayne decided that the organisation would be "radically open". And he means it. How it works:</p> <ul><li>our constantly evolving strategic direction is openly published as a Free Cultural Work,</li> <li>all decision making is done collectively with our partners and the results (and thinking behind them) is proactively published,</li> <li>all our meetings are streamed, videoed, and made available for posterity, along with post meeting reports summarising the discussion and any decisions reached, and</li> <li><a href="/proprietary-software-holdouts">wherever possible</a>, the OERF uses FOSS (we all use Linux internally), contribute to upstream projects, and release any internally developed software as FOSS.</li> </ul><p>Here's an example of what it looks like: <a href="https://wikieducator.org/OERu/OERu_18.11_Meeting/Report#OERu_milestones">our 2018 partner meeting in Port Macqurie, NSW, Australia.</a></p> <h2>Associated Policies</h2> <p>In addition to its commitment to openness, the OERF undertakes its activities in accordance with a few general principles:</p> <ol><li>All our curriculum materials should be available for any educator to incorporate into any course an educator chooses, in a way that respects their authorship and the provenance of the materials.</li> <li>Most higher education institutions choose a monolithic learning management system (LMS) to frame learner interactions, allowing only authenticated learners to participate, and on matriculation they lose access to their learning materials and artefacts. Instead, we prefer a more flexible, open, and persistent environment that is also more relevant to learners: the Internet. </li> <li>No learners studying, nor educators contributing to, our course materials should be disadvantaged because they cannot afford proprietary software to do so, nor should they need to entrust any of their creative efforts, or personal privacy to any third party with a commercial interest.</li> <li> <p>This entire process should be fiscally sustainable, not based on pure charity, but due to enlightened self-interest from educational institutions (who realise they are in imminent danger of having their business models permanently disrupted).</p> </li> </ol><h2>Open, Reusable, Editable, Distributable content</h2> <p>All our curriculum materials should be available for any educator to incorporate into any course an educator chooses. We use the powerful (but relatively transparent to non-technical users) versioning capabilities of MediaWiki to version all content and provide an educators' analogue to a git repository. Our curriculum material site is <a href="https://wikieducator.org">WikiEducator.org</a> and it is usually among the top 100k sites on the web.</p> <h2>Loosely Coupled, no Monoliths</h2> <p>Rather than an LMS, we adhere to the "<a href="https://en.wikipedia.org/wiki/Unix_philosophy">Unix philosophy</a>" - all of our learning materials are delivered by a <a href="/many-simple-tools-loosely-coupled">loosely coupled</a> selection of excellent FOSS applications via the open Internet, and can be viewed by anyone, anonymously.</p> <p>Collaborating with other learners does require authentication to manage behaviour and demonstrate participation, however this is almost entirely done using FOSS web services we host.</p> <p>We select the software components for our "<a href="/2018-update-oeru-technology-stack">Next Generation Digital Learning Environment</a>" based on being FOSS licensed, web-based, open standards compliant, fit for purpose, mobile-friendly, and flexible (e.g. with respect to authentication methods).</p> <p>We also consider scalability - although our current learner numbers are low, in the hundreds or thousands, we expect to scale to hundreds of thousands in the coming year or two, and eventually to millions of learners.</p> <p>Thanks to our component model, if we identify a better FOSS component to fill a particular niche, we simply replace what's there already. Our solutions can evolve quickly (especially compared to most of academia) and relatively painlessly from our learners' perspective.</p> <h2>Respect People</h2> <p>To ensure we do not open our learners or collaborators to exploitation by corporate interests outside of our control... we don't use any in our teaching process. We see them as largely incompatible with the <em>sharing</em> necessary for real teaching and learning. Our entire software stack is comprised of FOSS, including collaboration tools for both learners and educators.</p> <h2>Sustainable Model</h2> <p>Of course, as with any FOSS project, for the OERF's approach to succeed, it must appeal to the <em>enlightened self-interest</em> of its participants. Our aim, as a charitable foundation is not to profit, but to achieve sustainability while maximising our impact...</p> <p>Part of that is consistently achieving sufficient income and the other part is minimising costs. We have done the latter to the extent possible (our small staff overhead and an annual IT budget for software and infrastructure is $5000!). Other than a small but much appreciated grant from the Hewlett Foundation (we're a long time grantee), we are already self-sustaining.</p> <p>The major of our income comes from our partners' membership fees (which are about USD4000 per year, although some opt to pay somewhat more), which they choose to pay for a variety of different reasons.</p> <p>For some it's a matter of fulfilling their "greater good" mission. For others, it's a philosophical direction which appeals - improving access to education to those who currently have no realistic opportunities to pursue it due to their location, cultural restrictions, and personal circumstances.</p> <p>Many of our partners realise that their current situations are precarious - that they are in imminent danger of having their business models permanently disrupted by technologically savvy initiatives like ours  - it's in their best interest to be part of them.</p> <p>Many also recognise an opportunity in becoming familiar with the FOSS technologies that we're implementing and making available to our partners - if they choose to implement any one of our chosen technologies at their own institutions rather than a proprietary alternative, the cost savings alone (as well as increase capabilities) would easily compensate of many years worth of our membership fee.</p> <h2>The FOSS Stack</h2> <p>At a conference like this, people like to get specific about the technologies we use, so <a href="/2018-update-oeru-technology-stack">let's look into that</a> for the time we have left! Happy to talk to anyone further about all the details!</p> <h2> </h2> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=24&amp;2=field_blog_comments&amp;3=comment" token="mjUav4-YhA77NHR_0jlXW-3giZNhdMcjCjhc8ozAcyM"></drupal-render-placeholder> </div> </section> Mon, 21 Jan 2019 01:02:01 +0000 dave 24 at http://tech.oeru.org http://tech.oeru.org/democratising-higher-education-oers-foss#comments 2018 update on the OERu Technology Stack http://tech.oeru.org/2018-update-oeru-technology-stack <span class="field field--name-title field--type-string field--label-hidden">2018 update on the OERu Technology Stack</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Wed 31/10/2018 - 10:24</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>As I prepare for the <a href="https://course.oeru.org/2018-meetings/">2018 OERu Partners' Meeting</a>, I'm working how to convey the depth and breadth of the OERu open source technology stack - our infrastructure and applications - to our partners and other attendees.</p> <p>Over the past year, we've had our first live OERu courses allowing learners to work towards a formal exit qualification (our "1st year of study"), and our infrastructure has worked as intended throughout, so I'm chalking that up as a win.</p> <h2>OERu Infrastructure - NGDLE</h2> <p>At the OERu, in following our open principles, we have ended up creating a "Next Generation Digital Learning Environment" (NGDLE) to meet the needs of our learners, partners, and OER collaborators. It's a distributed, <a href="/many-simple-tools-loosely-coupled">loosely coupled component</a> model. It's also entirely made up of Free and Open Source software, from top to bottom.</p> <figure role="group" class="caption caption-img"><img alt="OERu Technology Service wheel - the basis for our Next Generation Digital Learning Environment" data-entity-type="file" data-entity-uuid="19dbe4e0-3256-42a3-a34a-575c518dc532" height="484" src="/sites/default/files/inline-images/OERu_technology_wheel.png" width="701" /><figcaption>Services making up parts of the OERu's NGDLE. These are either hosted by the community or on the OERu's fully open source technology infrastructure.</figcaption></figure><p>We believe our approach has a lot of advantages and, if emulated by our partners and other academic institutions, could revolutionise both the quality of digital services used in education, as well as vastly reducing costs and increasing the autonomy and resilience of technical solutions while providing unprecedented technology-related learning opportunities and agency for learners and educators alike. The purpose of this post is to describe our technology infrastructure, explain some of its advantages, and highlight the challenges it presents.</p> <p>A key takeaway: <em>if any of our partners adopted even one of the technologies we have incorporated into our NGDLE, they could easily save many times the value of their annual OERu subscription fees  in the first year, and in every subsequent year</em>. This blog exists to provide handy howtos to make it easy for our partners to trial and adopt these technologies in a maintainable way.</p> <h3>Geographic Diversity</h3> <p>In my role of Open Source Technologist, I maintain our global computing infrastructure, currently made up of <a href="https://www.mapcustomizer.com/map/OERu%20Global%20Infrastructure" title="Map of OERu's global hosting infrastructure.">four separate nodes</a>, on behalf of the <a href="https://oeru.org">OERu</a>:</p> <ol><li><a href="https://osm.org/go/0DvZ0tbZ--" title="The likely hosting facility for u.oerfoundation.org">u.oerfoundation.org</a> ( - a dedicated server hosted in an Hetzner Online facility in Gunzenhausen, Bavaria, Germany.</li> <li><a href=" https://osm.org/go/uG4HYrE-?m=">open.oerfoundation.org</a> ( - a compute instance hosted in a Microsoft Azure facility in Melbourne, Victoria, Australia (charitable organisation grant from TechSoup).</li> <li><a href="https://osm.org/go/urEwlGNt-?m=">containers.oerfoundation.org</a> ( - a compute instance hosted in a Catalyst Cloud facility in Wellington, New Zealand.</li> <li><a href="https://osm.org/go/WIDlXmu-?m=">we6.wikieducator.org</a> ( - a compute instance hosted in an Amazon Web Services facility in Portland, Oregon, USA.</li> </ol><figure role="group" class="caption caption-img"><a href="https://www.mapcustomizer.com/map/OERu%20Global%20Infrastructure" title="Link to a live map (using the open source OpenStreetMap platform!) - you can drill down to each hosting facility."><img alt="A map view of the OERu's open source tech infrastructure." data-entity-type="file" data-entity-uuid="225adadd-5811-47e3-8c03-a86d81f5ae78" src="/sites/default/files/inline-images/OERu_global_infrastructure_map.png" /></a> <figcaption>This image is linked to a live <a href="https://www.openstreetmap.org">OpenStreetMap</a> view (open source alternative to Google Maps) - there you can zoom into each facility in which our hosting infrastructure is held. This interactive map was created using <a href="https://www.mapcustomizer.com">MapCustomiser</a>.</figcaption></figure><h3>Functional Diversity and Containers</h3> <p>All of these servers run the open source operating system, <a href="https://en.wikipedia.org/wiki/Linux">Linux</a>, and with the exception of we6.wikieducator.org, run multiple services in <a href="https://www.docker.com/docker-community">Docker</a> containers. We6 is the original OERu server and it runs the technology on which our first initiative is built: a <a href="https://www.mediawiki.org/wiki/MediaWiki">MediaWiki</a> implementation answering to <a href="https://wikieducator.org">https://wikieducator.org</a> - it's where all our curriculum materials are assembled and maintained.</p> <p>The story on our other hosts is more complicated. Here's a run down of the <em>fully open source</em> services each provides:</p> <ol><li>u.oerfoundation.org is currently running a number of services, including 26 Docker containers, which collectively provide the following services: <ol><li>Our <a href="https://www.silverstripe.org/">SilverStripe</a> OERu website, <a href="https://oeru.org">https://oeru.org</a>, which is our main contact with prospective partners and our course advertising platform.</li> <li>Our <a href="https://wordpress.org">WordPress</a> <a href="https://codex.wordpress.org/Create_A_Network">Multisite</a>, <a href="https://course.oeru.org">https://course.oeru.org</a>, our primary course delivery platform (each course instance is a separate "sub-site" but a user on our course site can enrol in any course sub-site), to which we can automatically deploy fully formed courses from <a href="https://wikieducator.org">WikiEducator</a> via our Snapshotting system.</li> <li>Our <a href="https://www.discourse.org/">Discourse Forums</a>: <ol><li><a href="https://community.oeru.org">https://community.oeru.org</a> - for our educator and partner community</li> <li><a href="https://forums.oeru.org">https://forums.oeru.org</a> - for our learners</li> </ol></li> <li>Our <a href="https://rocket.chat">Rocket.Chat</a> instance, <a href="https://chat.oeru.org">https://chat.oeru.org</a>, for real-time communication - text &amp; media messages, audio and video calls - within our team, and with the broader partner and learner community. We also host a couple other Rocket.Chat instances for well aligned groups and partners trialling it. Or you could <a href="/docker-compose-better-way-deploy-rocketchat-wekan-and-mongodb">run your own Rocket.Chat</a>...</li> <li>Our <a href="/wikieducator-notes-oerus-course-feed-aggregation-and-messaging-system">WEnotes Course Feed</a> stack made up of several containers that monitor various social media and blogs, and then scan, store, and serve up our dynamic course feeds - <a href="https://course.oeru.org/lida101/interactions/course-feed/">for example</a>.</li> <li>Our <a href="https://matomo.org">Matomo</a> (formerly Piwik) instance, <a href="https://stats.oeru.org">https://stats.oeru.org</a>, allows us to track the use of our websites ourselves. It's functionally similar to Google Analytics, but without giving information on our web visitors to Google.</li> <li>Additionally this server hosts a number of native services (not in containers) including <ol><li><a href="https://yourls.org">YourLS</a>, our link shortener on <a href="https://oer.nz">https://oer.nz</a>, and</li> <li><a href="http://semanticscuttle.sourceforge.net/">Semantic Scuttle</a>, on <a href="https://bookmarks.oeru.org">https://bookmarks.oeru.org</a>.</li> </ol></li> <li>Finally, we run a number of test systems including replicas of WikiEducator and our Course site.</li> </ol></li> <li>open.oerfoundation.org runs 27 Docker containers which collectively provide <ol><li>Our <a href="https://joinmastodon.org/">Mastodon</a> instance, <a href="https://mastodon.oeru.org">https://mastodon.oeru.org</a>, our open education corner of the "<a href="https://en.wikipedia.org/wiki/Fediverse">Fediverse</a>" - it's a micro-blogging platform similar to Twitter, but distributed - without the centralised corporate control and surveillance capitalism business model. Here's how to <a href="/installing-mastodon-docker-compose-ubuntu-1604">run your own Mastodon</a>...</li> <li>Our <a href="https://www.collaboraoffice.com/">Collabora Office</a> + <a href="https://nextcloud.com/">NextCloud</a> instance, <a href="https://doc.oeru.org">https://doc.oeru.org</a>, which together are functionally similar to Google Drive or Dropbox + Google Docs, offering web based document storage, combined with the ability to collaboratively edit documents, spreadsheets, and presentations. Here's how to <a href="/installing-nextcloud-and-collabora-office-online-docker-ubuntu-1604">make your own</a>...</li> <li>Our <a href="http://etherpad.org/">Etherpad-Lite</a> instance, <a href="https://etherpad.oeru.org">https://etherpad.oeru.org</a>, for collaborative note taking.</li> <li>Our <a href="https://mautic.com/">Mautic</a> instance, <a href="https://mautic.oeru.org">https://mautic.oeru.org</a>, which manages and automates our email out to learners and partners, as well as  prospective partners and learners. It's an invaluable tool for magnifying the responsiveness and effectiveness of our small OER Foundation team. Replicate our approach to <a href="/installing-mautic-php7-fpm-docker-nginx-and-mariadb-ubuntu-1604">run your own</a>.</li> <li>Our <a href="https://www.limesurvey.org/">Lime Survey</a> instance, <a href="https://survey.oeru.org">https://survey.oeru.org</a>, where we manage our learner and partner surveys.</li> <li>Our <a href="https://moodle.org">Moodle</a> instance, <a href="https://moodle.oeru.org">https://moodle.oeru.org</a>, for various assessment related testing and other activities.</li> <li>Our <a href="http://wallabag.org/">Wallabag</a> instance, <a href="https://wallabag.oeru.org">https://wallabag.oeru.org</a>, a very slick shared bookmarking tool, which we're evaluating as a replacement for Semantic Scuttle.</li> <li>Our <a href="https://drupal.org">Drupal 8 CMS</a> instance, that powers this very Tech Blog!</li> <li>Our own <a href="https://bitwarden.com">BitWarden</a> instance, <a href="https://safe.oeru.org">https://safe.oeru.org</a>, as our password keeper for managing our passwords and ensuring we have <strong>strong</strong> passwords, that we can optionally share in specific organisation-level groups.</li> <li>We also run a couple of test websites built on a blogging/CMS platform called "<a href="https://getgrav.org">Grav</a>". Watch this space.</li> </ol></li> <li>containers.oerfoundation.org is our newest computing resource, based in New Zealand (made available through generous sponsorship from <a href="https://catalystcloud.nz">Catalyst Cloud</a>). It is intended to run containers hosting services that are local to us. Its current set of Docker containers runs our <a href="https://gitlab.org">Gitlab</a> instance, at <a href="https://git.oeru.org">https://git.oeru.org</a> - you can view <a href="https://git.oeru.org/explore/projects">our existing projects</a> you're welcome to use, learn from, and improve - to which we are moving all of our remote software code repositories. Any software developers are encouraged to have a look around and make use of our code! Contributions are always welcome.</li> </ol><h2>Agile and flexible</h2> <p>One of the best things about our "loosely coupled component" model is that we have the ability to quickly incorporate useful new capabilities, reinforce and update components, and readily replace weaker components with stronger ones without sentimentality as we identify them - our open source policy allows rapid evolution without the sentimentality that the <a href="https://en.wikipedia.org/wiki/Sunk_cost">sunk-cost fallacy</a> tends to create.</p> <p>In recent months, we have tested a number of new open source technologies, and introduced a few of those into our component mix, including Lime Survey, Gitlab, and BitWarden, as well as enjoyed steadily improving capabilities - features, security, privacy, and performance - by tracking community-driven updates to most of our existing components.</p> <p>On the back end, there have been even more improvements, like our move from normal Docker to <a href="https://docs.docker.com/compose/">Docker Compose</a> which has revolutionised our ability to rapidly deploy and <em>move around</em> services between hosting infrastructure.</p> <figure role="group" class="caption caption-img"><img alt="Side by side comparison of Semantic Scuttle and Wallabag." data-entity-type="file" data-entity-uuid="5a1a508e-bf3f-4932-9ec8-9437497f5d1f" height="384" src="/sites/default/files/inline-images/wallabag_and_bookmarks.png" width="714" /><figcaption>This is a side-by-side comparison of Semantic Scuttle (left) and Wallabag (right).</figcaption></figure><p>We have also been able to identify and test potentially better components for certain uses - for example,  our "Bookmarking" tool, provided by an instance of Semantic Scuttle, is not as modern as many of our other components, and its interface is decidedly dowdy by comparison. We have since found another social bookmarking tool, Wallabag, which promises to provide a more usable service, with high quality open source mobile apps available, as well as having in-built support for a variety of Single-Sign-On technologies which Semantic Scuttle lacks.</p> <h2>Capacity and scalability</h2> <p>The OERu is starting small - with tens or hundreds of learners participating in our courses. This places minimal load on our infrastructure, but allows us to validate that it is working as intended. But for many software implementations, those sorts of numbers might already be challenging the ability of normally available computing infrastructure to supply a usable service.</p> <p>One of the major advantages of our loosely coupled component model, making use of best-of-breed open source applications working together, is that each of those components is in active use in other contexts. They have all already had their "trial-by-fire", confronting the challenges of efficiently supporting large numbers of users (by large, I mean not tens or hundreds, but millions or tens of millions) and they have <em>already</em> evolved to meet those challenges.</p> <p>Although the applications we choose as components are all the products of different communities, different developers, and different technologies, all adhere to a set of well tested robust and <em>scalable</em> internet software service patterns. All have separate data stores (mostly databases, including MariaDB,  PostgreSQL, MongoDB, CouchDB, and SQLite) themselves decoupled from the containers doing the computing, usually running scripting engines (we use PHP, Ruby on Rails, Python, and Node.JS) that manipulate that data in a "stateless" way pushing and pulling data from users' browsers which employ open standards compliant HTML5 (comprising HTML markup, CSS for styling, and Javascript for in-browser client application functionality).</p> <p>That model makes it possible to scale up <em>all</em> of these services simply by adding more computing containers (which is facilitated by the use of Docker).</p> <h2>Costs</h2> <p>One way that the OERu maintains its capabilities with such a small infrastructure and development budget is that we adhere to a few key principles:</p> <ol><li>Use commodity open source hosting (we only run Linux), allowing for rapid movement between hosting providers with minimal trouble or disruption to our services.</li> <li>If using a Software-as-a-Service (SaaS) solution, strongly prefer open source options which gives us a safety valve if the pricing model/service doesn't suit our needs. (At this point, Zoom is the only proprietary software in our stack). This largely removes vendor lock-in.</li> <li>Ensure any external purchased service is fixed price and does not increase with number of users.</li> <li>Include internal maintenance time in cost of ownership.</li> </ol><p>As described above, we use four main hosting providers across the globe. All of our computing instances and dedicated machines are commodity system without proprietary features. We never exceed the (usually very generous) in-built data and storage allotments, so our prices are fixed and predictable.</p> <p>The OERu's entire annual infrastructure/IT costs can be summarised as follows (values approximate, in USD):</p> <ul><li>Four Servers: AWS ($4000) + Hetzner ($440) + Azure ($0) + Catalyst Cloud ($0) = $4440.</li> <li>SaaS: Zoom ($180) + <a href="https://kanboard.org" title="A SaaS offering built around an open source web-based Kanban application.">Kanboard</a> ($360)</li> </ul><p>Total <strong>annual</strong> software + infrastructure budget: <strong>$4800</strong></p> <p>Some of the OER Foundation's hosting infrastructure costs are covered by a "Charitable Organisation Grant" which includes $5000/year worth of Microsoft Azure hosting services. We also gratefully receive $500/month sponsored hosting services from the NZ-based hosting provider, Catalyst Cloud, who provide a fully open source cloud hosting infrastructure, and like the work the OERu is doing!</p> <h3>Commodity</h3> <p>We run Linux on all of them (Ubuntu or Debian) which is available at no cost. We can run as many or as few systems as we want at a fixed cost. Only the sunk cost of my time and the relative computing resource requirements are variable - but these vary in far less than a linear fashion with user numbers, e.g. for 10 times more users, we might require only 10% more of my time, and maybe 50% more computing resources.</p> <h3>Case Study: SaaS and the value of open source</h3> <p>At last year's Partners' meeting, we described Mautic, an impressive new open source "marketing automation" tool we were using to automate much of our email communication with learners and partners as well as prospective learners and partners.</p> <p>Initially, to test it, we opted to use the $30/month SaaS "entry-level" offering from Mautic.com which allowed us a single login to get started right away and test the Mautic system's fit to our requirements. It allowed us up to 2000 contacts, with an increased cost beyond that.</p> <p>Shortly after last partner meeting, however, the folks at Mautic.com contacted us to say that their pricing model was changing and that our new costs would go up by more than 10 times - $500/month, and with a much steeper increase for new contacts. For example, 10k contacts would cost us more like $1000/month.</p> <figure role="group" class="caption caption-img"><img alt="Mautic SaaS vs. Self-Hosted" data-entity-type="file" data-entity-uuid="e63e203a-1b37-40d9-bb70-804f7bdfc6c0" height="273" src="/sites/default/files/inline-images/MauticCostPlots.png" width="622" /><figcaption>Relative cost models of the hosted (SaaS) Mautic service vs. the self-hosted OERu Mautic instance.</figcaption></figure><p>Given the substantial price rise and unpleasant "scaling factor" for new contacts, combined with the fact that we found Mautic a very useful piece of software, we were in an uncomfortable position. Thankfully the Mautic platform itself is open source software. So <em>unlike all proprietary SaaS offerings </em>we had the option of hosting the service ourselves.</p> <p>In the course of a day or two of my time invested into creating an implementation on our hosting infrastructure (which <a href="/installing-mautic-php7-fpm-docker-nginx-and-mariadb-ubuntu-1604">I have documented</a>), we had <a href="https://mautic.oeru.org">https://mautic.oeru.org</a> up and running.</p> <p>Our new cost profile for Mautic is far more favourable. I spend perhaps an hour or so every month or two to update the Mautic system (it's improving continually thanks to the efforts of its Mautic.com-led developer community - which now includes the OERu!). Mautic places a negligible additional load on our AU-based infrastructure, but I have assigned it a small proportion of the (donated) hosting cost, a bit of my time, and a negligible amount for outgoing email costs (the 20k or so emails our Mautic has sent so far using the AWS "Simple Email Service" commodity SaaS have cost us a whopping $0.50 so far).</p> <p>Overall, the per-month cost-comparison for 10000 contacts looks like its about $1000 vs. $80, or an annual savings of approximately $11,040 off a total of $12,000. Massive 92% savings, <em>amounting to twice our total infrastructure budget</em>.</p> <p>And the savings only increases as we grow as we full intend to do! If, for example, we ended up with 100k contacts in the next year, the SaaS cost would be more like $4000/month. For 100k contacts, our self hosted system's costs might increase to $120/month (due to increased infrastructure resources being used)... The resulting savings would then be $46,560 off a total of $48,000 - a non-trivial savings of 97% or <em>almost 10 times our total infrastructure budget</em>!</p> <p>More importantly, the absolute cost of our environment is a tiny fraction, perhaps 5-10%, of that represented by a managed, outsourced SaaS offering. The open source self-hosted approach we've been able to validate represents a huge opportunity, particularly for higher education in emerging economies.</p> <h2>Progress since last year</h2> <p>In addition to building and maintaining the above open source software stack, our small team on occasionally identifies opportunities act decisively and strategically, to write our own code that bridges gaps or improves the usability of components in our NGDLE. Since the <a href="https://course.oeru.org/2017-meetings/">last OERu partner meeting</a>, we've rolled up our sleeves to create the <a href="/oeru-blog-feed-finder">Blog Feed Finder</a>, and we're about to release our new Registration and Enrolment plugin for WordPress MultiSite implementations (still a <a href="https://git.oeru.org/oeru/register-enrol">work-in-progress</a>).</p> <p>We have lots of other strategic initiatives stacked up and waiting to receive our full attention. The main constraint on that progress is our capacity, namely me (as the OER Foundation's <a href="/insight-what-does-open-source-technologist-oer-foundation-do">Open Source Technologist</a>). Mine is a very gratifying role, but a bit daunting at times - so much to learn and do!</p> <h2>What this means for OERu Partners</h2> <p>So, what should our OERu partners take from this description of the OERu's NGDLE? I think the most important thing is this: the status quo for higher learning institution IT conventions is neither the only way to do things, nor is it the best way.</p> <p>Because we are unbound to convention or historical decisions, we at the OERu are able to pioneer new approaches, from a "technology expert" perspective. We are driven by open principles and very tight resource constraints, but we also need fulfil our vision: to build a rich, fit-for-purpose infrastructure for learners and OER collaborators alike, which has the potential to scale to unprecedented global learner volumes.</p> <p>Implementing an open source end-to-end service gives the OERu a unique perspective and experience compared to organisations who only implement the occasional open source component in the midst of IT infrastructure dominated by proprietary software that is costly and extremely restrictive by comparison.</p> <p>We are also building (anonymous) monitoring systems into these services to ensure we can measure our success (without impinging on the privacy of our learners or collaborators). The insight we gain may well be of value to our partners.</p> <h3>Return on (open source technology) Investment</h3> <p>Return on Investment (ROI) can be achieved in a number of ways: investment can increase productivity and therefore value created (usually measured as <em>profit</em>) <em>or</em> it can be achieved by reducing costs. The best ROI achieves a combination of both.</p> <p>Our OERu partners can safely appraise our NGDLE and pick and choose from among the many (always improving) best-of-breed open source components that might fill an emerging need within their own institutional context. We can provide assistance and demonstration instances to help them initiate pilot implementations and work with local advocates and experts to provide ongoing support.</p> <p>Any partner interested in the capabilities of technologies we have included in our NGDLE has the potential to save their institutions many times the cost of their OERu partner membership if they adopt any one of them - along the lines of the Mautic case study above... And, of course, that savings increases with the number of technologies adopted. </p> <p> </p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <h2 class="comment-field__title">Blog comments</h2> <a id="comment-27"></a> <article data-comment-user-id="0" about="/comment/27" typeof="schema:Comment" class="comment js-comment by-anonymous has-title clearfix"> <div class="comment__container"> <h3 property="schema:name" datatype="" class="comment__title"> <a href="/comment/27#comment-27" class="permalink" rel="bookmark" hreflang="en">Swapping out the last proprietary layer of the stack</a> <span class="comment__new marker marker--success hidden" data-comment-timestamp="1546981578"></span> </h3> <div class="comment__meta"> <div class="comment__submitted"> <span class="comment__author"><span rel="schema:author"><span lang="" typeof="schema:Person" property="schema:name" datatype="">Danyl Strype (not verified)</span></span> </span> <span class="comment__pubdate">Fri 21/12/2018 - 07:34 <span property="schema:dateCreated" content="2018-12-20T18:34:52+00:00" class="rdf-meta hidden"></span> </span> </div> </div> <div class="comment__content"> <div property="schema:text" class="clearfix text-formatted field field-comment--comment-body field-name-comment-body field-type-text-long field-label-hidden"> <div class="field__items"> <div property="schema:text" class="field__item"><p>This is a very impressive accomplishment Dave. The only thing that wrinkles my brow is that a significant chunk of OERU spending is going to the very corporate datafarms we&#039;re trying to replace (Microsoft and Amazon), by developing software and services that respects their users&#039; software freedom. If we are swapping from gratis services managed by the datafarms, to taking on management work ourselves while starting to pay them for hosting, is this actually progress? They still make all the money, but now we pay them for doing some of their work for them ;-P</p> <p>I&#039;m much happier to see OERU money going to Catalyst. As you know, having worked for them, they are not and have never been datafarmers. They have been involved in free code development and deployment in both the commercial and governmental sectors in Aotearoa, and huge supporters of the libre commons communities in the country (including sponsoring and organizing the NZ Open Source Awards). It would be great to read your next annual report, and see that all the OERU infrastructure hosted on the datafarms has been moved to companies like Catalyst. It&#039;s the final piece of the puzzle :)</p> </div> </div> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=27&amp;1=default&amp;2=en&amp;3=" token="otqyAq8w5Pd0NDlWnsqJTdnPUTbkZ3076qwkZ7AXGc8"></drupal-render-placeholder> </div> </div> </article> <div class="indented"><a id="comment-34"></a> <article data-comment-user-id="1" about="/comment/34" typeof="schema:Comment" class="comment js-comment by-node-author has-title clearfix"> <div class="comment__container"> <h3 property="schema:name" datatype="" class="comment__title"> <a href="/comment/34#comment-34" class="permalink" rel="bookmark" hreflang="en">Fair comment, Danyl. Yes, we…</a> <span class="comment__new marker marker--success hidden" data-comment-timestamp="1546981964"></span> </h3> <div class="comment__meta comment__meta--has-user-picture"> <a href="/blog/1">View recent blog entries</a> <div class="comment__submitted"> <span class="comment__author"><span rel="schema:author"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> </span> <span class="comment__pubdate">Wed 09/01/2019 - 10:12 <span property="schema:dateCreated" content="2019-01-08T21:12:44+00:00" class="rdf-meta hidden"></span> </span> </div> </div> <div class="comment__content"> <p class="comment__parent visually-hidden">In reply to <a href="/comment/27#comment-27" class="permalink" rel="bookmark" hreflang="en">Swapping out the last proprietary layer of the stack</a> by <span lang="" typeof="schema:Person" property="schema:name" datatype="">Danyl Strype (not verified)</span></p> <div property="schema:text" class="clearfix text-formatted field field-comment--comment-body field-name-comment-body field-type-text-long field-label-hidden"> <div class="field__items"> <div property="schema:text" class="field__item"><p>Fair comment, Danyl. Yes, we agonised over the use of Microsoft's Azure cloud quite a bit... we decided in the end that, given our use of their service is within their USD5000 annual charitable organisation grant (so we're taxing their services rather than funding them) and our policy of actively avoiding the use of any service-specific proprietary tools (i.e. we treat all our hosting services as commodity services), we're effectively exploiting their efforts to lock organsations into their offering (and convert them to paying customers) and using them to promote their antithesis, namely fully open, community focused services... </p> <p>The use of Amazon is somewhat less contentious (given that Amazon hasn't historically been actively hostile to FOSS like Microsoft has been for most of its history), and although we're currently paying for their service (albeit not very much) we're similarly treating their services as commodity, so that <em>the instant</em> we're able to identify an alternative service that's fit for our purpose but is better aligned with our principle, we can move there toot-sweet! :)</p> <p>Keep on keeping us honest!</p> </div> </div> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=34&amp;1=default&amp;2=en&amp;3=" token="5Ndj6-NY2Rp4CIlcOJ9Py-2-MLvBNF34yZ1q6Vjwifg"></drupal-render-placeholder> </div> </div> </article> </div> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=23&amp;2=field_blog_comments&amp;3=comment" token="2zE34of-pPoUy4YnQR_I3RaAyQnePTxGQyL0a8q2djg"></drupal-render-placeholder> </div> </section> Tue, 30 Oct 2018 21:24:12 +0000 dave 23 at http://tech.oeru.org http://tech.oeru.org/2018-update-oeru-technology-stack#comments The OERu Blog Feed Finder http://tech.oeru.org/oeru-blog-feed-finder <span class="field field--name-title field--type-string field--label-hidden">The OERu Blog Feed Finder</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--lida101"> <span class="field__item-wrapper"><a href="/taxonomy/term/47" hreflang="en">lida101</a></span> </div> <div class="field__item field__item--wenotes"> <span class="field__item-wrapper"><a href="/taxonomy/term/41" hreflang="en">wenotes</a></span> </div> <div class="field__item field__item--blog-feed-finder"> <span class="field__item-wrapper"><a href="/taxonomy/term/53" hreflang="en">blog feed finder</a></span> </div> <div class="field__item field__item--free--open-source"> <span class="field__item-wrapper"><a href="/taxonomy/term/6" hreflang="en">free &amp; open source</a></span> </div> <div class="field__item field__item--rss"> <span class="field__item-wrapper"><a href="/taxonomy/term/54" hreflang="en">rss</a></span> </div> <div class="field__item field__item--atom"> <span class="field__item-wrapper"><a href="/taxonomy/term/55" hreflang="en">atom</a></span> </div> <div class="field__item field__item--json"> <span class="field__item-wrapper"><a href="/taxonomy/term/56" hreflang="en">json</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Mon 15/10/2018 - 11:40</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>One of the <a href="/many-simple-tools-loosely-coupled">distributed tools</a> that the <a href="https://oeru.org">OERu</a> makes available as part of its open source distributed digital learning environment is WEnotes. I've <a href="/wikieducator-notes-oerus-course-feed-aggregation-and-messaging-system">described what it is and how it works previously</a>. One of the most powerful distributed tools has also been the most complicated to achieve: the ability for learners to complete assignments and reflect on their learning journeys on their own blogs, tagging posts with relevant OERu course ids (e.g. <em>lida101</em> - see the <a href="https://course.oeru.org/lida101/interactions/course-feed/">Learning in a Digital Age 101 course feed</a> for example) and then having our blog feed scanner:</p> <ol><li>know to look at the learner's blog in a timely manner,</li> <li>identify the newly published appropriate blog post (i.e. with the relevant tag), and</li> <li>create a WEnotes message with a reference to the blog post</li> </ol><p>for the other learners participating in the course to see.</p> <p>There are a lot of moving parts in achieving this ambition, but the most unexpectedly difficult (from my perspective as a technologist) was solving the first problem: knowing how to find the learner blog feeds to monitor!</p> <h2>The mysterious blog feed</h2> <p>It turns out few people who use the web, even those who blog, are familiar with "feeds". A "<a href="https://en.wikipedia.org/wiki/Web_feed">feed</a>" is a web accessible file, adhering to a well-known and publicly defined "open standard" <em>machine readable </em>format, which summarises <em>and references</em> (links to), the content on a website, in this case a blog. Most blog platforms enable them right out-of-the-box, by default. This fact isn't widely appreciated, even among active bloggers!</p> <p>If you haven't seen a feed, here're some examples and an explanation of the two most common types: <a href="https://www.w3schools.com/XML/xml_rss.asp">RSS</a> (which stands for "Really Simple Syndication") and <a href="https://en.wikipedia.org/wiki/Atom_(Web_standard)">Atom</a>, both of which support "tagging" content.</p> <p>When we (here at the OER Foundation) first looked into offering the monitoring of learner (or educator) blogs for suitably tagged posts to include in course interaction feeds, I thought it would be a simple process of allowing learners to nominate a "Blog Feed URL" when they registered an account on our <a href="https://course.oeru.org" title="The OERu Course Site">Course Site</a>. I was quite mistaken.</p> <p>Over the course of six or so months after we made a "Blog Feed URL" field available in learner profiles, entered during the registration process, we had perhaps a hundred or so learners nominate a URL. (N.B. our model allows a learner to nominate a different blog feed URL for each course in which they're participating, allowing for the possibility that they might create a new blog site for each course. Naturally, they can nominate the <em>same</em> blog URL for multiple courses) Of those, perhaps one in 50 was an actual valid blog <em>feed</em> URL. Most were not even valid blog addresses. The most common entries were either facebook.com, google.com, a wikieducator.org user profile, or the course's own url - none of which are blogs, much less blog feeds. So this was useful market research: the market does not, in general, know what a blog is or what its feed might be, if it has one.</p> <p>Among other problems this creates is the fact that our automated feed monitoring code could not effectively sift the wheat from the chaff to find <em>valid</em> blog feeds to monitor. Clearly, a different approach was required.</p> <h2>Back to the drawing board</h2> <p>Turns out that if a learner<em> has a blog</em>, they generally know its web address. If provided clear instructions, they can fairly reliably go to it, and copy and paste the URL into a text field. So I decided to use this as a starting point for creating our "Blog Feed Finder" - a simple tool that helps us side step the issue of requiring a learner to find their blog feed URL - by finding it for them. Our approach was to provide a clear, well documented interface that embraces "<a href="http://www.instructionaldesign.org/theories/elaboration-theory/" title="A summary of elaboration theory">elaboration theory</a>", providing enough information at a glance to guide a learner, but offers further information to those who want to understand more.</p> <p>Here's what it looks like:</p> <figure role="group" class="caption caption-img"><img alt="The BFF with no blog URL entered yet. " data-entity-type="file" data-entity-uuid="b6e1d29c-ed8b-44bb-8842-e37d432f81e1" src="/sites/default/files/inline-images/bff_default_0.png" /><figcaption>Here's the default view of the BFF - you can enter your blog URL as per the instructions, or seek further instruction if you're not sure how to do that.</figcaption></figure><p>I'll copy in the URL of this very blog (and I won't even bother with the "https://" that appears at the start of it):</p> <figure role="group" class="caption caption-img"><img alt="BFF with block URL entered... " data-entity-type="file" data-entity-uuid="52be95bc-c609-4e8f-b966-04bbd4d8a2a1" src="/sites/default/files/inline-images/bff_with_blog_url_0.png" /><figcaption>The URL is pasted in. Now push the "Submit" button...</figcaption></figure><p>After a few seconds of behind-the-scenes whirring and grinding...</p> <figure role="group" class="caption caption-img"><img alt="BFF with an initial result." data-entity-type="file" data-entity-uuid="57caa975-4761-4369-bfba-71b2bde08802" src="/sites/default/files/inline-images/bff_with_result.png" /><figcaption>We have a result from the BFF. Note that the BFF also determines the type of feed, typically RSS or Atom. Some blog platforms offer both, in which case the BFF lets you choose which one to use.</figcaption></figure><p>If we want to know more about what the BFF had to do to find that blog feed, we can review its output, and even ask for more information by hovering over the info icons, fo example:</p> <figure role="group" class="caption caption-img"><img alt="Showing further elaboration of the blog feed finding process." data-entity-type="file" data-entity-uuid="da32b127-3e68-418a-8e1a-3ad8360fa218" src="/sites/default/files/inline-images/bff_with_result_and_elaboration_0.png" /><figcaption>Showing an elaborating pop-up, triggered by hovering over (or clicking on) an information icon.</figcaption></figure><p>And having found a feed, we can now update any of the courses for which we're registered to use that feed as the designated "feed to scan periodically" for blog posts tagged with the relevant course code (in brackets next to each course title, e.g. analyticsdev, lida101, mun).</p> <p>Here's what it looks like after I've replaced <em>all</em> of my blog feed associations with the new feed URL:</p> <figure role="group" class="caption caption-img"><img alt="After 'replacing' the feed URLs previously associated with my courses." data-entity-type="file" data-entity-uuid="a66ddf80-ec3b-46cf-b550-9dd4b4e1f83e" src="/sites/default/files/inline-images/bff_with_result_after_updating_all.png" /><figcaption>This is what the interface looks like after I've elected to "Replace" each of my blog feed URLs with the newly found URL.</figcaption></figure><p>And, what if, instead of tech.oeru.org, I'd mistakenly entered a URL like one of those I mentioned above? Here's how the BFF responds if someone enters, say, Facebook.com as their blog URL:</p> <figure role="group" class="caption caption-img"><img alt="BFF with the 'common accidental' blog URL message... " data-entity-type="file" data-entity-uuid="e699e2b2-9a83-4c58-bddf-a6c81ce0b1d3" src="/sites/default/files/inline-images/bff_with_common_accident_message_0.png" /><figcaption>This is what the BFF says to a learner if they mistakenly enter one of a handful of frequently entered non-blog URLs like facebook.com, google.com, or even course.oeru.org.</figcaption></figure><h2>Try it now, or adapt it! It's free (and open source)</h2> <p>Of course, you can <a href="https://course.oeru.org/blog-feed-finder" title="The OERu's Block Feed Finder. It's all open source.">try it yourself right now</a> without even needing to log in, although you get additional functionality if you're logged in an enrolled in one or more courses, like the ability to assign blog feed URLs to any courses in which you're enrolled.</p> <p>Even though we're only on our first iteration of it (any software developer will tell you: no software application is ever <em>finished</em>) our Blog Feed Finder (BFF for short) has also been <a href="http://cogdogblog.com/2018/06/better-magic-box/">assessed by others in the online learning realm</a>, who have given it a good going over and their seal of approval.</p> <p>If anyone is curious about how it works, the <a href="https://github.com/oeru/blog-feed-finder">entire source code is available</a> under the terms of the <a href="https://www.gnu.org/licenses/agpl-3.0.en.html">GNU Affero General Public License</a> (same as what's used by the Linux kernel, WordPress, Drupal, MediaWiki, and thousands of other well know free and open source projects) to make its review and reuse convenient and reliable!</p> <h2>OERu Blog Scanner is Go!</h2> <p>As of a few weeks ago, following the introduction of the BFF (and a bit of manual tidy-up of previously entered blog feed URL where OERu administrators used the BFF to find legitimate feed URLs where-ever possible) we now have a working scan which periodically checks all registered learner blog feeds and, if a suitably tagged post is found on any of them, incorporates a link to the post in the relevant course feed!</p> <p>The <a href="https://course.oeru.org/lida101/interactions/course-feed/">course interaction feed</a> (also linked above) includes some posts with a "blog" designator, which indicates they are references to blog posts which have been scanned by our WEnotes blog scanner (which is a different piece of software).</p> <h2>Loose Ends</h2> <p>As with any first iteration software release, there are quite a few known issues (a whole itemised, prioritised list, actually) we'd like to address for the next release. The first among them in the BFF's case is making the interface properly "responsive" (i.e. friendly for mobile device users)... At present, it doesn't do very well at that.</p> <p>If you have a go with the BFF and run into trouble related to the software itself, we encourage you to let us know in <a href="https://github.com/oeru/blog-feed-finder/issues">the project's Issue Queue</a>. </p> <p>Final note: we will be moving our source code repositories to a properly open source code repository (our own <a href="https://gitlab.com" title="Gitlab is an open source alternative to Github, which is not open source.">Gitlab</a> instance) in the coming weeks, so you may find you're being redirected from Github to a different site. Please don't be alarmed! It's by design.</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=22&amp;2=field_blog_comments&amp;3=comment" token="DjJy08iCX70uHh392eLxPjDOG3f9SoByCElo9m3CGxs"></drupal-render-placeholder> </div> </section> Sun, 14 Oct 2018 22:40:06 +0000 dave 22 at http://tech.oeru.org http://tech.oeru.org/oeru-blog-feed-finder#comments What is the OERu? - Pizza Thursday talk for Catalyst Christchurch http://tech.oeru.org/what-oeru-pizza-thursday-talk-catalyst-christchurch <span class="field field--name-title field--type-string field--label-hidden">What is the OERu? - Pizza Thursday talk for Catalyst Christchurch</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--lida101"> <span class="field__item-wrapper"><a href="/taxonomy/term/47" hreflang="en">lida101</a></span> </div> <div class="field__item field__item--oeru"> <span class="field__item-wrapper"><a href="/taxonomy/term/46" hreflang="en">oeru</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Thu 23/08/2018 - 09:31</span> <div class="float-none field field-node--field-image field-name-field-image field-type-image field-label-hidden has-single"> <figure class="field-type-image__figure image-count-1 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-08/OERu_technology_wheel.png?itok=SujshCxP" title="The wheel of interactive technologies OERu learners can work with that will be recruited into their course feed." data-colorbox-gallery="gallery-field_image-rUaT7HdTu4I" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;The wheel of interactive technologies OERu learners can work with that will be recruited into their course feed.&quot;}"><img src="/sites/default/files/styles/medium/public/2018-08/OERu_technology_wheel.png?itok=c-VCQ8aT" width="220" height="152" alt="The wheel of interactive technologies OERu learners can work with that will be recruited into their course feed." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> </div> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>I've been asked to speak about the OERu to my former colleagues at <a href="https://catalyst.net.nz" title="NZ's premier open source development and cloud hosting company">Catalyst IT</a>'s Christchurch office as part of their monthly "Pizza Thursday" talks. Amusingly, this talk will be on a Friday, and lunch will be curries rather than pizza :)</p> <h2>What is the OERu?</h2> <p>Last week, an <a href="https://opensource.com/article/18/8/oeru-courses">article and interview discussing exactly that</a> was published on <a href="https://opensource.com">OpenSource.com</a> which examined both the OERu and its relationship to free and open source software, which is core to the organisation's ethos. We're a "radically open" organisation... our terms of reference state that we</p> <ul><li>use exclusively openly licensed educational resources (OERs), i.e. written materials and digital artefacts that conform with the "<a href="https://freedomdefined.org/Definition">free cultural works</a>" criteria.</li> <li>we only use free and open source software for <em>everything</em> where-ever possible</li> <li>all our planning processes are conducted openly and with external participation encouraged (where-ever possible, they're streamed live and <a href="https://www.youtube.com/channel/UCCMLj7wLhiIYq_CH84D_TPg">recorded for posterity</a>, so others can see how we arrived at strategic decisions)</li> </ul><p>I work in the role of "<a href="/insight-what-does-open-source-technologist-oer-foundation-do">Open Source Technologist</a>" for the OER Foundation (a Dunedin-based charitable foundation, which in turn, is owned and formally hosted by the Otago Polytech, my official employer), who facilitate and enable the OER universitas (OERu), the global network of <a href="https://oeru.org/oeru-partners/">higher education institutions and related organsations</a> (like donors and sponsors), who are working together to meet the OERu's strategic goals. </p> <p>This is how we explain the OERu to prospective partners:</p> <p><iframe allow="autoplay; encrypted-media" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/Ia3g2qh-qjQ" width="560"></iframe></p> <p>Of course, as an educational provider, the OERu cannot make a lot of progress without "learners" (they're not "students" because that implies a contractual relationship with an institution - people choosing to learn with us can do so anonymously, without any formal relationship, so we've adopted the term "learners"). This is what the OERu looks like from a learner's perspective:<br /><iframe allow="autoplay; encrypted-media" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/F4q3CjOQARI" width="560"></iframe></p> <p>Earlier this year, we launched our "MVP", a "1st year of study" - a sequence of courses a learner can take (equivalent to a first year of undergraduate study) which, if they choose to be assessed (and meet the assessment criteria) can lead to a formal exit qualification.</p> <p>Our current "flagship" course is <a href="https://oeru.org/learning-in-a-digital-age/">Learning in a Digital Age</a>, a series of four "microcourses" which can be undertaking in a sequence or individually:</p> <ol><li><a href="https://oeru.org/oeru-partners/otago-polytechnic/digital-literacies-for-online-learning/">Digital literacies for online learning</a> (LiDA101)</li> <li><a href="https://oeru.org/oeru-partners/otago-polytechnic/digital-citizenship/">Digital citizenship</a> (LiDA102)</li> <li><a href="https://oeru.org/oeru-partners/otago-polytechnic/open-education-copyright-and-open-licensing-in-a-digital-world/">Open education, copyright and open licensing in a digital world</a> (LiDA103)</li> <li><a href="https://oeru.org/oeru-partners/otago-polytechnic/critical-media-literacies-and-associated-digital-skills/">Critical media literacies and associated digital skills</a> (LiDA104)</li> </ol><p>So, as a learner, what does a course look like? Let's have a look at <a href="https://course.oeru.org/lida101" title="Digital literacies for online learning">LiDA101</a>...</p> <p>What you see is it's a WordPress site holding all the course materials. These materials are not edited or altered by course designers within WordPress. Instead, they're created as a course outline on our collaborative OER assembly platform, <a href="https://wikieducator.org">WikiEducator</a>, an instance of the venerable MediaWiki platform (yes, it's high time for a facelift - never fear, in the works). Here's the <a href="https://wikieducator.org/Learning_in_a_digital_age/_Outline_LiDA101">course outline</a> used to build the LiDA101 course.</p> <p>You can see that the course outline is just a selection of content from elsewhere on the site, including reference materials, media objects (videos, images, audio), quizzes and assignments, and assessment rubrics. We use MediaWiki as the collaboration platform thanks to the (relatively) user-friendly and flexible version control of content it provides. (n.b. the WikiEducator site, one of the OER Foundation's main initiatives, the other being the OERu, is very heavily used by educators around the world, and <a href="https://www.alexa.com/siteinfo/wikieducator.org">hovers near the 100k sites on the web</a>).</p> <p>You'll notice the "Snapshot" button at the top of the outline - behind that button is a one line specification for a target WordPress site - clicking that button allows someone with "administrator" permissions on that site (or "sub site") to automatically push the course materials contained in that outline to the chosen WordPress site - it will be formatted as you see, with the selected branding (e.g. for our partner institutions), <em>particularly designed for use with mobile devices</em> (although that could use some refinement) which is our largest potential audience.</p> <p>Learners can access all course materials anonymously, however, if they want to <a href="https://wikieducator.org/File:OERu_technology_wheel.svg">interact with one another, and have their participation recorded</a>, they must create an account on the site.</p> <h2>OERu's FOSS Tech Stack</h2> <p>The technology stack on which the OERu is built has a few parts. We have created what we refer to as a "next generation digital learning environment" made up of a number of carefully selected free and open source tools all "<a href="/many-simple-tools-loosely-coupled">loosely coupled</a>" together (the "UNIX philosophy"), with some strategic custom glue...</p> <ul><li>educator collaboration tools (Discourse, Rocket.Chat, OnlineGroups, Jitsi Meet)</li> <li>educator collaborative course assembly tools (MediaWiki, NextCloud/CollaboraOffice, Kanboard)</li> <li>market-automation, learner instruction email tools, market research (Mautic, LimeSurvey)</li> <li>learner course sites (WordPress)</li> <li>learner assessment tools (Moodle)</li> <li>learner interaction (Mastodon, Discourse, Semantic Scuttle, Hypothes.is, other social media, learner blogs)</li> <li>learner <a href="https://course.oeru.org/lida101/interactions/course-feed/" title="An example of a course feed">interaction feed</a> (<a href="/wikieducator-notes-oerus-course-feed-aggregation-and-messaging-system">WEnotes</a>: Node.Js + Python + CouchDB + MediaWiki extensions + WordPress plugins)</li> <li>organisational info platforms (MediaWiki, Silverstripe, Drupal, Grav)</li> <li>reporting/promotional presentations (Reveal.JS)</li> <li>analytics (Matomo)</li> <li>infrastructure (Ubuntu/Debian Linux VMs/Instances, Git, Docker, Docker-Compose)</li> </ul><p>We use this website (tech.oeru.org) as a place to explain how we do what we do, and why. We include how-tos for many of the technologies we use in hopes that more organisations will start to use them inspired by our example - our underlying "enlightened self-interest" culture sees that we all fare better when we have an interest in making these tools the best they can be...</p> <h2>Why I do this</h2> <p>In my career as a technologist, I've been very aware of (and involved in activism for improving) the many problems faced by our society (and others around the world). There's one compelling, effective, and unequivocally <em>good</em> thing I've identified as a key component to all the solutions to our social ills:<strong> education</strong>. By that, I mean education (which depends on aptitude, motivation, and opportunity) from literacy to higher learning, across all of social strata, particularly for girls and women.</p> <p>As we in the OERu network build a platform for low cost higher education, we hope to create something that <em>anyone</em> can build upon, and scale up to meet the latent demand for education (globally, fewer than 10% of people ever have the opportunity to complete a tertiary qualification), providing people the <em>realistic</em> opportunity to reach their individual educational potentials. That opportunity depends only on people having access to suitable technology and Internet connectivity, both of which are spreading across the world faster than anyone could ever have imagined.</p> <p>The fact that I can do something I enjoy, have real passion for, and derive satisfaction from because it's the "right thing to do for our world" - also known as "enlightened self-interest" - with people I respect and admire (and get on with really well), means that I have found my calling and consider myself incredibly fortunate.</p> <h2>Demos</h2> <p>Time permitting, I'd be keen to show you a few of the tech things:</p> <ol><li>Demo of the WEnotes <a href="https://course.oeru.org/lida101/interactions/course-feed/">Course Feed </a>(git repos: <a href="https://bitbucket.org/wikieducator/wenotes-tools/src" title="The tools">tools</a>, <a href="https://github.com/oeru/wenotes" title="WordPress plugin and MediaWiki extension">plugins</a>, <a href="https://github.com/oeru/wenotes-docker">deployment</a>) <ol><li>The <a href="https://wikieducator.org/WENotesCompleteFeed">full noise</a></li> <li>A look at Hypothesis annotations</li> <li>A look at Mastodon</li> </ol></li> <li>A look at our <a href="https://wpms.oeru.org/blog-feed-finder">Blog Feed Finder</a> (git <a href="https://github.com/oeru/blog-feed-finder">repo</a>) which helps learners who want to submit their personal blog for scanning (to include suitably "tagged" posts in their course feed) actually find the feed and associate it with their course so our WEnotes scanner periodically checks it for new posts of interest.</li> <li>A tour of our <a href="https://mautic.oeru.org">Mautic</a> and our approach to keeping learners and partners in the loop!</li> </ol><p>I reckon many of these tools are the sort of thing Catalyst could offer as managed services commercially.</p> <p>N.B. we're not happy with Microsoft acquiring Github, and will be moving our repos from Github (and Bitbucket, too, while we're at it, because we're not overly excited to be using proprietary repos in general) to our own free and open source Git hosting as soon as possible. We will leave pointers to the new locations for these (and our many other) repos when they are created.</p> <p> </p> <p> </p> <p> </p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=21&amp;2=field_blog_comments&amp;3=comment" token="99KWcafILHKzzimelPVVtuDa5XZ3jjZyJ2-reyAbBAI"></drupal-render-placeholder> </div> </section> Wed, 22 Aug 2018 21:31:44 +0000 dave 21 at http://tech.oeru.org http://tech.oeru.org/what-oeru-pizza-thursday-talk-catalyst-christchurch#comments Creating Simple, Semantic HTML Markup from a Google Doc http://tech.oeru.org/creating-simple-semantic-html-markup-google-doc <span class="field field--name-title field--type-string field--label-hidden">Creating Simple, Semantic HTML Markup from a Google Doc</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Wed 23/05/2018 - 12:55</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>Google Docs are great for allowing people to collaboratively build a document, with the ability for people to suggest (and discuss) changes and view revisions and a variety of other useful behaviours. At present, I'm sad to admit, <a href="/installing-nextcloud-and-collabora-office-online-docker-ubuntu-1604">CollaboraOffice</a> isn't quite in the same level (although it's catching up quickly!).</p> <h2>The Problem</h2> <p>A big problem with Google Docs (shared, incidentally, with Microsoft Word and even LibreOffice/CollaboraOffice) is that it's almost impossible to convert a collaboratively created document into a form suitable for publishing on the web.</p> <p>Yes, you can go to File-&gt;Download as...-&gt;Web Page (html, zipped)... but that results in an HTML document that is chock full of badness. Google Docs (and the other word processing packages) all assume you want your document to render on the web <em>exactly</em> like it does in their app. In my experience, is<strong> impossible to get clean, semantic HTML out of any word processing platform</strong> without all the nasty in-line styling and formatting kruft that these applications assume you want.</p> <p>Anyone (like me) who's had to work with non-technical people wanting to manage their own websites, via a content management system (CMS) like Wordpress or Drupal or any one of hundreds of others, can attest to the fact that the average person's first tool for creating content is always their word processor. This, of course, is probably the least appropriate tool, because it</p> <ol><li>creates an expectation of  control and formatting that is invalid on the web, and</li> <li>tries to control the content that is copied and pasted into the CMS' editing interface, which is entirely counter-productive.</li> </ol><p>All you really want is very simple, semantically marked-up content, <em>without any styling whatsoever</em>. The styling for CMS content is typically applied by the theme, rather than the editor. The purpose behind that is to ensure that all the site's content is <em>consisten</em>t in its look-and-feel and <em>the whole lot can be changed with simple tweaks to that theme. </em>Problem is, most people neither understand that nor care - in fact, few people will even know what I mean by "semantically marked-up": simply put, it's the idea that the content is simply marked up to show what type of content it is, e.g. a title, a paragraph, a list, a table, something requiring emphasis (usually achieved by <em>italics</em>) or strong presentation (usually achieved by <strong>bold</strong>).</p> <h2>An Illustrated Example</h2> <p>Here's an example of a document</p> <figure role="group" class="caption caption-img align-center"><img alt="A screenshot of a Google Document used in this example." data-entity-type="file" data-entity-uuid="0a945b14-ffa1-45ae-aea7-b5685e3d6a36" src="/sites/default/files/inline-images/ScreenshotGoogleDocsSample.png" /><figcaption>Screenshot of a Google Doc containing some text with basic formatting.</figcaption></figure><p>Here's the simple, clean semantic mark-up we <em>want</em> to represent this content:</p> <p><blockcode style="html"><code>&lt;h1&gt;Privacy notice&lt;/h1&gt;<br /> &lt;p&gt;The Open Education Resource universitas (OERu) privacy notice provides a simple and concise summary to explain our treatment of personal information from OERu learners and users of the websites hosted by the OER Foundation (OERF).&lt;/p&gt;<br /> &lt;p&gt;This privacy notice complies with our Privacy Policy and Terms of Service Policy and applies to you when you decide to use our services.&lt;/p&gt;<br /> &lt;p&gt;We collect information when users visit websites hosted by the OERF. This includes IP addresses which are unique numbers that identify specific internet connections, and sometimes specific computers and devices that visit our sites.&lt;/p&gt;<br /> &lt;h2&gt;Definitions&lt;/h2&gt;<br /> &lt;p&gt;In the context of this policy&lt;/p&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;&lt;p&gt;Personal data refers to information related to an identified or identifiable natural person used for data processing by the OERF.&lt;/p&gt;&lt;/li&gt;<br /> &lt;/ul&gt; </code></blockcode></p> <p>Here's the mark-up we <em>get</em> if we copy and paste from Google Docs (you'll get similar results copy-and-pasting from any word processor):</p> <p><blockcode style="html"><code>&lt;p&gt;&amp;nbsp;&lt;/p&gt;<br /><br /> &lt;h1 dir="ltr" style="line-height:1.38;margin-top:20pt;margin-bottom:6pt;"&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:20pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;Privacy notice&lt;/span&gt;&lt;/b&gt;&lt;/h1&gt;<br /><br /> &lt;p&gt;&amp;nbsp;&lt;/p&gt;<br /><br /> &lt;p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;The Open Education Resource universitas (OERu) privacy notice provides a simple and concise summary to explain our treatment of personal information from OERu learners and users of the websites hosted by the OER Foundation (OERF).&lt;/span&gt;&lt;/b&gt;&lt;/p&gt;<br /><br /> &lt;p&gt;&amp;nbsp;&lt;/p&gt;<br /><br /> &lt;p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;This privacy notice complies with our Privacy Policy and Terms of Service Policy and applies to you when you decide to use our services. &lt;/span&gt;&lt;/b&gt;&lt;/p&gt;<br /><br /> &lt;p&gt;&amp;nbsp;&lt;/p&gt;<br /><br /> &lt;p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;We collect information when users visit websites hosted by the OERF. This includes IP addresses which are unique numbers that identify specific internet connections, and sometimes specific computers and devices that visit our sites.&lt;/span&gt;&lt;/b&gt;&lt;/p&gt;<br /><br /> &lt;h2 dir="ltr" style="line-height:1.38;margin-top:18pt;margin-bottom:6pt;"&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:16pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;Definitions&lt;/span&gt;&lt;/b&gt;&lt;/h2&gt;<br /><br /> &lt;p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;In the context of this policy &lt;/span&gt;&lt;/b&gt;&lt;/p&gt;<br /><br /> &lt;p&gt;&lt;b id="docs-internal-guid-614306f4-8a93-ba9b-7e04-91586b612f23" style="font-weight:normal;"&gt;&lt;span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt;Personal data&lt;/span&gt;&lt;span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"&gt; refers to information related to an identified or identifiable natural person used for data processing by the OERF. &lt;/span&gt;&lt;/b&gt;&lt;/p&gt;<br /><br /> &lt;p&gt;&amp;nbsp;&lt;/p&gt;<br /><br /> &lt;p&gt;&amp;nbsp;&lt;/p&gt;<br /><br /> &lt;p&gt;&amp;nbsp;&lt;/p&gt;</code></blockcode></p> <p>As you can see, it's fairly impenetrable (it's not written for human's eyes :) ). It doesn't even use proper semantic tags to represent the content. Instead it uses "ad hoc" styling to achieve a superficial resemblance to what we want to see, and in doing so, it also generates huge amounts of totally awful mark-up.</p> <h2>The (Free and Open Source) Solution</h2> <p>Luckily, after a lot of dives down rabbit holes, I have emerged with a free and open source software (FOSS) solution. It involves an incredible software tool called "<a href="https://pandoc.org" title="The Pandoc Free and Open Source tool">Pandoc</a>" that can be installed on any Windows, MacOS, or Linux computer. Because I use FOSS where ever possible, I'll describe how I achieved a solution on Linux. </p> <p>In addition to providing clean semantic HTML, preserving links and structure (including tables), Pandoc can be asked to build a "Table of Contents" for the HTML, too, based on titles (H1, H2, etc.) to the level desired.</p> <p>The steps:</p> <ol><li>download a "Web Page (html, zipped)" version of the Google Document and unzip it in to a file, in my case,<code> PrivacyNotice.html</code></li> <li>install Pandoc on my computer:<code> <code>sudo apt install pandoc</code></code></li> <li>in the directory in which I have<code> PrivacyNotice.html,</code> I ran<code><br /><br /><code>pandoc -r html -w docbook --email-obfuscation=none -S -s PrivacyNotice.html | pandoc -r docbook -w html --toc --toc-depth=2 --email-obfuscation=none -S -s -o PrivacyNotice-semantic.html -</code></code><br /><br /> Which produces a semantically marked up version of the file PrivacyNotice-semantic.html, and, in this case, with a linked Table of Contents automatically created, using headings up to a depth of "2" (i.e. H1 and H2) as the headings.<br /><br /> If you prefer not to have the Table of Contents, run this instead:<code><br /><br /><code>pandoc -r html -w docbook --email-obfuscation=none -S -s PrivacyNotice.html | pandoc -r docbook -w html --email-obfuscation=none -S -s -o PrivacyNotice-semantic.html -</code> </code><br /><br /> Note: the trailing "-" is important (it's a way of telling the script to take as input the output of the script described prior to the "pipe" character, "|")!</li> </ol><p>What this Pandoc process does, is to convert the nasty HTML provided by the Google Doc "Web Page" export into a different (and very semantically structured) format called DocBook, which strips out all the unnecessary styling. We then convert it back to HTML, preserving the semantic structure, and removing the stuff we don't want.</p> <p>To simplify matters, you can also write a script or shell alias which achieves the same thing in a single command. For example, here's the alias I created in my ~/.bashrc (the hidden bash shell configuration file in my home directory, "~"):</p> <p><blockcode><code><code>alias tosemtoc='pandoc -r html -w docbook --email-obfuscation=none -S -s $1 | pandoc -r docbook -w html --toc --toc-depth=2 --email-obfuscation=none -S -s -o $1 -'</code></code></blockcode></p> <p><blockcode><br /><code><code>alias tosem='pandoc -r html -w docbook --email-obfuscation=none -S -s $1 | pandoc -r docbook -w html --email-obfuscation=none -S -s -o $1 -'</code></code></blockcode></p> <p>(if you add this to your .bashrc, you can make your bash session aware of it either by logging out or in again, or by running <code>. ~/.bashrc</code> at your command prompt)</p> <p>With these two lines I can achieve the same thing as the above complex command like this (the semantic HTML output will be put into the same file I started with in this case, rather than a separate file - no big loss on that Google Doc-produced HTML travesty!):</p> <p>With Table of Contents (down to H2): <code>tosemtoc PrivacyNotice.html</code></p> <p>Without the Table of Contents: <code>tosem PrivacyNotice.html</code></p> <p>Pretty straightforward. And it takes less than a second to run. Note that the result is a "well formed" complete HTML document, so you can view it happily in your browser, e.g. type <code>file:///path/to/your/PrivacyNotice.html</code> file into your browser's location bar as a URL.</p> <p>If you want to see the final result of this, have a look at our new GDPR-aware <a href="https://oeru.org/privacy-notice" title="The OERu Privacy Notice">Privacy Notice</a>!</p> <p>Hope this saves someone a lot of frustration and unnecessary remedial mark-up editing!</p> <p><code> </code></p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=20&amp;2=field_blog_comments&amp;3=comment" token="Tk3zaP2726_ZI8MqlLbML0HqaI7MDjD7WmCCTtyS-RM"></drupal-render-placeholder> </div> </section> Wed, 23 May 2018 00:55:19 +0000 dave 20 at http://tech.oeru.org http://tech.oeru.org/creating-simple-semantic-html-markup-google-doc#comments Insight: what does the Open Source Technologist at the OER Foundation do? http://tech.oeru.org/insight-what-does-open-source-technologist-oer-foundation-do <span class="field field--name-title field--type-string field--label-hidden">Insight: what does the Open Source Technologist at the OER Foundation do?</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Fri 18/05/2018 - 13:17</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>A few weeks ago, I got an unexpected (but very flattering!) request from a friend and colleague's son, Felix. He wanted to interview me about my work (at 11, he thinks he might want to become a software developer, too, when he's older) as material he could present at an up-coming <a href="https://en.wikipedia.org/wiki/PechaKucha">PechaKucha</a> session at his school... Wow, we didn't do PechaKucha when I was a kid :)</p> <p>I agreed to respond, and, due to logistical challenges, that response took the form of an email answer his questions. After sending it to him, it occurred to me that it might be of interest to some folks in the OERu community and beyond, so I asked if he'd be ok with me publishing the results under a CC-By license. He was agreed, and so here it is (note, I've made a few minor edits and additions to my original response).</p> <dl><dt>1. What do you do in an average day at work?</dt> <dd> <p>My days are generally very diverse, but I can give you an idea of a typical "business as usual" day... (as opposed to, say, when I'm overseas at one of our very enjoyable international OERu partner meetings).</p> <p>Of late, my usual routine Usually, I make myself a coffee and go up to the garage loft in my house. I sit down at my computer and log in, immediately checking my email and various chat channels to see if there're any urgent problems to look into (my work is global, so there's always someone using the systems I manage).</p> <p>If there something urgent, like a system not running the way it's meant to, I get right onto making it right - our systems are meant to serve hundreds (and eventually tens of thousands or more!) learners daily, so they need to be reliable and responsive.</p> <p>On Mondays, I "go around the traps", applying updates to servers that I'm responsible for in Germany, the US, NZ, Australia, and Singapore - they support a bunch of websites and services that are part of our "Digital Learning Environment". I also maintain some web systems for the NZ Open Source Society (which I sometimes include in "the traps" - my job title with the OER Foundation who employ me, is "Open Source Technologist").</p> <p>In amongst the updates, I sometimes do some research on new technologies I need to deploy as part of my "continual improvement" process (I can do this without needing to concentrate intensely, so I can do it in short bursts well suited to the start and stop nature of the server update process) - finding ways to make it easier for me, as one half of a 2-person team trying to build a global service, to manage all the many systems I'm responsible for. Today, for instance, I'm looking at setting up an open source service monitoring system which will alert me if any of the systems I'm interested in have problems, go do, or even if they're slow... I'm also interested in monitoring whether any of them are running short of hard disk space, as recovering from a full hard disk is often very painful and messy, and can result in both long downtimes and lost user data (something we work <em>very</em> hard to avoid).</p> <p>Most days I also do quite a bit of software development. My role includes the responsibility for building strategic software capabilities. Today that will include working on a plugin for the <a href="https://wordpress.org">WordPress</a> blog platform (like <a href="https://drupal.org">Drupal</a>, but somewhat simpler, but also open source) that we use to provide our course materials to learners. When a learner logs in, and asks to take part in a particular course, the plugin I've written, automatically registers their details with a separate system we use, <a href="/installing-mautic-php7-fpm-docker-nginx-and-mariadb-ubuntu-1604">Mautic</a>, which is a "marketing automation" platform.</p> <p>Many people use Mautic to "spam" people or send out newsletters to people who subscribe to them via a website, for instance. We use it differently: we use it to automate the process of sending out hundreds of carefully timed emails to people taking each of our courses, so that they know when to start working on things, what they need to do, when their projects and assignments are due, and how to go about getting assessed after they finish a course (they need to be assessed to get formal academic credit for our courses).</p> <p>Using Mautic means that we can set up the rules and email templates for each course once, and then they're available any time we run a course with a cohort (a group of learners are going through the course together, who could be anywhere in the world - our last course had 310 learners from 58 countries!), or even for people who decide to take our courses via "independent study" in which case they can set their own schedule.</p> <p>The other thing I do quite a lot is to work out better ways to host the various services we offer - because the open source technologies we build our services on are constantly improving, there's a steady stream of new approaches to doing things. I have to use my judgment and experience to decide which improvements are worth making (because they all take my time, which is limited to 8 hrs a day), and which probably aren't worthwhile. I recently changed the way we host our main WordPress "multisite", <a href="https://course.oeru.org">https://course.oeru.org</a>, so that it's built as a collection of Docker containers, and I was able to improve the number of simultaneous learners we could support by 10 times without increasing our costs...</p> <p>Then, just about every afternoon, I have a video chat with my colleague, Wayne Mackintosh, who's based in Mosgiel (a town south of Dunedin), and is the only other employee of the OER Foundation (he's also the founder of it, and the main brains behind the operation). He's a keen supporter of open stuff - particularly open source software and open educational resources (the learning materials we make available to our learners as courses), which are the educational analogue to open source software. Wayne's main focus is on academic things - like coordinating educators at our many partner institutions globally to help contribute towards our open educational resources, to get their academic boards to push through accreditation processes for our courses (so that their institutions formally recognise our courses as being of a suitable quality for them to accept as if they were their own). He's got the really hard job, because it's more to do with people than technology. And, despite claiming he's not at every turn, he's remarkably switched on with the technology, too.</p> <p>We report on the things we've done that day, catch up on other news and developments, talk through problems we're working on, and make plans and prioritise things for the coming days and weeks.</p> <p>From time to time, I also write up "how tos" and explanations of how we use open source technologies at the OERu (the OERu - <a href="https://oeru.org">https://oeru.org</a> is the "network" made up of the OER Foundation and our higher education partners) - you can see it at <a href="https://tech.oeru.org">https://tech.oeru.org</a> - the introduction provides an overview of the mix of technologies we use if you're curious.</p> </dd> <dt>2. What is your favourite thing about being a software developer?</dt> <dd> <p>I love that I have a steady stream of interesting problems to solve, and that, when I solve them, I'm potentially making life better for hundreds or thousands of people who, in this case, might not have had the opportunity to reach their educational potential before.</p> <p>Plus, I really appreciate the fact that my current role allows me to learn new things constantly, and to actively engage with the communities of people who write the software that we use to make our learning materials available. For example, I frequently end up making minor improvements to software, like Mautic, or WordPress, which I then make available to other people (at places like this: <a href="https://github.com/kiwilightweight">https://github.com/kiwilightweight</a> or <a href="https://github.com/oeru">https://github.com/oeru</a>) as a way of contributing back.</p> <p>The fact that I can use my skills and experience do what I love, to benefit the world - people I've never even met - is very satisfying and fulfilling, and I consider myself very lucky.</p> </dd> <dt>3. What is the hardest part about being a software developer?</dt> <dd> <p>There're plenty of stressful things in managing systems that people use - for example, if there's a bug that means parts of the system (or all of it) don't work, I have to really scramble to make things right. I also have to to my very best to protect people's data! Very rarely (thankfully) something goes wrong, and people lose data (sometimes things they've invested a lot of time or energy in) and that's really tough.</p> <p>There're also "good-hard" things about developing software - making complex things easier for others to understand... understanding them myself... solving problems by doing a lot of detective work on the Internet - lots of times it requires hours of focused concentration, and really thinking hard... but if you never drop the ball, eventually you solve the problem, and that feels <em>great</em>.</p> </dd> <dt>4. What sort of training and education do you need to do before becoming a software developer?</dt> <dd> <p>Well, ultimately, as a software developer, you're a people interpreter: you're someone who learns what others want to do, how they want to do it, and then you work out how to "codify" that in software. This takes a LOT of time to do it properly. If you don't, you're likely to end up writing software that solves a problem... but one that's often quite different from what people want solved.  So learning about how people do things is very useful... not sure what formal education you can get here - good software developers are keen observers of human behaviour.</p> <p>Once you know roughly where you're going, software development boils down to a lot of problem solving and "micro-decision-making". You have to be the sort of person who, once you commit to solving a problem, you don't stop. Not sure you can learn that - it's just part of some people's nature...</p> <p>As far as formal education goes, I generally recommend this: <strong>get all the education you possibly can</strong> - in maths, science, and computing, but <em>also in other fields</em>. I think one of the best things I ever did was to do a "liberal arts" degree at university - I was privileged that my family and I were able to afford to do it (it was <em>very</em> expensive). I got to study many other fields besides, especially English. Being able to express myself in writing (be it code or English :) ) confidently is perhaps the best thing I learned. I'd also recommend learning other (human) languages!</p> </dd> <dt>5. What do you find least enjoyable about being a software developer?</dt> <dd> <p>There's always a bit of drudgery in every job. There're routine things that are a bit boring (the great thing about being a software developer is that you have the power to <em>automate</em> the boring stuff if it becomes a real drag - here's a <a href="/creating-simple-semantic-html-markup-google-doc">recent example</a>... ) - and there's also the frustration of having to work with (and keep happy) people who really <em>don't</em> understand technology at all, and see it as a threat.</p> <p>I think people who have a deep understanding of both how people work, <em>and</em> how software works have the best prospects in the world today, and I think that'll continue to be the case into the future.</p> </dd> </dl></div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <h2 class="comment-field__title">Blog comments</h2> <a id="comment-7"></a> <article data-comment-user-id="0" about="/comment/7" typeof="schema:Comment" class="comment js-comment by-anonymous has-title clearfix"> <div class="comment__container"> <h3 property="schema:name" datatype="" class="comment__title"> <a href="/comment/7#comment-7" class="permalink" rel="bookmark" hreflang="en">Why GH not GitLab?</a> <span class="comment__new marker marker--success hidden" data-comment-timestamp="1528242423"></span> </h3> <div class="comment__meta"> <div class="comment__submitted"> <span class="comment__author"><span rel="schema:author"><span lang="" typeof="schema:Person" property="schema:name" datatype="">Danyl Strype (not verified)</span></span> </span> <span class="comment__pubdate">Thu 31/05/2018 - 01:18 <span property="schema:dateCreated" content="2018-05-30T13:18:12+00:00" class="rdf-meta hidden"></span> </span> </div> </div> <div class="comment__content"> <div property="schema:text" class="clearfix text-formatted field field-comment--comment-body field-name-comment-body field-type-text-long field-label-hidden"> <div class="field__items"> <div property="schema:text" class="field__item"><p>Hi Dave, this is a great write-up, and hearing that even a pro like you sometimes loses people&#039;s data makes me feel much less existentially terrified about the idea of going pro as a sysadmin or developer, rather than just paddling around the edges of it. One question, why are you using GH for your code, rather than a self-hosted, free code replacement like GitLab? I know that you run a GL instance for NZOSS, so I&#039;m curious about why you the links you&#039;ve given are to GH.</p> </div> </div> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=7&amp;1=default&amp;2=en&amp;3=" token="gSITky6KmlYh8k3gzKegeQw5Ab0eMLcMkqN0j1ewtsc"></drupal-render-placeholder> </div> </div> </article> <div class="indented"><a id="comment-8"></a> <article data-comment-user-id="1" about="/comment/8" typeof="schema:Comment" class="comment js-comment by-node-author has-title clearfix"> <div class="comment__container"> <h3 property="schema:name" datatype="" class="comment__title"> <a href="/comment/8#comment-8" class="permalink" rel="bookmark" hreflang="en">Network Effect... for now</a> <span class="comment__new marker marker--success hidden" data-comment-timestamp="1528242760"></span> </h3> <div class="comment__meta comment__meta--has-user-picture"> <a href="/blog/1">View recent blog entries</a> <div class="comment__submitted"> <span class="comment__author"><span rel="schema:author"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> </span> <span class="comment__pubdate">Wed 06/06/2018 - 11:52 <span property="schema:dateCreated" content="2018-06-05T23:52:40+00:00" class="rdf-meta hidden"></span> </span> </div> </div> <div class="comment__content"> <p class="comment__parent visually-hidden">In reply to <a href="/comment/7#comment-7" class="permalink" rel="bookmark" hreflang="en">Why GH not GitLab?</a> by <span lang="" typeof="schema:Person" property="schema:name" datatype="">Danyl Strype (not verified)</span></p> <div property="schema:text" class="clearfix text-formatted field field-comment--comment-body field-name-comment-body field-type-text-long field-label-hidden"> <div class="field__items"> <div property="schema:text" class="field__item"><p>Very good question, Danyl. My predecessor used GitHub prior to my involvement. I recognise that GitHub enjoys a network effect, and, given our goal is to maximise the value of our code for "The Commons", I (with misgivings, to be sure) decided to stick with GitHub (and a few projects on BitBucket, too) for the short term. </p> <p>Of course, everything's changed a couple days ago with the revelations that GitHub is being purchased by one of the least trustworthy corporations in the tech world, Microsoft. We will be setting up our own Gitlab instance (or perhaps Gogs, not sure yet) and shifting all our repos away from GitHub and Bitbucket (which is very similar to GitHub). Watch this space.</p> </div> </div> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=8&amp;1=default&amp;2=en&amp;3=" token="roEwVhPI8x3guUlKO4gZlMfjatgWmh9ZcaE9ZvBUEwo"></drupal-render-placeholder> </div> </div> </article> <a id="comment-46"></a> <article data-comment-user-id="1" about="/comment/46" typeof="schema:Comment" class="comment js-comment by-node-author has-title clearfix"> <div class="comment__container"> <h3 property="schema:name" datatype="" class="comment__title"> <a href="/comment/46#comment-46" class="permalink" rel="bookmark" hreflang="en">OERu code repos now all on GitLab</a> <span class="comment__new marker marker--success hidden" data-comment-timestamp="1552341237"></span> </h3> <div class="comment__meta comment__meta--has-user-picture"> <a href="/blog/1">View recent blog entries</a> <div class="comment__submitted"> <span class="comment__author"><span rel="schema:author"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> </span> <span class="comment__pubdate">Tue 12/03/2019 - 10:53 <span property="schema:dateCreated" content="2019-03-11T21:53:57+00:00" class="rdf-meta hidden"></span> </span> </div> </div> <div class="comment__content"> <p class="comment__parent visually-hidden">In reply to <a href="/comment/7#comment-7" class="permalink" rel="bookmark" hreflang="en">Why GH not GitLab?</a> by <span lang="" typeof="schema:Person" property="schema:name" datatype="">Danyl Strype (not verified)</span></p> <div property="schema:text" class="clearfix text-formatted field field-comment--comment-body field-name-comment-body field-type-text-long field-label-hidden"> <div class="field__items"> <div property="schema:text" class="field__item"><p>Hi Danyl,</p> <p>Just a follow-up - all of the OERu's source code repositories (which is quite a few!) are now on <a href="https://git.oeru.org">https://git.oeru.org</a>, which is  a GitLab installation I set up and maintain here in New Zealand, on local cloud hosting infrastructure generously provided (sponsored) by <a href="https://catalystcloud.nz">CatalystCloud.nz!</a></p> </div> </div> </div> <drupal-render-placeholder callback="comment.lazy_builders:renderLinks" arguments="0=46&amp;1=default&amp;2=en&amp;3=" token="gedjSAkNeSxUx8t2wJaPnNQFELS_DwaigcB_qfBpLRs"></drupal-render-placeholder> </div> </div> </article> </div> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=19&amp;2=field_blog_comments&amp;3=comment" token="RWzhW49w2Ranf8cyf73ptbSAI2rTzCPypSc-MBKegxE"></drupal-render-placeholder> </div> </section> Fri, 18 May 2018 01:17:23 +0000 dave 19 at http://tech.oeru.org http://tech.oeru.org/insight-what-does-open-source-technologist-oer-foundation-do#comments Installing NextCloud and Collabora Office Online with Docker on Ubuntu 16.04 http://tech.oeru.org/installing-nextcloud-and-collabora-office-online-docker-ubuntu-1604 <span class="field field--name-title field--type-string field--label-hidden">Installing NextCloud and Collabora Office Online with Docker on Ubuntu 16.04</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--ubuntu-linux"> <span class="field__item-wrapper"><a href="/taxonomy/term/12" hreflang="en">ubuntu linux</a></span> </div> <div class="field__item field__item--mariadb"> <span class="field__item-wrapper"><a href="/taxonomy/term/48" hreflang="en">mariadb</a></span> </div> <div class="field__item field__item--docker"> <span class="field__item-wrapper"><a href="/taxonomy/term/16" hreflang="en">docker</a></span> </div> <div class="field__item field__item--docker-compose"> <span class="field__item-wrapper"><a href="/taxonomy/term/49" hreflang="en">docker-compose</a></span> </div> <div class="field__item field__item--php"> <span class="field__item-wrapper"><a href="/taxonomy/term/40" hreflang="en">php</a></span> </div> <div class="field__item field__item--collabora-office"> <span class="field__item-wrapper"><a href="/taxonomy/term/50" hreflang="en">collabora office</a></span> </div> <div class="field__item field__item--nextcloud"> <span class="field__item-wrapper"><a href="/taxonomy/term/51" hreflang="en">nextcloud</a></span> </div> <div class="field__item field__item--lets-encrypt"> <span class="field__item-wrapper"><a href="/taxonomy/term/17" hreflang="en">let&#039;s encrypt</a></span> </div> <div class="field__item field__item--redis"> <span class="field__item-wrapper"><a href="/taxonomy/term/21" hreflang="en">redis</a></span> </div> <div class="field__item field__item--productivity"> <span class="field__item-wrapper"><a href="/taxonomy/term/52" hreflang="en">productivity</a></span> </div> <div class="field__item field__item--nginx"> <span class="field__item-wrapper"><a href="/taxonomy/term/30" hreflang="en">nginx</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Mon 29/01/2018 - 17:29</span> <div class="float-none field field-node--field-image field-name-field-image field-type-image field-label-hidden has-multiple"> <figure class="field-type-image__figure image-count-1 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/Files%20-%20OERu%20NextCloud.png?itok=xQHlcyml" title="The NextCloud web interface for browsing your files" data-colorbox-gallery="gallery-field_image-rUaT7HdTu4I" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;The NextCloud web interface for browsing your files&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/Files%20-%20OERu%20NextCloud.png?itok=6v2Kuyct" width="220" height="122" alt="The NextCloud web interface for browsing your files" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-2 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/NextCloud-AppStore.png?itok=DPeCx5Rd" title="The central AppStore (note, almost all apps have no cost and are open source). You get a similar view within your own NextCloud instance." data-colorbox-gallery="gallery-field_image-rUaT7HdTu4I" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;The central AppStore (note, almost all apps have no cost and are open source). You get a similar view within your own NextCloud instance.&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/NextCloud-AppStore.png?itok=WqCJJdGj" width="220" height="175" alt="The central AppStore (note, almost all apps have no cost and are open source). You get a similar view within your own NextCloud instance." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-3 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/NextCloud-Calendar.png?itok=-j0Dq2rG" title="The NextCloud shared calendar plugin works with all major calendaring applications alongside your existing digital calendars." data-colorbox-gallery="gallery-field_image-rUaT7HdTu4I" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;The NextCloud shared calendar plugin works with all major calendaring applications alongside your existing digital calendars.&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/NextCloud-Calendar.png?itok=bP23WxDf" width="220" height="175" alt="The NextCloud shared calendar plugin works with all major calendaring applications alongside your existing digital calendars." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-4 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/Nextcloud-CollaboraSpreadsheet.png?itok=Ovp0KryQ" title="An example of a fairly complex spreadsheet in the Collabora spreadsheet interface." data-colorbox-gallery="gallery-field_image-rUaT7HdTu4I" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;An example of a fairly complex spreadsheet in the Collabora spreadsheet interface.&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/Nextcloud-CollaboraSpreadsheet.png?itok=CNhDR2y-" width="220" height="157" alt="An example of a fairly complex spreadsheet in the Collabora spreadsheet interface." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-5 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/Nextcloud-CollaboraWordprocessor.png?itok=IOyfA_M4" title="A fairly complex document, with variables, shown in the Collabora wordprocessor interface." data-colorbox-gallery="gallery-field_image-rUaT7HdTu4I" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;A fairly complex document, with variables, shown in the Collabora wordprocessor interface.&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/Nextcloud-CollaboraWordprocessor.png?itok=HPawBI-o" width="220" height="157" alt="A fairly complex document, with variables, shown in the Collabora wordprocessor interface." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-6 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/DavInFilemanager.png?itok=rCbwaUUY" title="This is what your NextCloud would look like in your desktop filemanager (this is the Nemo filemanager on a Linux desktop)" data-colorbox-gallery="gallery-field_image-rUaT7HdTu4I" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;This is what your NextCloud would look like in your desktop filemanager (this is the Nemo filemanager on a Linux desktop)&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/DavInFilemanager.png?itok=g2dNm33H" width="220" height="122" alt="This is what your NextCloud would look like in your desktop filemanager (this is the Nemo filemanager on a Linux desktop)" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-7 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-02/CollaboraAdminConsole.png?itok=1tNI9ZdJ" title="Collabora Office admin console" data-colorbox-gallery="gallery-field_image-rUaT7HdTu4I" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Collabora Office admin console&quot;}"><img src="/sites/default/files/styles/medium/public/2018-02/CollaboraAdminConsole.png?itok=iijjMrBK" width="220" height="149" alt="Collabora Office admin console" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> </div> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>Dropbox is the best known of the end-user "cloud storage" services for documents, backups, and synchronising data among multiple devices, although now Google's Drive and Microsoft's OneDrive are functionally similar and are being heavily promoted and tied into all sorts of services.</p> <p>Similarly the collaborative editing of documents, spreadsheets, and presentations in the browser, pioneered by Etherpad, but then adopted in a big way by Google Docs (and more recently, Microsoft Office 365), has revolutionised collective note taking, document preparation, and ease of access to these powerful tools by the mainstream of computer users. Only a browser is required, and no other software needs to be installed.</p> <p>But what about people who don't want to entrust all of their data to foreign corporations, holding their data in foreign jurisdiction, in formats that may or may not be retrievable in the event that the supplier fails or changes "strategic direction"? And many of these services involve "mining" their data to extract useful information that vendors sell to others to <em>help them advertise to us in a more targeted way. </em>Yeah, that's creepy.</p> <p>More-over, often if you want to <em>share</em> your data with others, <em>they</em> have to log into the same service, and accept the service's terms and conditions (usually substantially constraining the user's normal rights and freedoms, although who<em> actually</em> reads those, eh?!) in order to do so... so ones use of those services has a magnifying effect on the loss of privacy and control.</p> <p>Some people sensibly prefer to manage their own, or institution-specific, solutions on the infrastructure of their choosing, in a way that doesn't tie anyone into paying ever increasing amounts for data storage as the volumes increase perpetually, month on month.</p> <p>Some of us simply prefer to have control of our own destiny, without a dependence on, for example, file or data storage formats and practices that are completely opaque to them. Our data reflects our creativity energy, and it seems much more comfortable for many of us to be in charge of our own fates rather than entrusting it to a third party who simply sees us a profit centre.</p> <p>Thankfully, the open source world has created an array of possible equivalent systems, and this post describes how you, too, can set up your own equivalent to Dropbox + Google Docs using entirely open source software on any commodity virtual machine hosting system you want to use by adopting NextCloud and Collabora Office.</p> <h2>NextCloud</h2> <p><a href="https://nextcloud.com">NextCloud</a> is <a href="https://nextcloud.com/files/">functionally similar</a> to Dropbox, however, with its active development community and plug-in architecture, it can provide quite a lot more as well, like shared calendaring, email, video conferencing, contact syncing, image/sound/video galleries, <a href="https://nextcloud.com/files/">among many other services</a>.</p> <p>If you prefer not to organise and run your own server, you can purchase a supported server via their website for a cost similar to Dropbox (although, realise that NextCloud is relatively small by comparison and doesn't have the massive economies of scale enjoyed by the bigger players).</p> <p>For those with an interest in history: NextCloud is a fork created by the founder of OwnCloud, after he decided that the company which formed around his OwnCloud project was moving in a direction that was philosophical unpalatable for him. The beauty of open source is that developers can follow their consciences without requiring anyone's permission. The resulting "forks" in code bases and communities then thrive or die based on the strengths of the communities they can build and sustain. This fork is remarkably similar to that which occurred in the OpenOffice community which resulted in the founding of LibreOffice. LibreOffice has thrived and OpenOffice has faded into irrelevance. More on that below.</p> <p>For those with a technological interest, NextCloud is a mature PHP application (but with a modern architecture, including a command line interface, occ) which stores its data in an RDBMS like MySQL, MariaDB, PostgreSQL, or (usually for development purposes) the lightweight SQLite database. Here are <a href="https://docs.nextcloud.com/server/12/admin_manual/installation/index.html">details for would-be administrators</a>.</p> <h2>Collabora Office</h2> <p>Given how much companies like Google and Microsoft invest on Docs and Office 365 respectively, how is it possible for an open source community to create a credible competitor? Turns out it's not as hard as you might think if they leverage the power of open source.</p> <p>A small software company with headquarters in the UK (although their team appears to be from all over), Collabora Office, has taken on the ambitious mission of creating a "collaborative web interface" allowing users to collaborate using <a href="https://libreoffice.org">LibreOffice</a>, one of the most powerful and widely used office package available anywhere. We're currently at Collabora Office 3.0, and the front end is quite nice and functional, but still pretty simple - that can be a good thing for many users. Collabora is progressively re-imagining the user interface of LibreOffice as a collaborative web interface. This isn't easy, but it's <em>much</em> easier than it otherwise would be because the difficult job of creating the heavy-lifting application back-end is already done - LibreOffice is a mature widely used application (albeit with a desktop interface, not a web-based collaborative interface). So we can expect progress will be rapid, and large sets of new capabilities will be "unlocked" as they progress their efforts.</p> <h2>NextCloud and Collabora - better together!</h2> <p>The beauty of the open source software model is that we can connect NextCloud and Collabora office - completely separate and unrelated communities - thanks to a new integration standard, WOPI (Web-application Open Platform Interface) they form a well integrated component model - with the <em>major </em>added benefit of being able to swap in a better file management platform, or a better collaborative productivity package if one or the other emerges, without having to start from scratch.</p> <h2>Setting up your own NextCloud Collabora Server</h2> <p>If you're game to run your own (and, in my experience, it's a surprisingly well behaved system) here's how you do it.</p> <p>In preparation, you'll want to have the following ready:</p> <ul><li>a Linux virtual machine or "VM" (I recommend running the current Ubuntu LTS version, or current Debian) with a user with Sudo privileges...,</li> <li>your domain name for the NextCloud instance, pointing to the IP address of your VM,</li> <li>your domain name for the Collabora instance, also pointing to the IP of your VM, and</li> <li>credentials for an email address capable of sending from a remote server (usually termed an "authenticating SMTP email account")</li> </ul><h3>Secure access with SSH</h3> <p>First things first, make sure you're logged into your host (probably via SSH) as a user who has "sudo" capabilities! You need to log into the host from your local machine. We recommend setting up <a href="https://www.digitalocean.com/community/tutorials/how-to-configure-ssh-key-based-authentication-on-a-linux-server">key-based authentication</a>.</p> <h3>Firewall with UFW</h3> <p>No computer system is ever full secure - there're always exploits waiting to be found, so security is a process of maintaining vigilance. Part of that is reducing exposure - minimising your "attack surface". Use a firewall - "<a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-ubuntu-16-04" title="Uncomplicated FireWall">ufw</a>" is installed on Ubuntu by default. Make sure you've got exceptions for SSH (without them, you could lock yourself out of your machine! Doh!).</p> <p>Run the following commands to allow your Docker containers to talk to other services on your host.</p> <p><code>sudo ufw allow in on docker0<br /> sudo ufw allow from to any<br /> sudo ufw allow from to any<br /> sudo ufw allow from to any<br /> sudo ufw allow from to any</code></p> <p>Specifically for Docker's benefit, you need to tweak the default Forwarding rule (I use "vim" as my editor. If you don't know how to/want to use it, replace <strong>vim</strong> with <strong>nano</strong> everywhere you see it in the following - nano's easier to use for simple edits like this):</p> <p><code>sudo vim /etc/defaults/ufw</code></p> <p>and copy the line <code>DEFAULT_FORWARD_POLICY="DROP"</code> tweak it to look like this (commenting out the default, but leaving it there for future reference!):</p> <p><code>#DEFAULT_FORWARD_POLICY="DROP"<br /> DEFAULT_FORWARD_POLICY="ACCEPT"</code></p> <p>You also have to edit <code>/etc/ufw/sysctl.conf</code> and remove the "#" at the start of the following lines, so they look like this:</p> <p><code>sudo vim /etc/ufw/sysctl.conf</code></p> <p><code># Uncomment this to allow this host to route packets between interfaces<br /> net/ipv4/ip_forward=1<br /> net/ipv6/conf/default/forwarding=1<br /> net/ipv6/conf/all/forwarding=1</code></p> <p>and finally restart the network stack and ufw on your server<code> </code></p> <p><code>sudo service networking restart<br /> sudo service ufw restart</code></p> <h3>Installing the Nginx webserver</h3> <p>In the configuration I'm describing here, you'll need a webserver running on the server - it'll be acting as a "proxy" for the Docker-based Nginx instance described below. I like the efficiency of Nginx and clarity of Nginx configurations over those of Apache and other open source web servers. Here's how you install it.</p> <p><code>sudo apt-get install nginx-full</code></p> <p>To allow nginx to be visible via ports 80 and 443, run</p> <p><code>sudo ufw allow "Nginx Full"</code></p> <p><strong>Note</strong>: make sure your hosting service is not blocking these ports at some outer layer (depending on who's providing that hosting service you may have to set up port forwarding).</p> <h3>Installing MariaDB</h3> <p>MariaDB is effectively a drop-in alternative to MySQL and we prefer it because it's not controlled by Oracle and has a more active developer community. On Ubuntu, MariaDB pretends to be MySQL for compatibility purposes, so don't be weirded out by the interchangeable names below. Install the server and the client like this.</p> <p><code>sudo apt-get install mariadb-server-10.0 mariadb-client-10.0</code></p> <p>You need to set a root (admin) user password - you might want to create a /root/.my.cnf file containing the following (replacing YOURPASSWORD) to let you access MariaDB without a password from the commandline<code>:</code></p> <p><code>[client]<br /> user=root<br /> password=YOURPASSWORD</code></p> <p>You should now be able to type "mysql" at the command prompt</p> <p>Tweak the configuration so that it's listening on</p> <p><code>sudo vim /etc/mysql/mariadb.conf.d/50-server.cnf </code></p> <p>and copy the bind-address line and adjust so it looks like this - we want MariaDB to be listening on all interfaces, not just localhost (</p> <p><code># Instead of skip-networking the default is now to listen only on<br /> # localhost which is more compatible and is not less secure.<br /> #bind-address           =<br /> bind-address            =</code></p> <p>Then restart MariaDB:</p> <p><code>sudo service mysql restart</code></p> <p>It should now be listening on port 3306 on all interfaces, i.e.</p> <p>Now set up the database which will hold NextCloud's data. Log into the MySQL client on the host (if you've created a .my.cnf file in your home directory as describe above, you won't need to enter your username and password):</p> <p><code>mysql -u root -p</code></p> <p>Enter your root password when prompted. It's also a good idea to gin up a password for your "nextcloud" database user. I usually use pwgen (<code>sudo apt-get install pwgen</code>) - for example running this command will give you a single 12 character password without special characters (just numbers and letters):</p> <p><code>pwgen -s 12 1<br /> T7KR2osrMkyC</code></p> <p>At the prompt (which will look something like MariaDB [(none)]&gt;) enter the following lines (putting your password in place of [passwd]):</p> <p><code>CREATE DATABASE nextcloud CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;<br /> CREATE USER "nextcloud"@"%" IDENTIFIED BY "[passwd]";<br /> GRANT ALL ON nextcloud.* to "nextcloud"@"%";<br /> FLUSH PRIVILEGES;</code></p> <p>Then enter \q to exit.</p> <h2>NextCloud and Collabora Office with Docker</h2> <p>We make use of the NextCloud community's <a href="https://hub.docker.com/_/nextcloud/" title="Documentation for the reference NextCloud Docker container.">stable Docker container</a> which they keep up to date. Similarly, the Collabora community has created a <a href="https://hub.docker.com/collabora/code">reference Docker container</a>.</p> <p>The over all architecture consists of five Docker containers (note, done properly, you aim to ensure that each container runs only one service!):</p> <ol><li>the main NextCloud container (running the PHP-FPM service)</li> <li>an identical container to the PHP one which runs the cron service (which does periodic administrative tasks relevant to NextCloud)</li> <li>the self-contained Collabora Office container (running PHP with an Apache web server instance and a full instance of LibreOffice running in headless server mode (never fear, no servers were harmed in the building of this server!) - yes this server doesn't really adhere to the "one-service per container" convention, but I'm ok with that. It's just a convention after all.)</li> <li>a Redis container (which provides performance improving caching for NextCloud), and</li> <li>an Nginx webserver container which makes it easier to manage the configuration and paths of the NextCloud and Collabora servers via WOPI. It means that on the hosting server, we only need to run a proxying web server, which is easy.</li> </ol><p>The way I prefer to implement this set of containers is to use <a href="https://docs.docker.com/compose/">Docker Compose</a> (after first setting up <a href="https://docs.docker.com/install/linux/docker-ce/ubuntu/">Docker support</a> on your server - I'll assume you've followed the complete instructions including <a href="https://docs.docker.com/install/linux/linux-postinstall/">setting up Docker for your non-root user</a>). I suggest using the latest <a href="https://docs.docker.com/compose/install/#install-compose">installation instructions</a> provided by the Docker community. To be honest, I usually use the alternative instructions, <a href="https://docs.docker.com/compose/install/#install-using-pip">employing the "pip" approach</a>. You can upgrade an existing install by issuing (on your Linux VM's command line):</p> <p><code>sudo pip install -U docker-compose </code></p> <p>To set up your server, I recommend setting up a place for your Docker containers (replace "me" with your non-root username on the server) and the associated persistent data (your Docker containers should hold <em>no</em> important data - you should be able to delete and recreate them entirely without losing any important data or configuration):</p> <p><code>sudo mkdir /home/data</code><br /><code>sudo mkdir /home/data/nextcloud</code><br /><code>sudo mkdir /home/data/nextcloud/apps<br /> sudo mkdir /home/data/nextcloud/config<br /> sudo mkdir /home/data/nextcloud/data<br /> sudo mkdir /home/data/nextcloud/redis<br /> sudo mkdir /home/data/nextcloud/resources<br /> sudo mkdir /home/docker<br /> sudo mkdir /home/docker/nextcloud-collabora<br /> sudo chown -R me:me /home/docker<br /> cd /home/docker/nextcloud-collabora</code></p> <p>Here's an example of the required docker-compose.yml file (you can create this via a text editor like "nano" which should be pre-installed on any VM these days, or use my preferred, but less intuitive, editor, vim via <code>vim docker-compose.yml</code> in the /home/docker/nextcloud-collabora directory):</p> <p><code>version: '2'<br /> networks:<br />   back:<br />     driver: bridge<br /> services:<br />   web:<br />     image: nginx<br />     ports:<br />       -<br />     volumes:<br />       - ./nginx.conf:/etc/nginx/nginx.conf:ro<br />     links:<br />       - app<br />     volumes_from:<br />       - app<br />     environment:<br />       - VIRTUAL_HOST<br />     networks:<br />     - back<br />     restart: unless-stopped      <br />   app:<br />     image: nextcloud:12-fpm<br />     links:<br />       - redis<br />     volumes:<br />       - /home/data/nextcloud/apps:/var/www/html/apps<br />       - /home/data/nextcloud/config:/var/www/html/config<br />       - /home/data/nextcloud/resources:/var/www/html/resources<br />       - /home/data/nextcloud/data:/var/www/html/data<br />     networks:<br />     - back<br />     restart: unless-stopped      <br />   cron:<br />     image: nextcloud:12-fpm<br />     volumes_from:<br />       - app<br />     user: www-data<br />     entrypoint: |<br />       bash -c 'bash -s &lt;&lt;EOF<br />       trap "break;exit" SIGHUP SIGINT SIGTERM<br />       while /bin/true; do<br />         /usr/local/bin/php /var/www/html/cron.php<br />         sleep 900<br />       done<br />       EOF'<br />     networks:<br />       - back<br />     restart: unless-stopped      <br />   redis:<br />     image: redis:alpine<br />     volumes:<br />       - /home/data/nextcloud/redis:/data<br />     networks:<br />       - back<br />     restart: unless-stopped<br />   collab:<br />     image: collabora/code<br />     environment:</code><br /><code>      # put the domain name you select for your NextCloud instance<br />       # here! Escape any . in your domain name by preceding them with \\<br />       domain: your\\.domain\\.tld<br />       username: admin</code><br /><code>      # put your own strong password in here!<br />       password: some-good-password<br />     cap_add:<br />       - MKNOD<br />     networks:<br />       - back<br />     volumes_from:<br />       - app<br />     ports:<br />       -<br />     links:<br />       - app<br />     restart: unless-stopped</code></p> <p>You'll need to substitute the domain name you pick for your NextCloud instance - Collabora's container requires that you specify it so that it doesn't accept connections from other (potentially nefarious) containers elsewhere on the Internet!</p> <p>Also note, the "ports" specified above, 8082 for <code>nginx</code> and 9980 for <code>collab</code> are arbitrary - I picked these to ensure they don't conflict with ports being used by other containers on my server - you can use these if you want, or use <code>sudo netstat -punta</code> to see what ports are currently claimed by other services on your server (if there are any) and pick ones that don't clash! If it scroll past too fast, you can pipe it into less to allow you to scroll and search: <code>sudo netstat -punta | less</code> - hit "q" to exit or "/" to initiate a text search.</p> <p>You will also need to provide the "nginx.conf" file referenced in the nginx section of the file. Do that by using your editor, e.g. <code>vim nginx.conf</code>, and enter this content:</p> <p><code>user www-data;</code></p> <p><code>events {<br />   worker_connections 768;<br /> }</code></p> <p><code>http {<br />   upstream backend {</code><br /><code>      # if you don't call your NextCloud server "app" in your<br />       # docker-compose.yml, you'll need to change app below to </code><br /><code>      # whatever you end up calling it.<br />       server app:9000;<br />   }<br />   include /etc/nginx/mime.types;<br />   default_type application/octet-stream;</code></p> <p><code>  server {<br />     listen 80;<br />     <br />     # Add headers to serve security related headers<br />     add_header X-Content-Type-Options nosniff;<br />     add_header X-Frame-Options "SAMEORIGIN";<br />     add_header X-XSS-Protection "1; mode=block";<br />     add_header X-Robots-Tag none;<br />     add_header X-Download-Options noopen;<br />     add_header X-Permitted-Cross-Domain-Policies none;</code></p> <p><code>    root /var/www/html;</code></p> <p><code>    location = /robots.txt {<br />       allow all;<br />       log_not_found off;<br />       access_log off;<br />     }</code></p> <p><code>    location = /.well-known/carddav {<br />       return 301 $scheme://$host/remote.php/dav;<br />     }<br />     location = /.well-known/caldav {<br />       return 301 $scheme://$host/remote.php/dav;<br />     }</code></p> <p><code>    client_max_body_size 1G;<br />     fastcgi_buffers 64 4K;</code></p> <p><code>    gzip off;</code></p> <p><code>    index index.php;<br />     error_page 403 /core/templates/403.php;<br />     error_page 404 /core/templates/404.php;<br />  <br />     location / {<br />         rewrite ^ /index.php$uri;<br />     }</code></p> <p><code>    location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {<br />         deny all;<br />     }<br />     location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) {<br />         deny all;<br />     }</code></p> <p><code>    location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+|core/templates/40[34])\.php(?:$|/) {<br />         include fastcgi_params;<br />         fastcgi_split_path_info ^(.+\.php)(/.*)$;<br />         fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;<br />         fastcgi_param PATH_INFO $fastcgi_path_info;<br />         fastcgi_param HTTPS on;<br />         #Avoid sending the security headers twice<br />         fastcgi_param modHeadersAvailable true;<br />         fastcgi_param front_controller_active true;<br />         fastcgi_pass backend;<br />         fastcgi_intercept_errors on;<br />         fastcgi_request_buffering off;<br />     }</code></p> <p><code>    location ~ ^/(?:updater|ocs-provider)(?:$|/) {<br />         try_files $uri/ =404;<br />         index index.php;<br />     }</code></p> <p><code>    # Adding the cache control header for js and css files<br />     # Make sure it is BELOW the PHP block<br />     location ~* \.(?:css|js)$ {<br />         try_files $uri /index.php$uri$is_args$args;<br />         add_header Cache-Control "public, max-age=7200";<br />         # Add headers to serve security related headers (It is intended to<br />         # have those duplicated to the ones above)<br />         # Before enabling Strict-Transport-Security headers please read into<br />         # this topic first.<br />         # add_header Strict-Transport-Security "max-age=15768000;<br />         #  includeSubDomains; preload;";<br />         add_header X-Content-Type-Options nosniff;<br />         add_header X-Frame-Options "SAMEORIGIN";<br />         add_header X-XSS-Protection "1; mode=block";<br />         add_header X-Robots-Tag none;<br />         add_header X-Download-Options noopen;<br />         add_header X-Permitted-Cross-Domain-Policies none;<br />         # Optional: Don't log access to assets<br />         access_log off;<br />     }</code></p> <p><code>    location ~* \.(?:svg|gif|png|html|ttf|woff|ico|jpg|jpeg)$ {<br />         try_files $uri /index.php$uri$is_args$args;<br />         # Optional: Don't log access to other assets<br />         access_log off;<br />     }<br />   }<br /> }</code></p> <p>That should be all the configuration you need to make the Docker containers go.</p> <h2>Configuring Nginx to proxy NextCloud and Collabora</h2> <p>The next step is configuring the local nginx proxy servers for NextCloud and Collabora using the nginx instance you installed earlier. That's what responds to the domain name you choose for this service. In our case, the name is <a href="https://docs.oeru.org">https://docs.oeru.org</a> - you can have a look at it to see what you should be seeing when you first start things up! We use <a href="https://letsencrypt.org" title="This is an incredible free and open source service, that is single-handedly making the web a much safer place.">Let's Encrypt</a> to provide secure hosting - <a href="/protecting-your-users-lets-encrypt-ssl-certs">here're my Let's Encrypt instructions</a> on setting it up. The key thing to realise is that your "certificates" need to exist for Nginx to restart with the new configurations below - use the "commenting out the intervening lines" trick mentioned in my instructions to bootstrap the creation of your secure certificates!</p> <p>To configure the proxies, you need to create two configuration files in your /etc/nginx/sites-available/ directory.</p> <h3>NextCloud Proxy Configuration</h3> <p>Create a file with a meaningful name for your NextCloud Proxy, perhaps based on the domain name you've chosen (our file for docs.oeru.org is called "docs") using the same editing approach as the last few (although this is in a different directory) for example <code>sudo vim /etc/nginx/sites-available/docs</code> with the following contents, replacing "nextcloud.domain" with your selected domain name (and the port number 8082 if you've opted to change to a different one!):</p> <p><code>server {<br />     listen 80;<br />     server_name nextcloud.domain;</code></p> <p><code>    include /etc/nginx/includes/letsencrypt.conf;</code></p> <p><code>    # redirect all HTTP traffic to HTTPS.<br />     location / {<br />         return  302 https://nextcloud.domain$request_uri;<br />     }<br /> }</code></p> <p><code># This configuration assumes that there's an nginx container talking to the mautic PHP-fpm container,<br /> # and this is a reverse proxy for that Mautic instance.<br /> server {<br />     listen 443 ssl;<br />     server_name nextcloud.domain;</code></p> <p><code>    ssl_certificate /etc/letsencrypt/live/nextcloud.domain/fullchain.pem;<br />     ssl_certificate_key /etc/letsencrypt/live/nextcloud.domain/privkey.pem;<br />     ssl_protocols TLSv1 TLSv1.1 TLSv1.2;<br />     # to create this, see https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html<br />     ssl_dhparam /etc/ssl/certs/dhparam.pem;<br />     keepalive_timeout 20s;</code></p> <p><code>    include /etc/nginx/includes/letsencrypt.conf;<br />    <br />     location ^~ / {<br />         proxy_pass http://localhost:8082;<br />         proxy_set_header Upgrade $http_upgrade;<br />         proxy_set_header Connection "Upgrade";<br />         proxy_set_header Host $http_host;<br />         proxy_read_timeout 36000s;<br />     }<br />     client_max_body_size 1G;<br />     fastcgi_buffers 64 4K;</code></p> <p><code>    add_header X-Frame-Options "SAMEORIGIN";<br />     add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";<br /> }</code></p> <h3>Collab Proxy Configuration</h3> <p>Now create a collabora proxy configuration.</p> <p>Note: This will probably never by used by any user directly (there is a resource analysis service on the collabora system that might be of interest) - instead it'll be referenced by the NextCloud instance transparently to your users. </p> <p>In our case, we chose the domain collab.oeru.org and the file is called "collab", created via <code>sudo vim /etc/nginx/sites-available/collab</code> and containing (replace collab.domain with the one you've selected - similarly replace the port number 9980 with whatever you've selected if you've opted for a different one!):</p> <p><code>server {<br />     listen 80;<br />     server_name collab.domain;</code></p> <p><code>    # for let's encrypt renewals!<br />     include /etc/nginx/includes/letsencrypt.conf;</code></p> <p><code>    # redirect all HTTP traffic to HTTPS.<br />     location / {<br />         return  302 https://collab.domain$request_uri;<br />     }<br /> }</code></p> <p><code>server {<br />     listen 443 ssl;<br />     server_name collab.domain;</code></p> <p><code>    ssl_certificate /etc/letsencrypt/live/collab.domain/fullchain.pem;<br />     ssl_certificate_key /etc/letsencrypt/live/collab.domain/privkey.pem;<br />     ssl_protocols TLSv1 TLSv1.1 TLSv1.2;<br />     # to create this, see https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html<br />     ssl_dhparam /etc/ssl/certs/dhparam.pem;<br />     keepalive_timeout 20s;</code></p> <p><code>    # for let's encrypt renewals!<br />     include /etc/nginx/includes/letsencrypt.conf;</code></p> <p><code>    proxy_http_version 1.1;<br />     proxy_buffering off;</code></p> <p><code>    # static files<br />     location ^~ /loleaflet {<br />         proxy_pass https://localhost:9980;<br />         proxy_set_header Host $http_host;<br />     }</code></p> <p><code>    # WOPI discovery URL<br />     location ^~ /hosting/discovery {<br />         proxy_pass https://localhost:9980;<br />         proxy_set_header Host $http_host;<br />     }</code><br /><br /><code>    # download, presentation and image upload<br />     location ^~ /lool {<br />         proxy_pass https://localhost:9980;<br />         proxy_set_header Upgrade $http_upgrade;<br />         proxy_set_header Conection "upgrade";<br />         proxy_set_header Host $http_host;<br />     }<br /> }</code></p> <p>Once those are created, you have to make sure that they're "enabled" (replacing with your file names, of course):</p> <p><code>sudo cd /etc/nginx/sites-enabled<br /> sudo ln -sf ../sites-available/docs .<br /> sudo ln -sf ../sites-available/collab .</code></p> <p>To confirm that there aren't any typos or issues that might make nginx unhappy, run</p> <p><code>sudo nginx -t</code></p> <p>If all's well, get nginx to reread its configuration with the new files:</p> <p><code>sudo service nginx reload</code></p> <h2>Firing it all up!</h2> <p>Phew - congratulations on getting here! We've reached the moment of truth where we need to see if this whole thing will work!</p> <p>We need to make sure we're back in the Docker directory we set up:</p> <p><code>cd /home/docker/nextcloud-collabora</code></p> <p>and then we need to try running our docker-compose script to "pull" in the pre-built Docker containers we've specified in our docker-compose.yml file:</p> <p><code>docker-compose pull</code></p> <p>All going well, after a few minutes (longer or shorter depending on the speed of your server's connection) you should have download the Nginx, Redis, NextCloud and Collabora-CODE Docker images. Then you can run:</p> <p><code>docker-compose up -d &amp;&amp; docker-compose logs -f</code></p> <p>This will attempt to start up the containers (bringing them "up" in daemon mode, thus the -d) and then show you a stream of log messages from the containers, preceded by the container name. This should help you debug any problems that occur during the process (ideally, none).</p> <p>Once you see log messages streaming past, and no obvious "container exited" or other error messages (which will usually contain the word "error" a lot), you should be able to point your browser at your selected domain name and bring it up in your browser!</p> <h3>Setting up the database</h3> <p>On doing so, if all is well, you should be directed through the database set up process for your NextCloud instance. Your details should be:</p> <p>database IP: - this is the default IP of the Docker host server.<br /> database name: nextcloud<br /> database user: nextcloud<br /> database password: (the one you came up with above)</p> <h3>Setting the Admin user</h3> <p>Once that's set and working, NextCloud will install all the relevant database tables and initial data. You'll be asked to set up an <em>admin user</em> account, which can be "admin" (you could make it something different to help stymie nefarious probes that assume you've got a user called "admin" - but don't forget what you've called it!) and some strong password you create (you can use the pwgen utility you used earlier) - I'd recommend recording it somewhere. I would <em>not</em> recommend making your own account, in your name, the main admin account. I recommend creating a second account, <em>with administrator privileges</em> for yourself, but leave the admin account purely for administrative activities.</p> <h3>Configuring Outgoing Email</h3> <p>To allow your NextCloud instance to send outgoing email, so that your site can alert you to security updates that need to be applied, or so that users can request a replacement password if they've forgot theirs, you'll need an <em>authenticating SMTP account</em> somewhere. Most of you already have one. You'll probably want to set up a dedicated email address for this server somewhere, perhaps something like "<a href="mailto:nextcloud@your.domain">nextcloud@your.domain</a>" or similar, with a username (often just the email address) and a password. You'll need the following details:</p> <p>SMTP server : an IP address or a domain name<br /> SMTP username: a username or an email address<br /> SMTP password: a strong password already configured for the username on that server<br /> SMTP login security: whether login is via TLS, SSL, or unsecure (!!), and<br /> SMTP login method: plain, encrypted, "login" or some other value.</p> <p>You should be able to text your email settings to make sure the details you've entered are valid. If you need to adjust these settings later, you can go to the admin menu (top right of the web browser interface) and go to Admin-&gt;Additional Settings  - should have a path of <a href="https://your.domain/settings/admin/additional">https://your.domain/settings/admin/additional</a></p> <h3>Configuring Collabora Office Integration</h3> <p>Once you're logged in as your own user, looking at your own default folders, you can start having a look around. You should have an "admin" menu (assuming you've created your user with Administrator privileges) at the top right of the web interface. If you go to Apps, you can use the search box to search for "Collabora" or go to the "Office &amp; text" App category. You'll need to "enable" the Collabora Online "official" app, at which point it will download the latest version of the connector app and install it (it should appear in your /home/data/nextcloud/apps directory)</p> <p>Once you've done that, go to your top right menu again, selecting Admin, and you should see "Collabora Online" as an option in the left column (which starts with "Basic settings"). Selecting that, you'll need to enter  "<a href="https://collab.domain">https://collab.domain</a>" (replacing with your domain, of course). I don't have any of the other options ticked.</p> <p>If it works, you should have the ability to go back to the home of your NextCloud install, which should show you your top-level folders. If you click the "+" next to the home icon (top left of the folder pane) you should now have the option to create (in addition to "Upload file", "New folder", "New text file") a "New Document", "New Spreadsheet", and "New Presentation". Clicking those should give you the Collabora Office interface for the designated content type.</p> <p>Similarly, you can use the "Upload file" to upload a document in a format that is supported by Collabora Office, once uploaded clicking on the filename should open it for editing in the appropriate Collabora Office interface.</p> <p>It is saved as it is change, you shouldn't need to save it explicitly.</p> <h2>Upgrading it</h2> <p>So, as you're no doubt aware, both NextCloud and Collabora Office are always being improved and updated. I certainly encourage you to keep your installation up-to-date.</p> <p>While you'll periodically see that NextCloud apps have available updates (these can be upgraded through the browser interface) updates to the NextCloud and Collabora Office systems themselves need to be undertaken by upgrading the containers. Luckily it's easy to do (although I strongly urge you to ensure you have a very recent backup of both database and uploaded files - they're the files in /home/data/nextcloud/data:</p> <p>Updating the container should be as easy as either doing another</p> <p><code>docker pull oeru/mautic</code></p> <p>and then shutting down Docker container via a</p> <p><code>docker-compose stop</code></p> <p>removing the old containers (this won't remove any data you want to save if you followed the directions above! But remember to do it in the right directory!) via</p> <p><code>docker-compose rm -v</code></p> <p>and then restarting it via</p> <p><code>docker-compose up -d</code></p> <p>Use <code>docker-compose logs -f</code> to watch the logs - you'll likely see debugging information in the unlikely event that something goes wrong in the upgrade process.</p> <h2>Backing it up</h2> <p>To back up your instance on your server, you need two things: a file system backup of your /home/data/nextcloud directory, and database dumps of your database.</p> <p>There're lots of ways to back up your files (I personally use a bash script that I wrote in a past role, which uses <a href="http://www.nongnu.org/rdiff-backup/">rdiff-backup</a> to create versioned backups either locally or on a remote server, although there're <a href="https://www.howtoforge.com/linux_rdiff_backup">other documented approaches</a> - leave a comment below if you'd like to learn more about my approach!).</p> <p>Backing up your database is as easy installing automysqlbackups:</p> <p><code>sudo apt install automysqlbackups</code></p> <p>You'll find daily versioned dumps of your MariaDB database(s) in /var/lib/automysqlbackups. To run an ad hoc backup (which will replace the previous backup from that day, if there is one) just run</p> <p><code>sudo automysqlbackups</code></p> <h2>Collabora Admin Console</h2> <p>Once you've got everything set up, you can access the admin console of the Collabora Office instance at the collab.domain you specified above - it'll have the path <code>https://collab.domain/loleaflet/dist/admin/admin.html</code> (of course replacing collab.domain with your domain) which gives you useful info about the system resources being used, number of documents being edited and by whom, and some other interesting details. I've included a screen shot.</p> <p>When prompted for login details, use the collab username - "admin" if you used the default I provided, and the password you set in your docker-compose.yml file above.</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=17&amp;2=field_blog_comments&amp;3=comment" token="aF_f2aYW3SGRfmrsffb5fOJlN5wEXJibvMUiQKm2VjE"></drupal-render-placeholder> </div> </section> Mon, 29 Jan 2018 04:29:13 +0000 dave 17 at http://tech.oeru.org http://tech.oeru.org/installing-nextcloud-and-collabora-office-online-docker-ubuntu-1604#comments Installing LimeSurvey with Docker on Ubuntu 16.04 with Nginx and Mariadb http://tech.oeru.org/installing-limesurvey-docker-ubuntu-1604-nginx-and-mariadb <span class="field field--name-title field--type-string field--label-hidden">Installing LimeSurvey with Docker on Ubuntu 16.04 with Nginx and Mariadb</span> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Fri 19/01/2018 - 10:27</span> <div class="float-none field field-node--field-image field-name-field-image field-type-image field-label-hidden has-multiple"> <figure class="field-type-image__figure image-count-1 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-06/OERuSurvey-mainpage.png?itok=U9bb22aE" title="Limesurvey main page and menu" data-colorbox-gallery="gallery-field_image-rUaT7HdTu4I" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Limesurvey main page and menu&quot;}"><img src="/sites/default/files/styles/medium/public/2018-06/OERuSurvey-mainpage.png?itok=DUz2wzqw" width="220" height="165" alt="Limesurvey main page and menu" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-2 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-06/OERuSurvey-config.png?itok=0yRPUASD" title="Limesurvey administrator configuration" data-colorbox-gallery="gallery-field_image-rUaT7HdTu4I" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Limesurvey administrator configuration&quot;}"><img src="/sites/default/files/styles/medium/public/2018-06/OERuSurvey-config.png?itok=jx88srCu" width="220" height="168" alt="Limesurvey administrator configuration" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-3 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-06/OERuSurvey-question-edit.png?itok=8BB0VF1J" title="Limesurvey question editing interface" data-colorbox-gallery="gallery-field_image-rUaT7HdTu4I" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Limesurvey question editing interface&quot;}"><img src="/sites/default/files/styles/medium/public/2018-06/OERuSurvey-question-edit.png?itok=ERuWcUOS" width="209" height="220" alt="Limesurvey question editing interface" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-4 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-06/OERuSurvey-sample-survey_0.png?itok=OO_sL2D4" title="Limesurvey sample survey" data-colorbox-gallery="gallery-field_image-rUaT7HdTu4I" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Limesurvey sample survey&quot;}"><img src="/sites/default/files/styles/medium/public/2018-06/OERuSurvey-sample-survey_0.png?itok=jP-g_vyQ" width="220" height="219" alt="Limesurvey sample survey" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> </div> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p><a href="https://www.limesurvey.org/">LimeSurvey</a> is an open source survery tool, functionally similar to far more heavily marketed proprietary tools like SurveyMonkey and Google Forms. It's a very mature, fully-featured system. You can either use the reasonably priced hosted service available on the LimeSurvey site, or you can host your own, holding your own data and collecting your own analytics if you prefer. The only cost of hosting your own is your time to set it up and any costs associated with your hosting environment (which, today, can be negligible). Our instance is on <a href="https://survey.oeru.org">https://survey.oeru.org</a>...</p> <h2>Setup</h2> <p>It's as simple as having a server online somewhere (perhaps provided by your organisation or institution or if you're not sure where to go, I can recommend <a href="https://digitalocean.com" title="Their low-end Virtual Server would be sufficent, at USD5/month... ">Digital Ocean</a> - note: I have no affiliation with them other than being a happy customer)  and installing <a href="https://docs.docker.com/install/linux/docker-ce/ubuntu/">Docker-CE</a> (including <a href="https://docs.docker.com/install/linux/linux-postinstall/">use by non-root users</a>) and <a href="https://docs.docker.com/compose/install/#install-using-pip">Docker Compose</a> (I prefer to do this with 'pip' as described in the link).</p> <p>First create Docker and Data directories on your server:</p> <p><code>sudo mkdir /home/docker home/data<br /> sudo chown ${USER}:${USER} /home/docker home/data</code></p> <p>And we'll create a directory for Limesurvey's docker compose configuration and data respectively:</p> <p><code>mkdir /home/docker/limesurvey /home/data/limesurvey</code></p> <p>Then you simply have to create a <code>docker-compose.yml</code> file in /home/docker/limesurvey with your preferred editor (I use vim. If you don't know vim or vi, use nano which is easy to use and self-explanatory, e.g. <code>nano /home/docker/limesurvey/docker-compose.yml</code> ) containing the following:</p> <p><blockcode><code>version: '3'<br /> services:<br />   lime:<br />     restart: unless-stopped<br />     image: oeru/limesurvey<br />     ports:<br />       - ''<br />     volumes:<br />       - '/home/data/limesurvey/upload:/var/www/html/upload'<br />       - '/home/data/limesurvey/plugins:/var/www/html/plugins'<br />       - '/home/data/limesurvey/config:/var/www/html/application/config'<br />       - '/home/data/limesurvey/reference:/var/www/reference'</code></blockcode></p> <p>Last steps, you need your domain name you want to use, and to install Nginx on your server and configure it</p> <p><code>apt update &amp;&amp; apt install nginx-full</code></p> <p>and you then need to add a configuration file in /etc/nginx/sites-available/limesurvey that looks like this (let's say your domain is "survey.oeru.org" - obviously, replace with your own domain :) ):</p> <p><blockcode><code>server {<br />   listen 80;<br />   root /var/www/html;<br />   index index.html index.htm;<br />   server_name survey.oeru.org;</code></blockcode><br /><blockcode><code>  ## Access and error logs.<br />   access_log /var/log/nginx/survey.oeru.org_access.log;<br />   error_log /var/log/nginx/survey.oeru.org_error.log;</code></blockcode><blockcode><br /><code>#  include /etc/nginx/includes/letsencrypt.conf;</code></blockcode><blockcode><br /><code>#  location / {<br /> #     return 302 https://survey.oeru.org$request_uri;<br /> #  }<br /> #}</code></blockcode><blockcode><br /><code># server {<br /> #  listen 443 ssl;<br /> #  ssl on;<br /> #  ssl_certificate /etc/letsencrypt/live/survey.oeru.org/fullchain.pem;<br /> #  ssl_certificate_key /etc/letsencrypt/live/survey.oeru.org/privkey.pem;<br /> #  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;<br /> #  ssl_dhparam /etc/ssl/certs/dhparam.pem;</code></blockcode><blockcode><br /><code>#  keepalive_timeout 20s;</code></blockcode><blockcode><br /><code>#  root /var/www/html;<br /> #  index index.html index.htm;</code></blockcode><blockcode><br /><code>#  server_name survey.oeru.org;</code></blockcode><blockcode><br /><code>#  ## Access and error logs.<br /> #  access_log /var/log/nginx/survey.oeru.org_access.log;<br /> #  error_log /var/log/nginx/survey.oeru.org_error.log;</code></blockcode><blockcode><br /><code>  location / {<br />     proxy_set_header Host $host;<br />     proxy_set_header X-Forwarded-Host $host;<br />     proxy_set_header X-Real-IP $remote_addr;<br />     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br />     proxy_set_header X-Forwarded-Proto $scheme;<br />     add_header Front-End-Https  on;<br />     proxy_pass;<br />   }<br /> }</code></blockcode></p> <p>Note the port number 8080 - this is set in your <code>docker-compose.yml</code> file - it should be any free high port on your server (i.e. not already in use by another application - <code>netstat -punta</code> will help you work out what's in use). Change the port number to whatever you've set in the docker-compose.yml file.</p> <p>You'll note that many of the lines start with a # - these are commented out, particularly the info about SSL (secure sockets layer - encrypted transmission between users' computers and your server). Use our <a href="/protecting-your-users-lets-encrypt-ssl-certs">Lets Encrypt howto</a> to set that up - it's worth it for your users' security! Once you do that, you can remove the extra layer of # to create two server configurations - one for HTTP which redirects users to the other which implements HTTPS.</p> <p>You need to "enable" your nginx configuration like this:</p> <p><code>sudo ln -sf /etc/nginx/sites-available/limewire /etc/nginx/sites-enabled</code></p> <p>Then you can confirm that it's set up properly by running</p> <p><code>sudo nginx -t </code></p> <p>and if it returns without errors, you can implement your settings with (this should work even if you've never started nginx before!):</p> <p><code>sudo service nginx restart</code></p> <p>You should then be able to go to your site on <a href="http://survey.oeru.org">http://survey.oeru.org</a> (replaced, of course, with your own domain).</p> <h2>Upgrades</h2> <p>No recipe is very useful unless it gives you an idea how to provide ongoing maintenance for your application. Limesurvey is a wee bit tricky, but not too hard. This is how we do it when you see that there's an upgrade available (check <a href="https://hub.docker.com/r/oeru/limesurvey/">https://hub.docker.com/r/oeru/limesurvey/</a> to see!)</p> <ol><li>make a copy of your existing plugins and configuration: <code>sudo mkdir /home/data/limesurvey-backup &amp;&amp; cp -a /home/data/limesurvey/config /home/data/limesurvey/plugins /home/data/limesurvey-backup</code></li> <li>go to /home/docker/limesurvey (<code>cd /home/docker/limesurvey)</code> and run <code>docker-compose pull</code> to get the most recent Docker container I've produced.</li> <li>prepare your users for brief downtime.</li> <li>run docker-compose stop to stop the current Limesurvey site</li> <li>move the old config and plugin directories <code>sudo mv /home/data/limesurvey/config /home/data/limesurvey/config.old &amp;&amp; sudo mv /home/data/limesurvey/plugins /home/data/limesurvey/plugins.old</code> (this might overwrite previous old updates, which is hopefully ok! If not, choose something other than .old as a file suffix - perhaps the date)</li> <li>run <code>docker-compose up -d</code> This will start the new Limesurvey docker container.</li> <li>You can then run<code> docker-compose exec lime cp -a /var/www/html/application/config.orig /var/www/html/plugins.orig /var/www/reference/ </code>which is a sneaky way of running commands on the container itself. When I build the docker container you've "pulled" down, I copy the default config and plugins directories to config.orig and plugins.orig to ensure they're still there for you - this copies them somewhere on your local filesystem where you can usefully access them...</li> <li>You need to copy the reference configuration and plugins to the appropriate directories with <code>sudo cp -a /home/data/limesurvey/reference/config.orig/* /home/data/limesurvey/config/ &amp;&amp; sudo cp -a /home/data/limesurvey/reference/plugins.orig/* /home/data/limesurvey/plugins/</code></li> <li>Then you need to copy your previous config.php file to where it belongs: <code>sudo cp /home/data/limesurvey/config.old/config.php /home/data/limesurvey/config/</code> and the same with any plugins you have added to your instance. The latter will require you go copy them each individually from <code>/home/data/limesurvey/plugins.old/_____ to /home/data/limesurvey/plugins</code> - Just be sure not to overwrite plugins provided by the updated Limesurvey!</li> <li>make sure all the permissions are ok: <code>sudo chown -R www-data:${USER} /home/data/limesurvey/* &amp;&amp; sudo chmod -R g+w /home/data/limesurvey/* </code>which will allow both you to read and write all files and the webserver user (www-data) to read (and potentially write) the files it needs to.</li> <li>Finally, you need to run the database update in your browser (again, replacing with your domain name, and https if applicable): <code>http://survey.oeru.org/admin/upgrade</code> which should walk you through the process of upgrading. After that, you're good to go!</li> </ol><p> </p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=18&amp;2=field_blog_comments&amp;3=comment" token="Hm8NpVysFuvAd5MqABltr4y4y5jisiYytp23JzS6M-s"></drupal-render-placeholder> </div> </section> Thu, 18 Jan 2018 21:27:34 +0000 dave 18 at http://tech.oeru.org http://tech.oeru.org/installing-limesurvey-docker-ubuntu-1604-nginx-and-mariadb#comments WikiEducator Notes: OERu's course feed aggregation and messaging system http://tech.oeru.org/wikieducator-notes-oerus-course-feed-aggregation-and-messaging-system <span class="field field--name-title field--type-string field--label-hidden">WikiEducator Notes: OERu&#039;s course feed aggregation and messaging system</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--wenotes"> <span class="field__item-wrapper"><a href="/taxonomy/term/41" hreflang="en">wenotes</a></span> </div> <div class="field__item field__item--couchdb"> <span class="field__item-wrapper"><a href="/taxonomy/term/42" hreflang="en">couchdb</a></span> </div> <div class="field__item field__item--faye"> <span class="field__item-wrapper"><a href="/taxonomy/term/43" hreflang="en">faye</a></span> </div> <div class="field__item field__item--javascript"> <span class="field__item-wrapper"><a href="/taxonomy/term/44" hreflang="en">javascript</a></span> </div> <div class="field__item field__item--nodejs"> <span class="field__item-wrapper"><a href="/taxonomy/term/39" hreflang="en">node.js</a></span> </div> <div class="field__item field__item--docker"> <span class="field__item-wrapper"><a href="/taxonomy/term/16" hreflang="en">docker</a></span> </div> <div class="field__item field__item--docker-compose"> <span class="field__item-wrapper"><a href="/taxonomy/term/25" hreflang="en">docker compose</a></span> </div> <div class="field__item field__item--mastodon"> <span class="field__item-wrapper"><a href="/taxonomy/term/31" hreflang="en">mastodon</a></span> </div> <div class="field__item field__item--hypothesis"> <span class="field__item-wrapper"><a href="/taxonomy/term/45" hreflang="en">hypothesis</a></span> </div> <div class="field__item field__item--oeru"> <span class="field__item-wrapper"><a href="/taxonomy/term/46" hreflang="en">oeru</a></span> </div> <div class="field__item field__item--lida101"> <span class="field__item-wrapper"><a href="/taxonomy/term/47" hreflang="en">lida101</a></span> </div> <div class="field__item field__item--mediawiki"> <span class="field__item-wrapper"><a href="/taxonomy/term/38" hreflang="en">mediawiki</a></span> </div> <div class="field__item field__item--wikieducator"> <span class="field__item-wrapper"><a href="/taxonomy/term/34" hreflang="en">wikieducator</a></span> </div> <div class="field__item field__item--wordpress"> <span class="field__item-wrapper"><a href="/taxonomy/term/35" hreflang="en">wordpress</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Thu 24/08/2017 - 09:21</span> <div class="float-none field field-node--field-image field-name-field-image field-type-image field-label-hidden has-multiple"> <figure class="field-type-image__figure image-count-1 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-08/WEnotes_diagram.png?itok=_aB7Lt-Y" title="A diagram describing the function of WikiEducator Notes (WEnotes)" data-colorbox-gallery="gallery-field_image-rUaT7HdTu4I" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;A diagram describing the function of WikiEducator Notes (WEnotes)&quot;}"><img src="/sites/default/files/styles/medium/public/2017-08/WEnotes_diagram.png?itok=EncLVnTc" width="174" height="220" alt="A diagram describing the function of WikiEducator Notes (WEnotes)" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-2 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-09/WENotesPostMW.png?itok=sAdOfXNT" title="Example of a WEnotes feed and post widget on WikiEducator - MediaWiki" data-colorbox-gallery="gallery-field_image-rUaT7HdTu4I" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Example of a WEnotes feed and post widget on WikiEducator - MediaWiki&quot;}"><img src="/sites/default/files/styles/medium/public/2017-09/WENotesPostMW.png?itok=0QqAZWQf" width="220" height="182" alt="Example of a WEnotes feed and post widget on WikiEducator - MediaWiki" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-3 align-none"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-09/WENotesPostWP.png?itok=_58SHpuz" title="Example of a WEnotes feed and post widget on Course - WordPress Multisite" data-colorbox-gallery="gallery-field_image-rUaT7HdTu4I" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Example of a WEnotes feed and post widget on Course - WordPress Multisite&quot;}"><img src="/sites/default/files/styles/medium/public/2017-09/WENotesPostWP.png?itok=QQl6B5hF" width="220" height="182" alt="Example of a WEnotes feed and post widget on Course - WordPress Multisite" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> </div> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>Here at the OERu, we provide a service, attached to all of our online courses (and available to all of our partners - or anyone else for that matter) which allows anyone involved in those courses to communicate with one another from any one of a dazzling array of online "places" with WikiEducator Notes (aka WEnotes). The entire system is free and open source software (FOSS). </p> <p>The magic glue is "tags" - most social media and online systems support "tagging" in one way or another. Tools like Twitter, <a href="https://github.com/tootsuite/mastodon" title="A FOSS alternative to Twitter, but distributed rather than centralised. Ruby-on-Rails and React.js are the underlying technology.">Mastodon</a>, and G+ support "hashtags" like "#OERu" as shorthand. Other tools like blog engines (<a href="https://medium.com" title="A proprietary blogging cloud service and aggregator">Medium</a>, <a href="https://wordpress.org" title="The worlds most widely used web platform. FOSS blogging engine used by 25% of websites.">WordPress</a>, and others), forums (like <a href="https://discourse.org" title="A next generation forum engine, FOSS, built on Ruby on Rails and Ember.js for the front end.">Discourse</a>), content annotation services (like <a href="https://web.hypothes.is/" title="FOSS website annotation system - review any web page, sentence by sentence. ">Hypothesis</a>), social bookmarking services (like <a href="http://semanticscuttle.sourceforge.net/" title="The software underlying our https://bookmarks.oeru.org service">Semantic Scuttle</a>), and instant messaging services (like <a href="https://rocket.chat" title="A FOSS private messaging system built on Meteor and React.JS, with MongoDB">Rocket.Chat</a>, <a href="https://about.riot.im/" title="A FOSS React.js-based private messaging client system built on top of the Matrix (matrix.org) messaging standard and the &quot;Synapse&quot; server (Python + Twisted).">Riot</a> + <a href="https://matrix.org/docs/projects/server/synapse.html" title="The Matrix server used by the Riot messaging client. Written in Python + Twisted.">Synapse</a>, <a href="https://zulip.org" title="A FOSS private messaging service originally created by Dropbox for internal use. Written in Python with the Django framework and Electron clients.">Zulip</a>, and <a href="https://about.mattermost.com" title="A FOSS private messaging service built on Go (aka Golang)">Mattermost</a>, among others) support adding  "tags", "labels" or "categories" to any given post. WEnotes is sensitive to those tags, and thanks to the modern trend towards services providing feeds, it can happily "harvest" relevant content from all over the web, in real time (or nearly, depending on the technology).</p> <h2>How WEnotes components fit together</h2> <p>WEnotes is a collection of a significant number of separate FOSS components. See the graphical diagram of WEnotes alongside this article for an overview of the WEnotes component architecture. To provide further explanation, WEnotes is made up of several functional components:</p> <ol><li>A set of feed harvesting scripts that are part of the <a href="https://bitbucket.org/wikieducator/wenotes-tools">WEnotes-tools project</a> (written in Javascript and Python, for those interested in that sort of detail, the former ideal for those sites producing JSON feeds and real time updates, the latter our choice for periodic scans - every 10 minutes by default at the moment - of RSS and ATOM feeds)</li> <li>A central <a href="http://couchdb.apache.org/">CouchDB</a> instance, which stores all posts harvested, along with metadata like their source, a unique ID (to ensure we don't re-harvest it) and some useful info to ensure we can push the messages to subscribers effectively.</li> <li>The WEnotes client for MediaWiki or WordPress, part of the <a href="https://bitbucket.org/wikieducator/wenotes">WEnotes project</a>, which provides a real-time feed of posts, either a full feed (for example <a href="https://wikieducator.org/WENotesCompleteFeed">this one</a>) or specific to a tag (for example a specific <a href="https://course.oeru.org/lida101/interactions/course-feed/">OERu Course</a>). The client also offers a posting interface, allowing logged in users to post new posts (up to 300 characters) automatically tagged in the relevant context. See two screenshots attached.</li> <li>A "publish-subscribe" or "Pub-Sub" service, <a href="https://bitbucket.org/wikieducator/wenotes-server">WEnotes-server project</a>, a Node.JS based <a href="https://faye.jcoglan.com/">Faye</a> implementation of the <a href="https://docs.cometd.org/current/reference/index.html#_bayeux">Bayeux protocol</a>. It uses <a href="https://en.wikipedia.org/wiki/WebSocket">websockets</a> to push out real-time updates to subscribed WEnotes clients.</li> <li> <p>The "couchwatch.js" script, also part of the WEnotes-tools project, watches the CouchDB and alerts Faye whenever a new post is received, which in turn pushes it out to subscribed WENote clients.</p> </li> </ol><h2>Whence WEnotes posts come</h2> <p>Here are three scenarios which outline the process by which a WEnote post is created.</p> <h3>Handy WEnotes post form</h3> <p>The most obvious way to create a WEnotes post is to go to a WEnote feed page - for <a href="https://course.oeru.org/lida101/interactions/course-feed/">instance</a> - and log into the system (assuming you have authentication credentials or can create some). If you're logged in, you should see a simple form as illustrated in the two attached screen shots. The system then knows who you are so that your post can be properly attributed, and it knows the "context" for the feed you're looking at (namely the associated tag). You can enter a message of up to 300 characters, and hit the "Post a WEnote" button, and this will send - direct from your browser - a suitably formatted post (a simple JSON object - example below) into the WEnotes CouchDB where it is then available for inclusion in relevant feed displays.</p> <h3>Personal blog post, appropriately tagged</h3> <p>If you register for our <a href="https://course.oeru.org" title="The main OERu course delivery platform for OERu itself and many partner institutions.">Course site</a>, or any course on that site, you have the option of adding a "Blog URL" - that's the web address of your blog (the address of <em>this </em>blog is <code>https://tech.oeru.org</code>, for instance). Because most widely used blogging platforms, like the open source <a href="https://wordpress.org">WordPress</a> platform (which is the most widely use web platform in the world, the basis for <a href="https://w3techs.com/blog/entry/wordpress-powers-25-percent-of-all-websites">25% of <em>all</em> websites</a>!! For the record, this website uses <a href="https://drupal.org">Drupal</a>, which is the 3rd biggest web platform, and is also open source), and proprietary cloud blogging platforms like Blogger and Medium, provide both "tagging" of posts and a built-in <a href="http://www.whatisrss.com/">RSS feed</a>. When you know what to look for, you'll start to see them all over the place (this blog has <a href="/blog/feed">one</a> - the logo is the little grey icon on the bottom left corner of <a href="/blog" title="The Tech Blog index page.">this page</a>).</p> <p>When you create a blog post on your registered blog and tag it with a course-specific tag (each course we run has one, it's usually the short code immediately after the main course website URL: <code>https://course.oeru.org/[course code here]/...</code>) the WEnotes feed harvesters will periodically visit your blog's RSS feed and look for new posts tagged with a course tag. If it finds one, it will create a new post in CouchDB with all the necessary info to provide an informative post in the WEnotes feed, proper author attribution (and an "avatar" picture of you where available!), and a way for interested readers to get to your original post.</p> <h3>Tagged/Hashtagged social media reference</h3> <p>Because the attention of many people is heavily focused on social media - dominated by "network effect" companies like Twitter and Facebook, for better or worse, but also by an array of more open and egalitarian services (which we, on principle, prefer, but recognise that they're far less well marketed due mostly to not taking anyone's money) - we also provide a variety of social media specific "feed harvesters".  Some of these periodically scan specific social media technologies for new relevant posts, others perpetually monitor their sources providing instant updates.</p> <p>If you</p> <ul><li><a href="https://mastodon.oeru.org" title="OERu's Instance of the open source Mastodon social media platform.">Toot</a> including a relevant hashtag (e.g. #oeru or #lida101 for the "Learning in a Digital Age 101" course),</li> <li>use <a href="https://hypothes.is">Hypothesis</a> to annotate a web page, tagging it appropriately (e.g. oeru or lida101),</li> <li>add a <a href="http://semanticscuttle.sourceforge.net/">Semantic Scuttle</a> "<a href="https://bookmarks.oeru.org" title="Our Semantic Scuttle instance.">Bookmark</a>" to a web reference of interest, tagging it appropriately,</li> <li>add a topic or reply on our <a href="https://community.oeru.org" title="Our Discourse forum for educators and OER developers">Community</a> or <a href="https://forums.oeru.org" title="The OERu's Discourse forum for learners">Forums</a> Discourse instances also suitably tagged, or</li> <li>create a <a href="https://moodle.org/" title="Market leading learning management system (it's also open source).">Moodle</a> post on a Moodle site running a suitable "student bot",</li> </ul><p>the WEnotes feed harvesters will find it and create a suitable post in the CouchDB.</p> <h2>How WEnotes feeds are made</h2> <p>Once you have a bunch of WEnotes posts stored in CouchDB, how do we create a feed? At present, we have "WEnotes client" scripts that run either on MediaWiki instances, like Wikieducator, or on WordPress sites, like our Course multisite, for which content is usually provisioned per-course using our "<a href="/oeru-mediawiki-wordpress-snapshot-toolchain">Snapshot</a>" toolchain. In both cases, the client is invoked via the inclusion of a "Widget" in the MediaWiki content that will either be viewed on the <a href="https://wikieducator.org/WENotesCompleteFeed" title="An example of the WENotes Widget, in this case a full feed, not restricted to one or more tags.">MediaWiki instance itself</a>, or on a target <a href="https://course.oeru.org/lida101/interactions/course-feed/" title="This is an example of the widget being transferred to a WordPress course subsite. Note the 'Content' link at the bottom of the page, which points to the Wikieducator page which contains the actual Widget.">WordPress course subsite</a>.</p> <p>After you log into the Wikieducator site, and create or find a page on which you want a feed, you "Edit source" and you can add both a <a href="https://wikieducator.org/Widget:WEnotesPost" title="The WENotes Post Widget how-to documentation.">WEnotes Post template</a> and <a href="https://wikieducator.org/Widget:WEnotes" title="The WENotes Feed Widget documentation">WEnotes Feed template</a> two simple one line recipes like this:</p> <p><code>{{#widget:WEnotesPost|tag=lida101}}<br /> {{#widget:WEnotes|tag=lida101|count=20}} </code></p> <p>This invocation will create a page showing a WEnotes Post Widget, where any submitted post will automatically be tagged by "lida101", and directly below that, a feed showing up to 20 posts from the CouchDB tagged with "lida101". Note, any <em>new</em> suitably tagged posts will pop into the feed in realtime as they're added to CouchDB thanks to couchwatch.js and Faye - invoking a WEnotes Feed Widget automatically subscribes you to realtime updates for that tag.</p> <p>You can also create a cross-tag feed using the magic <code>"_"</code> designator. WEnotes Posts require a tag (the default, if none is specified, is "wikieducator"):</p> <p><code>{{#widget:WEnotes|tag=_|count=20}} </code></p> <h2>WEnotes technical details</h2> <p>This is a tech-blog, so I'm not going to shy away from the technical details. Here're some useful bits and pieces for folks wanting to get a better understanding of this system. If you're scared of technical details, you can stop reading here.</p> <h3>WEnotes technology infrastructure</h3> <p>We implement the services making up the WEnotes stack using a set of three Docker containers as well as our main MediaWiki implementation, WikiEducator, and our main WordPress multisite implementation, the OERu Course multisite (we also maintain development instance of each of these). The Docker containers are managed with <a href="/docker-compose-better-way-deploy-rocketchat-wekan-and-mongodb">Docker Compose</a> and the roles are divided  We've created a separate <a href="https://github.com/oeru/wenotes-docker">code repository on GitHub </a>for our evolving WEnotes stack to facilitate others making use of some or all of it! The three containers currently being deployed do the following:</p> <ol><li>run CouchDB version 2.0 or later, as well as its "Fauxton" interactive web front-end.</li> <li>run the Faye Pub-Sub websocket destination for realtime feed updates, implemented as a Node.JS service running under <a href="http://pm2.keymetrics.io/">pm2</a>.</li> <li>run the Javascript (using pm2) and Python-based (using cron to run periodically) feed harvesters and the Javascript "couchwatch.js" script which alerts Faye to the addition of new posts.</li> </ol><p>Once configured (we're working on making that a straight forward "out-of-the-box" experience, although it's not quite there yet), the three containers can be deployed together with a single invocation of Docker Compose - at the moment, though, this system is quite dependent on a properly configured "options.json" file, the format of which is still in a state of flux.</p> <h3>WENotes JSON object</h3> <p>A WEnotes post is stored in our CouchDB as a JSON object (JSON = <a href="json.org" title="JSON is rapidly superseding XML as a way of storing and propogating machine-readable structured data. ">JavaScript Object Notation</a>). This is an example of a Post as it's stored in our CouchDB:</p> <p>{<br />   "_id": "3e1c16eb1c99ae59899761a680006f4a",<br />   "_rev": "1-7462f14a0515d98b5985b9f4b8c39dd5",<br />   "media_attachments": [<br />     {<br />       "url": "<a href="https://mastodon.oeru.org/system/media_attachments/files/000/000/017/original/c3e85e56142c5c4d.png?1505016473">https://mastodon.oeru.org/system/media_attachments/files/000/000/017/or…</a>",<br />       "text_url": "<a href="https://mastodon.oeru.org/media/jvUoIpj6OAt-e5t-H8k">https://mastodon.oeru.org/media/jvUoIpj6OAt-e5t-H8k</a>",<br />       "preview_url": "<a href="https://mastodon.oeru.org/system/media_attachments/files/000/000/017/small/c3e85e56142c5c4d.png?1505016473">https://mastodon.oeru.org/system/media_attachments/files/000/000/017/sm…</a>",<br />       "meta": {<br />         "small": {<br />           "width": 400,<br />           "size": "400x321",<br />           "aspect": 1.2461059190031152,<br />           "height": 321<br />         },<br />         "original": {<br />           "width": 1148,<br />           "size": "1148x921",<br />           "aspect": 1.246471226927253,<br />           "height": 921<br />         }<br />       },<br />       "type": "image",<br />       "id": 17,<br />       "remote_url": ""<br />     }<br />   ],<br />   "truncated": true,<br />   "reblog": null,<br />   "id": 21,<br />   "in_reply_to_id": null,<br />   "content": "&lt;p&gt;Productive weekend fixing hashtags and instructions for &lt;a href=\"<a href="https://mastodon.oeru.org/tags/lida101">https://mastodon.oeru.org/tags/lida101</a>\" class=\"mention hashtag\" rel=\"tag\"&gt;#&lt;span&gt;LiDA101&lt;/span&gt;&lt;/a&gt; photo challenges . Super cool to see mastodon.oeru,org toots in the course feed. &lt;a href=\"<a href="https://mastodon.oeru.org/tags/lida101photo">https://mastodon.oeru.org/tags/lida101photo</a>\" class=\"mention hashtag\" rel=\"tag\"&gt;#&lt;span&gt;lida101photo&lt;/span&gt;&lt;/a&gt; &lt;a href=\"<a href="https://mastodon.oeru.org/media/jvUoIpj6OAt-e5t-H8k">https://mastodon.oeru.org/media/jvUoIpj6OAt-e5t-H8k</a>\" rel=\"nofollow noopener\" target=\"_blank\"&gt;&lt;span class=\"invisible\"&gt;https://&lt;/span&gt;&lt;span class=\"ellipsis\"&gt;mastodon.oeru.org/media/jvUoIp&lt;/span&gt;&lt;span class=\"invisible\"&gt;j6OAt-e5t-H8k&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;",<br />   "application": null,<br />   "text": "Productive weekend fixing hashtags and instructions for #LiDA101 photo challenges . Super cool to see mastodon.oeru,org toots in the course feed. #lida101photo...",<br />   "profile_url": "<a href="https://mastodon.oeru.org/@mackiwg">https://mastodon.oeru.org/@mackiwg</a>",<br />   "tags": [<br />     {<br />       "url": "<a href="https://mastodon.oeru.org/tags/lida101">https://mastodon.oeru.org/tags/lida101</a>",<br />       "name": "lida101"<br />     },<br />     {<br />       "url": "<a href="https://mastodon.oeru.org/tags/lida101photo">https://mastodon.oeru.org/tags/lida101photo</a>",<br />       "name": "lida101photo"<br />     }<br />   ],<br />   "visibility": "public",<br />   "we_tags": [<br />     "lida101"<br />   ],<br />   "user": {<br />     "screen_name": "mackiwg",<br />     "profile_image_url": "<a href="https://mastodon.oeru.org/system/accounts/avatars/000/000/002/original/1575afda498ac49c.jpg?1494995233">https://mastodon.oeru.org/system/accounts/avatars/000/000/002/original/…</a>",<br />     "name": "Wayne Mackintosh"<br />   },<br />   "spoiler_text": "",<br />   "we_timestamp": "2017-09-10T04:09:05.000Z",<br />   "account": {<br />     "username": "mackiwg",<br />     "display_name": "Wayne Mackintosh",<br />     "statuses_count": 10,<br />     "following_count": 1,<br />     "url": "<a href="https://mastodon.oeru.org/@mackiwg">https://mastodon.oeru.org/@mackiwg</a>",<br />     "locked": false,<br />     "created_at": "2017-04-30T01:51:59.778Z",<br />     "avatar_static": "<a href="https://mastodon.oeru.org/system/accounts/avatars/000/000/002/original/1575afda498ac49c.jpg?1494995233">https://mastodon.oeru.org/system/accounts/avatars/000/000/002/original/…</a>",<br />     "note": "&lt;p&gt;Open sourcing education at OERu&lt;/p&gt;",<br />     "header": "<a href="https://mastodon.oeru.org/headers/original/missing.png">https://mastodon.oeru.org/headers/original/missing.png</a>",<br />     "followers_count": 1,<br />     "avatar": "<a href="https://mastodon.oeru.org/system/accounts/avatars/000/000/002/original/1575afda498ac49c.jpg?1494995233">https://mastodon.oeru.org/system/accounts/avatars/000/000/002/original/…</a>",<br />     "header_static": "<a href="https://mastodon.oeru.org/headers/original/missing.png">https://mastodon.oeru.org/headers/original/missing.png</a>",<br />     "acct": "mackiwg",<br />     "id": 2<br />   },<br />   "favourites_count": 0,<br />   "language": "en",<br />   "url": "<a href="https://mastodon.oeru.org/@mackiwg/21">https://mastodon.oeru.org/@mackiwg/21</a>",<br />   "we_source": "mastodon",<br />   "in_reply_to_account_id": null,<br />   "uri": "tag:mastodon.oeru.org,2017-09-10:objectId=21:objectType=Status",<br />   "reblogs_count": 0,<br />   "sensitive": false,<br />   "mentions": [],<br />   "created_at": "2017-09-10T04:09:58.197Z"<br /> }</p> <p>The most crucial things for WEnotes are the values that start with '<code>we_</code>'. They are used by the system which filters and displays WEnotes posts. </p> <h3>The OERu Tag List</h3> <p>The list of tags which OERu's WEnotes looks for changes from time-to-time, usually growing as we add new course tags or other tags of business. Currently, the list is being updated manually, but it is <a href="http://wenotes.oeru.org/resources/feed-sources.json" title="WENotes Tag List">available as a web feed</a> in JSON format. We intend to provide both a simple editing interface for this list and to provide an API to automate adding new tags. Eventually, when we create a new course on our Course multisite, we expect to be able to automatically registered the new course "tag" with the tag list.</p> <h2>Credits</h2> <p>The entire underlying mechanics and most of the cleverness of WikiEducator Notes are thanks to the ingenuity and hard work of <a href="https://wikieducator.org/User:JimTittsler">Jim Tittsler</a>, my predecessor as OER Foundation Technical Lead.</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=16&amp;2=field_blog_comments&amp;3=comment" token="PxBEEjMNnN6bxJwA5uTvkIiXXSU76SK-WRBHv4dgcXc"></drupal-render-placeholder> </div> </section> Wed, 23 Aug 2017 21:21:04 +0000 dave 16 at http://tech.oeru.org http://tech.oeru.org/wikieducator-notes-oerus-course-feed-aggregation-and-messaging-system#comments