node.js

WikiEducator Notes: OERu's course feed aggregation and messaging system

dave — Wed, 23 Aug 2017 21:21:04 +0000

WikiEducator Notes: OERu's course feed aggregation and messaging system

Blog tags

wenotes

couchdb

faye

javascript

node.js

docker

docker compose

mastodon

hypothesis

oeru

lida101

mediawiki

wikieducator

wordpress

dave Thu 24/08/2017 - 09:21

Here at the OERu, we provide a service, attached to all of our online courses (and available to all of our partners - or anyone else for that matter) which allows anyone involved in those courses to communicate with their peers from any one of a dazzling array of online "places" with WikiEducator Notes (aka WEnotes). The entire system is free and open source software (FOSS).

The magic glue is "tags" - most social media and online systems support "tagging" in one way or another. Tools like Twitter, Mastodon, and G+ support "hashtags" like "#OERu" as shorthand. Other tools like blog engines (Medium, WordPress, and others), forums (like Discourse), content annotation services (like Hypothesis), social bookmarking services (like Semantic Scuttle), and instant messaging services (like Rocket.Chat, Riot + Synapse, Zulip, and Mattermost, among others) support adding "tags", "labels" or "categories" to any given post. WEnotes is sensitive to those tags, and thanks to the modern trend towards services providing feeds, it can happily "harvest" relevant content from all over the web, in real time (or nearly, depending on the technology).

How WEnotes components fit together

WEnotes is a collection of a significant number of separate FOSS components. See the graphical diagram of WEnotes alongside this article for an overview of the WEnotes component architecture. To provide further explanation, WEnotes is made up of several functional components:

A set of feed harvesting scripts that are part of the WEnotes-tools project (written in Javascript and Python, for those interested in that sort of detail, the former ideal for those sites producing JSON feeds and real time updates, the latter our choice for periodic scans - every 10 minutes by default at the moment - of RSS and ATOM feeds)
A central CouchDB instance, which stores all posts harvested, along with metadata like their source, a unique ID (to ensure we don't re-harvest it) and some useful info to ensure we can push the messages to subscribers effectively.
The WEnotes client for MediaWiki or WordPress, part of the WEnotes project, which provides a real-time feed of posts, either a full feed (for example this one) or specific to a tag (for example a specific OERu Course). The client also offers a posting interface, allowing logged in users to post new posts (up to 300 characters) automatically tagged in the relevant context. See two screenshots attached.
A "publish-subscribe" or "Pub-Sub" service, WEnotes-server project, a Node.JS based Faye implementation of the Bayeux protocol. It uses websockets to push out real-time updates to subscribed WEnotes clients.
The "couchwatch.js" script, also part of the WEnotes-tools project, watches the CouchDB and alerts Faye whenever a new post is received, which in turn pushes it out to subscribed WENote clients.

Whence WEnotes posts come

Here are three scenarios which outline the process by which a WEnote post is created.

Handy WEnotes post form

The most obvious way to create a WEnotes post is to go to a WEnote feed page - for instance - and log into the system (assuming you have authentication credentials or can create some). If you're logged in, you should see a simple form as illustrated in the two attached screen shots. The system then knows who you are so that your post can be properly attributed, and it knows the "context" for the feed you're looking at (namely the associated tag). You can enter a message of up to 300 characters, and hit the "Post a WEnote" button, and this will send - direct from your browser - a suitably formatted post (a simple JSON object - example below) into the WEnotes CouchDB where it is then available for inclusion in relevant feed displays.

Personal blog post, appropriately tagged

If you register for our Course site, or any course on that site, you have the option of adding a "Blog URL" - that's the web address of your blog (the address of this blog is https://tech.oeru.org, for instance). Because most widely used blogging platforms, like the open source WordPress platform (which is the most widely use web platform in the world, the basis for 25% of all websites!! For the record, this website uses Drupal, which is the 3rd biggest web platform, and is also open source), and proprietary cloud blogging platforms like Blogger and Medium, provide both "tagging" of posts and a built-in RSS feed. When you know what to look for, you'll start to see them all over the place (this blog has one - the logo is the little grey icon on the bottom left corner of this page).

When you create a blog post on your registered blog and tag it with a course-specific tag (each course we run has one, it's usually the short code immediately after the main course website URL: https://course.oeru.org/[course code here]/...) the WEnotes feed harvesters will periodically visit your blog's RSS feed and look for new posts tagged with a course tag. If it finds one, it will create a new post in CouchDB with all the necessary info to provide an informative post in the WEnotes feed, proper author attribution (and an "avatar" picture of you where available!), and a way for interested readers to get to your original post.

Tagged/Hashtagged social media reference

Because the attention of many people is heavily focused on social media - dominated by "network effect" companies like Twitter and Facebook, for better or worse, but also by an array of more open and egalitarian services (which we, on principle, prefer, but recognise that they're far less well marketed due mostly to not taking anyone's money) - we also provide a variety of social media specific "feed harvesters". Some of these periodically scan specific social media technologies for new relevant posts, others perpetually monitor their sources providing instant updates.

If you

Toot including a relevant hashtag (e.g. #oeru or #lida101 for the "Learning in a Digital Age 101" course),
use Hypothesis to annotate a web page, tagging it appropriately (e.g. oeru or lida101),
add a Semantic Scuttle "Bookmark" to a web reference of interest, tagging it appropriately,
add a topic or reply on our Community or Forums Discourse instances also suitably tagged, or
create a Moodle post on a Moodle site running a suitable "student bot",

the WEnotes feed harvesters will find it and create a suitable post in the CouchDB.

How WEnotes feeds are made

Once you have a bunch of WEnotes posts stored in CouchDB, how do we create a feed? At present, we have "WEnotes client" scripts that run either on MediaWiki instances, like Wikieducator, or on WordPress sites, like our Course multisite, for which content is usually provisioned per-course using our "Snapshot" toolchain. In both cases, the client is invoked via the inclusion of a "Widget" in the MediaWiki content that will either be viewed on the MediaWiki instance itself, or on a target WordPress course subsite.

After you log into the Wikieducator site, and create or find a page on which you want a feed, you "Edit source" and you can add both a WEnotes Post template and WEnotes Feed template two simple one line recipes like this:

{{#widget:WEnotesPost|tag=lida101}} {{#widget:WEnotes|tag=lida101|count=20}}

This invocation will create a page showing a WEnotes Post Widget, where any submitted post will automatically be tagged by "lida101", and directly below that, a feed showing up to 20 posts from the CouchDB tagged with "lida101". Note, any new suitably tagged posts will pop into the feed in realtime as they're added to CouchDB thanks to couchwatch.js and Faye - invoking a WEnotes Feed Widget automatically subscribes you to realtime updates for that tag.

You can also create a cross-tag feed using the magic "_" designator. WEnotes Posts require a tag (the default, if none is specified, is "wikieducator"):

{{#widget:WEnotes|tag=_|count=20}}

WEnotes technical details

This is a tech-blog, so I'm not going to shy away from the technical details. Here're some useful bits and pieces for folks wanting to get a better understanding of this system. If you're scared of technical details, you can stop reading here.

WEnotes technology infrastructure

We implement the services making up the WEnotes stack using a set of three Docker containers as well as our main MediaWiki implementation, WikiEducator, and our main WordPress multisite implementation, the OERu Course multisite (we also maintain development instance of each of these). The Docker containers are managed with Docker Compose and the roles are divided We've created a separate code repository on GitHub for our evolving WEnotes stack to facilitate others making use of some or all of it! The three containers currently being deployed do the following:

run CouchDB version 2.0 or later, as well as its "Fauxton" interactive web front-end.
run the Faye Pub-Sub websocket destination for realtime feed updates, implemented as a Node.JS service running under pm2.
run the Javascript (using pm2) and Python-based (using cron to run periodically) feed harvesters and the Javascript "couchwatch.js" script which alerts Faye to the addition of new posts.

Once configured (we're working on making that a straight forward "out-of-the-box" experience, although it's not quite there yet), the three containers can be deployed together with a single invocation of Docker Compose - at the moment, though, this system is quite dependent on a properly configured "options.json" file, the format of which is still in a state of flux.

WENotes JSON object

A WEnotes post is stored in our CouchDB as a JSON object (JSON = JavaScript Object Notation). This is an example of a Post as it's stored in our CouchDB:

{
"_id": "3e1c16eb1c99ae59899761a680006f4a",
"_rev": "1-7462f14a0515d98b5985b9f4b8c39dd5",
"media_attachments": [
 {
 "url": "https://mastodon.oeru.org/system/media_attachments/files/000/000/017/or…",
 "text_url": "https://mastodon.oeru.org/media/jvUoIpj6OAt-e5t-H8k",
 "preview_url": "https://mastodon.oeru.org/system/media_attachments/files/000/000/017/sm…",
 "meta": {
 "small": {
 "width": 400,
 "size": "400x321",
 "aspect": 1.2461059190031152,
 "height": 321
 },
 "original": {
 "width": 1148,
 "size": "1148x921",
 "aspect": 1.246471226927253,
 "height": 921
 }
 },
 "type": "image",
 "id": 17,
 "remote_url": ""
 }
],
"truncated": true,
"reblog": null,
"id": 21,
"in_reply_to_id": null,
"content": "Productive weekend fixing hashtags and instructions for <a href=\"https://mastodon.oeru.org/tags/lida101\" class=\"mention hashtag\" rel=\"tag\">#LiDA101</a> photo challenges . Super cool to see mastodon.oeru,org toots in the course feed. <a href=\"https://mastodon.oeru.org/tags/lida101photo\" class=\"mention hashtag\" rel=\"tag\">#lida101photo</a> <a href=\"https://mastodon.oeru.org/media/jvUoIpj6OAt-e5t-H8k\" rel=\"nofollow noopener\" target=\"_blank\">https://mastodon.oeru.org/media/jvUoIpj6OAt-e5t-H8k</a>",
"application": null,
"text": "Productive weekend fixing hashtags and instructions for #LiDA101 photo challenges . Super cool to see mastodon.oeru,org toots in the course feed. #lida101photo...",
"profile_url": "https://mastodon.oeru.org/@mackiwg",
"tags": [
 {
 "url": "https://mastodon.oeru.org/tags/lida101",
 "name": "lida101"
 },
 {
 "url": "https://mastodon.oeru.org/tags/lida101photo",
 "name": "lida101photo"
 }
],
"visibility": "public",
"we_tags": [
 "lida101"
],
"user": {
 "screen_name": "mackiwg",
 "profile_image_url": "https://mastodon.oeru.org/system/accounts/avatars/000/000/002/original/…",
 "name": "Wayne Mackintosh"
},
"spoiler_text": "",
"we_timestamp": "2017-09-10T04:09:05.000Z",
"account": {
 "username": "mackiwg",
 "display_name": "Wayne Mackintosh",
 "statuses_count": 10,
 "following_count": 1,
 "url": "https://mastodon.oeru.org/@mackiwg",
 "locked": false,
 "created_at": "2017-04-30T01:51:59.778Z",
 "avatar_static": "https://mastodon.oeru.org/system/accounts/avatars/000/000/002/original/…",
 "note": "Open sourcing education at OERu",
 "header": "https://mastodon.oeru.org/headers/original/missing.png",
 "followers_count": 1,
 "avatar": "https://mastodon.oeru.org/system/accounts/avatars/000/000/002/original/…",
 "header_static": "https://mastodon.oeru.org/headers/original/missing.png",
 "acct": "mackiwg",
 "id": 2
},
"favourites_count": 0,
"language": "en",
"url": "https://mastodon.oeru.org/@mackiwg/21",
"we_source": "mastodon",
"in_reply_to_account_id": null,
"uri": "tag:mastodon.oeru.org,2017-09-10:objectId=21:objectType=Status",
"reblogs_count": 0,
"sensitive": false,
"mentions": [],
"created_at": "2017-09-10T04:09:58.197Z"
}

The most crucial things for WEnotes are the values that start with 'we_'. They are used by the system which filters and displays WEnotes posts.

The OERu Tag List

The list of tags which OERu's WEnotes looks for changes from time-to-time, usually growing as we add new course tags or other tags of business. Currently, the list is being updated manually, but it is available as a web feed in JSON format. We intend to provide both a simple editing interface for this list and to provide an API to automate adding new tags. Eventually, when we create a new course on our Course multisite, we expect to be able to automatically register the new course "tag" with the tag list.

Credits

The entire underlying mechanics and most of the cleverness of WikiEducator Notes are thanks to the ingenuity and hard work of Jim Tittsler, my predecessor as OER Foundation Technical Lead.

Add new comment

OERu MediaWiki to WordPress Snapshot Toolchain

dave — Thu, 15 Jun 2017 23:19:06 +0000

OERu MediaWiki to WordPress Snapshot Toolchain

Blog tags

wikieducator

wordpress

multisite

snapshot

mediawiki

node.js

php

dave Fri 16/06/2017 - 11:19

One of the OERu's most compelling technological capabilities is the set of internally developed open source tools that allow us to automatically transform and inject a collection of MediaWiki content on WikiEducator (usually in form of micro-courses) making up a course into a mobile-friendly, easy-to-navigate WordPress site, usually situated on OERu's Course WordPress Multisite implementation.

How it works

Educators, usually working with one of our OERu partner institutions, can assemble the materials (either written by them or recruited from existing OERs on Wikieducator) into an "outline" which references all the content in one place. If they want to make the course available (either a cohort-based group or a "rolling" enrolment, either affiliated with their institution or under the OERu banner) they can do so either on a subsite on OERu's WordPress instance or some other WordPress, for example one hosted by their institution.

MediaWiki Widget

The process of linking an outline to a WordPress site and pushing a "snapshot" of the course content outlined to the WordPress site is remarkably simple. First, the author of the outline page includes a one line MediaWiki template or "widget" - which is managed by the open source OERu MediaWiki plugin (written in PHP, like the rest of MediaWiki) developed by Jim Tittsler - that will look something like this (usually in one line, expanded here for clarity):

{{ #widget:Snapshot | url=http://course.oeru.org/shortcode/ | logo=https://wikieducator.org/images/f/f1/MyInstitutionLogo.png | link=http://oeru.org/oeru-partners/my-institution-profile/}}

This defines

the URL of the WordPress site, using the "shortcode" the educator has defined for that course (like the institution's course code),
a logo image to display at the bottom of each page of the WordPress course site to identify the relevant institution, and
the link to which a user will be sent if they click on the logo.

There is a handy instruction page for the widget if you want a bit more detail.

Course Snapshot Script

When the page containing this template is saved, the resulting page will have a "Request snapshot" button at the top of it (see attached screenshot). Clicking this will popup a modal dialog box into which the user will be asked to enter their login and password for an admin user on the listed WordPress site (either multisite or stand-alone implementation).

If the WordPress site in question accepts these credentials, a new Snapshot task is added to WikiEducator's task list, handing over control to Jim's rather clever open source Course Snapshot script which runs in the background, and checks the task list ever minute or two. When it identifies as snapshot request, it uses the outline page and WikiEducator's API to grab all the relevant content automatically, and it builds a structured data archive (in a variant of XML) which, when completed, is imported directly into a WordPress site via WordPress' XMLRPC capabilities, replacing any existing content.

Note: the default export format used by the Course Snapshot script is a WordPress-importable data object, however for historical reasons, it also has the ability to generate (more manually at this stage) a static, internally linked HTML version of the content.

WordPress Mobile-Friendly Course Site

The resulting WordPress course site uses the OERu-created open source mobile-friendly Course WordPress theme - see the sample page screen shot. Each page includes a link back to the original page in WikiEducator from which it was generated - see sample page screen shot with the link identified. Also see the sample image of the same page as it would look on a mobile device with a narrower (lower pixel resolution) screen width.

Note: anyone can copy an outline page written by someone else and use it as the starting point for assembling their own version of the course - they will, however, have to have admin rights on the target WordPress site (and log in to it to initiate the process) protecting any existing snapshot sites from being overwritten by someone else's course content.

More examples

Here's a list of some of the currently available course outlines on WikiEducator with corresponding links to the "rendered" WordPress courses so you can see what it looks like.