ubuntu linux http://tech.oeru.org/ en Automatic versioned backups of Sqlite in a Docker Compose container http://tech.oeru.org/automatic-versioned-backups-sqlite-docker-compose-container <span class="field field--name-title field--type-string field--label-hidden">Automatic versioned backups of Sqlite in a Docker Compose container</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--sqlite"> <span class="field__item-wrapper"><a href="/taxonomy/term/72" hreflang="en">sqlite</a></span> </div> <div class="field__item field__item--bash"> <span class="field__item-wrapper"><a href="/taxonomy/term/58" hreflang="en">bash</a></span> </div> <div class="field__item field__item--backup"> <span class="field__item-wrapper"><a href="/taxonomy/term/57" hreflang="en">backup</a></span> </div> <div class="field__item field__item--ubuntu-linux"> <span class="field__item-wrapper"><a href="/taxonomy/term/12" hreflang="en">ubuntu linux</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Tue 04/02/2020 - 12:17</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>For relatively lightweight applications that are either in development, or single user, or have limited requirements for concurrency and massive data sets, <a href="https://en.wikipedia.org/wiki/SQLite">SQLite</a> is a superb, full-function, but compact, <em>almost ubiquitous</em> database (it's used on every mobile device, for example).</p> <p>What's more, it's also Free and Open Source Software (FOSS) - its code has been dedicated to the public domain - making it an ideal tool for a principled open organisation like the OER Foundation to incorporate in its fully Free and Open Source Software stack.</p> <p>We use SQLite for quite a few things here at the OER Foundation. For example, this implementation of the backup script was developed for the very impressive fully FOSS version of the <a href="/setting-your-own-bitwarden-password-keeper-and-sync-server">password manager BitWarden </a>we use to manage our secrets at the OER Foundation. Because the data it holds, although very compact, is very precious, we want to back it up right, and ensure we can go back in time and recreate the data from the past (like, if we realised at some point that in the past we'd deleted the wrong secrets and needed to recover them! It hasn't happened yet, but it's inevitable it <em>will happen</em> at some point). That's why we've created this backup solution that creates dated dumps of an SQLite database that it keeps to ensure that relevant past versions of the data can be recovered, but without storing more data than necessary (a full hard disk is never fun).</p> <p>Those among you who know about SQLite might rightly point out that an SQLite database (unlike more powerful databases) is simply a single file that can be copied to make a backup. In this case, we're using SQLite's ".backup" command to create the database backup, which we see as a mechanism for future-proofing - at some future time, it might be that the option to copy the file without corrupting the database may no longer be available, but we'd expect that the .backup command would continue to work.</p> <p>This backup process involves 3 files: the backup BASH script itself, which has all the smarts, the .conf file which defines the file paths and other details specific to your installation - like where the SQLite database actually is, and where to put the backup files - and a cron file which runs the script automatically when you tell it to run on your server.</p> <p>We've got the <a href="https://git.oeru.org/dave/sqlite-backup">whole thing</a> in our <a href="https://git.oeru.org">Gitlab instance</a>, which is the best place to learn more about how to use it - see the <a href="https://git.oeru.org/dave/sqlite-backup/blob/master/README.md">README.md</a> for installation instructions!</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=30&amp;2=field_blog_comments&amp;3=comment" token="LUsqyib8Nl62guCFKZHLxak9ymOGimis4ahdLWzFAfE"></drupal-render-placeholder> </div> </section> Mon, 03 Feb 2020 23:17:20 +0000 dave 30 at http://tech.oeru.org Installing NextCloud Hub with OnlyOffice on Ubuntu 18.04 http://tech.oeru.org/installing-nextcloud-hub-onlyoffice-ubuntu-1804 <span class="field field--name-title field--type-string field--label-hidden">Installing NextCloud Hub with OnlyOffice on Ubuntu 18.04</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--ubuntu-linux"> <span class="field__item-wrapper"><a href="/taxonomy/term/12" hreflang="en">ubuntu linux</a></span> </div> <div class="field__item field__item--_804"> <span class="field__item-wrapper"><a href="/taxonomy/term/68" hreflang="en">18.04</a></span> </div> <div class="field__item field__item--nextcloud"> <span class="field__item-wrapper"><a href="/taxonomy/term/51" hreflang="en">nextcloud</a></span> </div> <div class="field__item field__item--onlyoffice"> <span class="field__item-wrapper"><a href="/taxonomy/term/69" hreflang="en">onlyoffice</a></span> </div> <div class="field__item field__item--mariadb"> <span class="field__item-wrapper"><a href="/taxonomy/term/48" hreflang="en">mariadb</a></span> </div> <div class="field__item field__item--docker-compose"> <span class="field__item-wrapper"><a href="/taxonomy/term/25" hreflang="en">docker compose</a></span> </div> <div class="field__item field__item--docker"> <span class="field__item-wrapper"><a href="/taxonomy/term/16" hreflang="en">docker</a></span> </div> <div class="field__item field__item--php"> <span class="field__item-wrapper"><a href="/taxonomy/term/40" hreflang="en">php</a></span> </div> <div class="field__item field__item--redis"> <span class="field__item-wrapper"><a href="/taxonomy/term/21" hreflang="en">redis</a></span> </div> <div class="field__item field__item--polls"> <span class="field__item-wrapper"><a href="/taxonomy/term/70" hreflang="en">polls</a></span> </div> <div class="field__item field__item--scheduling"> <span class="field__item-wrapper"><a href="/taxonomy/term/71" hreflang="en">scheduling</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Tue 04/02/2020 - 09:41</span> <div class="field field-node--field-image field-name-field-image field-type-image field-label-hidden has-multiple"> <figure class="field-type-image__figure image-count-1"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2020-02/Screenshot_2020-02-11-2%20file-sample_1MB%20docx%20-%20Lane%20NextCloud.png?itok=bqhSZGni" title="Sample DOCX file being edited in open source OnlyOffice" data-colorbox-gallery="gallery-field_image--TK6YVjLOro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Sample DOCX file being edited in open source OnlyOffice&quot;}"><img src="/sites/default/files/styles/medium/public/2020-02/Screenshot_2020-02-11-2%20file-sample_1MB%20docx%20-%20Lane%20NextCloud.png?itok=OW_2e1WM" width="220" height="140" alt="Sample DOCX file being edited in open source OnlyOffice" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-2"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2020-02/Screenshot_2020-02-11%20Digital%20Storage%20Weight%20and%20Volume%20xlsx%20-%20Lane%20NextCloud.png?itok=luaOg5rt" title="Sample XLSX file being edited in open source OnlyOffice" data-colorbox-gallery="gallery-field_image--TK6YVjLOro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Sample XLSX file being edited in open source OnlyOffice&quot;}"><img src="/sites/default/files/styles/medium/public/2020-02/Screenshot_2020-02-11%20Digital%20Storage%20Weight%20and%20Volume%20xlsx%20-%20Lane%20NextCloud.png?itok=CZcvoi7c" width="220" height="140" alt="Sample XLSX file being edited in open source OnlyOffice" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-3"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2020-02/Screenshot_2020-02-11%20file-sample_1MB%20docx%20-%20Lane%20NextCloud.png?itok=GzQZ26uC" title="Another example of DOCX file being edited in open source OnlyOffice" data-colorbox-gallery="gallery-field_image--TK6YVjLOro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Another example of DOCX file being edited in open source OnlyOffice&quot;}"><img src="/sites/default/files/styles/medium/public/2020-02/Screenshot_2020-02-11%20file-sample_1MB%20docx%20-%20Lane%20NextCloud.png?itok=9KcMLR4t" width="220" height="140" alt="Another example of DOCX file being edited in open source OnlyOffice" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-4"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2020-02/Screenshot_2020-02-11%20Files%20-%20OERu%20NextCloud.png?itok=Zm3uP_wT" title="Sample of web-view of NextCloud files and folders." data-colorbox-gallery="gallery-field_image--TK6YVjLOro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Sample of web-view of NextCloud files and folders.&quot;}"><img src="/sites/default/files/styles/medium/public/2020-02/Screenshot_2020-02-11%20Files%20-%20OERu%20NextCloud.png?itok=Bpnx2qg6" width="220" height="155" alt="Sample of web-view of NextCloud files and folders." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-5"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2020-02/Screenshot_2020-02-11%20Polls%20-%20NZOSS%20Nextcloud.png?itok=Fp17si_J" title="A sample NextCloud Poll (alternative to Doodle Polls) for scheduling. Yes, it&#039;s timezone-aware!" data-colorbox-gallery="gallery-field_image--TK6YVjLOro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;A sample NextCloud Poll (alternative to Doodle Polls) for scheduling. Yes, it&#039;s timezone-aware!&quot;}"><img src="/sites/default/files/styles/medium/public/2020-02/Screenshot_2020-02-11%20Polls%20-%20NZOSS%20Nextcloud.png?itok=8rpIpQne" width="220" height="154" alt="A sample NextCloud Poll (alternative to Doodle Polls) for scheduling. Yes, it&#039;s timezone-aware!" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-6"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2020-02/Screenshot_2020-02-12%20ONLYOFFICE%E2%84%A2.png?itok=04xrGCN1" title="If your onlyoffice.domain server is working, this is what you should see in your browser!" data-colorbox-gallery="gallery-field_image--TK6YVjLOro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;If your onlyoffice.domain server is working, this is what you should see in your browser!&quot;}"><img src="/sites/default/files/styles/medium/public/2020-02/Screenshot_2020-02-12%20ONLYOFFICE%E2%84%A2.png?itok=1I3F7Cbi" width="220" height="187" alt="If your onlyoffice.domain server is working, this is what you should see in your browser!" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-7"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2020-02/Screenshot_2020-02-13%20Apps%20-%20NZOSS%20Nextcloud.png?itok=Io3v-pzT" title="The NextCloud &quot;app&quot; configuration page" data-colorbox-gallery="gallery-field_image--TK6YVjLOro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;The NextCloud &quot;app&quot; configuration page&quot;}"><img src="/sites/default/files/styles/medium/public/2020-02/Screenshot_2020-02-13%20Apps%20-%20NZOSS%20Nextcloud.png?itok=VGORRtXl" width="103" height="220" alt="The NextCloud &quot;app&quot; configuration page" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-8"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2020-02/Screenshot_2020-02-13%20Settings%20-%20NZOSS%20Nextcloud.png?itok=8sZGBrtG" title="The NextCloud settings page. " data-colorbox-gallery="gallery-field_image--TK6YVjLOro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;The NextCloud settings page. &quot;}"><img src="/sites/default/files/styles/medium/public/2020-02/Screenshot_2020-02-13%20Settings%20-%20NZOSS%20Nextcloud.png?itok=98LUc9zv" width="220" height="163" alt="The NextCloud settings page. " typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-9"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2020-02/Screenshot_2020-02-13%20Settings%20-%20OnlyOffice%20-%20NZOSS%20Nextcloud_0.png?itok=KwWWWKUT" title="The &quot;OnlyOffice&quot; app configuration page for NextCloud" data-colorbox-gallery="gallery-field_image--TK6YVjLOro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;The &quot;OnlyOffice&quot; app configuration page for NextCloud&quot;}"><img src="/sites/default/files/styles/medium/public/2020-02/Screenshot_2020-02-13%20Settings%20-%20OnlyOffice%20-%20NZOSS%20Nextcloud_0.png?itok=JdKSA022" width="220" height="189" alt="The &quot;OnlyOffice&quot; app configuration page for NextCloud" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-10"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2020-02/Screenshot_2020-02-21%20App%20Bundles-%20OERu%20NextCloud.png?itok=3N1GtflH" title="NextCloud App Bundle configuration page." data-colorbox-gallery="gallery-field_image--TK6YVjLOro" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;NextCloud App Bundle configuration page.&quot;}"><img src="/sites/default/files/styles/medium/public/2020-02/Screenshot_2020-02-21%20App%20Bundles-%20OERu%20NextCloud.png?itok=c1_0PwiH" width="220" height="141" alt="NextCloud App Bundle configuration page." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> </div> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>I have previously provided an <a href="/installing-nextcloud-and-collabora-office-online-docker-ubuntu-1604">in-depth explanation about NextCloud with Collabora Office Online and how we've installed it on Ubuntu 16.04</a>. This is an update both of the process, and of the technology. NextCloud is leaping from strength to strength, and seems to be <a href="https://nextcloud.com/blog/eu-governments-choose-independence-from-us-cloud-providers-with-nextcloud/">benefiting from</a> the well-founded concern held by many in the EU about data sovereignty and the market domination (and exploitation) of US-based multinationals like Amazon, Google, Microsoft, Dropbox, and others. As a collaborative, web-based front end to LibreOffice, Collabora shows great potential... but it's not anywhere near the capabilities of Google Docs...</p> <p>The same, however, is not true of a relatively new entry into the web-based collaborative productivity application space: <a href="https://www.onlyoffice.com/">OnlyOffice</a>. The application itself (for the tech focused reader, they've built an entirely <a href="https://github.com/ONLYOFFICE/">new application ecosystem</a> primarily using modern Javascript frameworks)  is impressive in both capabilities and polish. The only real caveat I've come across is that it uses, by default, the <a href="https://openstandards.nz/case-study-microsofts-ooxml-standard">fauxpen standard formats</a> developed by Microsoft rather than the true open standard formats of <a href="https://en.wikipedia.org/wiki/OpenDocument">OpenDocumentFormat</a>. But in a world where, sadly, most people don't even know what a file format is, any software that doesn't read and write the incumbent monopolist's format with great fidelity is dead in the water.  On that count, OnlyOffice is impressive.</p> <h2>NextCloud and OnlyOffice - even better together!</h2> <p>The beauty of the open source software model is that we can connect complementary applications, like NextCloud and OnlyOffice - developed by completely separate communities - to create a tightly integrated, highly functional, diverse computing platform. This combination, along with a bunch of other NextCloud "apps", is the equal of something like Google Apps (which includes Google Docs and Google Drive), but is <em>under your control, not Google's.</em> To me, that's a crucial difference. </p> <p>With the release of NextCloud 18.0.1, NextCloud has bundled OnlyOffice with it, creating something called "<a href="https://nextcloud.com/hub/">NextCloud Hub</a>". It's pretty impressive. That's what we're setting up here!</p> <h2>Setting up your own NextCloud Hub!</h2> <p>Yes, NextCloud and OnlyOffice servers on the same host.</p> <p>If you're game to run your own (and, in my experience, it's a surprisingly well behaved system) here's how you do it.</p> <p>In preparation, you'll want to have the following ready:</p> <ul><li>a Linux virtual machine or "VM" (I recommend running the current Ubuntu LTS version, or current Debian) with an external IP address and a user with sudo privileges - <a href="/setting-your-own-bitwarden-password-keeper-and-sync-server">more info on that</a>...,</li> <li>your domain name for the NextCloud instance, pointing to the IP address of your VM,</li> <li>credentials for an email address capable of sending from a remote server (usually termed an "<a href="/configuring-linux-server-send-email-postfix-smtp-server">authenticating SMTP email account</a>")</li> </ul><p>Please note: the images accompanying this howto have been pulled from several different NextCloud and OnlyOffices I maintain.</p> <h3>Secure access with SSH</h3> <p>First things first, make sure you're logged into your host (probably via SSH) as a user who has "sudo" capabilities! You need to log into the host from your local machine. We recommend setting up <a href="https://www.digitalocean.com/community/tutorials/how-to-configure-ssh-key-based-authentication-on-a-linux-server">key-based authentication</a>.</p> <h3>Firewall with UFW</h3> <p>No computer system is ever full secure - there're always exploits waiting to be found, so security is a process of maintaining vigilance. Part of that is reducing exposure - minimising your "attack surface". Use a firewall - "<a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-ubuntu-16-04" title="Uncomplicated FireWall">ufw</a>" is installed on Ubuntu by default. Make sure you've got exceptions for SSH (without them, you could lock yourself out of your machine! Doh!).</p> <p>Run the following commands to allow your Docker containers to talk to other services on your host.</p> <p><code>sudo ufw allow in on docker0<br /> sudo ufw allow from 172.0.0.0/8 to any</code></p> <p>Specifically for Docker's benefit, you need to tweak the default Forwarding rule (I use "vim" as my editor. If you don't know how to/want to use it, replace <strong>vim</strong> with <strong>nano</strong> everywhere you see it in the following - nano's easier to use for simple edits like this):</p> <p><code>sudo vim /etc/default/ufw</code></p> <p>and copy the line <code>DEFAULT_FORWARD_POLICY="DROP"</code> tweak it to look like this (commenting out the default, but leaving it there for future reference!):</p> <p><code>#DEFAULT_FORWARD_POLICY="DROP"<br /> DEFAULT_FORWARD_POLICY="ACCEPT"</code></p> <p>You also have to edit <code>/etc/ufw/sysctl.conf</code> and remove the "#" at the start of the following lines, so they look like this:</p> <p><code>sudo vim /etc/ufw/sysctl.conf</code></p> <p><code># Uncomment this to allow this host to route packets between interfaces<br /> net/ipv4/ip_forward=1<br /> net/ipv6/conf/default/forwarding=1<br /> net/ipv6/conf/all/forwarding=1</code></p> <p>and finally restart the network stack and ufw on your server<code> </code></p> <p><code>sudo service networking restart<br /> sudo service ufw restart</code></p> <h3>Installing the Nginx webserver</h3> <p>In the configuration I'm describing here, you'll need a webserver running on the server - it'll be acting as a "proxy" for the Docker-based Nginx instance described below. I like the efficiency of Nginx and clarity of Nginx configurations over those of Apache and other open source web servers. Here's how you install it.</p> <p><code>sudo apt-get install nginx-full</code></p> <p>To allow nginx to be visible via ports 80 and 443, run</p> <p><code>sudo ufw allow "Nginx Full"</code></p> <p><strong>Note</strong>: make sure your hosting service is not blocking these ports at some outer layer (depending on who's providing that hosting service you may have to set up port forwarding).</p> <h3>Installing MariaDB</h3> <p>MariaDB is effectively a drop-in alternative to MySQL and we prefer it because it's not controlled by Oracle and has a more active developer community. On Ubuntu, MariaDB pretends to be MySQL for compatibility purposes, so don't be weirded out by the interchangeable names below. Install the server and the client like this.</p> <p><code>sudo apt-get install mariadb-server-10.0 mariadb-client-10.0</code></p> <p>You need to set a root (admin) user password - you might want to create a /root/.my.cnf file containing the following (replacing YOURPASSWORD) to let you access MariaDB without a password from the commandline<code>:</code></p> <p><code>[client]<br /> user=root<br /> password=YOURPASSWORD</code></p> <p>You should now be able to type "mysql" at the command prompt</p> <p>Tweak the configuration so that it's listening on</p> <p><code>sudo vim /etc/mysql/mariadb.conf.d/50-server.cnf </code></p> <p>and copy the bind-address line and adjust so it looks like this - we want MariaDB to be listening on all interfaces, not just localhost (127.0.0.1)...</p> <p><code># Instead of skip-networking the default is now to listen only on<br /> # localhost which is more compatible and is not less secure.<br /> #bind-address           = 127.0.0.1<br /> bind-address            = 0.0.0.0</code></p> <p>Then restart MariaDB:</p> <p><code>sudo service mysql restart</code></p> <p>It should now be listening on port 3306 on all interfaces, i.e. 0.0.0.0.</p> <p>Now set up the database which will hold NextCloud's data. Log into the MySQL client on the host (if you've created a .my.cnf file in your home directory as describe above, you won't need to enter your username and password):</p> <p><code>mysql -u root -p</code></p> <p>Enter your root password when prompted. It's also a good idea to gin up a password for your "nextcloud" database user. I usually use pwgen (<code>sudo apt-get install pwgen</code>) - for example running this command will give you a single 19 character password without special characters (just numbers and letters):</p> <p><code>pwgen -s 19 1</code></p> <p>Giving you something like this (but if it's truly random, almost <em>certainly not exactly </em>this):</p> <p>bYIOSrvR9aGwL5FRGFU</p> <p>At the prompt (which will look something like <code>MariaDB [(none)]&gt;</code>) enter the following lines (putting your password in place of [passwd]):</p> <p><code>CREATE DATABASE nextcloud CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;<br /> CREATE USER "nextcloud"@"%" IDENTIFIED BY "[passwd]";<br /> GRANT ALL ON nextcloud.* to "nextcloud"@"%";<br /> FLUSH PRIVILEGES;</code></p> <p>Then enter \q to exit.</p> <h2>Prepare your Docker Compose host</h2> <p>We make use of the NextCloud community's <a href="https://hub.docker.com/_/nextcloud/" title="Documentation for the reference NextCloud Docker container.">stable Docker container</a> which they keep up to date. Similarly, the OnlyOffice developers maintain a Docker container, too. We will run them both on this same server as separate services via <a href="https://docs.docker.com/compose/">Docker Compose</a>. The two sets of Docker containers will look like this:</p> <ol><li>a suite of NextCloud containers: <ol><li>the main PHP-FPM container (which provides most of the functionality for NextCloud using the PHP scripting engine,</li> <li>an identical container to the PHP one which runs the cron service (which does periodic administrative tasks relevant to NextCloud)</li> <li>a Redis container (which provides performance improving caching for NextCloud), and</li> <li>an Nginx webserver container which makes it easier to manage the configuration and paths of the NextCloud instance. It means that on the hosting server, we only need to run a proxying web server, which is easy.</li> </ol></li> <li>the single OnlyOffice container which, despite the Docker convention of each container running only a single services, runs the whole OnlyOffice stack, which includes PostgreSQL, Nginx, Rabbit-MQ, Python, and NodeJS.</li> </ol><p>The way I prefer to implement this set of containers is to use:</p> <p><code>sudo apt-get install docker-compose </code></p> <p>to set up the entire Docker and Docker Compose system on your server.</p> <p>Then set up a place for your Docker containers (replace "me" with your non-root username on the server) and the associated persistent data (your Docker containers should hold <em>no</em> important data - you should be able to delete and recreate them entirely without losing any important data or configuration):</p> <p><code>sudo mkdir /home/data</code><br /><code>sudo mkdir /home/data/nextcloud<br /> sudo mkdir /home/data/nextcloud-nginx<br /> sudo mkdir /home/data/nextcloud-redis</code><br /><code>sudo mkdir /home/data/onlyoffice</code><br /><code>sudo mkdir /home/docker<br /> sudo mkdir /home/docker/nextcloud</code><br /><code>sudo chown -R me:me /home/docker</code></p> <h2>NextCloud Install</h2> <p>First, let's set up NextCloud (this also installs the OnlyOffice server):</p> <p><code>cd /home/docker/nextcloud</code></p> <p>Here's an example of the required docker-compose.yml file (you can create this via a text editor like "nano" which should be pre-installed on any VM these days (or use my preferred, but less intuitive, editor, vim) <code>nano docker-compose.yml</code> in the /home/docker/nextcloud directory):</p> <p><code>version: '3'<br /> services:<br />   nginx:<br />     container_name: nginx-server<br />     image: nginx<br />     ports:<br />       - 127.0.0.1:8082:80<br />     volumes:<br />       - /home/data/nextcloud-nginx/nginx/nginx.conf:/etc/nginx/nginx.conf:ro<br />       - /home/data/nextcloud:/var/www/html<br />     links:<br />       - app<br />     environment:<br />       - VIRTUAL_HOST<br />     restart: unless-stopped      <br />   app:<br />     container_name: app-server<br />     image: nextcloud:fpm<br />     stdin_open: true<br />     tty: true<br />     links:<br />       - redis<br />     expose:<br />       - '80'<br />       - '9000'<br />     volumes:<br />       - /home/data/nextcloud:/var/www/html<br />     restart: unless-stopped      <br />   cron:<br />     image: nextcloud:fpm<br />     volumes:<br />       - /home/data/nextcloud:/var/www/html<br />     user: www-data<br />     entrypoint: |<br />       bash -c 'bash -s &lt;&lt;EOF<br />       trap "break;exit" SIGHUP SIGINT SIGTERM<br />       while /bin/true; do<br />         /usr/local/bin/php /var/www/html/cron.php<br />         sleep 900<br />       done<br />       EOF'<br />     restart: unless-stopped      <br />   redis:<br />     image: redis:alpine<br />     volumes:<br />       - /home/data/nextcloud-redis:/data<br />     restart: unless-stopped<br />   onlyoffice-document-server:<br />     container_name: onlyoffice-document-server<br />     image: onlyoffice/documentserver:latest<br />     stdin_open: true<br />     tty: true<br />     restart: unless-stopped<br />     expose:<br />       - '80'<br />       - '443'<br />     volumes:<br />       - /home/data/onlyoffice/data:/var/www/onlyoffice/Data<br />       - /home/data/onlyoffice/log:/var/log/onlyoffice</code></p> <p>The "port" specified above, 8082 for <code>nginx</code> is arbitrary - I picked it to ensure it doesn't don't conflict with ports being used by other containers on my server - you can use these if you want, or use <code>sudo netstat -punta</code> to see what ports are currently claimed by other services on your server (if there are any) and pick one that doesn't clash! If it scroll past too fast, you can pipe it into less to allow you to scroll and search like this: <code>sudo netstat -punta | less</code> - hit "q" to exit or "/" to initiate a text search.</p> <p>You will also need to provide the "nginx.conf" file referenced in the nginx section of the file. Do that by using your editor, e.g. <code>nano nginx.conf</code>, and enter this content (you shouldn't need to alter anything):</p> <p><code>user  www-data;</code></p> <p><code>worker_processes  1;</code></p> <p><code>error_log  /var/log/nginx/error.log warn;<br /> pid        /var/run/nginx.pid;</code></p> <p><code>events {<br />     worker_connections  1024;<br /> }</code></p> <p><code>http {<br />     upstream backend {<br />         server app-server:9000;<br />     }</code></p> <p><code>    include       /etc/nginx/mime.types;<br />     default_type  application/octet-stream;</code></p> <p><code>    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '<br />                       '$status $body_bytes_sent "$http_referer" '<br />                       '"$http_user_agent" "$http_x_forwarded_for"';</code></p> <p><code>    access_log  /var/log/nginx/access.log  main;</code></p> <p><code>    sendfile        on;<br />     #tcp_nopush     on;</code></p> <p><code>    keepalive_timeout  65;</code></p> <p><code>    map $http_host $this_host {<br />         "" $host;<br />         default $http_host;<br />     }</code></p> <p><code>    map $http_x_forwarded_proto $the_scheme {<br />         default $http_x_forwarded_proto;<br />         "" $scheme;<br />     }</code></p> <p><code>    map $http_x_forwarded_host $the_host {<br />         default $http_x_forwarded_host;<br />         "" $this_host;<br />     }</code></p> <p><code>    server {<br />         listen 80;</code></p> <p><code>        # Add headers to serve security related headers<br />         add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";<br />         add_header X-Content-Type-Options nosniff;<br />         add_header X-XSS-Protection "1; mode=block";<br />         add_header X-Robots-Tag none;<br />         add_header X-Download-Options noopen;<br />         add_header X-Permitted-Cross-Domain-Policies none;</code></p> <p><code>        root /var/www/html;<br />         client_max_body_size 10G; # 0=unlimited - set max upload size<br />         fastcgi_buffers 64 4K;</code></p> <p><code>        gzip off;</code></p> <p><code>        index index.php;<br />         error_page 403 /core/templates/403.php;<br />         error_page 404 /core/templates/404.php;</code></p> <p><code>        rewrite ^/.well-known/carddav /remote.php/dav/ permanent;<br />         rewrite ^/.well-known/caldav /remote.php/dav/ permanent;</code></p> <p><code>        location = /robots.txt {<br />             allow all;<br />             log_not_found off;<br />             access_log off;<br />         }</code></p> <p><code>        location ~ ^/(build|tests|config|lib|3rdparty|templates|data)/ {<br />             deny all;<br />         }</code></p> <p><code>        location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) {<br />             deny all;<br />         }</code></p> <p><code>        location / {<br />             rewrite ^/remote/(.*) /remote.php last;<br />             rewrite ^(/core/doc/[^\/]+/)$ $1/index.html;<br />             try_files $uri $uri/ =404;<br />         }</code></p> <p><code>        location ~* ^/ds-vpath/ {<br />             rewrite /ds-vpath/(.*) /$1  break;<br />             proxy_pass http://onlyoffice-document-server;<br />             proxy_redirect     off;</code></p> <p><code>            client_max_body_size 100m;</code></p> <p><code>            proxy_http_version 1.1;<br />             proxy_set_header Upgrade $http_upgrade;<br />             proxy_set_header Connection "upgrade";</code></p> <p><code>            proxy_set_header Host $http_host;<br />             proxy_set_header X-Real-IP $remote_addr;<br />             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br />             proxy_set_header X-Forwarded-Host $the_host/ds-vpath;<br />             proxy_set_header X-Forwarded-Proto $the_scheme;<br />             #proxy_set_header X-Forwarded-Proto 'https';<br />         }</code></p> <p><code>        location ~ \.php(?:$|/) {<br />             fastcgi_split_path_info ^(.+\.php)(/.+)$;<br />             include fastcgi_params;<br />             fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;<br />             fastcgi_param PATH_INFO $fastcgi_path_info;<br />             fastcgi_param HTTPS off;<br />             fastcgi_param modHeadersAvailable true; #Avoid sending the security headers twice<br />             fastcgi_pass backend;<br />             fastcgi_intercept_errors on;<br />         }</code></p> <p><code>        # Adding the cache control header for js and css files<br />         # Make sure it is BELOW the location ~ \.php(?:$|/) { block<br />         location ~* \.(?:css|js)$ {<br />             add_header Cache-Control "public, max-age=7200";<br />             # Add headers to serve security related headers<br />             add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";<br />             add_header X-Content-Type-Options nosniff;<br />             add_header X-Frame-Options "SAMEORIGIN";<br />             add_header X-XSS-Protection "1; mode=block";<br />             add_header X-Robots-Tag none;<br />             add_header X-Download-Options noopen;<br />             add_header X-Permitted-Cross-Domain-Policies none;<br />             # Optional: Don't log access to assets<br />             access_log off;<br />         }</code></p> <p><code>        # Optional: Don't log access to other assets<br />         location ~* \.(?:jpg|jpeg|gif|bmp|ico|png|swf)$ {<br />             access_log off;<br />         }</code><br /><code>    }<br /> }</code></p> <p>That should be all the configuration you need to make the Docker containers go.</p> <h2>Configuring Nginx to proxy NextCloud and OnlyOffice</h2> <p>The next step is configuring the local nginx proxy servers for NextCloud and OnlyOffice using the nginx instance you installed earlier. That's what responds to the domain name you choose for this service. In our case, the name is <a href="https://docs.oeru.org">https://docs.oeru.org</a> - you can have a look at it to see what you should be seeing when you first start things up! We use <a href="https://letsencrypt.org" title="This is an incredible free and open source service, that is single-handedly making the web a much safer place.">Let's Encrypt</a> to provide secure hosting - <a href="/protecting-your-users-lets-encrypt-ssl-certs">here're my Let's Encrypt instructions</a> on setting it up. The key thing to realise is that your "certificates" need to exist for Nginx to restart with the new configurations below - use the "commenting out the intervening lines" trick mentioned in my instructions to bootstrap the creation of your secure certificates!</p> <p>To configure the proxy, you need to create this configuration file in your /etc/nginx/sites-available/ directory.</p> <h3>NextCloud Proxy Configuration</h3> <p>Create a file with a meaningful name for your NextCloud Proxy, perhaps based on the domain name you've chosen (our file for docs.oeru.org is called "docs") using the same editing approach as the last few (although this is in a different directory) for example <code>sudo nano /etc/nginx/sites-available/nextcloud</code> with the following contents, replacing <code>[nextcloud.domain]</code> with your selected domain name, but leave off the [ ] (those are just there to make sure nginx errors if you've missed replacing any) - and the port number 8082 if you've opted to change to a different one!:</p> <p><code>server {<br />     listen 80;<br />     listen [::]:80;<br />     server_name <strong>[nextcloud.domain]</strong>;</code></p> <p><code>    include includes/letsencrypt.conf;</code></p> <p><code>    # enforce https<br />     location / {<br />         return 302 https://$server_name$request_uri;<br />     }<br /> }</code></p> <p><code>server {<br />     listen 443 ssl;<br />     listen [::]:443 ssl;<br />     #listen 127.0.0.1:443 ssl;</code></p> <p><code>    server_name <strong>[nextcloud.domain]</strong>;</code></p> <p><code>    ## Access and error logs.<br />     access_log /var/log/nginx/<strong>[nextcloud.domain]</strong>_access.log;<br />     error_log /var/log/nginx/<strong>[nextcloud.domain]</strong>_error.log;</code></p> <p><code>    ssl_certificate /etc/letsencrypt/live/<strong>[nextcloud.domain]</strong>/fullchain.pem;<br />     ssl_certificate_key /etc/letsencrypt/live/<strong>[nextcloud.domain]</strong>/privkey.pem;</code></p> <p><code>    ssl on;<br />     # from http://axiacore.com/blog/enable-perfect-forward-secrecy-nginx/<br />     ssl_session_cache shared:SSL:10m;<br />     ssl_session_timeout  10m;<br />     # limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s;<br />     # forward secrecy settings<br />     ssl_protocols TLSv1 TLSv1.1 TLSv1.2;<br />     ssl_prefer_server_ciphers on;<br />     ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4";<br />     ssl_dhparam /etc/ssl/certs/dhparam.pem;</code></p> <p><code>    #location = /robots.txt {<br />     #    allow all;<br />     #    log_not_found off;<br />     #    access_log off;<br />     #}</code></p> <p><code>    # The following 2 rules are only needed for the user_webfinger app.<br />     # Uncomment it if you're planning to use this app.<br />     rewrite ^/.well-known/host-meta /public.php?service=host-meta last;<br />     rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json last;</code></p> <p><code>    # The following rule is only needed for the Social app.<br />     # Uncomment it if you're planning to use this app.<br />     rewrite ^/.well-known/webfinger /public.php?service=webfinger last;</code></p> <p><code>    location ^~ / {<br />         proxy_pass http://127.0.0.1:<strong>8082</strong>;<br />         proxy_set_header Upgrade $http_upgrade;<br />         proxy_set_header Connection "Upgrade";<br />         proxy_set_header Host $http_host;<br />         proxy_read_timeout 36000s;<br />         proxy_buffering off;<br />         proxy_max_temp_file_size 15000m;<br />     }<br />     client_max_body_size 1G;<br />     fastcgi_buffers 64 4K;<br />     add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";<br />     # Remove X-Powered-By, which is an information leak<br />     fastcgi_hide_header X-Powered-By;<br /> }</code></p> <p>Note: you'll need to create the file cited in the proxy configration: <code>/etc/ssl/certs/dhparam.pem </code></p> <p>You can do this as follows (install the necessary software, backup any possible existing version as a matter of prudence, and create a new one):</p> <p><code>sudo apt update &amp;&amp; sudo apt install openssl<br /><span class="pun">sudo [<span class="pln"> </span>-</span><span class="pln">f </span><span class="str">"</span>/etc/ssl/certs/dhparam.pem<span class="str">"</span><span class="pln"> </span><span class="pun">]</span><span class="pln"> </span><span class="pun">&amp;&amp;</span><span class="pln"> sudo mv </span>/etc/ssl/certs/dhparam.pem /etc/ssl/certs/dhparam.pem</code>.bak<br /><span style="font-family:monospace"><span style="color:#000000;background-color:#ffffff;">sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048</span></span><br />  </p> <p>Once those are created, you have to make sure that they're "enabled" (replacing with your file names, of course):</p> <p><code>sudo cd /etc/nginx/sites-enabled<br /> sudo ln -sf ../sites-available/nextcloud .</code></p> <p>To confirm that there aren't any typos or issues that might make nginx unhappy, run</p> <p><code>sudo nginx -t</code></p> <p>If all's well, get nginx to reread its configuration with the new files (if not, it might be because you missed replacing one of the [tokens]):</p> <p><code>sudo service nginx reload</code></p> <h2>Firing up your NextCloud!</h2> <p>Phew - congratulations on getting here! We've reached the moment of truth where we need to see if this whole thing will work!</p> <p>We need to make sure we're back in the NextCloud Docker directory we set up:</p> <p><code>cd /home/docker/nextcloud</code></p> <p>and then we need to try running our docker-compose script to "pull" in the pre-built Docker containers we've specified in our docker-compose.yml file:</p> <p><code>docker-compose pull</code></p> <p>All going well, after a few minutes (longer or shorter depending on the speed of your server's connection) you should have download the Nginx, Redis, and NextCloud Docker images. Then you can run:</p> <p><code>docker-compose up -d &amp;&amp; docker-compose logs -f</code></p> <p>This will attempt to start up the containers (bringing them "up" in daemon mode, thus the -d) and then show you a stream of log messages from the containers, preceded by the container name. This should help you debug any problems that occur during the process (ideally, none).</p> <p>Once you see log messages streaming past, and no obvious "container exited" or other error messages (which will usually contain the word "error" a lot), you should be able to point your browser at your selected domain name and bring it up in your browser! Just point your web browser at <code>https://nextcloud.domain</code> (replacing with your domain, of course - the https assumes you've got your Let's Encrypt certificate set up - I recommend doing that first).</p> <h3>Configuring database access</h3> <p>On doing so, if all is well, you should be directed through the database set up process for your NextCloud instance. Your details should be:</p> <p>database IP: 172.17.0.1 - this is the default IP of the Docker host server.<br /> database name: nextcloud<br /> database user: nextcloud<br /> database password: (the one you came up with above)</p> <h3>Configuring the Admin user</h3> <p>Once that's set and working, NextCloud will install all the relevant database tables and initial data. You'll be asked to set up an <em>admin user</em> account, which can be "admin" (you could make it something different to help stymie nefarious probes that assume you've got a user called "admin" - but don't forget what you've called it!) and some strong password you create (you can use the pwgen utility you used earlier) - I'd recommend recording it somewhere. I would <em>not</em> recommend making your own account, in your name, the main admin account. Instead, I recommend creating a second account, <em>with administrator privileges</em>, for yourself, but leave the admin account purely for administrative activities.</p> <h3>Configuring Outgoing Email</h3> <p>To allow your NextCloud instance to send outgoing email, so that your site can alert you to security updates that need to be applied, or so that users can request a replacement password if they've forgot theirs, you'll need an <em>authenticating SMTP account</em> somewhere. Most of you already have one. You'll probably want to set up a dedicated email address for this server somewhere, perhaps something like "<a href="mailto:nextcloud@your.domain">nextcloud@your.domain</a>" or similar, with a username (often just the email address) and a password. You'll need the following details:</p> <p>SMTP server : an IP address or a domain name<br /> SMTP username: a username or an email address<br /> SMTP password: a strong password already configured for the username on that server<br /> SMTP login security: whether login is via TLS, SSL, or unsecure (!!), and<br /> SMTP login method: plain, encrypted, "login" or some other value.</p> <p>You should be able to test your email settings to make sure the details you've entered are valid. If you need to adjust these settings later, you can go to the admin menu (top right of the web browser interface) and go to Admin-&gt;Additional Settings  - should have a path of <code>https://nextcloud.domain/settings/admin/additional</code></p> <h2>Setting up OnlyOffice</h2> <p>The OnlyOffice server should already be running - if you point your browser at <code>https://nextcloud.domain/ds-vpath/</code> you should see something like the "Document Server is running" (with a big green "tick") page included in the images accompanying this article.</p> <h3>Configuring OnlyOffice Integration with NextCloud</h3> <p>Once you're logged in to NextCloud as your own user, looking at your own default folders, you can start having a look around. You should have an "admin" menu (assuming you've created your user with Administrator privileges) at the top right of the web interface. If you go to Apps, you can install the new "Hub bundle" available under the "App bundles" option (see attached image). If you don't want the whole bundle you can just use the search box to search for "OnlyOffice" or go to the "Office &amp; text" App category and enable the OnlyOffice "official" app, at which point it will automatically download the latest version of the connector app and install it (it should appear in your /home/data/nextcloud/apps directory)</p> <p>Once you've done that, go to your top right menu again, selecting Admin, and you should see "OnlyOffice" as an option in the left column (which starts with "Basic settings"). Selecting that, you'll need to enter the following:</p> <ul><li> "Document Editing Service address":<code> /ds-vpath/</code></li> <li><code>"</code>Secret key": (leave blank)</li> <li> Under "Advanced server settings" <ul><li> <p class="onlyoffice-header">"Document Editing Service address for internal requests from the server": <code>http://onlyoffice-document-server/</code></p> </li> <li> <p class="onlyoffice-header">"Server address for internal requests from the Document Editing Service": <code>http://nginx-server/</code></p> </li> </ul></li> </ul><p>When you're done, click "Save".</p> <p>You can also select formats you'd like OnlyOffice to open and edit files of those types are clicked or created. I've selected the following: doc, docx, odp, ods, odt, ppt, pptx, xls, xlsx, and in the second section: csv and txt.</p> <p>You can also make other editor customisations as you desire. The only Editor customisation setting I <em>haven't</em> selected is "Display Chat menu button" because NextCloud Hub provides an integrated Chat service, making this one within OnlyOffice an unnecessary distraction.</p> <p>Once finished configuring, you should have the ability to go back to the home of your NextCloud install, which should show you your top-level folders. If you click the "+" next to the home icon (top left of the folder pane) you should now have the option to create (in addition to "Upload file", "New folder", "New text file") a "New Document", "New Spreadsheet", and "New Presentation". Clicking those should give you the OnlyOffice interface for the designated content type.</p> <p>Similarly, you can use the "Upload file" to upload a document in a format that is supported by OnlyOffice. Once uploaded, clicking on the filename should open it for editing in the appropriate OnlyOffice interface.</p> <p>It is saved as it is changed, so you shouldn't need to save it explicitly.</p> <h2>Keeping the whole thing up-to-date</h2> <p>So, as you're no doubt aware, both NextCloud and OnlyOffice are always being improved and updated. I certainly encourage you to keep your installations up-to-date.</p> <p>While you'll periodically be alerted that NextCloud <strong>apps</strong> have available updates (these can be upgraded through the browser interface) updates to the NextCloud and OnlyOffice systems themselves need to be undertaken by upgrading their containers. Luckily it's easy to do (although I strongly urge you to ensure you have a very recent backup of both database and uploaded files - they're the files in /home/data/nextcloud/data and /home/data/onlyoffice/ (note, backups of OnlyOffice are complicated somewhat by the fact that you can't reliably back up running PostgreSQL instance simply by backing up its files - see a solution below):</p> <p>Updating the container should be as easy as either doing another</p> <p><code>docker-compose pull </code></p> <p>and then shutting down Docker container via a</p> <p><code>docker-compose up -d</code></p> <p>which will remove any old containers (this won't remove any data you want to save if you followed the directions above! But remember to do it in the right directory!) and start up the new versions you've just pulled.</p> <p>Use <code>docker-compose logs -f</code> to watch the logs - you'll likely see useful debugging information in the unlikely event that something goes wrong in the upgrade process.</p> <h2>Backing up NextCloud</h2> <p>To back up your instance on your server, you need two things: a file system backup of your /home/data/nextcloud directory, and database dumps of your database.</p> <p>There're lots of ways to back up your files (I've recently updated to using a system called Restic to make off-server incremental encrypted backups - I plan to document this in a future howto! - although there're <a href="https://www.howtoforge.com/linux_rdiff_backup">other documented approaches</a> - leave a comment below if you'd like to learn more about my approach!).</p> <p>Backing up your MariaDB databases is as easy installing automysqlbackups:</p> <p><code>sudo apt install automysqlbackups</code></p> <p>You'll find daily versioned dumps of your MariaDB database(s) in /var/lib/automysqlbackups on your VM host's filesystem. To run an ad hoc backup (which will replace the previous backup from that day, if there is one) just run</p> <p><code>sudo automysqlbackups</code></p> <h2>Backup OnlyOffice</h2> <p>Along with backing up the files in your /home/data/onlyoffice directory, you'll also want a proper "dump" of your PostgreSQL backup (you can write simple bash scripts to do this regularly, automatically), particularly prior to doing an upgrade (to allow for recovery if something goes badly wrong, which is always possible). You can achieve this by going to</p> <p><code>cd /home/docker/onlyoffice</code></p> <p>and running this</p> <p><code>DATE=`date +%Y%m%d` &amp;&amp; FILE=/home/data/onlyoffice/backup/fullbackup-${DATE}.sql &amp;&amp; docker-compose exec onlyoffice sudo -u postgres pg_dumpall &gt; ${FILE} &amp;&amp; gzip ${FILE}</code></p> <p>which will assign the current date to DATE, the relevant filename to FILE, and then put the backup SQL into a dated file called $FILE and compress the result with gzip :)</p> <p>At some point, I'll modify my normal versioned dated database backup scripts to cater for this solution and make the result available on <a href="https://git.oeru.org">https://git.oeru.org</a> - in the meantime, you can use the above before you do a backup and manually delete older backups if they start taking up too much space (or, better still, write your own clever script that does it automatically and let me know about it!).</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=29&amp;2=field_blog_comments&amp;3=comment" token="9kYJqJN_nL4QNtjWT8YsPb_jXY2Bv9NrASHDdb5feiU"></drupal-render-placeholder> </div> </section> Mon, 03 Feb 2020 20:41:16 +0000 dave 29 at http://tech.oeru.org Configuring a Linux server to send email via the Postfix SMTP server using an external authenticating SMTP host http://tech.oeru.org/configuring-linux-server-send-email-postfix-smtp-server-using-external-authenticating-smtp-host <span class="field field--name-title field--type-string field--label-hidden">Configuring a Linux server to send email via the Postfix SMTP server using an external authenticating SMTP host</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--ubuntu-linux"> <span class="field__item-wrapper"><a href="/taxonomy/term/12" hreflang="en">ubuntu linux</a></span> </div> <div class="field__item field__item--postfix"> <span class="field__item-wrapper"><a href="/taxonomy/term/66" hreflang="en">postfix</a></span> </div> <div class="field__item field__item--smtp"> <span class="field__item-wrapper"><a href="/taxonomy/term/67" hreflang="en">smtp</a></span> </div> <div class="field__item field__item--_804"> <span class="field__item-wrapper"><a href="/taxonomy/term/68" hreflang="en">18.04</a></span> </div> <div class="field__item field__item--free--open-source"> <span class="field__item-wrapper"><a href="/taxonomy/term/6" hreflang="en">free &amp; open source</a></span> </div> <div class="field__item field__item--foss"> <span class="field__item-wrapper"><a href="/taxonomy/term/10" hreflang="en">foss</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Fri 20/09/2019 - 14:08</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>Just about any and every server needs to be able to send email - whether it's end-user-email, like password recovery services for a website to emails to system administrators reporting on the status of system backups and errors. The problem is that it's <em>non trivial</em> (understatement) to set up a mail server properly.</p> <p>This howto assumes you have a Linux server (these instructions are for Ubuntu 18.04 and 20.04, although it should work on earlier versions of Ubuntu server and Debian Linux with minor changes, and the concepts will be very similar on other Linuxen) with a static IP address, with one or more fully-qualified-domain-names (fdqn) pointing at that address, and you have SSH-based access to it. I've <a href="/setting-your-own-bitwarden-password-keeper-and-sync-server">previously provided tips</a> on how to get to this stage.</p> <h2>MailCow for authenticating SMTP</h2> <p>To send email, you need access to a server, somewhere on the Internet, that provides the <a href="https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol">Simple Mail Transfer Protocol</a> (SMTP) service. It's an open standard, and for most of the history of the Internet, email services have been mostly provided by Free and Open Source Software (FOSS) tools - the first SMTP was called "<a href="https://en.wikipedia.org/wiki/Sendmail">Sendmail</a>" and it was fully FOSS, and it's still in use today (although it has mostly been superseded by faster, more secure systems, the best of which are also FOSS).</p> <p>At the OERu, we use the <a href="https://mailcow.github.io/mailcow-dockerized-docs/" title="Dockerised MailCow">Docker-based installation of the amazing, completely FOSS MailCow project</a> to provide our organisational email services. I might cover that set up in a future howto here, because MailCow makes an otherwise almost intractable problem... much more tractable. Having set up MailCow set up means we can offer "full service" email for any number of domains and users and aliases with all the bells and whistles including incoming and outgoing mail with all the virus scanning (we don't really need it because we use Linux desktops, but for other folks it's useful I guess) and dynamic spam filtering services you'd expect from a much larger operation: <a href="https://mailcow.email/">Team MailCow</a> have done an amazing job in pulling together a comprehensive set of FOSS applications to provide all the conceivable requirements of a full-fledged, multi-domain email system, including shared calendaring, contacts, and webmail. A great companion to your organisation's MailCow server would be a <a href="/setting-your-own-bitwarden-password-keeper-and-sync-server">BitWarden password safe</a> server... just sayin'. With a tiny bit of configuration cleverness, you could even run them both simultaneously on the same server... but that's a story for another day.</p> <p>So, now, assuming that we have a MailCow server or a functionally equivalent SMTP service available, we have the option of "authenticated SMTP" for outgoing email using credentials we can set up. For example, in MailCow, we can specify a domain we host, like say <strong>oeru.org</strong> (and for which we've defined an MX record and a few other relevant records as guided by MailCow administrative web interface). On top of that, we can specify a mailbox for a dedicated "send stuff from remote relay hosts" email address using that domain, like <strong><a href="mailto:smtp@oeru.org">smtp@oeru.org</a></strong>, with a strong password. With that, we can <em>securely </em>send email using that email address as the username and that password from <em>anywhere we have access to the Internet</em>.</p> <p>The <strong>only tricky part</strong> is that we have to ensure that whatever "reply to" email address we specify from our applications, say <strong><a href="mailto:notifications@tech.oeru.org">notifications@tech.oeru.org</a></strong>, is using a domain we <em>also host on the same server, </em>and that there's an <em>email alias</em> of that email address defined and set as "allow to send from <a href="mailto:smtp@oeru.org">smtp@oeru.org</a>" in the MailCow interface. If we haven't made sure of that, our mail server is likely to reject sending emails with that "mismatching" email address. This is a basic spam deterrence measure, which is for the best, despite sometimes making a email system administrator's life harder.</p> <p>Once we've got that (and it's easy once you've done it once or twice - I'm mostly writing this down now so I don't have to try to re-remember every time I need to set up a new server - and I hope it helps others, too), we can set up any server we control to send secure (and spam-filter-resilient) email. For what it's worth, too, MailCow uses Postfix as its SMTP server component (there're a bunch of other components, too).</p> <h2>Postfix SMTP with SmartHost</h2> <p>The first thing you need to do to create a postfix smarthost is to install the postfix application on a new server (this assumes you're logged in with a user who has "sudo" - aka admin - permissions):</p> <p><code>sudo apt update &amp;&amp; sudo apt install postfix bsd-mailx</code></p> <p>During the install, you'll be asked to select a bunch of configuration parameters. Select the defaults except:</p> <ul><li>Select "Internet Site with Smarthost",</li> <li>fill in the domain name for your server,</li> <li>the domain name and port (in the form <code>[smtp server domain]:[port]</code>, e.g. <code>smtp.oeru.org:587</code> ) of your "smarthost" who'll be doing the authenticating SMTP for you, and</li> <li>the email address to which you want to receive system-related messages.</li> </ul><p>After that's done, you can proceed.</p> <h2>Next Steps</h2> <p>For the rest of this howto, you'll need to do the following. First, select your text editor. I use vim, but if you're new to the command line, I recommend using nano - it's more straightforward:</p> <p><code>EDIT=`which nano`</code> or <code>EDIT=`which vim`</code></p> <p><code>sudo $EDIT /etc/aliases</code></p> <p>We need to make sure the "root" user points to a real email address. Add a line at the bottom which says (replacing [your email] with <em>your email :) </em>)</p> <p><code>root: [your email]</code></p> <p>After which you'll need to convert the aliases file into a form that postfix can process, simply by running this:</p> <p><code>sudo newaliases</code></p> <p>Then we have to define the authentication credentials required to convince your mail server that you're you!</p> <p><code>sudo $EDIT /etc/postfix/relay_password</code></p> <p>The resulting file only needs one line with three bits of information:</p> <p><code>[smtp server domain] [user name]:[password]</code></p> <p>for example:</p> <p><code>smtp.oeru.org smtp@oeru.org:SomeObscurePassw0rd</code></p> <p>Then save the file and, like the aliases file, run the conversion process (which uses a slightly different mechanism):</p> <p><code>sudo postmap /etc/postfix/relay_password</code></p> <p>Finally, we'll edit the main configuration file for Postfix to tell it about all this stuff:</p> <p><code>sudo $EDIT /etc/postfix/main.cf</code></p> <p>If your SMTP server uses port 25 (the default for <em>unencrypted</em> SMTP) you don't have to change anything, although most people nowadays prefer to use StartTLS or otherwise encrypted transport to at least ensure that your SMTP authentication details (<em>at least</em>) are transferred encrypted. That means using port 587 or 465. If you're using either of those ports, find the "relayhost = [your server name]" line... and add your port number after a colon, like this <code>relayhost = smtp.oeru.org:465 </code></p> <p>Next, add the following lines at the bottom of the file:</p> <p><code># added to configure accessing the relay host via authenticating SMTP<br /> smtp_sasl_auth_enable = yes<br /> smtp_sasl_password_maps = hash:/etc/postfix/relay_password<br /> smtp_sasl_security_options =</code><br /><code>smtp_tls_security_level = encrypt</code></p> <p><code># add this if you're using Ubuntu 20.04, and comment out (with a "#") the </code><br /><code># earlier line smtp_tls_security_level = may to save errors in 'postfix check'<br /> #smtp_tls_wrappermode = yes</code><br />  </p> <p>Save the file, and then check that your syntax is correct:</p> <p><code>sudo postfix check</code></p> <p>If it is (running the command returns no errors, and it might not return anything at all - that's a good thing!), then you can run</p> <p><code>sudo postfix reload</code></p> <p>to get postfix to reload its configurations and you can test out your new smarthost-configured SMTP server!</p> <p>If not, the output of the check command will usually give you a helpful insight into what is wrong with your configuration... you'll also find that looking at the mail log is very helpful and offers great insights:</p> <p><code>sudo less +G /var/log/mail.log</code></p> <p>and if you're not able to fix it based on those, you'll find postfix is widely documented and has rich set of easily discoverable resources out there on the web - a search engine is your best resource!</p> <h2>Testing your outgoing email</h2> <p>By default, a command line application called "mail" is installed as part of the bsd-mailx package we installed alongside postfix. You can use it to send test email from the command line on your host to verify you've got things working correctly! The stuff in &lt;&gt; are the keys to hit at the end of the line...</p> <p><code>$ mail you@email.domain&lt;ENTER&gt;</code></p> <p><code>Subject: Testing from your.relay.server.domain&lt;ENTER&gt;<br /> Testing postfix remote host&lt;ENTER&gt;<br /> &lt;CTRL-D&gt;<br /> Cc:&lt;ENTER&gt;</code></p> <p>Typing &lt;CTRL-D&gt; (hold down the Control or Ctrl key on your keyboard and press the "d" key) will finish your message, showing you a "CC:" field, in which you can type in other email addresses if you want to test sending to multiple addresses. When you then hit &lt;ENTER&gt;, it will attempt to send this email. It might take a few minutes to work its way through to the receiving email system (having to run the gauntlet of spam and virus filters on the way).</p> <p>You can also always check the postfix system logs to see what postfix thinks about it using the command above. Hit &lt;SHIFT-F&gt; to have the log update in real time.</p> <h2>Done</h2> <p>Now you've got working outgoing email from your server. That means many higher-level web applications you might install on your infrastructure will work out-of-the-box, because what you've set up, for example, enables the default PHP email service and that used by other stacks.</p> <h2>Sending from Docker Containers</h2> <p>You can configure your server so you can reference it from services you run from Docker containers on your host. You do this by referencing the host, like via an ad hoc SMTP server on your container like <a href="https://marlam.de/msmtp/">msmtp</a>, and you can just reference it as 172.17.0.1, which is the default base IP for Docker hosts from the perspective of Docker containers. You might find it's different on your particular install. In that case, you have to make your Postfix SmartHost accept email for sending from the Docker containers on that server. There're quite a few examples of that among <a href="https://git.oeru.org/explore/projects?utf8=%E2%9C%93&amp;name=docker&amp;sort=latest_activity_desc">my Docker recipes on the OERu's git repository</a>.</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=28&amp;2=field_blog_comments&amp;3=comment" token="lLRkGAi5P6j9iM99_jKG1YxvJLITyB02GoT7oM-A7oA"></drupal-render-placeholder> </div> </section> Fri, 20 Sep 2019 02:08:28 +0000 dave 28 at http://tech.oeru.org Upgrading RocketChat to 1.0.x and MongoDB to 4.0 http://tech.oeru.org/upgrading-rocketchat-10x-and-mongodb-40 <span class="field field--name-title field--type-string field--label-hidden">Upgrading RocketChat to 1.0.x and MongoDB to 4.0</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--rocketchat"> <span class="field__item-wrapper"><a href="/taxonomy/term/18" hreflang="en">rocket.chat</a></span> </div> <div class="field__item field__item--mongodb"> <span class="field__item-wrapper"><a href="/taxonomy/term/14" hreflang="en">mongodb</a></span> </div> <div class="field__item field__item--docker"> <span class="field__item-wrapper"><a href="/taxonomy/term/16" hreflang="en">docker</a></span> </div> <div class="field__item field__item--docker-compose"> <span class="field__item-wrapper"><a href="/taxonomy/term/49" hreflang="en">docker-compose</a></span> </div> <div class="field__item field__item--ubuntu-linux"> <span class="field__item-wrapper"><a href="/taxonomy/term/12" hreflang="en">ubuntu linux</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Mon 29/04/2019 - 14:39</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>With the recent release of Rocket.Chat 1.0.x (after a couple years undergoing development at a fairly blistering pace), it's time for many of us to upgrade!</p> <p>Previously, I showed how to <a href="/docker-compose-better-way-deploy-rocketchat-wekan-and-mongodb">install Rocket.Chat via Docker Compose</a> but that was a much earlier version of Rocket.Chat and version 3.4 of MongoDB, which is now quite old (by FOSS standards at least). And it turns out upgrading everything has a few gotchas, so here's how I managed to do it.</p> <p>Before you do <em>anything</em> <a href="/hourly-versioned-mongodb-backup">do a backup of your MongoDB</a>!</p> <p>The first thing you need to do is upgrade <em>the way</em> in which you're running MongoDB. You have to enable a capability called "Local Replication".</p> <h2>Update your Docker Compose configuration</h2> <p>My first step, after logging into my virtual machine via SSH as the <em>unprivileged user</em> that I created to run docker commands, was to update my <code>docker-compose.yml</code> file (if you followed my previous instructions, you'll find it in <code>/home/www/docker-rocketchat-wekan-mongo</code>). </p> <p>First, make a backup of it nearby...</p> <p>cd <code>/home/www/docker-rocketchat-wekan-mongo</code><br /> cp docker-compose.yml docker-compose.yml-mongo3.4</p> <p>and then edit the file to say this:</p> <p><code>version: '2'<br /> services:<br />   mongo:<br />     restart: unless-stopped<br />     image: mongo<strong>:3.4</strong><br />     volumes:<br />       - [data directory path]:/data/db<br />       - [backup directory path]:/backups<br />     command: --smallfiles <strong>--oplogSize 128 --replSet rs0</strong><br /><strong>  # this container's job is just run the command to initialize the replica set.<br />   # it will run the command and remove himself (it will not stay running)<br />   mongo-init-replica:<br />     image: mongo:3.4<br />     command: 'bash -c "for i in `seq 1 30`; do mongo mongo/rocketchat --eval \"rs.initiate({ _id: ''rs0'', members: [ { _id: 0, host: ''localhost:27017'' } ]})\" &amp;&amp; s=$$? &amp;&amp; break || s=$$?; echo \"Tried $$i times. Waiting 5 secs...\"; sleep 5; done; (exit $$s)"'<br />     depends_on:<br />       - mongo</strong><br />   rocketchat:<br />     restart: unless-stopped<br />     image: rocketchat/rocket.chat<strong>:latest</strong></code><br /><code><strong>    command: bash -c 'for i in `seq 1 30`; do node main.js &amp;&amp; s=$$? &amp;&amp; break || s=$$?; echo "Tried $$i times. Waiting 5 secs..."; sleep 5; done; (exit $$s)'</strong><br />     ports:<br />       - "127.0.0.1:[port number]:3000" # should be a free port above 1024<br />     depends_on:<br />       - mongo<br />     environment:<br />       - MONGO_URL=mongodb://mongo/rocket<br /><strong>      - MONGO_OPLOG_URL=mongodb://mongo/local</strong><br />       - ROOT_URL=[domain name (including schema, e.g. http://)]<br />     volumes:<br />       - [upload directory path]:/var/www/rocket.chat/uploads</code><br /><strong><code>    labels:<br />       - "traefik.backend=rocketchat"<br />       - "traefik.frontend.rule=Host: [your domain name (<em>not </em>including schema)]"</code></strong></p> <p>Now, having updated your docker-compose.yml file, you have to do a couple other things. To do the upgrade from MongoDB 3.4 to 4.0, you have to do the interim upgrade to 3.6 first.</p> <h2>Enabling Local Replication</h2> <p>First you need to check what version of MongoDB you're <em>currently</em> using - both the version you're running <em>and</em> the "Feature Compatibility Version" (you can run a newer version of MongoDB, but configure it to only run features from some previous version to avoid breaking older software that depends on old features)... Do this as follows.</p> <p>Access your MongoDB instance:</p> <p><code>docker-compose exec mongo bash</code></p> <p>That should give you a command prompt that looks like this:</p> <p><code>root@a56eefe9f352: # </code></p> <p>but the container identifier (after the @) will be different (but the same length). At that prompt, you can run this command:</p> <p><code>mongo --eval "db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )"</code></p> <p>Tip: if you're on a Linux desktop, you can copy this command (via CTRL-C) from this document and past it into your SSH terminal window (via CTRL+SHIFT-V).</p> <p>It should tell you you're either running "featureCompatibilityVersion" 3.2 or 3.4. If it's the latter, skip this next step. If not, run this next:</p> <p><code>mongo --eval "db.adminCommand( { setFeatureCompatibilityVersion: '3.4' } )"</code></p> <p>to set the version to 3.4. If the command succeeds you'll likely see something like</p> <p><code>{ "ok" : 1 }</code></p> <p>as the response.</p> <p>Now you can upgrade your Mongo 3.4 is the latest version (should be 3.4.20 at the time of this writing). Get out of the container (back to your Docker host) via CTRL-D (or "exit" - they're synonymous for logging out of a terminal session). Then you can run:</p> <p><code>docker-compose pull mongo</code></p> <p>That should update both your Mongo docker container to the latest version in the 3.4 series.</p> <p>The final step to enabling local replication is to run</p> <p><code>docker-compose up -d mongo mongo-init-replica &amp;&amp; docker-compose logs -f</code></p> <p>That will restart MongoDB and drop you into the stream of logging from all the containers (including the rocket.chat container). It'll also start the "mongo-init-replica" container.  That container should run briefly <em>and then exit cleanly</em> having set up the local replication that you'll need for subsequent upgrades to MongoDB!</p> <p>Check for any errors in the output... there might be a couple if it takes your MongoDB a bit of time to accept connections... as long as it eventually stops showing errors, you should be ok!</p> <p> </p> <h2>Upgrading Rocket.Chat to 1.0.x</h2> <p>Now that you're fully on version 3.4, running in local replica mode, you can update your Rocket.Chat instance.   Rocket.Chat still supports Mongo 3.4 (it won't for long, thus this tutorial!), so you can now upgrade the Rocket.Chat container as well as make sure your Mongo 3.4 is the latest version (should be 3.4.20 at the time of this writing).</p> <p>Note that the latest version of the Rocket.Chat docker container could be quite a lot higher when you read this... if it's beyond, say, 1.1 it might be unsafe to use the approach I'm describing. You can <a href="https://github.com/RocketChat/Rocket.Chat/releases">check the current version release status</a>. To protect yourself, you can alter the rocket.chat image line in your docker-compose.yml file to explicitly tell it to use the 1.0.x series for which these instructions should continue to apply... pick the highest 1.0.x version you can find in the releases and alter the line in docker-compose.yml to specify that version:</p> <p><code>    image: rocketchat/rocket.chat<strong>:1.0.1</strong></code></p> <p>or whatever the latest 1.0.x version is. Then you can run:</p> <p><code>docker-compose pull</code> rocketchat</p> <p>which will update your Rocket.Chat from the current version to the one specified.</p> <p>Then restart your Rocket.Chat instance:</p> <p><code>docker-compose up -d &amp;&amp; docker-compose logs -f</code></p> <p>That should restart both MongoDB and Rocket.Chat, and drop you into the stream of logging from both containers. It'll also start the "mongo-init-replica" container again, but having done its job it should exit happily again.</p> <p>Check for any errors in the output... there might be a couple if it takes your MongoDB a bit of time to accept connections... as long as it eventually stops showing errors, you should be ok! Eventually, you should see something similar to (with version details updated appropriately):</p> <p> </p> <p><code>+----------------------------------------------+<br /> |                SERVER RUNNING                |<br /> +----------------------------------------------+<br /> |                                              |<br /> |  Rocket.Chat Version: 1.0.1                  |<br /> |       NodeJS Version: 8.11.4 - x64           |<br /> |      MongoDB Version: 3.4.20                 |<br /> |       MongoDB Engine: wiredTiger             |<br /> |             Platform: linux                  |<br /> |         Process Port: 3000                   |<br /> |             Site URL: https://chat.oeru.org  |<br /> |     ReplicaSet OpLog: Enabled                |<br /> |          Commit Hash: 60f1a4afd6             |<br /> |        Commit Branch: HEAD                   |<br /> |                                              |<br /> +----------------------------------------------+</code></p> <p>Your instance is now running the right version! Time to tidy things up by upgrading Mongo the rest of the way to 4.0!</p> <p> </p> <h2>Upgrading to MongoDB 3.6</h2> <p>Now you can upgrade Mongo to 3.6. First, adjust your docker-compose.yml file.  Update both occurances of this line:</p> <p><code>image: mongo:3.4</code></p> <p>to</p> <p><code>image: mongo:3.6</code></p> <p>Then you can do another</p> <p><code>docker-compose pull mongo</code></p> <p>which will download the newer Mongo 3.6 docker container. Then you can again run</p> <p><code>docker-compose up -d &amp;&amp; docker-compose logs -f</code></p> <p>Again check for errors. If there are none (other than perhaps a brief set of "mongo is not accepting connections" errors), you should be fine to update the "compatibility version" from 3.4 to 3.6... Get a session on your Mongo container via</p> <p><code>docker-compose exec mongo bash</code></p> <p>and then (as above) run this:</p> <p><code>mongo --eval "db.adminCommand( { setFeatureCompatibilityVersion: '3.6' } )"</code></p> <p>which should give you a more complicated response than that for the 3.4 transition, but it should still more or less say "ok"... To make sure everything's happy with the change, it's wise to run</p> <p><code>docker-compose up -d &amp;&amp; docker-compose logs -f</code></p> <p>again and make sure there're no obvious errors. If not (after making another database backup for safety!!) we can proceed to Mongo 4.0!</p> <h2>Final push to MongoDB 4.0</h2> <p>Finally, you can again edit your docker-compose.yml and change both occurrences of</p> <p> </p> <p><code>image: mongo:3.6</code></p> <p>to</p> <p><code>image: mongo:4.0</code></p> <p>Then you can do a final</p> <p><code>docker-compose pull mongo</code></p> <p>which will download the newer Mongo 4.0 docker container. Then you can again run</p> <p><code>docker-compose up -d &amp;&amp; docker-compose logs -f</code></p> <p>And, assuming you don't see any errors, you can push the feature compatibility to 4.0:</p> <p><code>docker-compose exec mongo bash</code></p> <p>and then run:</p> <p><code>mongo --eval "db.adminCommand( { setFeatureCompatibilityVersion: '4.0' } )"</code></p> <p>followed by a final</p> <p><code>docker-compose up -d &amp;&amp; docker-compose logs -f</code></p> <p>And, again, if you don't see any errors... you should get something a bit like this:</p> <p><code>+----------------------------------------------+<br /> |                SERVER RUNNING                |<br /> +----------------------------------------------+<br /> |                                              |<br /> |  Rocket.Chat Version: 1.0.1                  |<br /> |       NodeJS Version: 8.11.4 - x64           |<br /> |      MongoDB Version: 4.0.9                  |<br /> |       MongoDB Engine: wiredTiger             |<br /> |             Platform: linux                  |<br /> |         Process Port: 3000                   |<br /> |             Site URL: https://chat.oeru.org  |<br /> |     ReplicaSet OpLog: Enabled                |<br /> |          Commit Hash: 60f1a4afd6             |<br /> |        Commit Branch: HEAD                   |<br /> |                                              |<br /> +----------------------------------------------+</code></p> <p>you're done and future proofed!</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=26&amp;2=field_blog_comments&amp;3=comment" token="VK6tqiyjVB7HYLx9xMKgF1KeXWiFxiQ-XNY71DIaaF0"></drupal-render-placeholder> </div> </section> Mon, 29 Apr 2019 02:39:33 +0000 dave 26 at http://tech.oeru.org Installing NextCloud and Collabora Office Online with Docker on Ubuntu 16.04 http://tech.oeru.org/installing-nextcloud-and-collabora-office-online-docker-ubuntu-1604 <span class="field field--name-title field--type-string field--label-hidden">Installing NextCloud and Collabora Office Online with Docker on Ubuntu 16.04</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--ubuntu-linux"> <span class="field__item-wrapper"><a href="/taxonomy/term/12" hreflang="en">ubuntu linux</a></span> </div> <div class="field__item field__item--mariadb"> <span class="field__item-wrapper"><a href="/taxonomy/term/48" hreflang="en">mariadb</a></span> </div> <div class="field__item field__item--docker"> <span class="field__item-wrapper"><a href="/taxonomy/term/16" hreflang="en">docker</a></span> </div> <div class="field__item field__item--docker-compose"> <span class="field__item-wrapper"><a href="/taxonomy/term/49" hreflang="en">docker-compose</a></span> </div> <div class="field__item field__item--php"> <span class="field__item-wrapper"><a href="/taxonomy/term/40" hreflang="en">php</a></span> </div> <div class="field__item field__item--collabora-office"> <span class="field__item-wrapper"><a href="/taxonomy/term/50" hreflang="en">collabora office</a></span> </div> <div class="field__item field__item--nextcloud"> <span class="field__item-wrapper"><a href="/taxonomy/term/51" hreflang="en">nextcloud</a></span> </div> <div class="field__item field__item--lets-encrypt"> <span class="field__item-wrapper"><a href="/taxonomy/term/17" hreflang="en">let&#039;s encrypt</a></span> </div> <div class="field__item field__item--redis"> <span class="field__item-wrapper"><a href="/taxonomy/term/21" hreflang="en">redis</a></span> </div> <div class="field__item field__item--productivity"> <span class="field__item-wrapper"><a href="/taxonomy/term/52" hreflang="en">productivity</a></span> </div> <div class="field__item field__item--nginx"> <span class="field__item-wrapper"><a href="/taxonomy/term/30" hreflang="en">nginx</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Mon 29/01/2018 - 17:29</span> <div class="field field-node--field-image field-name-field-image field-type-image field-label-hidden has-multiple"> <figure class="field-type-image__figure image-count-1"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/Files%20-%20OERu%20NextCloud.png?itok=xQHlcyml" title="The NextCloud web interface for browsing your files" data-colorbox-gallery="gallery-field_image-lB2TEYU3oiM" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;The NextCloud web interface for browsing your files&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/Files%20-%20OERu%20NextCloud.png?itok=6v2Kuyct" width="220" height="122" alt="The NextCloud web interface for browsing your files" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-2"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/NextCloud-AppStore.png?itok=DPeCx5Rd" title="The central AppStore (note, almost all apps have no cost and are open source). You get a similar view within your own NextCloud instance." data-colorbox-gallery="gallery-field_image-lB2TEYU3oiM" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;The central AppStore (note, almost all apps have no cost and are open source). You get a similar view within your own NextCloud instance.&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/NextCloud-AppStore.png?itok=WqCJJdGj" width="220" height="175" alt="The central AppStore (note, almost all apps have no cost and are open source). You get a similar view within your own NextCloud instance." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-3"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/NextCloud-Calendar.png?itok=-j0Dq2rG" title="The NextCloud shared calendar plugin works with all major calendaring applications alongside your existing digital calendars." data-colorbox-gallery="gallery-field_image-lB2TEYU3oiM" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;The NextCloud shared calendar plugin works with all major calendaring applications alongside your existing digital calendars.&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/NextCloud-Calendar.png?itok=bP23WxDf" width="220" height="175" alt="The NextCloud shared calendar plugin works with all major calendaring applications alongside your existing digital calendars." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-4"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/Nextcloud-CollaboraSpreadsheet.png?itok=Ovp0KryQ" title="An example of a fairly complex spreadsheet in the Collabora spreadsheet interface." data-colorbox-gallery="gallery-field_image-lB2TEYU3oiM" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;An example of a fairly complex spreadsheet in the Collabora spreadsheet interface.&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/Nextcloud-CollaboraSpreadsheet.png?itok=CNhDR2y-" width="220" height="157" alt="An example of a fairly complex spreadsheet in the Collabora spreadsheet interface." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-5"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/Nextcloud-CollaboraWordprocessor.png?itok=IOyfA_M4" title="A fairly complex document, with variables, shown in the Collabora wordprocessor interface." data-colorbox-gallery="gallery-field_image-lB2TEYU3oiM" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;A fairly complex document, with variables, shown in the Collabora wordprocessor interface.&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/Nextcloud-CollaboraWordprocessor.png?itok=HPawBI-o" width="220" height="157" alt="A fairly complex document, with variables, shown in the Collabora wordprocessor interface." typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-6"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-01/DavInFilemanager.png?itok=rCbwaUUY" title="This is what your NextCloud would look like in your desktop filemanager (this is the Nemo filemanager on a Linux desktop)" data-colorbox-gallery="gallery-field_image-lB2TEYU3oiM" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;This is what your NextCloud would look like in your desktop filemanager (this is the Nemo filemanager on a Linux desktop)&quot;}"><img src="/sites/default/files/styles/medium/public/2018-01/DavInFilemanager.png?itok=g2dNm33H" width="220" height="122" alt="This is what your NextCloud would look like in your desktop filemanager (this is the Nemo filemanager on a Linux desktop)" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-7"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2018-02/CollaboraAdminConsole.png?itok=1tNI9ZdJ" title="Collabora Office admin console" data-colorbox-gallery="gallery-field_image-lB2TEYU3oiM" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Collabora Office admin console&quot;}"><img src="/sites/default/files/styles/medium/public/2018-02/CollaboraAdminConsole.png?itok=iijjMrBK" width="220" height="149" alt="Collabora Office admin console" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> </div> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>Dropbox is the best known of the end-user "cloud storage" services for documents, backups, and synchronising data among multiple devices, although now Google's Drive and Microsoft's OneDrive are functionally similar and are being heavily promoted and tied into all sorts of services.</p> <p>Similarly the collaborative editing of documents, spreadsheets, and presentations in the browser, pioneered by Etherpad, but then adopted in a big way by Google Docs (and more recently, Microsoft Office 365), has revolutionised collective note taking, document preparation, and ease of access to these powerful tools by the mainstream of computer users. Only a browser is required, and no other software needs to be installed.</p> <p>But what about people who don't want to entrust all of their data to foreign corporations, holding their data in foreign jurisdiction, in formats that may or may not be retrievable in the event that the supplier fails or changes "strategic direction"? And many of these services involve "mining" their data to extract useful information that vendors sell to others to <em>help them advertise to us in a more targeted way. </em>Yeah, that's creepy.</p> <p>More-over, often if you want to <em>share</em> your data with others, <em>they</em> have to log into the same service, and accept the service's terms and conditions (usually substantially constraining the user's normal rights and freedoms, although who<em> actually</em> reads those, eh?!) in order to do so... so ones use of those services has a magnifying effect on the loss of privacy and control.</p> <p>Some people sensibly prefer to manage their own, or institution-specific, solutions on the infrastructure of their choosing, in a way that doesn't tie anyone into paying ever increasing amounts for data storage as the volumes increase perpetually, month on month.</p> <p>Some of us simply prefer to have control of our own destiny, without a dependence on, for example, file or data storage formats and practices that are completely opaque to them. Our data reflects our creativity energy, and it seems much more comfortable for many of us to be in charge of our own fates rather than entrusting it to a third party who simply sees us a profit centre.</p> <p>Thankfully, the open source world has created an array of possible equivalent systems, and this post describes how you, too, can set up your own equivalent to Dropbox + Google Docs using entirely open source software on any commodity virtual machine hosting system you want to use by adopting NextCloud and Collabora Office.</p> <h2>NextCloud</h2> <p><a href="https://nextcloud.com">NextCloud</a> is <a href="https://nextcloud.com/files/">functionally similar</a> to Dropbox, however, with its active development community and plug-in architecture, it can provide quite a lot more as well, like shared calendaring, email, video conferencing, contact syncing, image/sound/video galleries, <a href="https://nextcloud.com/files/">among many other services</a>.</p> <p>If you prefer not to organise and run your own server, you can purchase a supported server via their website for a cost similar to Dropbox (although, realise that NextCloud is relatively small by comparison and doesn't have the massive economies of scale enjoyed by the bigger players).</p> <p>For those with an interest in history: NextCloud is a fork created by the founder of OwnCloud, after he decided that the company which formed around his OwnCloud project was moving in a direction that was philosophical unpalatable for him. The beauty of open source is that developers can follow their consciences without requiring anyone's permission. The resulting "forks" in code bases and communities then thrive or die based on the strengths of the communities they can build and sustain. This fork is remarkably similar to that which occurred in the OpenOffice community which resulted in the founding of LibreOffice. LibreOffice has thrived and OpenOffice has faded into irrelevance. More on that below.</p> <p>For those with a technological interest, NextCloud is a mature PHP application (but with a modern architecture, including a command line interface, occ) which stores its data in an RDBMS like MySQL, MariaDB, PostgreSQL, or (usually for development purposes) the lightweight SQLite database. Here are <a href="https://docs.nextcloud.com/server/12/admin_manual/installation/index.html">details for would-be administrators</a>.</p> <h2>Collabora Office</h2> <p>Given how much companies like Google and Microsoft invest on Docs and Office 365 respectively, how is it possible for an open source community to create a credible competitor? Turns out it's not as hard as you might think if they leverage the power of open source.</p> <p>A small software company with headquarters in the UK (although their team appears to be from all over), Collabora Office, has taken on the ambitious mission of creating a "collaborative web interface" allowing users to collaborate using <a href="https://libreoffice.org">LibreOffice</a>, one of the most powerful and widely used office package available anywhere. We're currently at Collabora Office 3.0, and the front end is quite nice and functional, but still pretty simple - that can be a good thing for many users. Collabora is progressively re-imagining the user interface of LibreOffice as a collaborative web interface. This isn't easy, but it's <em>much</em> easier than it otherwise would be because the difficult job of creating the heavy-lifting application back-end is already done - LibreOffice is a mature widely used application (albeit with a desktop interface, not a web-based collaborative interface). So we can expect progress will be rapid, and large sets of new capabilities will be "unlocked" as they progress their efforts.</p> <h2>NextCloud and Collabora - better together!</h2> <p>The beauty of the open source software model is that we can connect NextCloud and Collabora office - completely separate and unrelated communities - thanks to a new integration standard, WOPI (Web-application Open Platform Interface) they form a well integrated component model - with the <em>major </em>added benefit of being able to swap in a better file management platform, or a better collaborative productivity package if one or the other emerges, without having to start from scratch.</p> <h2>Setting up your own NextCloud Collabora Server</h2> <p>If you're game to run your own (and, in my experience, it's a surprisingly well behaved system) here's how you do it.</p> <p>In preparation, you'll want to have the following ready:</p> <ul><li>a Linux virtual machine or "VM" (I recommend running the current Ubuntu LTS version, or current Debian) with a user with Sudo privileges...,</li> <li>your domain name for the NextCloud instance, pointing to the IP address of your VM,</li> <li>your domain name for the Collabora instance, also pointing to the IP of your VM, and</li> <li>credentials for an email address capable of sending from a remote server (usually termed an "authenticating SMTP email account")</li> </ul><h3>Secure access with SSH</h3> <p>First things first, make sure you're logged into your host (probably via SSH) as a user who has "sudo" capabilities! You need to log into the host from your local machine. We recommend setting up <a href="https://www.digitalocean.com/community/tutorials/how-to-configure-ssh-key-based-authentication-on-a-linux-server">key-based authentication</a>.</p> <h3>Firewall with UFW</h3> <p>No computer system is ever full secure - there're always exploits waiting to be found, so security is a process of maintaining vigilance. Part of that is reducing exposure - minimising your "attack surface". Use a firewall - "<a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-ubuntu-16-04" title="Uncomplicated FireWall">ufw</a>" is installed on Ubuntu by default. Make sure you've got exceptions for SSH (without them, you could lock yourself out of your machine! Doh!).</p> <p>Run the following commands to allow your Docker containers to talk to other services on your host.</p> <p><code>sudo ufw allow in on docker0<br /> sudo ufw allow from 172.0.0.0/8 to any</code></p> <p>Specifically for Docker's benefit, you need to tweak the default Forwarding rule (I use "vim" as my editor. If you don't know how to/want to use it, replace <strong>vim</strong> with <strong>nano</strong> everywhere you see it in the following - nano's easier to use for simple edits like this):</p> <p><code>sudo vim /etc/default/ufw</code></p> <p>and copy the line <code>DEFAULT_FORWARD_POLICY="DROP"</code> tweak it to look like this (commenting out the default, but leaving it there for future reference!):</p> <p><code>#DEFAULT_FORWARD_POLICY="DROP"<br /> DEFAULT_FORWARD_POLICY="ACCEPT"</code></p> <p>You also have to edit <code>/etc/ufw/sysctl.conf</code> and remove the "#" at the start of the following lines, so they look like this:</p> <p><code>sudo vim /etc/ufw/sysctl.conf</code></p> <p><code># Uncomment this to allow this host to route packets between interfaces<br /> net/ipv4/ip_forward=1<br /> net/ipv6/conf/default/forwarding=1<br /> net/ipv6/conf/all/forwarding=1</code></p> <p>and finally restart the network stack and ufw on your server<code> </code></p> <p><code>sudo service networking restart<br /> sudo service ufw restart</code></p> <h3>Installing the Nginx webserver</h3> <p>In the configuration I'm describing here, you'll need a webserver running on the server - it'll be acting as a "proxy" for the Docker-based Nginx instance described below. I like the efficiency of Nginx and clarity of Nginx configurations over those of Apache and other open source web servers. Here's how you install it.</p> <p><code>sudo apt-get install nginx-full</code></p> <p>To allow nginx to be visible via ports 80 and 443, run</p> <p><code>sudo ufw allow "Nginx Full"</code></p> <p><strong>Note</strong>: make sure your hosting service is not blocking these ports at some outer layer (depending on who's providing that hosting service you may have to set up port forwarding).</p> <h3>Installing MariaDB</h3> <p>MariaDB is effectively a drop-in alternative to MySQL and we prefer it because it's not controlled by Oracle and has a more active developer community. On Ubuntu, MariaDB pretends to be MySQL for compatibility purposes, so don't be weirded out by the interchangeable names below. Install the server and the client like this.</p> <p><code>sudo apt-get install mariadb-server-10.0 mariadb-client-10.0</code></p> <p>You need to set a root (admin) user password - you might want to create a /root/.my.cnf file containing the following (replacing YOURPASSWORD) to let you access MariaDB without a password from the commandline<code>:</code></p> <p><code>[client]<br /> user=root<br /> password=YOURPASSWORD</code></p> <p>You should now be able to type "mysql" at the command prompt</p> <p>Tweak the configuration so that it's listening on</p> <p><code>sudo vim /etc/mysql/mariadb.conf.d/50-server.cnf </code></p> <p>and copy the bind-address line and adjust so it looks like this - we want MariaDB to be listening on all interfaces, not just localhost (127.0.0.1)...</p> <p><code># Instead of skip-networking the default is now to listen only on<br /> # localhost which is more compatible and is not less secure.<br /> #bind-address           = 127.0.0.1<br /> bind-address            = 0.0.0.0</code></p> <p>Then restart MariaDB:</p> <p><code>sudo service mysql restart</code></p> <p>It should now be listening on port 3306 on all interfaces, i.e. 0.0.0.0.</p> <p>Now set up the database which will hold NextCloud's data. Log into the MySQL client on the host (if you've created a .my.cnf file in your home directory as describe above, you won't need to enter your username and password):</p> <p><code>mysql -u root -p</code></p> <p>Enter your root password when prompted. It's also a good idea to gin up a password for your "nextcloud" database user. I usually use pwgen (<code>sudo apt-get install pwgen</code>) - for example running this command will give you a single 12 character password without special characters (just numbers and letters):</p> <p><code>pwgen -s 12 1<br /> T7KR2osrMkyC</code></p> <p>At the prompt (which will look something like MariaDB [(none)]&gt;) enter the following lines (putting your password in place of [passwd]):</p> <p><code>CREATE DATABASE nextcloud CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;<br /> CREATE USER "nextcloud"@"%" IDENTIFIED BY "[passwd]";<br /> GRANT ALL ON nextcloud.* to "nextcloud"@"%";<br /> FLUSH PRIVILEGES;</code></p> <p>Then enter \q to exit.</p> <h2>NextCloud and Collabora Office with Docker</h2> <p>We make use of the NextCloud community's <a href="https://hub.docker.com/_/nextcloud/" title="Documentation for the reference NextCloud Docker container.">stable Docker container</a> which they keep up to date. Similarly, the Collabora community has created a <a href="https://hub.docker.com/collabora/code">reference Docker container</a>.</p> <p>The over all architecture consists of five Docker containers (note, done properly, you aim to ensure that each container runs only one service!):</p> <ol><li>the main NextCloud container (running the PHP-FPM service)</li> <li>an identical container to the PHP one which runs the cron service (which does periodic administrative tasks relevant to NextCloud)</li> <li>the self-contained Collabora Office container (running PHP with an Apache web server instance and a full instance of LibreOffice running in headless server mode (never fear, no servers were harmed in the building of this server!) - yes this server doesn't really adhere to the "one-service per container" convention, but I'm ok with that. It's just a convention after all.)</li> <li>a Redis container (which provides performance improving caching for NextCloud), and</li> <li>an Nginx webserver container which makes it easier to manage the configuration and paths of the NextCloud and Collabora servers via WOPI. It means that on the hosting server, we only need to run a proxying web server, which is easy.</li> </ol><p>The way I prefer to implement this set of containers is to use <a href="https://docs.docker.com/compose/">Docker Compose</a> (after first setting up <a href="https://docs.docker.com/install/linux/docker-ce/ubuntu/">Docker support</a> on your server - I'll assume you've followed the complete instructions including <a href="https://docs.docker.com/install/linux/linux-postinstall/">setting up Docker for your non-root user</a>). I suggest using the latest <a href="https://docs.docker.com/compose/install/#install-compose">installation instructions</a> provided by the Docker community. To be honest, I usually use the alternative instructions, <a href="https://docs.docker.com/compose/install/#install-using-pip">employing the "pip" approach</a>. You can upgrade an existing install by issuing (on your Linux VM's command line):</p> <p><code>sudo pip install -U docker-compose </code></p> <p>To set up your server, I recommend setting up a place for your Docker containers (replace "me" with your non-root username on the server) and the associated persistent data (your Docker containers should hold <em>no</em> important data - you should be able to delete and recreate them entirely without losing any important data or configuration):</p> <p><code>sudo mkdir /home/data</code><br /><code>sudo mkdir /home/data/nextcloud</code><br /><code>sudo mkdir /home/data/nextcloud/apps<br /> sudo mkdir /home/data/nextcloud/config<br /> sudo mkdir /home/data/nextcloud/data<br /> sudo mkdir /home/data/nextcloud/redis<br /> sudo mkdir /home/data/nextcloud/resources<br /> sudo mkdir /home/docker<br /> sudo mkdir /home/docker/nextcloud-collabora<br /> sudo chown -R me:me /home/docker<br /> cd /home/docker/nextcloud-collabora</code></p> <p>Here's an example of the required docker-compose.yml file (you can create this via a text editor like "nano" which should be pre-installed on any VM these days, or use my preferred, but less intuitive, editor, vim via <code>vim docker-compose.yml</code> in the /home/docker/nextcloud-collabora directory):</p> <p><code>version: '2'<br /> networks:<br />   back:<br />     driver: bridge<br /> services:<br />   web:<br />     image: nginx<br />     ports:<br />       - 127.0.0.1:8082:80<br />     volumes:<br />       - ./nginx.conf:/etc/nginx/nginx.conf:ro<br />     links:<br />       - app<br />     volumes_from:<br />       - app<br />     environment:<br />       - VIRTUAL_HOST<br />     networks:<br />     - back<br />     restart: unless-stopped      <br />   app:<br />     image: nextcloud:12-fpm<br />     links:<br />       - redis<br />     volumes:<br />       - /home/data/nextcloud/apps:/var/www/html/apps<br />       - /home/data/nextcloud/config:/var/www/html/config<br />       - /home/data/nextcloud/resources:/var/www/html/resources<br />       - /home/data/nextcloud/data:/var/www/html/data<br />     networks:<br />     - back<br />     restart: unless-stopped      <br />   cron:<br />     image: nextcloud:12-fpm<br />     volumes_from:<br />       - app<br />     user: www-data<br />     entrypoint: |<br />       bash -c 'bash -s &lt;&lt;EOF<br />       trap "break;exit" SIGHUP SIGINT SIGTERM<br />       while /bin/true; do<br />         /usr/local/bin/php /var/www/html/cron.php<br />         sleep 900<br />       done<br />       EOF'<br />     networks:<br />       - back<br />     restart: unless-stopped      <br />   redis:<br />     image: redis:alpine<br />     volumes:<br />       - /home/data/nextcloud/redis:/data<br />     networks:<br />       - back<br />     restart: unless-stopped<br />   collab:<br />     image: collabora/code<br />     environment:</code><br /><code>      # put the domain name you select for your NextCloud instance<br />       # here! Escape any . in your domain name by preceding them with \\<br />       domain: your\\.domain\\.tld<br />       username: admin</code><br /><code>      # put your own strong password in here!<br />       password: some-good-password<br />     cap_add:<br />       - MKNOD<br />     networks:<br />       - back<br />     volumes_from:<br />       - app<br />     ports:<br />       - 127.0.0.1:9980:9980<br />     links:<br />       - app<br />     restart: unless-stopped</code></p> <p>You'll need to substitute the domain name you pick for your NextCloud instance - Collabora's container requires that you specify it so that it doesn't accept connections from other (potentially nefarious) containers elsewhere on the Internet!</p> <p>Also note, the "ports" specified above, 8082 for <code>nginx</code> and 9980 for <code>collab</code> are arbitrary - I picked these to ensure they don't conflict with ports being used by other containers on my server - you can use these if you want, or use <code>sudo netstat -punta</code> to see what ports are currently claimed by other services on your server (if there are any) and pick ones that don't clash! If it scroll past too fast, you can pipe it into less to allow you to scroll and search: <code>sudo netstat -punta | less</code> - hit "q" to exit or "/" to initiate a text search.</p> <p>You will also need to provide the "nginx.conf" file referenced in the nginx section of the file. Do that by using your editor, e.g. <code>vim nginx.conf</code>, and enter this content:</p> <p><code>user www-data;</code></p> <p><code>events {<br />   worker_connections 768;<br /> }</code></p> <p><code>http {<br />   upstream backend {</code><br /><code>      # if you don't call your NextCloud server "app" in your<br />       # docker-compose.yml, you'll need to change app below to </code><br /><code>      # whatever you end up calling it.<br />       server app:9000;<br />   }<br />   include /etc/nginx/mime.types;<br />   default_type application/octet-stream;</code></p> <p><code>  server {<br />     listen 80;<br />     <br />     # Add headers to serve security related headers<br />     add_header X-Content-Type-Options nosniff;<br />     add_header X-Frame-Options "SAMEORIGIN";<br />     add_header X-XSS-Protection "1; mode=block";<br />     add_header X-Robots-Tag none;<br />     add_header X-Download-Options noopen;<br />     add_header X-Permitted-Cross-Domain-Policies none;</code></p> <p><code>    root /var/www/html;</code></p> <p><code>    location = /robots.txt {<br />       allow all;<br />       log_not_found off;<br />       access_log off;<br />     }</code></p> <p><code>    location = /.well-known/carddav {<br />       return 301 $scheme://$host/remote.php/dav;<br />     }<br />     location = /.well-known/caldav {<br />       return 301 $scheme://$host/remote.php/dav;<br />     }</code></p> <p><code>    client_max_body_size 1G;<br />     fastcgi_buffers 64 4K;</code></p> <p><code>    gzip off;</code></p> <p><code>    index index.php;<br />     error_page 403 /core/templates/403.php;<br />     error_page 404 /core/templates/404.php;<br />  <br />     location / {<br />         rewrite ^ /index.php$uri;<br />     }</code></p> <p><code>    location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {<br />         deny all;<br />     }<br />     location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) {<br />         deny all;<br />     }</code></p> <p><code>    location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+|core/templates/40[34])\.php(?:$|/) {<br />         include fastcgi_params;<br />         fastcgi_split_path_info ^(.+\.php)(/.*)$;<br />         fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;<br />         fastcgi_param PATH_INFO $fastcgi_path_info;<br />         fastcgi_param HTTPS on;<br />         #Avoid sending the security headers twice<br />         fastcgi_param modHeadersAvailable true;<br />         fastcgi_param front_controller_active true;<br />         fastcgi_pass backend;<br />         fastcgi_intercept_errors on;<br />         fastcgi_request_buffering off;<br />     }</code></p> <p><code>    location ~ ^/(?:updater|ocs-provider)(?:$|/) {<br />         try_files $uri/ =404;<br />         index index.php;<br />     }</code></p> <p><code>    # Adding the cache control header for js and css files<br />     # Make sure it is BELOW the PHP block<br />     location ~* \.(?:css|js)$ {<br />         try_files $uri /index.php$uri$is_args$args;<br />         add_header Cache-Control "public, max-age=7200";<br />         # Add headers to serve security related headers (It is intended to<br />         # have those duplicated to the ones above)<br />         # Before enabling Strict-Transport-Security headers please read into<br />         # this topic first.<br />         # add_header Strict-Transport-Security "max-age=15768000;<br />         #  includeSubDomains; preload;";<br />         add_header X-Content-Type-Options nosniff;<br />         add_header X-Frame-Options "SAMEORIGIN";<br />         add_header X-XSS-Protection "1; mode=block";<br />         add_header X-Robots-Tag none;<br />         add_header X-Download-Options noopen;<br />         add_header X-Permitted-Cross-Domain-Policies none;<br />         # Optional: Don't log access to assets<br />         access_log off;<br />     }</code></p> <p><code>    location ~* \.(?:svg|gif|png|html|ttf|woff|ico|jpg|jpeg)$ {<br />         try_files $uri /index.php$uri$is_args$args;<br />         # Optional: Don't log access to other assets<br />         access_log off;<br />     }<br />   }<br /> }</code></p> <p>That should be all the configuration you need to make the Docker containers go.</p> <h2>Configuring Nginx to proxy NextCloud and Collabora</h2> <p>The next step is configuring the local nginx proxy servers for NextCloud and Collabora using the nginx instance you installed earlier. That's what responds to the domain name you choose for this service. In our case, the name is <a href="https://docs.oeru.org">https://docs.oeru.org</a> - you can have a look at it to see what you should be seeing when you first start things up! We use <a href="https://letsencrypt.org" title="This is an incredible free and open source service, that is single-handedly making the web a much safer place.">Let's Encrypt</a> to provide secure hosting - <a href="/protecting-your-users-lets-encrypt-ssl-certs">here're my Let's Encrypt instructions</a> on setting it up. The key thing to realise is that your "certificates" need to exist for Nginx to restart with the new configurations below - use the "commenting out the intervening lines" trick mentioned in my instructions to bootstrap the creation of your secure certificates!</p> <p>To configure the proxies, you need to create two configuration files in your /etc/nginx/sites-available/ directory.</p> <h3>NextCloud Proxy Configuration</h3> <p>Create a file with a meaningful name for your NextCloud Proxy, perhaps based on the domain name you've chosen (our file for docs.oeru.org is called "docs") using the same editing approach as the last few (although this is in a different directory) for example <code>sudo vim /etc/nginx/sites-available/docs</code> with the following contents, replacing "nextcloud.domain" with your selected domain name (and the port number 8082 if you've opted to change to a different one!):</p> <p><code>server {<br />     listen 80;<br />     server_name nextcloud.domain;</code></p> <p><code>    include /etc/nginx/includes/letsencrypt.conf;</code></p> <p><code>    # redirect all HTTP traffic to HTTPS.<br />     location / {<br />         return  302 https://nextcloud.domain$request_uri;<br />     }<br /> }</code></p> <p><code># This configuration assumes that there's an nginx container talking to the mautic PHP-fpm container,<br /> # and this is a reverse proxy for that Mautic instance.<br /> server {<br />     listen 443 ssl;<br />     server_name nextcloud.domain;</code></p> <p><code>    ssl_certificate /etc/letsencrypt/live/nextcloud.domain/fullchain.pem;<br />     ssl_certificate_key /etc/letsencrypt/live/nextcloud.domain/privkey.pem;<br />     ssl_protocols TLSv1 TLSv1.1 TLSv1.2;<br />     # to create this, see https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html<br />     ssl_dhparam /etc/ssl/certs/dhparam.pem;<br />     keepalive_timeout 20s;</code></p> <p><code>    include /etc/nginx/includes/letsencrypt.conf;<br />    <br />     location ^~ / {<br />         proxy_pass http://localhost:8082;<br />         proxy_set_header Upgrade $http_upgrade;<br />         proxy_set_header Connection "Upgrade";<br />         proxy_set_header Host $http_host;<br />         proxy_read_timeout 36000s;<br />     }<br />     client_max_body_size 1G;<br />     fastcgi_buffers 64 4K;</code></p> <p><code>    add_header X-Frame-Options "SAMEORIGIN";<br />     add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";<br /> }</code></p> <h3>Collab Proxy Configuration</h3> <p>Now create a collabora proxy configuration.</p> <p>Note: This will probably never by used by any user directly (there is a resource analysis service on the collabora system that might be of interest) - instead it'll be referenced by the NextCloud instance transparently to your users. </p> <p>In our case, we chose the domain collab.oeru.org and the file is called "collab", created via <code>sudo vim /etc/nginx/sites-available/collab</code> and containing (replace collab.domain with the one you've selected - similarly replace the port number 9980 with whatever you've selected if you've opted for a different one!):</p> <p><code>server {<br />     listen 80;<br />     server_name collab.domain;</code></p> <p><code>    # for let's encrypt renewals!<br />     include /etc/nginx/includes/letsencrypt.conf;</code></p> <p><code>    # redirect all HTTP traffic to HTTPS.<br />     location / {<br />         return  302 https://collab.domain$request_uri;<br />     }<br /> }</code></p> <p><code>server {<br />     listen 443 ssl;<br />     server_name collab.domain;</code></p> <p><code>    ssl_certificate /etc/letsencrypt/live/collab.domain/fullchain.pem;<br />     ssl_certificate_key /etc/letsencrypt/live/collab.domain/privkey.pem;<br />     ssl_protocols TLSv1 TLSv1.1 TLSv1.2;<br />     # to create this, see https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html<br />     ssl_dhparam /etc/ssl/certs/dhparam.pem;<br />     keepalive_timeout 20s;</code></p> <p><code>    # for let's encrypt renewals!<br />     include /etc/nginx/includes/letsencrypt.conf;</code></p> <p><code>    proxy_http_version 1.1;<br />     proxy_buffering off;</code></p> <p><code>    # static files<br />     location ^~ /loleaflet {<br />         proxy_pass https://localhost:9980;<br />         proxy_set_header Host $http_host;<br />     }</code></p> <p><code>    # WOPI discovery URL<br />     location ^~ /hosting/discovery {<br />         proxy_pass https://localhost:9980;<br />         proxy_set_header Host $http_host;<br />     }</code><br /><br /><code>    # download, presentation and image upload<br />     location ^~ /lool {<br />         proxy_pass https://localhost:9980;<br />         proxy_set_header Upgrade $http_upgrade;<br />         proxy_set_header Conection "upgrade";<br />         proxy_set_header Host $http_host;<br />     }<br /> }</code></p> <p>Once those are created, you have to make sure that they're "enabled" (replacing with your file names, of course):</p> <p><code>sudo cd /etc/nginx/sites-enabled<br /> sudo ln -sf ../sites-available/docs .<br /> sudo ln -sf ../sites-available/collab .</code></p> <p>To confirm that there aren't any typos or issues that might make nginx unhappy, run</p> <p><code>sudo nginx -t</code></p> <p>If all's well, get nginx to reread its configuration with the new files:</p> <p><code>sudo service nginx reload</code></p> <h2>Firing it all up!</h2> <p>Phew - congratulations on getting here! We've reached the moment of truth where we need to see if this whole thing will work!</p> <p>We need to make sure we're back in the Docker directory we set up:</p> <p><code>cd /home/docker/nextcloud-collabora</code></p> <p>and then we need to try running our docker-compose script to "pull" in the pre-built Docker containers we've specified in our docker-compose.yml file:</p> <p><code>docker-compose pull</code></p> <p>All going well, after a few minutes (longer or shorter depending on the speed of your server's connection) you should have download the Nginx, Redis, NextCloud and Collabora-CODE Docker images. Then you can run:</p> <p><code>docker-compose up -d &amp;&amp; docker-compose logs -f</code></p> <p>This will attempt to start up the containers (bringing them "up" in daemon mode, thus the -d) and then show you a stream of log messages from the containers, preceded by the container name. This should help you debug any problems that occur during the process (ideally, none).</p> <p>Once you see log messages streaming past, and no obvious "container exited" or other error messages (which will usually contain the word "error" a lot), you should be able to point your browser at your selected domain name and bring it up in your browser!</p> <h3>Setting up the database</h3> <p>On doing so, if all is well, you should be directed through the database set up process for your NextCloud instance. Your details should be:</p> <p>database IP: 172.17.0.1 - this is the default IP of the Docker host server.<br /> database name: nextcloud<br /> database user: nextcloud<br /> database password: (the one you came up with above)</p> <h3>Setting the Admin user</h3> <p>Once that's set and working, NextCloud will install all the relevant database tables and initial data. You'll be asked to set up an <em>admin user</em> account, which can be "admin" (you could make it something different to help stymie nefarious probes that assume you've got a user called "admin" - but don't forget what you've called it!) and some strong password you create (you can use the pwgen utility you used earlier) - I'd recommend recording it somewhere. I would <em>not</em> recommend making your own account, in your name, the main admin account. I recommend creating a second account, <em>with administrator privileges</em> for yourself, but leave the admin account purely for administrative activities.</p> <h3>Configuring Outgoing Email</h3> <p>To allow your NextCloud instance to send outgoing email, so that your site can alert you to security updates that need to be applied, or so that users can request a replacement password if they've forgot theirs, you'll need an <em>authenticating SMTP account</em> somewhere. Most of you already have one. You'll probably want to set up a dedicated email address for this server somewhere, perhaps something like "<a href="mailto:nextcloud@your.domain">nextcloud@your.domain</a>" or similar, with a username (often just the email address) and a password. You'll need the following details:</p> <p>SMTP server : an IP address or a domain name<br /> SMTP username: a username or an email address<br /> SMTP password: a strong password already configured for the username on that server<br /> SMTP login security: whether login is via TLS, SSL, or unsecure (!!), and<br /> SMTP login method: plain, encrypted, "login" or some other value.</p> <p>You should be able to test your email settings to make sure the details you've entered are valid. If you need to adjust these settings later, you can go to the admin menu (top right of the web browser interface) and go to Admin-&gt;Additional Settings  - should have a path of <a href="https://your.domain/settings/admin/additional">https://your.domain/settings/admin/additional</a></p> <h3>Configuring Collabora Office Integration</h3> <p>Once you're logged in as your own user, looking at your own default folders, you can start having a look around. You should have an "admin" menu (assuming you've created your user with Administrator privileges) at the top right of the web interface. If you go to Apps, you can use the search box to search for "Collabora" or go to the "Office &amp; text" App category. You'll need to "enable" the Collabora Online "official" app, at which point it will download the latest version of the connector app and install it (it should appear in your /home/data/nextcloud/apps directory)</p> <p>Once you've done that, go to your top right menu again, selecting Admin, and you should see "Collabora Online" as an option in the left column (which starts with "Basic settings"). Selecting that, you'll need to enter  "<a href="https://collab.domain">https://collab.domain</a>" (replacing with your domain, of course). I don't have any of the other options ticked.</p> <p>If it works, you should have the ability to go back to the home of your NextCloud install, which should show you your top-level folders. If you click the "+" next to the home icon (top left of the folder pane) you should now have the option to create (in addition to "Upload file", "New folder", "New text file") a "New Document", "New Spreadsheet", and "New Presentation". Clicking those should give you the Collabora Office interface for the designated content type.</p> <p>Similarly, you can use the "Upload file" to upload a document in a format that is supported by Collabora Office, once uploaded clicking on the filename should open it for editing in the appropriate Collabora Office interface.</p> <p>It is saved as it is change, you shouldn't need to save it explicitly.</p> <h2>Upgrading it</h2> <p>So, as you're no doubt aware, both NextCloud and Collabora Office are always being improved and updated. I certainly encourage you to keep your installation up-to-date.</p> <p>While you'll periodically see that NextCloud apps have available updates (these can be upgraded through the browser interface) updates to the NextCloud and Collabora Office systems themselves need to be undertaken by upgrading the containers. Luckily it's easy to do (although I strongly urge you to ensure you have a very recent backup of both database and uploaded files - they're the files in /home/data/nextcloud/data:</p> <p>Updating the container should be as easy as either doing another</p> <p><code>docker pull oeru/mautic</code></p> <p>and then shutting down Docker container via a</p> <p><code>docker-compose stop</code></p> <p>removing the old containers (this won't remove any data you want to save if you followed the directions above! But remember to do it in the right directory!) via</p> <p><code>docker-compose rm -v</code></p> <p>and then restarting it via</p> <p><code>docker-compose up -d</code></p> <p>Use <code>docker-compose logs -f</code> to watch the logs - you'll likely see debugging information in the unlikely event that something goes wrong in the upgrade process.</p> <h2>Backing it up</h2> <p>To back up your instance on your server, you need two things: a file system backup of your /home/data/nextcloud directory, and database dumps of your database.</p> <p>There're lots of ways to back up your files (I personally use a bash script that I wrote in a past role, which uses <a href="http://www.nongnu.org/rdiff-backup/">rdiff-backup</a> to create versioned backups either locally or on a remote server, although there're <a href="https://www.howtoforge.com/linux_rdiff_backup">other documented approaches</a> - leave a comment below if you'd like to learn more about my approach!).</p> <p>Backing up your database is as easy installing automysqlbackups:</p> <p><code>sudo apt install automysqlbackups</code></p> <p>You'll find daily versioned dumps of your MariaDB database(s) in /var/lib/automysqlbackups. To run an ad hoc backup (which will replace the previous backup from that day, if there is one) just run</p> <p><code>sudo automysqlbackups</code></p> <h2>Collabora Admin Console</h2> <p>Once you've got everything set up, you can access the admin console of the Collabora Office instance at the collab.domain you specified above - it'll have the path <code>https://collab.domain/loleaflet/dist/admin/admin.html</code> (of course replacing collab.domain with your domain) which gives you useful info about the system resources being used, number of documents being edited and by whom, and some other interesting details. I've included a screen shot.</p> <p>When prompted for login details, use the collab username - "admin" if you used the default I provided, and the password you set in your docker-compose.yml file above.</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=17&amp;2=field_blog_comments&amp;3=comment" token="aF_f2aYW3SGRfmrsffb5fOJlN5wEXJibvMUiQKm2VjE"></drupal-render-placeholder> </div> </section> Mon, 29 Jan 2018 04:29:13 +0000 dave 17 at http://tech.oeru.org Installing Mastodon with Docker-Compose on Ubuntu 16.04 http://tech.oeru.org/installing-mastodon-docker-compose-ubuntu-1604 <span class="field field--name-title field--type-string field--label-hidden">Installing Mastodon with Docker-Compose on Ubuntu 16.04</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--mastodon"> <span class="field__item-wrapper"><a href="/taxonomy/term/31" hreflang="en">mastodon</a></span> </div> <div class="field__item field__item--ubuntu-linux"> <span class="field__item-wrapper"><a href="/taxonomy/term/12" hreflang="en">ubuntu linux</a></span> </div> <div class="field__item field__item--_604"> <span class="field__item-wrapper"><a href="/taxonomy/term/27" hreflang="en">16.04</a></span> </div> <div class="field__item field__item--nginx"> <span class="field__item-wrapper"><a href="/taxonomy/term/30" hreflang="en">nginx</a></span> </div> <div class="field__item field__item--lets-encrypt"> <span class="field__item-wrapper"><a href="/taxonomy/term/17" hreflang="en">let&#039;s encrypt</a></span> </div> <div class="field__item field__item--ruby-on-rails"> <span class="field__item-wrapper"><a href="/taxonomy/term/22" hreflang="en">ruby on rails</a></span> </div> <div class="field__item field__item--docker"> <span class="field__item-wrapper"><a href="/taxonomy/term/16" hreflang="en">docker</a></span> </div> <div class="field__item field__item--docker-compose"> <span class="field__item-wrapper"><a href="/taxonomy/term/25" hreflang="en">docker compose</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Fri 02/06/2017 - 15:02</span> <div class="field field-node--field-image field-name-field-image field-type-image field-label-hidden has-multiple"> <figure class="field-type-image__figure image-count-1"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-06/mastodon_userscreen2.png?itok=pyxlNcbL" title="The Mastodon webapp, showing a federated timeline (right panel - and yes, Mastodon is popular in Japan)" data-colorbox-gallery="gallery-field_image-9tvkkGSirDM" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;The Mastodon webapp, showing a federated timeline (right panel - and yes, Mastodon is popular in Japan)&quot;}"><img src="/sites/default/files/styles/medium/public/2017-06/mastodon_userscreen2.png?itok=P5YMy5Yf" width="220" height="141" alt="The Mastodon webapp, showing a federated timeline (right panel - and yes, Mastodon is popular in Japan)" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-2"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-06/mastodon_user_settings.png?itok=1CcpPZP0" title="Mastodon user settings" data-colorbox-gallery="gallery-field_image-9tvkkGSirDM" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Mastodon user settings&quot;}"><img src="/sites/default/files/styles/medium/public/2017-06/mastodon_user_settings.png?itok=w2QahwB4" width="220" height="141" alt="Mastodon user settings" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-3"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-06/mastodon_admin_settings.png?itok=cRSdDIKu" title="Mastodon administrator settings" data-colorbox-gallery="gallery-field_image-9tvkkGSirDM" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Mastodon administrator settings&quot;}"><img src="/sites/default/files/styles/medium/public/2017-06/mastodon_admin_settings.png?itok=jrYKu26i" width="220" height="141" alt="Mastodon administrator settings" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-4"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-06/mastodon_social_info.png?itok=_LbvxOiq" title="Info on a heavily used Mastodon node (Mastodon.Social, the reference node)" data-colorbox-gallery="gallery-field_image-9tvkkGSirDM" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Info on a heavily used Mastodon node (Mastodon.Social, the reference node)&quot;}"><img src="/sites/default/files/styles/medium/public/2017-06/mastodon_social_info.png?itok=N2x5gmY2" width="220" height="143" alt="Info on a heavily used Mastodon node (Mastodon.Social, the reference node)" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-5"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-06/mastodon_nzoss_info.png?itok=mT-9glYy" title="Info on a more humble node (the NZ Open Source Society node I run)" data-colorbox-gallery="gallery-field_image-9tvkkGSirDM" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Info on a more humble node (the NZ Open Source Society node I run)&quot;}"><img src="/sites/default/files/styles/medium/public/2017-06/mastodon_nzoss_info.png?itok=93rckq5L" width="220" height="141" alt="Info on a more humble node (the NZ Open Source Society node I run)" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> </div> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>Not long ago, <a href="https://github.com/tootsuite/mastodon" title="Scroll down to see the write up - the source code is front and centre.">Mastodon</a>, an open source, <em>federated</em> alternative to the proprietary network-effect wunderkind, Twitter, came out of no where. Actually, it came out of an insane amount of work done by free and open source powerhouse Eugen Rochko aka <a href="https://github.com/Gargron">Gargron</a> and a small elite developer community, and many predecessors who are part of the <a href="https://www.coactivate.org/projects/disintermedia/blog/2017/04/01/a-brief-history-of-the-gnu-social-fediverse-and-the-federation/">GNU Social Fediverse</a> (kudos to <a href="http://www.coactivate.org/people/strypey/profile">Danyl Strype</a> for compiling that excellent history).</p> <p>Mastodon, unlike Twitter, is entirely community driven - there are no ads, there are no privacy threats, there are no corporate Terms and Conditions to blindly "I Accept". And your Mastodon "persona" can be on a server you control (or that is controlled by someone you trust). Despite being distributed, you're still part of a global network, but one made resilient by its federated, decoupled nature.</p> <p>Instead of "Tweeting" in 140 characters like on Twitter, your "Toots" are limited to 500 characters (a lot more information can usefully be passed). You can follow people (and they you) by learning their handle - which looks like an email. I've got a couple Mastodon accounts, but my main one right now is <a href="mailto:lightweight@mastodon.social">lightweight@mastodon.social</a> (I set it up quite a while back, before I set up my first couple Mastodon servers). Actually, Mastodon's biggest problem (in my opinion) right now is that you can't easily migrate your "main" persona from one server to another without losing a lot of its value (historical toots, followers, those you follow, etc.). You can migrate some things, like those you're following, and any users you've "blocked" but it's still fairly rudimentary.</p> <p>Mastodon includes a nice web interface which will look somewhat familiar to anyone who's used Twitter's "Tweetdeck" web application. Similarly, the GNU Social community has rallied to provide at least 2 separate open source mobile apps (I run <a href="https://play.google.com/store/apps/details?id=com.keylesspalace.tusky&amp;hl=en">Tusky</a> on my <a href="https://lineageos.org" title="Open Source Android - the way it was supposed to be before the OEMs messed it up.">LineageOS</a> powered phone at the moment) - I think there're some for iOS, too, although Apple's not as amenable to open source apps. </p> <p>There's a useful <a href="https://github.com/tootsuite/documentation/blob/master/Using-Mastodon/FAQ.md">Mastodon FAQ</a>.</p> <h2>Running with the Mastodon Herd</h2> <p>The way I implement a complex Ruby on Rails app like Mastodon is to do as much as possible to keep it at arms length (and stop it from getting anything gooey on my virtual machine). To achieve that comforting isolation, I employ Docker Compose on Ubuntu Linux 16.04. See our <a href="/docker-compose-better-way-deploy-rocketchat-wekan-and-mongodb">Docker Compose article</a> on how to install it (and its dependencies, like Docker itself).</p> <p>Once you've got Docker Compose running, you can do what I did. </p> <p>A couple notes:</p> <ul><li>I have an unprivileged user on my server, "ubuntu". You can use any unprivileged users - I'd encourage you to use sudo rather than login as root.</li> <li>I use "vim" as my terminal-based text editor below. I think it's a great tool, but it does have a learning curve. If you're daunted (no shame in that), I recommend using "nano" instead - it'll probably installed on most Ubuntu 16.04 instances. If someone suggests you use "emacs" instead, they're jerkin' yer chain (I used emacs for over a decade, I know what I'm talking about).</li> <li>make sure you have the "git" VCS system installed... <code>sudo apt install git</code> should do it.</li> <li>you'll need nginx installed, too... <code>sudo apt install nginx-full</code> will do that for you.</li> </ul><p>After logging into my server (via SSH remotely) as the ubuntu user (you might have different non-privileged user name, that's ok), I did the following (to avoid permissions problems later on, we'll create a "mastodon" group and user with the id 991, used by the Mastodon app by default, on the hosting platform):</p> <p><code>groupadd -g 991 mastodon<br /> useradd -u 991 -g 991 -c "Mastodon User" -s /usr/bin/nologin -d /home/data/mastodon mastodon</code><br /><code>sudo mkdir -p /home/docker /home/data/mastodon<br /> sudo chown -R ubuntu:ubuntu /home/docker<br /> sudo chown -R mastodon:mastodon /home/data/mastodon</code><br /><code>cd /home/docker<br /> git clone https://github.com/tootsuite/mastodon.git docker-mastodon<br /> cd docker-mastodon</code></p> <p>What you then need to do is ensure you're using the current "tagged" release (it'll make your life easiest). You can find out what tags are available:</p> <p><code>git tag -l </code></p> <p>At present, the latest tag is "v1.4.7" - to use it do this:</p> <p><code>git checkout tags/v1.4.7</code></p> <p>Obviously, replace this with the most recent tag (note, you might have to look through the whole list to find it!). Then you're using the specific collection of files corresponding to the v1.4.7 tagged release. We can carry on...</p> <p><code>cp .env.production.sample .env.production<br /> vim .env.production</code></p> <p>Edit this file to look like the .env.production sample below, but replacing the [tokens] with your values. Then run this:</p> <p><code>vim docker-compose.yml</code></p> <p>Edit this file to look like docker-compose.yml below.</p> <p><code>docker-compose run --rm web rake secret </code></p> <p>Run this last command 3 times - to get 3 secrets - long random strings - for .env.production! Copy and paste your 3 secrets into your .env.production file with your preferred editor as shown below.</p> <p><code>docker-compose build<br /> docker-compose up</code></p> <p>That should download the required Docker images (might take quite a while depending on how fast your server's network connection is) and result in starting 5 different Docker containers, and you'll be able to watch them put out status (and error) messages as they boot and find their various dependencies. If there're no obvious errors, you can hit CTRL-C to shut them down again and restart them in a mode that keeps them running after you log out</p> <p><code>docker-compose up -d</code></p> <p>Note, you can always stop the containers by running docker-compose stop in that directory. You can check their status by running</p> <p><code>docker ps</code></p> <p>which should show you something like this:</p> <p><code>CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                NAMES<br /> c6be9f3eef1e        gargron/mastodon         "bundle exec rails s "   13 days ago         Up 13 days          127.0.0.1:3000-&gt;3000/tcp, 4000/tcp   dockermastodon_web_1<br /> 6a123d9b1843        gargron/mastodon         "bundle exec sidekiq "   13 days ago         Up 13 days          3000/tcp, 4000/tcp                   dockermastodon_sidekiq_1<br /> f06c4a9bc479        gargron/mastodon         "npm run start"          13 days ago         Up 13 days          3000/tcp, 127.0.0.1:4000-&gt;4000/tcp   dockermastodon_streaming_1<br /> 6dbfad0669f8        postgres:alpine          "docker-entrypoint.sh"   2 weeks ago         Up 13 days          5432/tcp                             dockermastodon_postgres_1<br /> 8026b79e976d        redis:alpine             "docker-entrypoint.sh"   4 weeks ago         Up 13 days          6379/tcp                             dockermastodon_redis_1</code></p> <p>You can use the 12 digit IDs to run other Docker commands, like <code>docker inspect [ID]</code> or <code>docker exec -it [ID] bash</code> to log into the container itself and get a bash prompt. After all that's running, you can do some final housekeeping:</p> <p><code>docker-compose run --rm web rails db:migrate<br /> docker-compose run --rm web rails assets:precompile<br /> sudo vim /etc/nginx/sites-available/mastodon</code></p> <p>Edit this to look like the mastodon nginx config file below.</p> <p><code>sudo cd /etc/nginx/sites-enabled<br /> sudo ln -sf ../sites-available/mastodon .</code></p> <p>to enable the new configuration...</p> <p><code>sudo nginx -t</code></p> <p>To check for typos in you file. If you get no errors, you can restart nginx:</p> <p><code>sudo service nginx restart</code></p> <p>When that's done,  to [your domain] in your browser, which should take you to https://[your domain] and create a new user. If your email is set up properly, you'll get an email confirmation, and this will allow you to log in. If that works, I'd encourage you to modify your configuration to use a Let's Encrypt SSL certificate to protect your users' (and your server's) security. <a href="/protecting-your-users-lets-encrypt-ssl-certs">We provide this dedicated howto</a>! The .env.production template below <em>assumes you've done this</em>, so if your Mastodon isn't working, that might be why (you can try turning <code>LOCAL_HTTPS=false</code> temporarily if that's helpful).</p> <p>You will want to create an admin user - create the user first through the web interface, and then on the command line run (replacing <code>[admin username] </code>with the username you set up:</p> <p><code>cd /home/docker</code><code>/docker-mastodon<br /> docker-compose run --rm web rails mastodon:make_admin USERNAME=[admin username]</code></p> <p>Then go to that user's Mastodon preferences and define the relevant info for your instance (see the administration options).</p> <h2>Debugging</h2> <p>If you run in to problems, a very useful Docker Compose option to use (from within the docker-mastodon directory) is </p> <p><code>docker-compose logs -f</code></p> <p>It will provide you with the automatically updating integrated logs of all the containers you've unleashed!</p> <h3>Sample .env.production</h3> <p>Here's a sample with (hopefully obviously named) [placeholders]</p> <p><code># Service dependencies<br /> REDIS_HOST=redis<br /> REDIS_PORT=6379<br /> DB_HOST=postgres<br /> DB_USER=postgres<br /> DB_NAME=postgres<br /> DB_PASS=<br /> DB_PORT=5432</code></p> <p><code># Federation<br /> LOCAL_DOMAIN=[your domain]<br /> LOCAL_HTTPS=true</code></p> <p><code># Application secrets<br /> # Generate each with the `rake secret` task (`docker-compose run --rm web rake secret` if you use docker compose)<br /> PAPERCLIP_SECRET=[first secret]<br /> SECRET_KEY_BASE=[second secret]<br /> OTP_SECRET=</code>[third secret]</p> <p><code># Registrations<br /> # Single user mode will disable registrations and redirect frontpage to the first profile<br /> # SINGLE_USER_MODE=true<br /> # Prevent registrations with following e-mail domains<br /> # EMAIL_DOMAIN_BLACKLIST=example1.com|example2.de|etc</code></p> <p><code># E-mail configuration<br /> SMTP_SERVER=[smtp server domain name]<br /> SMTP_PORT=587<br /> SMTP_LOGIN=[smtp user name]<br /> SMTP_PASSWORD=[smtp user password]<br /> SMTP_FROM_ADDRESS=[sender address for outgoing mastodon emails]<br /> SMTP_DOMAIN=[your site's base domain]<br /> SMTP_OPENSSL_VERIFY_MODE=none</code></p> <p><code># Optional asset host for multi-server setups<br /> # CDN_HOST=assets.example.com</code></p> <p><code># S3 (optional)<br /> # S3_ENABLED=true<br /> # S3_BUCKET=<br /> # AWS_ACCESS_KEY_ID=<br /> # AWS_SECRET_ACCESS_KEY=<br /> # S3_REGION=<br /> # S3_PROTOCOL=http<br /> # S3_HOSTNAME=192.168.1.123:9000</code></p> <p><code># Optional alias for S3 if you want to use Cloudfront or Cloudflare in front<br /> # S3_CLOUDFRONT_HOST=</code></p> <p><code># Streaming API integration<br /> # STREAMING_API_BASE_URL=</code></p> <h3>Sample docker-compose.yml</h3> <p>Here's a sample with [placeholders]. Note - this generates <strong>five Docker containers. </strong>Yeah, like I said, this is a serious, complex app.</p> <p><code>version: '2'<br /> services:<br />   postgres:<br />     restart: unless-stopped<br />     image: postgres:alpine<br />     volumes:<br />      - /home/data/mastodon/postgres:/var/lib/postgresql/data<br />   redis:<br />     restart: unless-stopped<br />     image: redis:alpine<br />     volumes:<br />      - /home/data/mastodon/redis:/data<br />   web:<br />     restart: unless-stopped<br />     build: .<br />     image: gargron/mastodon<br />     env_file: .env.production<br />     command: bundle exec rails s -p 3000 -b '0.0.0.0'<br />     ports:<br />       - "127.0.0.1:3000:3000"<br />     depends_on:<br />       - postgres<br />       - redis<br />     volumes:<br />       - /home/data/mastodon/packs:/mastodon/public/packs</code><br /><code>      - /home/data/mastodon/assets:/mastodon/public/assets</code><br /><code>      - /home/data/mastodon/system:/mastodon/public/system<br />   streaming:<br />     restart: unless-stopped<br />     build: .<br />     image: gargron/mastodon<br />     env_file: .env.production<br />     command: npm run start<br />     ports:<br />       - "127.0.0.1:4000:4000"<br />     depends_on:<br />       - postgres<br />       - redis<br />   sidekiq:<br />     restart: unless-stopped<br />     build: .<br />     image: gargron/mastodon<br />     env_file: .env.production<br />     command: bundle exec sidekiq -q default -q mailers -q pull -q push<br />     depends_on:<br />       - postgres<br />       - redis<br />     volumes:<br />       - /home/data/mastodon/system:/mastodon/public/system</code></p> <h3>Sample nginx mastodon config file</h3> <p>Here's a copy of the nginx configuration file I use (with [placeholders], obviously) - it's the result of quite a lot of tweaking. Have fun!</p> <p><code>map $http_upgrade $connection_upgrade {<br />     default upgrade;<br />     ''      close;<br /> }</code></p> <p><code>server {<br />     listen 80;<br /> #    listen [::]:80;<br />     server_name [your domain];<br />     root /var/www/html;</code></p> <p><code>    # for let's encrypt renewals!<br />     location /.well-known/acme-challenge/ {<br />         default_type text/plain;<br />         root /var/www/html;<br />    }</code></p> <p><code>    # redirect all HTTP traffic to HTTPS.<br />     location / {<br />         return 302 https://[your domain]$request_uri;<br />     }<br /> }</code></p> <p><code>server {<br />     listen 443 ssl;<br /> #    listen [::]:443 ssl;<br />     server_name [your domain];</code></p> <p><code>    ssl_certificate /etc/letsencrypt/live/[your domain]/fullchain.pem;<br />     ssl_certificate_key /etc/letsencrypt/live/[your domain]/privkey.pem;<br />     ssl_protocols TLSv1 TLSv1.1 TLSv1.2;<br />     # from https://0x39b.fr/post/nginx_security/<br />     ssl_session_timeout 1d;<br />     ssl_session_cache shared:SSL:50m;<br />     #ssl_session_tickets off;<br />     ssl_prefer_server_ciphers on;<br />     ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';<br />     # OCSP Stapling ---<br />     # fetch OCSP records from URL in ssl_certificate and cache them<br />     ssl_stapling on;<br />     ssl_stapling_verify on;<br />     # to create this, see https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html<br />     ssl_dhparam /etc/ssl/certs/dhparam.pem;</code></p> <p><code>    # for let's encrypt renewals!<br />     location /.well-known/acme-challenge/ {<br />         default_type text/plain;<br />         root /var/www/html;<br />     }</code></p> <p><code>    keepalive_timeout    70;<br />     sendfile             on;<br />     client_max_body_size 0;<br />    </code>  <code># update from https://github.com/tootsuite/documentation/blob/master/Running-Mastodon/Production-guide.md<br />     gzip on;<br />     gzip_vary on;<br />     gzip_proxied any;<br />     gzip_comp_level 6;<br />     gzip_buffers 16 8k;<br />     gzip_http_version 1.1;<br />     gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;</code></p> <p><code>    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";</code></p> <p><code>    location / {<br />         try_files $uri @proxy;<br />     }</code></p> <p><code>    location ~ ^/(packs|system/media_attachments/files|system/accounts/avatars) {<br />         add_header Cache-Control "public, max-age=31536000, immutable";<br />         try_files $uri @proxy;<br />     }</code></p> <p> </p> <p><code>    location @proxy {<br />         proxy_set_header Host $host;<br />         proxy_set_header X-Real-IP $remote_addr;<br />         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br />         proxy_set_header X-Forwarded-Proto https;</code><br /><code>        proxy_set_header Proxy "";</code><br /><code>        proxy_pass_header Server;<br />         proxy_pass http://localhost:3000;<br />         proxy_buffering off;<br />         proxy_redirect off;<br />         proxy_http_version 1.1;<br />         proxy_set_header Upgrade $http_upgrade;<br />         proxy_set_header Connection $connection_upgrade;<br />         tcp_nodelay on;<br />     }</code></p> <p><code>    location /api/v1/streaming {<br />         proxy_set_header Host $host;<br />         proxy_set_header X-Real-IP $remote_addr;<br />         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br />         proxy_set_header X-Forwarded-Proto https;<br />         proxy_set_header Proxy "";<br />         proxy_pass http://localhost:4000;<br />         proxy_buffering off;<br />         proxy_redirect off;<br />         proxy_http_version 1.1;<br />         proxy_set_header Upgrade $http_upgrade;<br />         proxy_set_header Connection $connection_upgrade;<br />         tcp_nodelay on;<br />     }</code></p> <p><code>    error_page 500 501 502 503 504 /500.html;<br />     # this should give you an A+ rating on https://instances.mastodon.xyz/<br />     add_header X-XSS-Protection "1; mode=block";<br />     add_header Content-Security-Policy "default-src 'none'; font-src 'self'; media-src 'self'; style-src 'self' 'unsafe-inline'; script-src 'self'; img-src 'self' data: blob:; connect-src 'self' wss://[your domain]";<br /> }</code></p> <p>Enjoy!</p> <h2>Keeping Mastodon up to date</h2> <p>To ensure your Mastodon doesn't become a run down abandoned trailerpark node bit rotting quietly in the ether, I recommend you keep it up to date! There is a useful <a href="https://github.com/tootsuite/documentation/blob/master/Running-Mastodon/Docker-Guide.md">Mastodon Administrator's guide for Docker instances</a> that I consult every time I want to update. Note, if the "git stash" part of it is too hard, I recommend that any time you change your docker-compose.yml file, you copy it to</p> <p><code>cp docker-compose.yml docker-compose.yml-backup</code></p> <p>That way, you can simply remove docker-compose.yml (double check your docker-compose.yml-backup is up-to-date first!), do the <code>git checkout TAG_NAME</code>, and then</p> <p><code>cp docker-compose.yml-backup docker-compose.yml </code></p> <p>and you're done. Welcome the Fediverse!</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=14&amp;2=field_blog_comments&amp;3=comment" token="MWkKHP2AmYlJPPoRbiZwT_Yf5xOfApOMw_Kb2WBZEpo"></drupal-render-placeholder> </div> </section> Fri, 02 Jun 2017 03:02:31 +0000 dave 14 at http://tech.oeru.org Docker Compose: A better way to deploy Rocketchat, Wekan, and MongoDB http://tech.oeru.org/docker-compose-better-way-deploy-rocketchat-wekan-and-mongodb <span class="field field--name-title field--type-string field--label-hidden">Docker Compose: A better way to deploy Rocketchat, Wekan, and MongoDB</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--docker-compose"> <span class="field__item-wrapper"><a href="/taxonomy/term/49" hreflang="en">docker-compose</a></span> </div> <div class="field__item field__item--rocketchat"> <span class="field__item-wrapper"><a href="/taxonomy/term/18" hreflang="en">rocket.chat</a></span> </div> <div class="field__item field__item--wekan"> <span class="field__item-wrapper"><a href="/taxonomy/term/15" hreflang="en">wekan</a></span> </div> <div class="field__item field__item--ubuntu-linux"> <span class="field__item-wrapper"><a href="/taxonomy/term/12" hreflang="en">ubuntu linux</a></span> </div> <div class="field__item field__item--_604"> <span class="field__item-wrapper"><a href="/taxonomy/term/27" hreflang="en">16.04</a></span> </div> <div class="field__item field__item--docker"> <span class="field__item-wrapper"><a href="/taxonomy/term/16" hreflang="en">docker</a></span> </div> <div class="field__item field__item--mongodb"> <span class="field__item-wrapper"><a href="/taxonomy/term/14" hreflang="en">mongodb</a></span> </div> <div class="field__item field__item--nginx"> <span class="field__item-wrapper"><a href="/taxonomy/term/30" hreflang="en">nginx</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Tue 23/05/2017 - 11:03</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>A few months back, I posted instructions on deploying <a href="/installing-rocketchat-docker-ubuntu-linux-1404">Rocket.Chat</a> and <a href="/installing-wekan-docker-ubuntu-linux-1404">Wekan</a> instances (and their mutual dependency, <a href="/installing-mongodb-docker-ubuntu-linux-1404">MongoDB</a>) individually. Since then, I've spent some time with Docker Compose, a set of scripts which help you to define, build, and manage a set of Docker containers. Docker Compose is a thing of beauty. This is the way I now deploy Rocket.Chat, Wekan, and MongoDB together.</p> <h2>Install Docker and Docker Compose</h2> <p>Install <a href="https://docs.docker.com/engine/installation/linux/ubuntu/">Docker</a> (including the "<a href="https://docs.docker.com/engine/installation/linux/linux-postinstall/">post-installation</a>" steps to allow non-root users to run Docker) and <a href="https://docs.docker.com/compose/install/#alternative-install-options" title="We recommend the &quot;pip&quot; install method">Docker Compose</a> on your server (we recommend Ubuntu 16.04 or the older 14.04). We recommend using the "pip" (Python package manager) to do the install.</p> <h2>Create your Docker Compose recipe</h2> <p>We recommend creating a directory with an obvious name - in my case, it's <code>/home/www/docker-rocketchat-wekan-mongo</code></p> <p>In that directory, I create a file called <code>docker-compose.yml</code> containing (I've removed implementation specific details and replaced them with [placeholders]):</p> <p><code>version: '2'<br /> services:<br />   mongo:<br />     restart: unless-stopped<br />     image: mongo<br />     volumes:<br />       - [data directory path]:/data/db<br />       - [backup directory path]:/backups<br />     command: --smallfiles<br />   rocketchat-oeru:<br />     restart: unless-stopped<br />     image: rocketchat/rocket.chat<br />     ports:<br />       - "127.0.0.1:[port number]:3000" # should be a free port above 1024<br />     depends_on:<br />       - mongo<br />     environment:<br />       - MONGO_URL=mongodb://mongo/rocket<br />       - ROOT_URL=[domain name (including schema, e.g. http://)]<br />     volumes:<br />       - [upload directory path]:/var/www/rocket.chat/uploads<br />   wekan:<br />     restart: unless-stopped<br />     image: mquandalle/wekan<br />     ports:<br />       - "127.0.0.1:[port number]:80" # should be a free port above 1024<br />     depends_on:<br />       - mongo<br />     environment:<br />       - VIRTUAL_HOST=[domain name (don't include schema, e.g. https://)]<br />       - MONGO_URL=mongodb://mongo/plan<br />       - ROOT_URL=[domain name (include schema, e.g. https://)]<br />       - MAIL_URL=smtp://[smtp username]:[smtp password]@[server name or IP]:[port: 25, 465, or 587]/<br />     volumes:<br />       - [path to public content]:/built_app/programs/web.browser/app</code></p> <p>Note, you can include multiple instances of either Rocket.Chat or Wekan simply by providing a new name (e.g. rocketchat2 or wekan2 or similar) and a new set of properties - just make sure you're using a unique (and otherwise unused) port number! You can check what's on your server's ports using <code>netstat -punta | less </code>to make sure you're not doubling up. </p> <p>In case it's not obvious, you can leave out either the rocketchat or wekan definitions if you don't want to run those services!</p> <h2>Creating and Running the Docker Containers</h2> <p>It's easy to create the containers: simply run</p> <p><code>docker pull mongo<br /> docker pull rocket.chat<br /> docker pull mquandalle/wekan</code></p> <p>and when it's finished, run</p> <p><code>docker-compose up </code></p> <p>which should start all your containers, but leave you with a running log - this is great for testing, but when you're happy it's all running you hit CTRL-C (to shut down the current set of containers) and then run</p> <p><code>docker-compose up -d </code></p> <p>which runs the containers in daemon mode, without the running log. You can then log out of your server, and your containers will continue running!</p> <p>Based on the configuration above (the "unless-stopped" directive), your containers will restart automatically if your server is rebooted. If you <em>do</em> want to stop them for some reason, you can via</p> <p><code>docker-compose stop</code></p> <p>Easy.</p> <h2>Serving them to the Web</h2> <p>Once you've got your containers running, you need to make sure you've got a web server running on your host to act as the reverse proxy so that external requests get sent to them reliably! We use <a href="nginx.org">Nginx</a>.</p> <h3>RocketChat Nginx</h3> <p>Here's our configuration (with appropriate [substitutions]) - you can create it as <code>/etc/nginx/sites-available/[domain name]</code>:</p> <p><code>server {<br />     listen 80;<br />     server_name [domain name];</code></p> <p><code>  ## Access and error logs.<br />   access_log /var/log/nginx/[domain name]_access.log;<br />   error_log /var/log/nginx/[domain name]_error.log;</code></p> <p><code>  # see https://tech.oeru.org/protecting-your-users-lets-encrypt-ssl-certs<br />   include /etc/nginx/includes/letsencrypt.conf</code></p> <p><code>  # we use a 302 temporary redirect rather than a 301 permanent redir</code><br /><code>  location / {<br />     return 302 https://[domain name]$request_uri;<br />   }<br /> }<br /><br /> server {<br />     listen 443 ssl;<br />     ssl on;</code><br />      <code>  # see https://tech.oeru.org/protecting-your-users-lets-encrypt-ssl-certs<br />     ssl_certificate /etc/letsencrypt/live/[domain name]/fullchain.pem;<br />     ssl_certificate_key /etc/letsencrypt/live/[domain name]/privkey.pem;<br />     ssl_protocols TLSv1 TLSv1.1 TLSv1.2;<br />     ssl_dhparam /etc/ssl/certs/dhparam.pem;</code></p> <p><code>    keepalive_timeout 20s;</code></p> <p><code>    root /var/www/html;<br />     index index.html index.htm;</code></p> <p><code>    server_name [domain name];</code></p> <p><code>    ## Access and error logs.<br />     access_log /var/log/nginx/[domain name]_access.log;<br />     error_log /var/log/nginx/[domain name]_error.log;</code></p> <p><code>    location / {<br />         proxy_pass http://127.0.0.1:[your rocketchat port];<br />         proxy_http_version 1.1;<br />         proxy_set_header Upgrade $http_upgrade;<br />         proxy_set_header Connection "upgrade";<br />         proxy_set_header Host $http_host;<br />         proxy_set_header X-Forwarded-Host $host;<br />         proxy_set_header X-Real-IP $remote_addr;<br />         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br />         proxy_set_header X-Forward-Proto http;<br />         proxy_set_header X-Nginx-Proxy true;<br />         proxy_redirect off;<br />     }<br /> }</code></p> <p> </p> <h3>Wekan Nginx</h3> <p>Here's our configuration (with appropriate [substitutions]) - you can create it as <code>/etc/nginx/sites-available/[domain name]</code>: </p> <p><code># from https://github.com/wekan/wekan/wiki/Install-Wekan-Docker-in-production<br /> upstream websocket {<br />     server 127.0.0.1:[wekan port];<br /> }</code></p> <p><code>map $http_upgrade $connection_upgrade {<br />     default upgrade;<br />     '' close;<br /> }</code><br /><br /><code>server {<br />     listen    80;<br /><br />     root /var/www/html;<br />     index index.html index.htm;</code></p> <p><code>    # Make site accessible from http://localhost/<br />     server_name [domain name];</code></p> <p><code>    access_log /var/log/nginx/[domain name]_access.log;<br />     error_log /var/log/nginx/[domain name]_error.log;</code></p> <p><code>    # see https://tech.oeru.org/protecting-your-users-lets-encrypt-ssl-certs<br />     include /etc/nginx/includes/letsencrypt.conf</code></p> <p><code>    location / {<br />         return 302 https://[domain name]$request_uri; <br />     }<br /> }</code></p> <p><code>server {<br />     listen 443 ssl;<br />     ssl on;</code><br />      <code>  # see https://tech.oeru.org/protecting-your-users-lets-encrypt-ssl-certs<br />     ssl_certificate /etc/letsencrypt/live/[domain name]/fullchain.pem;<br />     ssl_certificate_key /etc/letsencrypt/live/[domain name]/privkey.pem;<br />     ssl_protocols TLSv1 TLSv1.1 TLSv1.2;<br />     ssl_dhparam /etc/ssl/certs/dhparam.pem;</code></p> <p><code>    keepalive_timeout 20s;</code></p> <p><code>    access_log /var/log/nginx/[domain name]_access.log;<br />     error_log /var/log/nginx/[domain name]_error.log;</code></p> <p><code>    root /var/www/html;<br />     index index.html index.htm;</code></p> <p><code>    server_name [domain name];</code></p> <p><code>    location / {<br />         proxy_read_timeout 300;<br />         proxy_connect_timeout 300;<br />         proxy_redirect off;<br />         proxy_set_header Host $http_host;<br />         proxy_set_header X-Real-IP $remote_addr;<br />         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br />         proxy_set_header X-Forwarded-Proto scheme;<br />         proxy_pass http://127.0.0.1:[your wekan port];<br />         proxy_set_header Host $host;<br />     }</code></p> <p><code>    location ~ websocket$ {<br />         proxy_pass http://websocket;<br />         proxy_http_version 1.1;<br />         proxy_set_header Upgrade $http_upgrade;<br />         proxy_set_header Connection $connection_upgrade;<br />     }</code><br /><code>}</code></p> <h3>Enable Nginx Configuration</h3> <p>To make your configurations active, do the following for each of your Nginx configurations:</p> <p><code>cd /etc/nginx/sites-enabled</code></p> <p>Do this for each file:<br /><code>ln -sf ../sites-available/[filename] .</code></p> <p>To check if there are any errors in the files, run</p> <p><code>nginx -t</code></p> <p>If not, you can restart Nginx to incorporate the new configuration files:</p> <p><code>sudo service nginx reload</code></p> <p>You can check for errors in the relevant log files specified in your nginx configurations above in <code>/var/log/nginx/*_error.log</code> or <code>/var/log/nginx/*_access.log</code>.</p> <h2>Protecting your users and reputation with encryption</h2> <p>We encourage you to ensure that these services are made available with full encryption to protect your users' privacy. It's <a href="/protecting-your-users-lets-encrypt-ssl-certs">easy (and no cost) to set up</a>!  The "include" directive in the Nginx configuration files above are examples of this approach.</p> <h2>Upgrades and Backups</h2> <p>We also encourage you to keep your services upgraded. It's easy to do and you'll experience little if any perceptible down time!</p> <p>Simply re-pull the containers and restart them - the updated containers will be launched without loss of data!</p> <p><code>docker pull mongo<br /> docker pull rocket.chat<br /> docker pull mquandalle/wekan</code></p> <p><code>docker-compose up -d</code></p> <p>If you want to back up your data - you need to do normal file backups of the directories on your local server that you've configured in the <code>docker-compose.yml</code> file, and you can do MongoDB backups based on <a href="/installing-mongodb-docker-ubuntu-linux-1404">our previous article</a> on the topic!</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=13&amp;2=field_blog_comments&amp;3=comment" token="IKqqNGzna0isfyEknk_oRE2sfWThSeFDbglHdf4AmjA"></drupal-render-placeholder> </div> </section> Mon, 22 May 2017 23:03:29 +0000 dave 13 at http://tech.oeru.org Many simple tools, loosely coupled http://tech.oeru.org/many-simple-tools-loosely-coupled <span class="field field--name-title field--type-string field--label-hidden">Many simple tools, loosely coupled</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--philosophy"> <span class="field__item-wrapper"><a href="/taxonomy/term/28" hreflang="en">philosophy</a></span> </div> <div class="field__item field__item--unix"> <span class="field__item-wrapper"><a href="/taxonomy/term/29" hreflang="en">unix</a></span> </div> <div class="field__item field__item--ubuntu-linux"> <span class="field__item-wrapper"><a href="/taxonomy/term/12" hreflang="en">ubuntu linux</a></span> </div> <div class="field__item field__item--docker"> <span class="field__item-wrapper"><a href="/taxonomy/term/16" hreflang="en">docker</a></span> </div> <div class="field__item field__item--lets-encrypt"> <span class="field__item-wrapper"><a href="/taxonomy/term/17" hreflang="en">let&#039;s encrypt</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Mon 08/05/2017 - 15:10</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>Our approach to technology here at the <a href="https://oeru.org" title="The Open Education Resource universitas">OERu</a> is inspired by the <a href="http://www.faqs.org/docs/artu/ch01s06.html">UNIX tool philosophy</a> which can be summarised as follows:</p> <blockquote> <p>"create simple tools that do one job well, and make it easy to combine them to work together"</p> </blockquote> <p>In the UNIX (and, somewhat more recently, the Linux) computing environment, this originally meant a lot of small commandline applications like "ls" for listing the contents of file directories, and "grep" for searching directories of files for words and other snippets of content, and "diff" for showing the difference between two files, and many many more. These all output text, and they also accept text as an input - you can <em>chain</em> all of these simple little applications together to create, on the fly, remarkably complex capabilities. This is one of the things that makes Linux and the commandline so powerful for those who have learned its lore (and so intimidating for those who haven't yet done so). </p> <p>This idea of "loosely coupled" tools, working together is also a good way to describe both the OERu technology and documentation philosophy.</p> <p>On this website, the way it manifests is interesting - each time I write a howto article, there're certain common tasks - things like setting up Docker, or creating secure SSL certificates for encrypting user interactions with a web service.</p> <p>Initially, I wrote howtos with all of those details contained in one document, however the instructions fairly quickly become outdated, for example, the install process for Docker or Let's Encrypt is changed by its community (usually to make it faster and more convenient) or to reflect the release of new software dependencies. It doesn't sit well with me to be leaving outdated or inaccurate resources on the web - I feel a responsibility for curating them to improve the signal-to-noise ratio of the 'net. Also, it rapidly becomes an intractable problem to go through old howtos to update all the slight variations on the same instructions to something new (the problem grows exponentially as more howtos are added).</p> <p>So, taking the UNIX approach, I use my experience writing a few howtos to provide insight into parts of each that are repeated. Any section repeated (more or less unchanged) in each howto is a candidate for replacement with a stand-alone howto.</p> <p>As it turns out, the community that's building the Docker container technology has done a good job of keeping their <a href="https://docs.docker.com/engine/installation/linux/ubuntu/">installation documentation</a> up-to-date and making it easy to find the relevant info for our target platforms, Ubuntu Linux 14.04 and 16.04. As a result, there's no point in my repeating their instructions. Instead, I just point my readers there when it's time to install Docker.</p> <p>My first candidate for documentation "<a href="https://en.wikipedia.org/wiki/Code_refactoring">refactoring</a>" was the "<a href="/protecting-your-users-lets-encrypt-ssl-certs">Let's Encrypt</a>" SSL Certificate process (it's a feature of almost all my howtos to date). Having now created that stand-alone howto, I can replace the largely repeated sections of several howtos with a single link to the same place.</p> <p>Yes, this change adds the overhead of the reader of a howto needing to go to a different article on this site, but I think this is greatly outweighed by the benefits: if I need to update or improve the description of this operation, I can simply update one document and ensure it's got the "best of" tips across all the different howtos. Also, if people leave questions or comments, all relevant ones will be in the same place, making it easier for other site visitors to find them.</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=12&amp;2=field_blog_comments&amp;3=comment" token="OgsykCUI1TUoj8ce8ZZ9L5Ax7Aq8mwkhmaCmxsLyqQI"></drupal-render-placeholder> </div> </section> Mon, 08 May 2017 03:10:35 +0000 dave 12 at http://tech.oeru.org Protecting your users with Let's Encrypt SSL Certs http://tech.oeru.org/protecting-your-users-lets-encrypt-ssl-certs <span class="field field--name-title field--type-string field--label-hidden">Protecting your users with Let&#039;s Encrypt SSL Certs</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--lets-encrypt"> <span class="field__item-wrapper"><a href="/taxonomy/term/17" hreflang="en">let&#039;s encrypt</a></span> </div> <div class="field__item field__item--install"> <span class="field__item-wrapper"><a href="/taxonomy/term/11" hreflang="en">install</a></span> </div> <div class="field__item field__item--ubuntu-linux"> <span class="field__item-wrapper"><a href="/taxonomy/term/12" hreflang="en">ubuntu linux</a></span> </div> <div class="field__item field__item--_404"> <span class="field__item-wrapper"><a href="/taxonomy/term/13" hreflang="en">14.04</a></span> </div> <div class="field__item field__item--_604"> <span class="field__item-wrapper"><a href="/taxonomy/term/27" hreflang="en">16.04</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Mon 08/05/2017 - 14:23</span> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p><a class="visually-hidden focusable skip-link" href="https://tech.oeru.org/node/add/blog_post#main-content">Sk</a>For any website that requires anyone (users or even just a few admins) to transfer secrets to and from it, you want to ensure the data is end-to-end encrypted. Today various browsers (like Firefox) give warnings when you're sending secret data (like passwords) "in the clear", namely unencrypted. In early 2017, Google <a href="http://searchengineland.com/google-starts-giving-ranking-boost-secure-httpsssl-sites-199446">added further urgency</a> to doing the right thing for your users. </p> <p>In the past, getting an SSL certificate to achieve encryption for your domain (the little "lock" icon in browser address bar indicating that your communication with the site is encrypted), was a complicated, expensive proposition, requiring a lot of annoying and time consuming "identity verification" (sometimes via post in the dark old days) and a fee of, in some cases, a couple hundred dollars per year paid to your "SSL Cert Provider" to pay for those administrative services along with the software needed to gin up a long prime number to act as your encryption key (the long string of characters making up your SSL certificate).</p> <p>Thankfully, thanks to the efforts of the <a href="https://letsencrypt.org" title="Let's Encrypt - democratising SSL and making it ubquitous.">Let's Encrypt</a> community, the process is both far far easier, and free of cost. Now there really isn't an excuse for not having an SSL certificate on your site.</p> <p>Members of the Let's Encrypt community have provided a range of useful open source tools you can use to create and maintain certificates on your hosting infrastructure (e.g. the Virtual Machine (VM) on which you're installing web services detailed in the howtos on this site!). In this case we're going to use a tool, "<a href="https://certbot.eff.org/">certbot</a>" provided by the good folks at the <a href="https://eff.org">Electronic Frontier Foundation</a>. For VMs running Ubuntu 14.04 or 16.04 (the Long Term Support (LTS) versions of the Ubuntu Linux platform) which are what we use, the install is easy - at your VM command line, run:</p> <pre> <code>sudo add-apt-repository ppa:certbot/certbot sudo apt-get update sudo apt-get install certbot </code></pre> <p>We run Nginx on our VMs, which makes one or more hosted services (normally running in Docker containers) available on the Internet. Strictly speaking, we don't use full "end-to-end" encryption - in our case, on the server-end the encryption terminates at the Nginx server. We, perhaps cavalierly, assume that transfer between the host machine and a Docker container running on that host will be implicitly secure... The only way it could be compromised is if the VM itself is compromised, in which case, the Docker containers running on it could be, too. Avoiding having secure transfer between Nginx on the VM host and the various Docker containers also substantially simplifies setting up each application.</p> <p>Thanks to a service which Nginx provides SNI (or <a href="https://en.wikipedia.org/wiki/Server_Name_Indication">Server Name Indication</a>) - Apache and a few other web servers also provide this - which removes the historical limitation that meant you could only have one SSL certificate per IP address on a web server. The only downside of SNI is that some older browsers (and platforms) don't support it. Since those older technologies are rapidly dying out and it's quite expensive and difficult to have a single IP address for each SSL service on a server, we accept this compromise.</p> <h2>The Let's Encrypt Cert Process</h2> <p>Here's (roughly) how the process works:</p> <ol><li>Point your domain (via A or CNAME record) to point to the/an external IP address on your VM.</li> <li>Set up a domain (or domains) for non-secure hosting (on port 80) via your Nginx instance.</li> <li>The domain's configuration must include a special directory reserved for Let's Encrypt verification content.</li> <li>You request that certbot (on the VM) acquires a certificate for that domain (or domains) at which point <ol><li>the certbot writes a file with a hard-to-guess name to that special directory and requests that the Let's Encrypt infrastructure checks the domain name from outside</li> <li>Let's Encrypt checks that domain name and special directory to see that the expected number appears there, thus verifying that the requesting party actually has the ability to set content at this hard-to-guess filename, and therefore has legitimate claim to being the party controlling the domain name and server.</li> <li>Let's Encrypt registers the certificate request in the name of the party running the certbot (so it can, for example, send emails to the administrator warning them that the certificate needs to be renewed - which happens every 8 weeks or so), and</li> <li>Let's Encrypt sends verification back to your VM's certbot telling it to complete the certificate generation, which it then (digitally) signs in the name of the Let's Encrypt Certificate Authority (which, in turn, is recognised by almost all web browsers out-of-the-box - no mean feat, I can tell you).</li> </ol></li> <li>You get an alert telling you that you have created a valid SSL certificate.</li> <li>You alter your Nginx domain configuration to <ol><li>redirect connections to port 80 (un-encrypted) to port 443 (encrypted), and</li> <li>you set up the 443 configuration including your new certificates.</li> </ol></li> <li>You reload your Nginx configuration, and your site will now be end-to-end encrypted.</li> </ol><h2>The Let's Encrypt Snippet</h2> <p>To make it easy to include the relevant directory info, I recommend that you create a new file in your Nginx configuration (substitute your preferred text editor for "vim" in the following - "nano" is a good choice if you haven't already got a preference):</p> <p><code>sudo mkdir /etc/nginx/includes<br /> sudo vim /etc/nginx/includes/letsencrypt.conf</code></p> <p>and make sure it has the following content (note, I learned this thanks to <a href="https://community.letsencrypt.org/t/how-to-nginx-configuration-to-enable-acme-challenge-support-on-all-http-virtual-hosts/5622">someone else's howto</a> on the global Internet knowledge commons :))</p> <p><code># Rule for legitimate ACME Challenge requests<br /> location ^~ /.well-known/acme-challenge/ {<br />     default_type "text/plain";<br />     # this can be any directory, but this name keeps it clear<br />     root /var/www/letsencrypt;<br /> }</code></p> <p><code># Hide /acme-challenge subdirectory and return 404 on all requests.<br /> # It is somewhat more secure than letting Nginx return 403.<br /> # Ending slash is important!<br /> location = /.well-known/acme-challenge/ {<br />     return 404;<br /> }</code></p> <p>Next, make sure your directory exists (note - you only need to do this once per VM) - it shouldn't need an special permissions - it'll be written by the "root" user, and needs to be readable by the Nginx user, usually "www-data" on a Debian or Ubuntu Linux instance.</p> <p><code>mkdir /var/www/letsencrypt</code></p> <h2>Example Nginx Domain Configuration - unencrypted</h2> <p>Here's an example of a pre-cert Nginx domain configuration for example.org and <a href="http://www.example.org">www.example.org</a> (I usually name the configuration file after the main domain it concerns, so my file would be /etc/nginx/sites-available/example.org) - this should also let you do initial test of your app to make sure it works, before adding the additional complexity of SSL. (<em>Replace example.com (and <a href="http://www.example.com">www.example.com</a>) with your own domain!</em>):</p> <p><code>server {</code></p> <p><code>    listen 80; # this is one of our external IPs on the server.<br />     #listen   [::]:80 default ipv6only=on; ## listen for ipv6<br /><br />     # this root directory isn't really relevant in a proxy situation</code><br /><code>    # so I usually set it to the system default<br />     root /usr/share/nginx/www;<br />     index index.html index.htm;<br /><br />     server_name example.org www.example.org;<br /><br />     access_log /var/log/nginx/example.org_access.log;<br />     error_log /var/log/nginx/example.org_error.log;</code></p> <p><code>    # this is where we include the snippet<br />     include /etc/nginx/includes/letsencrypt.conf;</code></p> <p><code>    # this is just an example of a "proxy" configuration<br />     # for, say, a Docker-based service, exposed on the VM's<br />     # local port 8081<br />     location / {<br />         proxy_read_timeout      300;<br />         proxy_connect_timeout   300;<br />         proxy_redirect          off;<br />         proxy_set_header    Host                $http_host;<br />         proxy_set_header    X-Real-IP           $remote_addr;<br />         proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;<br />         proxy_set_header    X-Forwarded-Proto   $scheme;<br />         proxy_pass      http://127.0.0.1:8081;<br />     }</code><br /><code>}</code></p> <p>You can make sure that the configuration is visible to Nginx by adding it into the "sites-enabled" directory via a file link</p> <p><code>sudo ln -sf /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled </code></p> <p>Test things to make sure the new configuration doesn't have typos or bad references:</p> <p><code>sudo nginx -t</code></p> <p>and if not, make it live:</p> <p><code>sudo service nginx reload</code></p> <p>You should now be able to go to <a href="http://example.com">http://example.com</a> (or your domain) and you'll hopefully get your proxied application (if it's set up) or an Nginx error (see you nginx error file for more info!).</p> <p>Now it's time to request the certificate!</p> <h2>Example Certbot invocation</h2> <p>Once cerbot is installed, and a domain is configured, it's pretty straightforward to get a certificate.</p> <p>On the first invocation of certbot, you might get a coloured interface that requests your user details (e.g. name and email address) so that Let's Encrypt can register them for the purposes of future emails. They email if one of your certificates is on the verge of expiring, or if there's been a change to Let's Encrypt policy or process. It's worth being on the list. </p> <p>You can request your certificate with the following:</p> <p><code>sudo certbot certonly --webroot -w /var/www/letsencrypt -d example.org -d www.example.org</code></p> <p>If it works, it gratifyingly results in a message that starts with "Congratulations"!</p> <h2>Example Nginx Domain Configuration - unencrypted</h2> <p>Once you've got your certificate, you can reference it in your configuration. We normally set up a redirect from the unencrypted version of the site to the encrypted on (except for the Let's Encrypt verification directory!):</p> <p><code>server {</code></p> <p><code>    listen 80; # this is one of our external IPs on the server.<br />  <br />     root /var/www/html;<br />     index index.html index.htm;<br /><br />     server_name example.org www.example.org;<br /><br />     access_log /var/log/nginx/example.org_access.log;<br />     error_log /var/log/nginx/example.org_error.log;</code></p> <p><code>    include /etc/nginx/includes/letsencrypt.conf;</code></p> <p><code>    # a 302 is a "soft" redirect. A 301 can never be reversed.<br />     location / {<br />         return 302 https://chat.oeru.org$request_uri;<br />     }       <br /> }<br /><br /> server {<br />     listen 443 ssl;<br />     ssl on;<br />     ssl_certificate /etc/letsencrypt/live/example.org/fullchain.pem;<br />     ssl_certificate_key /etc/letsencrypt/live/example.org/privkey.pem;</code><br /><code>    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;<br />     ssl_dhparam /etc/ssl/certs/dhparam.pem;<br />     keepalive_timeout 20s;</code></p> <p><code>    access_log /var/log/nginx/example.org_access.log;<br />     error_log /var/log/nginx/example.org_error.log;</code></p> <p><code>    root /var/www/html;<br />     index index.html index.htm;</code><br /><br /><code>    server_name example.org www.example.org;<br />    <br />     # this is just an example of a "proxy" configuration<br />     # for, say, a Docker-based service, exposed on the VM's<br />     # local port 8081<br />     location / {<br />         proxy_read_timeout      300;<br />         proxy_connect_timeout   300;<br />         proxy_redirect          off;<br />         proxy_set_header    Host                $http_host;<br />         proxy_set_header    X-Real-IP           $remote_addr;<br />         proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;<br />         proxy_set_header    X-Forwarded-Proto   $scheme;<br />         proxy_pass      http://127.0.0.1:8081;<br />     }</code><br /><code>}</code></p> <p>Note - you also need to set up the <code>ssl_dhparam</code> file for this configuration to work. You can do this based on <a href="https://michael.lustfield.net/nginx/getting-a-perfect-ssl-labs-score" title="Setting hp ssl_dhparam">these instructions</a> after installing OpenSSL tools:</p> <p><code>sudo apt-get install openssl</code></p> <p>by running (warning - this can take quite a long time - the system needs to generate sufficient <a href="https://en.wikipedia.org/wiki/Entropy">entropy</a> to achieve acceptable randomness):</p> <pre class="literal-block"> <code>openssl dhparam -out /etc/ssl/certs/dhparam.pem 4096</code> </pre> <p>When you've set up the file, you can enable it:</p> <p><code>sudo ln -sf /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled</code></p> <p>Test the file to ensure there aren't any syntax errors before reloading nginx:</p> <p><code>sudo nginx -t</code></p> <p>If this shows an error, you'll need to fix the file. If all's well, reload nginx to include the new configuration:</p> <p><code>sudo service nginx reload</code></p> <p>You should now be able to point your browser at your domain name, and it should automatically redirect you to <a href="https://example.org">https://example.org</a> - and (based on the above configuration, <a href="https://www.example.org">https://www.example.org</a> should work too. You might want to redirect <a href="http://www.example.org">www.example.org</a> to example.org or visa versa).</p> <p>A word to the wise - if it doesn't work, check your firewall settings!</p> <p><strong>Update:</strong> discovered this <a href="https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-14-04">very well done how-to</a> on Let's Encrypt that offers additional background to this one.</p> <h2>Ongoing Certificate Maintenance</h2> <p>One of the nice things about EFF's certbot is that when it's installed, it also installs a nightly cron task (see <code>/etc/cron.d/certbot</code>) which checks all domains registered on the server for renewals. Assuming your domains have been configured in Nginx as described above, renewals should occur automatically, and you'll just receive a periodic email to let you know that they've happened.</p> <p>If you want to check your renewal status, you can run this:</p> <pre> <code>sudo certbot renew --dry-run</code></pre> <p>Good on you for securing your users and your site!</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=11&amp;2=field_blog_comments&amp;3=comment" token="u_cC5Zw8SKo5_CwG7txdoRLRmx6G2xIvWYWTc95IdY0"></drupal-render-placeholder> </div> </section> Mon, 08 May 2017 02:23:06 +0000 dave 11 at http://tech.oeru.org Multiple Discourse Forums on the same server http://tech.oeru.org/multiple-discourse-forums-same-server <span class="field field--name-title field--type-string field--label-hidden">Multiple Discourse Forums on the same server</span> <div class="field field-node--field-blog-tags field-name-field-blog-tags field-type-entity-reference field-label-above"> <h3 class="field__label">Blog tags</h3> <div class="field__items"> <div class="field__item field__item--discourse"> <span class="field__item-wrapper"><a href="/taxonomy/term/19" hreflang="en">discourse</a></span> </div> <div class="field__item field__item--ubuntu-linux"> <span class="field__item-wrapper"><a href="/taxonomy/term/12" hreflang="en">ubuntu linux</a></span> </div> <div class="field__item field__item--docker"> <span class="field__item-wrapper"><a href="/taxonomy/term/16" hreflang="en">docker</a></span> </div> <div class="field__item field__item--postgresql"> <span class="field__item-wrapper"><a href="/taxonomy/term/20" hreflang="en">postgresql</a></span> </div> <div class="field__item field__item--redis"> <span class="field__item-wrapper"><a href="/taxonomy/term/21" hreflang="en">redis</a></span> </div> <div class="field__item field__item--ruby-on-rails"> <span class="field__item-wrapper"><a href="/taxonomy/term/22" hreflang="en">ruby on rails</a></span> </div> <div class="field__item field__item--oauth2"> <span class="field__item-wrapper"><a href="/taxonomy/term/23" hreflang="en">oauth2</a></span> </div> <div class="field__item field__item--lets-encrypt"> <span class="field__item-wrapper"><a href="/taxonomy/term/17" hreflang="en">let&#039;s encrypt</a></span> </div> </div> </div> <span class="field field--name-uid field--type-entity-reference field--label-hidden"><span lang="" about="/user/1" typeof="schema:Person" property="schema:name" datatype="">dave</span></span> <span class="field field--name-created field--type-created field--label-hidden">Fri 24/03/2017 - 13:20</span> <div class="field field-node--field-image field-name-field-image field-type-image field-label-hidden has-multiple"> <figure class="field-type-image__figure image-count-1"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-03/CommunityDiscourse_frontpage.png?itok=zflBNfcG" title="Our Discourse front page from a user&#039;s perspective" data-colorbox-gallery="gallery-field_image-9tvkkGSirDM" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Our Discourse front page from a user&#039;s perspective&quot;}"><img src="/sites/default/files/styles/medium/public/2017-03/CommunityDiscourse_frontpage.png?itok=PKmMwvtz" width="220" height="125" alt="Our Discourse front page from a user&#039;s perspective" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-2"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-03/CommunityDiscourse_badgepage.png?itok=wB27ZtNy" title="&quot;Social credit&quot;: A user&#039;s &quot;badges&quot; which reflect the user&#039;s participation and trust levels within the forum community" data-colorbox-gallery="gallery-field_image-9tvkkGSirDM" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;&quot;Social credit&quot;: A user&#039;s &quot;badges&quot; which reflect the user&#039;s participation and trust levels within the forum community&quot;}"><img src="/sites/default/files/styles/medium/public/2017-03/CommunityDiscourse_badgepage.png?itok=j_6aa6dT" width="220" height="125" alt="&quot;Social credit&quot;: A user&#039;s &quot;badges&quot; which reflect the user&#039;s participation and trust levels within the forum community" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> <figure class="field-type-image__figure image-count-3"> <div class="field-type-image__item"> <a href="http://tech.oeru.org/sites/default/files/styles/max_1300x1300/public/2017-03/CommunityDiscourse_topicediting.png?itok=UKvo5xRW" title="Editing topics - Discourse uses markdown for editing with side-by-side rendering and a simple &quot;rich text&quot; interface to assist new users" data-colorbox-gallery="gallery-field_image-9tvkkGSirDM" class="colorbox" data-cbox-img-attrs="{&quot;alt&quot;:&quot;Editing topics - Discourse uses markdown for editing with side-by-side rendering and a simple &quot;rich text&quot; interface to assist new users&quot;}"><img src="/sites/default/files/styles/medium/public/2017-03/CommunityDiscourse_topicediting.png?itok=osrNHUAn" width="220" height="164" alt="Editing topics - Discourse uses markdown for editing with side-by-side rendering and a simple &quot;rich text&quot; interface to assist new users" typeof="foaf:Image" class="image-style-medium" /> </a> </div> </figure> </div> <div class="clearfix text-formatted field field-node--body field-name-body field-type-text-with-summary field-label-hidden"> <div class="field__items"> <div class="field__item"><p>At the OERu we have two separate instances of market category leading <a href="https://discourse.org" title="The Discourse Forum Community"> Discourse Forum</a>: one for <a href="https://community.oeru.org" title="The OERu Partner and Contributor Collaboration Forum">OER partner and contributor collaborators</a> and the <a href="https://forum.oeru.org" title="The OERu Learner Discourse Forum">other for learners</a>. These days, online forums are seen as a bit old-school: fuddy-duddy. From my point of view, however, Discourse is "Forum-NG" (a Next Generation forum). I think it's both very cool and innovative - not at all fuddy-duddy. Even better, Discourse also happens to be free and open source. Its active development community is storming ahead with updates and improvements at an impressive pace.</p> <p>Discourse is what we developers refer to as a "non-trivial" application. It's complex, no question, but it's also very mature and well engineered. It's built entirely on open source components. It uses the <a href="https://rubyonrails.org">Ruby on Rails</a> framework and pulls in a bunch of external systems including <a href="https://redis.io">Redis</a> (for caching and queuing) and <a href="https://www.postgresql.org/">PostgreSQL</a> for persistent data storage. The most common mode for running Discourse is via a single Docker container which includes PostgreSQL, Redis, and the full Ruby on Rails stack and Discourse application. Typically, an organisation only deploys a single Discourse instance. We, however, identified the need to segment our audiences and so decided to deploy the two instances on our main hosting server. This was much more challenging deployment, and not overly well documented. It took a while to get it right. I wrote up a blow-by-blow of how I did it in hopes it would benefit others in my position! See these two threads:</p> <ul><li><a href="Multiple Discoursen, multiple Docker containers, one server with one nginx">Multiple Discourses, multiple containers, one server</a> for the whole story (and some community comments)</li> <li><a class="fancy-title" data-ember-action="" data-ember-action-932="932" href="https://meta.discourse.org/t/discourse-in-docker-nginx-reverse-proxy-ssl-everywhere-oauth2-custom/52280">Discourse in Docker + NGINX reverse proxy + SSL everywhere + OAuth2 Custom</a> - for protecting the privacy and security of our users, and making it quick and easy for them to log in using existing credentials (but preferably not ones controlled by foreign corporations)</li> </ul><p>Discourse is impressive. It <a href="https://meta.discourse.org/t/benefits-of-discourse-have-i-missed-anything/39849">offers a lot more</a> than I've described so far. I recommend your organisation has a look - if you don't want to manage it in-house (it's easy once it's set up), by all means support the developers by buying their hosted service!</p> </div> </div> </div> <section class="field field-node--field-blog-comments field-name-field-blog-comments field-type-comment field-label-above comment-wrapper"> <a name="comments"></a> <div class="comment-form-wrapper"> <h2 class="comment-form__title">Add new comment</h2> <drupal-render-placeholder callback="comment.lazy_builders:renderForm" arguments="0=node&amp;1=9&amp;2=field_blog_comments&amp;3=comment" token="q1H1hJgcXmYH07bFGODqnJapU96HbkXSo3XOYNpq97w"></drupal-render-placeholder> </div> </section> Fri, 24 Mar 2017 00:20:39 +0000 dave 9 at http://tech.oeru.org