18.04 Configuring a Linux server to send email via the Postfix SMTP server using an external authenticating SMTP host

dave — Mon, 02 Aug 2021 02:08:28 +0000

Configuring a Linux server to send email via the Postfix SMTP server using an external authenticating SMTP host

Blog tags

ubuntu linux

postfix

smtp

18.04

free & open source

foss

20.04

22.04

dave Mon 02/08/2021 - 14:08

Just about any and every server needs to be able to send email - whether it's end-user-email, like password recovery services for a website to emails to system administrators reporting on the status of system backups and errors. The problem is that it's non trivial (understatement) to set up a mail server properly.

This howto assumes you have a Linux server (these instructions are for Ubuntu 22.04 and 20.04, although it should work on earlier versions of Ubuntu server and Debian Linux with minor changes, and the concepts will be very similar on other Linuxen) with a static IP address, with one or more fully-qualified-domain-names (fdqn) pointing at that address, and you have SSH-based access to it. I've previously provided tips on how to get to this stage.

Authenticating SMTP

To send email, you need access to a server, somewhere on the Internet, that provides the Simple Mail Transfer Protocol (SMTP) service. It's an open standard, and for most of the history of the Internet, email services have been mostly provided by Free and Open Source Software (FOSS) tools - the first SMTP was called "Sendmail" and it was fully FOSS, and it's still in use today (although it has mostly been superseded by faster, more secure systems, the best of which are also FOSS).

At the OERu, we use the Docker-based installation of the amazing, completely FOSS MailCow project to provide our organisational email services. I might cover that set up in a future tutorial here, because MailCow makes an otherwise almost intractable problem - hosting your own email service - much more tractable. Having a MailCow set up means we can offer "full service" email for any number of domains and users and aliases with all the bells and whistles including incoming and outgoing mail with all the virus scanning (we don't really need it because we use Linux desktops, but for other folks it's useful) and dynamic spam filtering services you'd expect from a much larger operation: Team MailCow have done an amazing job in pulling together a comprehensive set of FOSS applications to provide all the conceivable requirements of a full-fledged, multi-domain email system, including shared calendaring, contacts, and webmail. A great companion to your organisation's MailCow server would be a BitWarden password safe server (also FOSS)... just sayin'.

So, now, assuming that we have a MailCow server or some other functionally equivalent SMTP service available (apparently you can do this with Gmail, if you're a paying using although because of Google's terms of use, we recommend finding a more trustworthy solution), we have the option of "authenticated SMTP" for outgoing email using credentials we can set up. For example, in MailCow, we can specify a domain we host, like say oeru.org (and for which we've defined an MX record and a few other relevant records as guided by MailCow administrative web interface). On top of that, we can specify a mailbox for a dedicated "send stuff from remote relay hosts" email address using that domain, like smtp@oeru.org, with a strong password. With that, we can securely send email using that email address as the username and that password from anywhere we have access to the Internet.

The only tricky part is that we have to ensure that whatever "reply to" email address we specify from our applications, say notifications@tech.oeru.org, is using a domain we also host on the same server, and that there's an email alias of that email address defined and set as "allow to send from smtp@oeru.org" in the MailCow interface. If we haven't made sure of that, our mail server is likely to reject sending emails with that "mismatching" email address. This is a basic spam deterrence measure, which is for the best, despite sometimes making a email system administrator's life harder.

Once we've got that (and it's easy once you've done it once or twice - I'm mostly writing this down now so I don't have to try to re-remember every time I need to set up a new server - and I hope it helps others, too), we can set up any server we control to send secure (and spam-filter-resilient) email. For what it's worth, too, MailCow uses Postfix as its SMTP server component (there're a bunch of other components, too).

Postfix SMTP with SmartHost

The first thing you need to do to create a postfix smarthost is to install the postfix application on a new server (this assumes you're logged in with a user who has "sudo" - aka admin - permissions):

sudo apt update && sudo apt install postfix bsd-mailx

During the install, you'll be asked to select a bunch of configuration parameters. Select the defaults except:

Select "Internet Site with Smarthost",
fill in the domain name for your server,
the domain name and port (in the form [smtp server domain]:[port], e.g. smtp.oeru.org:587 ) of your "smarthost" who'll be doing the authenticating SMTP for you, and
the email address to which you want to receive system-related messages.

After that's done, you can proceed.

Next Steps

For the rest of this tutorial, you'll need to do the following. First, select your text editor. I use vim, but if you're new to the command line, I recommend using nano - it's more straightforward:

EDIT=`which nano` or EDIT=`which vim`

sudo $EDIT /etc/aliases

We need to make sure the "root" user points to a real email address. Add a line at the bottom which says (replacing [your email] with your email :) )

root: [your email]

After which you'll need to convert the aliases file into a form that postfix can process, simply by running this:

sudo newaliases

Then we have to define the authentication credentials required to convince your mail server that you're you!

sudo $EDIT /etc/postfix/relay_password

The resulting file only needs one line with three bits of information:

[smtp server domain] [user name]:[password]

for example:

smtp.oeru.org smtp@oeru.org:SomeObscurePassw0rd

Then save the file and, like the aliases file, run the conversion process (which uses a slightly different mechanism):

sudo postmap /etc/postfix/relay_password

Finally, we'll edit the main configuration file for Postfix to tell it about all this stuff:

sudo $EDIT /etc/postfix/main.cf

If your SMTP server uses port 25 (the default for unencrypted SMTP) you don't have to change anything, although most people nowadays prefer to use StartTLS or otherwise encrypted transport to at least ensure that your SMTP authentication details (at least) are transferred encrypted. That means using port 587 or 465. If you're using either of those ports, find the "relayhost = [your server name]" line... and add your port number after a colon, like this

relayhost = [your server name]:[server port]

or, for example:

relayhost = smtp.oeru.org:465

Next, add the following lines at the bottom of the file:

# added to configure accessing the relay host via authenticating SMTP smtp_sasl_auth_enable = yes smtp_sasl_password_maps = hash:/etc/postfix/relay_password smtp_sasl_security_options = noanonymous
smtp_tls_security_level = encrypt

# if you're using Ubuntu prior to 20.04, uncomment (remove the #) the
# earlier line smtp_tls_security_level = may to save errors in 'postfix check' # and comment this line (by adding a # at the start) smtp_tls_wrappermode = yes

And, finally, comment out the line smtp_tls_security_level = may higher in the file - careful not to confuse it with the very similar smtpd_tls_security_level variable (note the extra '''d''' in `smtpd...`) line.

Save the file, and then check that your syntax is correct:

sudo postfix check

If it is (running the command returns no errors, and it might not return anything at all - that's a good thing!), then you can run

sudo postfix reload

to get postfix to reload its configurations and you can test out your new smarthost-configured SMTP server!

If not, the output of the check command will usually give you a helpful insight into what is wrong with your configuration... you'll also find that looking at the mail log is very helpful and offers great insights:

sudo less +G /var/log/mail.log

and if you're not able to fix it based on those, you'll find postfix is widely documented and has rich set of easily discoverable resources out there on the web - a search engine is your best resource!

Testing your outgoing email

By default, a command line application called "mail" is installed as part of the bsd-mailx package we installed alongside postfix. You can use it to send test email from the command line on your host to verify you've got things working correctly! The stuff in <> are the keys to hit at the end of the line...

$ mail you@email.domain<ENTER>

Subject: Testing from your.relay.server.domain<ENTER> Testing postfix remote host<ENTER> <CTRL-D> Cc:<ENTER>

Typing <CTRL-D> (hold down the Control or Ctrl key on your keyboard and press the "d" key) will finish your message, showing you a "CC:" field, in which you can type in other email addresses if you want to test sending to multiple addresses. When you then hit <ENTER>, it will attempt to send this email. It might take a few minutes to work its way through to the receiving email system (having to run the gauntlet of spam and virus filters on the way).

You can also always check the postfix system logs to see what postfix thinks about it using the command above. Hit <SHIFT-F> to have the log update in real time.

Done

Now you've got working outgoing email from your server. That means many higher-level web applications you might install on your infrastructure will work out-of-the-box, because what you've set up, for example, enables the default PHP email service and that used by other stacks.

Sending from Docker Containers

You can configure your server so you can reference it from services you run from Docker containers on your host. You do this by referencing the host, like via an ad hoc SMTP server on your container like msmtp, and you can just reference it as 172.17.0.1, which is the default base IP for Docker hosts from the perspective of Docker containers. You might find it's different on your particular install. In that case, you have to make your Postfix SmartHost accept email for sending from the Docker containers on that server. There're quite a few examples of that among my Docker recipes on the OERu's git repository.

Add new comment

Protecting your users with Let's Encrypt SSL Certs

dave — Thu, 08 Jul 2021 02:23:06 +0000

Protecting your users with Let's Encrypt SSL Certs

Blog tags

let's encrypt

install

ubuntu linux

14.04

16.04

18.04

20.04

dave Thu 08/07/2021 - 14:23

SkFor any website that requires anyone (users or even just a few admins) to transfer secrets to and from it, you want to ensure the data is end-to-end encrypted. Today various browsers (like Firefox) give warnings when you're sending secret data (like passwords) "in the clear", namely unencrypted. In early 2017, Google added further urgency to doing the right thing for your users.

In the past, getting an SSL certificate to achieve encryption for your domain (the little "lock" icon in browser address bar indicating that your communication with the site is encrypted), was a complicated, expensive proposition, requiring a lot of annoying and time consuming "identity verification" (sometimes via post in the dark old days) and a fee of, in some cases, a couple hundred dollars per year paid to your "SSL Cert Provider" to pay for those administrative services along with the software needed to gin up a long prime number to act as your encryption key (the long string of characters making up your SSL certificate).

Thankfully, thanks to the efforts of the Let's Encrypt community, the process is both far far easier, and free of cost. Now there really isn't an excuse for not having an SSL certificate on your site.

Members of the Let's Encrypt community have provided a range of useful open source tools you can use to create and maintain certificates on your hosting infrastructure (e.g. the Virtual Machine (VM) on which you're installing web services detailed in the howtos on this site!). In this case we're going to use a tool, "certbot" provided by the good folks at the Electronic Frontier Foundation.

For VMs running Ubuntu 18.04 or more recent Long Term Support (LTS) versions of the Ubuntu Linux platform, you should be able to just run:

sudo apt-get install letsencrypt

For VMs running Ubuntu 14.04 or 16.04 which are what we use, the install is easy - at your VM command line, run:

sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install certbot

We run Nginx on our VMs, which makes one or more hosted services (normally running in Docker containers) available on the Internet. Strictly speaking, we don't use full "end-to-end" encryption - in our case, on the server-end the encryption terminates at the Nginx server. We, perhaps cavalierly, assume that transfer between the host machine and a Docker container running on that host will be implicitly secure... The only way it could be compromised is if the VM itself is compromised, in which case, the Docker containers running on it could be, too. Avoiding having secure transfer between Nginx on the VM host and the various Docker containers also substantially simplifies setting up each application.

Thanks to a service which Nginx provides SNI (or Server Name Indication) - Apache and a few other web servers also provide this - which removes the historical limitation that meant you could only have one SSL certificate per IP address on a web server. The only downside of SNI is that some older browsers (and platforms) don't support it. Since those older technologies are rapidly dying out and it's quite expensive and difficult to have a single IP address for each SSL service on a server, we accept this compromise.

The Let's Encrypt Cert Process

Here's (roughly) how the process works:

Point your domain (via A or CNAME record) to point to the/an external IP address on your VM.
Set up a domain (or domains) for non-secure hosting (on port 80) via your Nginx instance.
The domain's configuration must include a special directory reserved for Let's Encrypt verification content.
You request that certbot (on the VM) acquires a certificate for that domain (or domains) at which point
1. the certbot writes a file with a hard-to-guess name to that special directory and requests that the Let's Encrypt infrastructure checks the domain name from outside
2. Let's Encrypt checks that domain name and special directory to see that the expected number appears there, thus verifying that the requesting party actually has the ability to set content at this hard-to-guess filename, and therefore has legitimate claim to being the party controlling the domain name and server.
3. Let's Encrypt registers the certificate request in the name of the party running the certbot (so it can, for example, send emails to the administrator warning them that the certificate needs to be renewed - which happens every 8 weeks or so), and
4. Let's Encrypt sends verification back to your VM's certbot telling it to complete the certificate generation, which it then (digitally) signs in the name of the Let's Encrypt Certificate Authority (which, in turn, is recognised by almost all web browsers out-of-the-box - no mean feat, I can tell you).
You get an alert telling you that you have created a valid SSL certificate.
You alter your Nginx domain configuration to
1. redirect connections to port 80 (un-encrypted) to port 443 (encrypted), and
2. you set up the 443 configuration including your new certificates.
You reload your Nginx configuration, and your site will now be end-to-end encrypted.

The Let's Encrypt Snippet

To make it easy to include the relevant directory info, I recommend that you create a new file in your Nginx configuration (substitute your preferred text editor for "vim" in the following - "nano" is a good choice if you haven't already got a preference):

sudo mkdir /etc/nginx/includes sudo vim /etc/nginx/includes/letsencrypt.conf

and make sure it has the following content (note, I learned this thanks to someone else's howto on the global Internet knowledge commons :))

# Rule for legitimate ACME Challenge requests location ^~ /.well-known/acme-challenge/ { default_type "text/plain"; # this can be any directory, but this name keeps it clear root /var/www/letsencrypt; }

# Hide /acme-challenge subdirectory and return 404 on all requests. # It is somewhat more secure than letting Nginx return 403. # Ending slash is important! location = /.well-known/acme-challenge/ { return 404; }

Next, make sure your directory exists (note - you only need to do this once per VM) - it shouldn't need an special permissions - it'll be written by the "root" user, and needs to be readable by the Nginx user, usually "www-data" on a Debian or Ubuntu Linux instance.

mkdir /var/www/letsencrypt

Example Nginx Domain Configuration - unencrypted

Here's an example of a pre-cert Nginx domain configuration for example.org and www.example.org (I usually name the configuration file after the main domain it concerns, so my file would be /etc/nginx/sites-available/example.org) - this should also let you do initial test of your app to make sure it works, before adding the additional complexity of SSL. (Replace example.com (and www.example.com) with your own domain!):

server {

listen 80; # this is one of our external IPs on the server. #listen [::]:80 default ipv6only=on; ## listen for ipv6 # this root directory isn't really relevant in a proxy situation
# so I usually set it to the system default root /usr/share/nginx/www; index index.html index.htm; server_name example.org www.example.org; access_log /var/log/nginx/example.org_access.log; error_log /var/log/nginx/example.org_error.log;

# this is where we include the snippet include /etc/nginx/includes/letsencrypt.conf;

# this is just an example of a "proxy" configuration # for, say, a Docker-based service, exposed on the VM's # local port 8081 location / { proxy_read_timeout 300; proxy_connect_timeout 300; proxy_redirect off; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_pass http://127.0.0.1:8081; }
}

You can make sure that the configuration is visible to Nginx by adding it into the "sites-enabled" directory via a file link

sudo ln -sf /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled

Test things to make sure the new configuration doesn't have typos or bad references:

sudo nginx -t

and if not, make it live:

sudo service nginx reload

You should now be able to go to http://example.com (or your domain) and you'll hopefully get your proxied application (if it's set up) or an Nginx error (see you nginx error file for more info!).

Now it's time to request the certificate!

Example Certbot invocation

Once cerbot is installed, and a domain is configured, it's pretty straightforward to get a certificate.

On the first invocation of certbot, you might get a coloured interface that requests your user details (e.g. name and email address) so that Let's Encrypt can register them for the purposes of future emails. They email if one of your certificates is on the verge of expiring, or if there's been a change to Let's Encrypt policy or process. It's worth being on the list.

You can request your certificate with the following:

sudo certbot certonly --webroot -w /var/www/letsencrypt -d example.org -d www.example.org

If it works, it gratifyingly results in a message that starts with "Congratulations"!

Example Nginx Domain Configuration - unencrypted

Once you've got your certificate, you can reference it in your configuration. We normally set up a redirect from the unencrypted version of the site to the encrypted on (except for the Let's Encrypt verification directory!):

server {     listen 80; # listen on port 80 on all IPv4 addresses
    listen [::]:80; # listen on port 80 on all IPv6 addresses
    root /var/www/html;     index index.html index.htm;     server_name example.org www.example.org;     access_log /var/log/nginx/example.org_access.log;     error_log /var/log/nginx/example.org_error.log;

include /etc/nginx/includes/letsencrypt.conf;

    # a 302 is a "soft" redirect. A 301 can never be reversed.     location / {         return 302 https://example.org$request_uri;     }      } server {
    listen 443 ssl; # listen on port 443 on all IPv4 addresses
    listen [::]:443 ssl; # listen on port 443 on all IPv6 addresses

    # this is a deprecated rule, not required in Ubuntu 20.04's nginx     # ssl on;     ssl_certificate /etc/letsencrypt/live/example.org/fullchain.pem;     ssl_certificate_key /etc/letsencrypt/live/example.org/privkey.pem;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_dhparam /etc/ssl/certs/dhparam.pem;     keepalive_timeout 20s;

access_log /var/log/nginx/example.org_access.log; error_log /var/log/nginx/example.org_error.log;

root /var/www/html; index index.html index.htm;

server_name example.org www.example.org; # this is just an example of a "proxy" configuration # for, say, a Docker-based service, exposed on the VM's # local port 8081 location / { proxy_read_timeout 300; proxy_connect_timeout 300; proxy_redirect off; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_pass http://127.0.0.1:8081; }
}

Note - you also need to set up the ssl_dhparam file for this configuration to work. You can do this based on these instructions after installing OpenSSL tools:

sudo apt-get install openssl

by running (warning - this can take quite a long time, 5-15 minutes in my experience - the system needs to generate sufficient entropy to achieve acceptable randomness):

openssl dhparam -out /etc/ssl/certs/dhparam.pem 4096

When you've set up the file, you can enable it:

sudo ln -sf /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled

Test the file to ensure there aren't any syntax errors before reloading nginx:

sudo nginx -t

If this shows an error, you'll need to fix the file. If all's well, reload nginx to enable the new configuration:

sudo service nginx reload

You should now be able to point your browser at your domain name, and it should automatically redirect you to https://example.org - and (based on the above configuration, https://www.example.org should work too. You might want to redirect www.example.org to example.org or visa versa).

A word to the wise - if it doesn't work, check your firewall settings!

Update: discovered this very well done how-to on Let's Encrypt that offers additional background to this one.

Ongoing Certificate Maintenance

One of the nice things about EFF's certbot is that when it's installed, it also installs a nightly cron task (see /etc/cron.d/certbot) which checks all domains registered on the server for renewals. Assuming your domains have been configured in Nginx as described above, renewals should occur automatically, and you'll just receive a periodic email to let you know that they've happened.

If you want to check your renewal status, you can run this:

sudo certbot renew --dry-run

Good on you for securing your users and your site!

Add new comment

Installing NextCloud Hub with OnlyOffice on Ubuntu 18.04

dave — Mon, 03 Feb 2020 20:41:16 +0000

Installing NextCloud Hub with OnlyOffice on Ubuntu 18.04

Blog tags

ubuntu linux

18.04

nextcloud

onlyoffice

mariadb

docker compose

docker

php

redis

polls

scheduling

dave Tue 04/02/2020 - 09:41

Update 2023-06-14: Here's a new tutorial for Ubuntu 22.04! Consider this older tutorial obsolete (but possibly helpful for people with older systems?)

I have previously provided an in-depth explanation about NextCloud with Collabora Office Online and how we've installed it on Ubuntu 16.04. This is an update both of the process, and of the technology. NextCloud is leaping from strength to strength, and seems to be benefiting from the well-founded concern held by many in the EU about data sovereignty and the market domination (and exploitation) of US-based multinationals like Amazon, Google, Microsoft, Dropbox, and others. As a collaborative, web-based front end to LibreOffice, Collabora shows great potential... but it's not anywhere near the capabilities of Google Docs...

The same, however, is not true of a relatively new entry into the web-based collaborative productivity application space: OnlyOffice. The application itself (for the tech focused reader, they've built an entirely new application ecosystem primarily using modern Javascript frameworks) is impressive in both capabilities and polish. The only real caveat I've come across is that it uses, by default, the fauxpen standard formats developed by Microsoft rather than the true open standard formats of OpenDocumentFormat. But in a world where, sadly, most people don't even know what a file format is, any software that doesn't read and write the incumbent monopolist's format with great fidelity is dead in the water. On that count, OnlyOffice is impressive.

NextCloud and OnlyOffice - even better together!

The beauty of the open source software model is that we can connect complementary applications, like NextCloud and OnlyOffice - developed by completely separate communities - to create a tightly integrated, highly functional, diverse computing platform. This combination, along with a bunch of other NextCloud "apps", is the equal of something like Google Apps (which includes Google Docs and Google Drive), but is under your control, not Google's. To me, that's a crucial difference.

With the release of NextCloud 18.0.1, NextCloud has bundled OnlyOffice with it, creating something called "NextCloud Hub". It's pretty impressive. That's what we're setting up here!

Setting up your own NextCloud Hub!

Yes, NextCloud and OnlyOffice servers on the same host.

If you're game to run your own (and, in my experience, it's a surprisingly well behaved system) here's how you do it.

In preparation, you'll want to have the following ready:

a Linux virtual machine or "VM" (I recommend running the current Ubuntu LTS version, or current Debian) with an external IP address and a user with sudo privileges - more info on that...,
your domain name for the NextCloud instance, pointing to the IP address of your VM,
credentials for an email address capable of sending from a remote server (usually termed an "authenticating SMTP email account")

Please note: the images accompanying this howto have been pulled from several different NextCloud and OnlyOffices I maintain.

Secure access with SSH

First things first, make sure you're logged into your host (probably via SSH) as a user who has "sudo" capabilities! You need to log into the host from your local machine. We recommend setting up key-based authentication.

Firewall with UFW

No computer system is ever full secure - there're always exploits waiting to be found, so security is a process of maintaining vigilance. Part of that is reducing exposure - minimising your "attack surface". Use a firewall - "ufw" is installed on Ubuntu by default. Make sure you've got exceptions for SSH (without them, you could lock yourself out of your machine! Doh!).

Run the following commands to allow your Docker containers to talk to other services on your host.

sudo ufw allow in on docker0 sudo ufw allow from 172.0.0.0/8 to any

Specifically for Docker's benefit, you need to tweak the default Forwarding rule (I use "vim" as my editor. If you don't know how to/want to use it, replace vim with nano everywhere you see it in the following - nano's easier to use for simple edits like this):

sudo vim /etc/default/ufw

and duplicate (copy and paste it) the line DEFAULT_FORWARD_POLICY="DROP" tweak it to look like this (commenting out the default, but leaving it there for future reference, so you know what you've changed!):

#DEFAULT_FORWARD_POLICY="DROP" DEFAULT_FORWARD_POLICY="ACCEPT"

You also have to edit /etc/ufw/sysctl.conf and remove the "#" at the start of the following lines, so they look like this:

sudo vim /etc/ufw/sysctl.conf

# Uncomment this to allow this host to route packets between interfaces net/ipv4/ip_forward=1 net/ipv6/conf/default/forwarding=1 net/ipv6/conf/all/forwarding=1

and finally restart the network stack and ufw on your server

sudo service networking restart sudo service ufw restart

Installing the Nginx webserver

In the configuration I'm describing here, you'll need a webserver running on the server - it'll be acting as a "proxy" for the Docker-based Nginx instance described below. I like the efficiency of Nginx and clarity of Nginx configurations over those of Apache and other open source web servers. Here's how you install it.

sudo apt-get install nginx-full

To allow nginx to be visible via ports 80 and 443, run

sudo ufw allow "Nginx Full"

Note: make sure your hosting service is not blocking these ports at some outer layer (depending on who's providing that hosting service you may have to set up port forwarding).

Installing MariaDB

MariaDB is effectively a drop-in alternative to MySQL and we prefer it because it's not controlled by Oracle and has a more active developer community. On Ubuntu, MariaDB pretends to be MySQL for compatibility purposes, so don't be weirded out by the interchangeable names below. Install the server and the client like this.

sudo apt-get install mariadb-server-10.0 mariadb-client-10.0

You need to set a root (admin) user password - you might want to create a /root/.my.cnf file containing the following (replacing YOURPASSWORD) to let you access MariaDB without a password from the commandline:

[client] user=root password=YOURPASSWORD

You should now be able to type "mysql" at the command prompt

Tweak the configuration so that it's listening on

sudo vim /etc/mysql/mariadb.conf.d/50-server.cnf

and copy the bind-address line and adjust so it looks like this - we want MariaDB to be listening on all interfaces, not just localhost (127.0.0.1)...

# Instead of skip-networking the default is now to listen only on # localhost which is more compatible and is not less secure. #bind-address = 127.0.0.1 bind-address = 0.0.0.0

Then restart MariaDB:

sudo service mysql restart

It should now be listening on port 3306 on all interfaces, i.e. 0.0.0.0.

Now set up the database which will hold NextCloud's data. Log into the MySQL client on the host (if you've created a .my.cnf file in your home directory as describe above, you won't need to enter your username and password):

mysql -u root -p

Enter your root password when prompted. It's also a good idea to gin up a password for your "nextcloud" database user. I usually use pwgen (sudo apt-get install pwgen) - for example running this command will give you a single 19 character password without special characters (just numbers and letters):

pwgen -s 19 1

Giving you something like this (but if it's truly random, almost certainly not exactly this):

bYIOSrvR9aGwL5FRGFU

At the prompt (which will look something like MariaDB [(none)]>) enter the following lines (putting your password in place of [passwd]):

CREATE DATABASE nextcloud CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; CREATE USER "nextcloud"@"%" IDENTIFIED BY "[passwd]"; GRANT ALL ON nextcloud.* to "nextcloud"@"%"; FLUSH PRIVILEGES;

Then enter \q to exit.

Prepare your Docker Compose host

We make use of the NextCloud community's stable Docker container which they keep up to date. Similarly, the OnlyOffice developers maintain a Docker container, too. We will run them both on this same server as separate services via Docker Compose. The two sets of Docker containers will look like this:

a suite of NextCloud containers:
1. the main PHP-FPM container (which provides most of the functionality for NextCloud using the PHP scripting engine,
2. an identical container to the PHP one which runs the cron service (which does periodic administrative tasks relevant to NextCloud)
3. a Redis container (which provides performance improving caching for NextCloud), and
4. an Nginx webserver container which makes it easier to manage the configuration and paths of the NextCloud instance. It means that on the hosting server, we only need to run a proxying web server, which is easy.
the single OnlyOffice container which, despite the Docker convention of each container running only a single services, runs the whole OnlyOffice stack, which includes PostgreSQL, Nginx, Rabbit-MQ, Python, and NodeJS.

The way I prefer to implement this set of containers is to use:

sudo apt-get install docker-compose

to set up the entire Docker and Docker Compose system on your server.

Then set up a place for your Docker containers (replace "me" with your non-root username on the server) and the associated persistent data (your Docker containers should hold no important data - you should be able to delete and recreate them entirely without losing any important data or configuration):

sudo mkdir /home/data
sudo mkdir /home/data/nextcloud sudo mkdir /home/data/nextcloud-nginx sudo mkdir /home/data/nextcloud-redis
sudo mkdir /home/data/onlyoffice
sudo mkdir /home/docker sudo mkdir /home/docker/nextcloud
sudo chown -R me:me /home/docker

NextCloud Install

First, let's set up NextCloud (this also installs the OnlyOffice server):

cd /home/docker/nextcloud

Here's an example of the required docker-compose.yml file (you can create this via a text editor like "nano" which should be pre-installed on any VM these days (or use my preferred, but less intuitive, editor, vim) nano docker-compose.yml in the /home/docker/nextcloud directory):

version: '3' services: nginx: container_name: nginx-server image: nginx ports: - 127.0.0.1:8082:80 volumes: - /home/data/nextcloud-nginx/nginx/nginx.conf:/etc/nginx/nginx.conf:ro - /home/data/nextcloud:/var/www/html links: - app environment: - VIRTUAL_HOST restart: unless-stopped app: container_name: app-server image: nextcloud:fpm stdin_open: true tty: true links: - redis expose: - '80' - '9000' volumes: - /home/data/nextcloud:/var/www/html restart: unless-stopped cron: image: nextcloud:fpm volumes: - /home/data/nextcloud:/var/www/html user: www-data entrypoint: | bash -c 'bash -s <<EOF trap "break;exit" SIGHUP SIGINT SIGTERM while /bin/true; do /usr/local/bin/php /var/www/html/cron.php sleep 900 done EOF' restart: unless-stopped redis: image: redis:alpine volumes: - /home/data/nextcloud-redis:/data restart: unless-stopped onlyoffice-document-server: container_name: onlyoffice-document-server image: onlyoffice/documentserver:latest stdin_open: true tty: true restart: unless-stopped expose: - '80' - '443' volumes: - /home/data/onlyoffice/data:/var/www/onlyoffice/Data - /home/data/onlyoffice/log:/var/log/onlyoffice

The "port" specified above, 8082 for nginx is arbitrary - I picked it to ensure it doesn't don't conflict with ports being used by other containers on my server - you can use these if you want, or use sudo netstat -punta to see what ports are currently claimed by other services on your server (if there are any) and pick one that doesn't clash! If it scroll past too fast, you can pipe it into less to allow you to scroll and search like this: sudo netstat -punta | less - hit "q" to exit or "/" to initiate a text search.

You will also need to provide the "nginx.conf" file referenced in the nginx section of the file. Do that by using your editor, e.g. nano nginx.conf, and enter this content (you shouldn't need to alter anything):

user www-data;

worker_processes 1;

error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid;

events { worker_connections 1024; }

http { upstream backend { server app-server:9000; }

include /etc/nginx/mime.types; default_type application/octet-stream;

log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"';

access_log /var/log/nginx/access.log main;

sendfile on; #tcp_nopush on;

keepalive_timeout 65;

map $http_host $this_host { "" $host; default $http_host; }

map $http_x_forwarded_proto $the_scheme { default $http_x_forwarded_proto; "" $scheme; }

map $http_x_forwarded_host $the_host { default $http_x_forwarded_host; "" $this_host; }

server { listen 80;

# Add headers to serve security related headers add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;"; add_header X-Content-Type-Options nosniff; add_header X-XSS-Protection "1; mode=block"; add_header X-Robots-Tag none; add_header X-Download-Options noopen; add_header X-Permitted-Cross-Domain-Policies none;

root /var/www/html; client_max_body_size 10G; # 0=unlimited - set max upload size fastcgi_buffers 64 4K;

gzip off;

index index.php; error_page 403 /core/templates/403.php; error_page 404 /core/templates/404.php;

rewrite ^/.well-known/carddav /remote.php/dav/ permanent; rewrite ^/.well-known/caldav /remote.php/dav/ permanent;

location = /robots.txt { allow all; log_not_found off; access_log off; }

location ~ ^/(build|tests|config|lib|3rdparty|templates|data)/ { deny all; }

location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) { deny all; }

location / { rewrite ^/remote/(.*) /remote.php last; rewrite ^(/core/doc/[^\/]+/)$ $1/index.html; try_files $uri $uri/ =404; }

location ~* ^/ds-vpath/ { rewrite /ds-vpath/(.*) /$1 break; proxy_pass http://onlyoffice-document-server; proxy_redirect off;

client_max_body_size 100m;

proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade";

proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Host $the_host/ds-vpath; proxy_set_header X-Forwarded-Proto $the_scheme; #proxy_set_header X-Forwarded-Proto 'https'; }

location ~ \.php(?:$|/) { fastcgi_split_path_info ^(.+\.php)(/.+)$; include fastcgi_params; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; fastcgi_param PATH_INFO $fastcgi_path_info; fastcgi_param HTTPS off; fastcgi_param modHeadersAvailable true; #Avoid sending the security headers twice fastcgi_pass backend; fastcgi_intercept_errors on; }

# Adding the cache control header for js and css files # Make sure it is BELOW the location ~ \.php(?:$|/) { block location ~* \.(?:css|js)$ { add_header Cache-Control "public, max-age=7200"; # Add headers to serve security related headers add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;"; add_header X-Content-Type-Options nosniff; add_header X-Frame-Options "SAMEORIGIN"; add_header X-XSS-Protection "1; mode=block"; add_header X-Robots-Tag none; add_header X-Download-Options noopen; add_header X-Permitted-Cross-Domain-Policies none; # Optional: Don't log access to assets access_log off; }

# Optional: Don't log access to other assets location ~* \.(?:jpg|jpeg|gif|bmp|ico|png|swf)$ { access_log off; }
} }

That should be all the configuration you need to make the Docker containers go.

Configuring Nginx to proxy NextCloud and OnlyOffice

The next step is configuring the local nginx proxy servers for NextCloud and OnlyOffice using the nginx instance you installed earlier. That's what responds to the domain name you choose for this service. In our case, the name is https://docs.oeru.org - you can have a look at it to see what you should be seeing when you first start things up! We use Let's Encrypt to provide secure hosting - here're my Let's Encrypt instructions on setting it up. The key thing to realise is that your "certificates" need to exist for Nginx to restart with the new configurations below - use the "commenting out the intervening lines" trick mentioned in my instructions to bootstrap the creation of your secure certificates!

To configure the proxy, you need to create this configuration file in your /etc/nginx/sites-available/ directory.

NextCloud Proxy Configuration

Create a file with a meaningful name for your NextCloud Proxy, perhaps based on the domain name you've chosen (our file for docs.oeru.org is called "docs") using the same editing approach as the last few (although this is in a different directory) for example sudo nano /etc/nginx/sites-available/nextcloud with the following contents, replacing [nextcloud.domain] with your selected domain name, but leave off the [ ] (those are just there to make sure nginx errors if you've missed replacing any) - and the port number 8082 if you've opted to change to a different one!:

server { listen 80; listen [::]:80; server_name [nextcloud.domain];

include includes/letsencrypt.conf;

# enforce https location / { return 302 https://$server_name$request_uri; } }

server { listen 443 ssl; listen [::]:443 ssl; #listen 127.0.0.1:443 ssl;

server_name [nextcloud.domain];

## Access and error logs. access_log /var/log/nginx/[nextcloud.domain]_access.log; error_log /var/log/nginx/[nextcloud.domain]_error.log;

ssl_certificate /etc/letsencrypt/live/[nextcloud.domain]/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/[nextcloud.domain]/privkey.pem;

ssl on; # from http://axiacore.com/blog/enable-perfect-forward-secrecy-nginx/ ssl_session_cache shared:SSL:10m; ssl_session_timeout 10m; # limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s; # forward secrecy settings ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_prefer_server_ciphers on; ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4"; ssl_dhparam /etc/ssl/certs/dhparam.pem;

#location = /robots.txt { # allow all; # log_not_found off; # access_log off; #}

# The following 2 rules are only needed for the user_webfinger app. # Uncomment it if you're planning to use this app. rewrite ^/.well-known/host-meta /public.php?service=host-meta last; rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json last;

# The following rule is only needed for the Social app. # Uncomment it if you're planning to use this app. rewrite ^/.well-known/webfinger /public.php?service=webfinger last;

location ^~ / { proxy_pass http://127.0.0.1:8082; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "Upgrade"; proxy_set_header Host $http_host; proxy_read_timeout 36000s; proxy_buffering off; proxy_max_temp_file_size 15000m; } client_max_body_size 1G; fastcgi_buffers 64 4K; add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;"; # Remove X-Powered-By, which is an information leak fastcgi_hide_header X-Powered-By; }

Note: you'll need to create the file cited in the proxy configration: /etc/ssl/certs/dhparam.pem

You can do this as follows (install the necessary software, backup any possible existing version as a matter of prudence, and create a new one):

sudo apt update && sudo apt install openssl sudo [ -f "/etc/ssl/certs/dhparam.pem" ] && sudo mv /etc/ssl/certs/dhparam.pem /etc/ssl/certs/dhparam.pem.bak
sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048

Once those are created, you have to make sure that they're "enabled" (replacing with your file names, of course):

sudo cd /etc/nginx/sites-enabled sudo ln -sf ../sites-available/nextcloud .

To confirm that there aren't any typos or issues that might make nginx unhappy, run

sudo nginx -t

If all's well, get nginx to reread its configuration with the new files (if not, it might be because you missed replacing one of the [tokens]):

sudo service nginx reload

Firing up your NextCloud!

Phew - congratulations on getting here! We've reached the moment of truth where we need to see if this whole thing will work!

We need to make sure we're back in the NextCloud Docker directory we set up:

cd /home/docker/nextcloud

and then we need to try running our docker-compose script to "pull" in the pre-built Docker containers we've specified in our docker-compose.yml file:

docker-compose pull

All going well, after a few minutes (longer or shorter depending on the speed of your server's connection) you should have download the Nginx, Redis, and NextCloud Docker images. Then you can run:

docker-compose up -d && docker-compose logs -f

This will attempt to start up the containers (bringing them "up" in daemon mode, thus the -d) and then show you a stream of log messages from the containers, preceded by the container name. This should help you debug any problems that occur during the process (ideally, none).

Once you see log messages streaming past, and no obvious "container exited" or other error messages (which will usually contain the word "error" a lot), you should be able to point your browser at your selected domain name and bring it up in your browser! Just point your web browser at https://nextcloud.domain (replacing with your domain, of course - the https assumes you've got your Let's Encrypt certificate set up - I recommend doing that first).

Configuring database access

On doing so, if all is well, you should be directed through the database set up process for your NextCloud instance. Your details should be:

database IP: 172.17.0.1 - this is the default IP of the Docker host server.
database name: nextcloud
database user: nextcloud
database password: (the one you came up with above)

Configuring the Admin user

Once that's set and working, NextCloud will install all the relevant database tables and initial data. You'll be asked to set up an admin user account, which can be "admin" (you could make it something different to help stymie nefarious probes that assume you've got a user called "admin" - but don't forget what you've called it!) and some strong password you create (you can use the pwgen utility you used earlier) - I'd recommend recording it somewhere. I would not recommend making your own account, in your name, the main admin account. Instead, I recommend creating a second account, with administrator privileges, for yourself, but leave the admin account purely for administrative activities.

Configuring Outgoing Email

To allow your NextCloud instance to send outgoing email, so that your site can alert you to security updates that need to be applied, or so that users can request a replacement password if they've forgot theirs, you'll need an authenticating SMTP account somewhere. Most of you already have one. You'll probably want to set up a dedicated email address for this server somewhere, perhaps something like "nextcloud@your.domain" or similar, with a username (often just the email address) and a password. You'll need the following details:

SMTP server : an IP address or a domain name
SMTP username: a username or an email address
SMTP password: a strong password already configured for the username on that server
SMTP login security: whether login is via TLS, SSL, or unsecure (!!), and
SMTP login method: plain, encrypted, "login" or some other value.

You should be able to test your email settings to make sure the details you've entered are valid. If you need to adjust these settings later, you can go to the admin menu (top right of the web browser interface) and go to Admin->Additional Settings - should have a path of https://nextcloud.domain/settings/admin/additional

Setting up OnlyOffice

The OnlyOffice server should already be running - if you point your browser at https://nextcloud.domain/ds-vpath/ you should see something like the "Document Server is running" (with a big green "tick") page included in the images accompanying this article.

Configuring OnlyOffice Integration with NextCloud

Once you're logged in to NextCloud as your own user, looking at your own default folders, you can start having a look around. You should have an "admin" menu (assuming you've created your user with Administrator privileges) at the top right of the web interface. If you go to Apps, you can install the new "Hub bundle" available under the "App bundles" option (see attached image). If you don't want the whole bundle you can just use the search box to search for "OnlyOffice" or go to the "Office & text" App category and enable the OnlyOffice "official" app, at which point it will automatically download the latest version of the connector app and install it (it should appear in your /home/data/nextcloud/apps directory)

Once you've done that, go to your top right menu again, selecting Admin, and you should see "OnlyOffice" as an option in the left column (which starts with "Basic settings"). Selecting that, you'll need to enter the following:

"Document Editing Service address": /ds-vpath/
"Secret key": (leave blank)
Under "Advanced server settings"
- "Document Editing Service address for internal requests from the server": http://onlyoffice-document-server/
- "Server address for internal requests from the Document Editing Service": http://nginx-server/

When you're done, click "Save".

You can also select formats you'd like OnlyOffice to open and edit files of those types are clicked or created. I've selected the following: doc, docx, odp, ods, odt, ppt, pptx, xls, xlsx, and in the second section: csv and txt.

You can also make other editor customisations as you desire. The only Editor customisation setting I haven't selected is "Display Chat menu button" because NextCloud Hub provides an integrated Chat service, making this one within OnlyOffice an unnecessary distraction.

Once finished configuring, you should have the ability to go back to the home of your NextCloud install, which should show you your top-level folders. If you click the "+" next to the home icon (top left of the folder pane) you should now have the option to create (in addition to "Upload file", "New folder", "New text file") a "New Document", "New Spreadsheet", and "New Presentation". Clicking those should give you the OnlyOffice interface for the designated content type.

Similarly, you can use the "Upload file" to upload a document in a format that is supported by OnlyOffice. Once uploaded, clicking on the filename should open it for editing in the appropriate OnlyOffice interface.

It is saved as it is changed, so you shouldn't need to save it explicitly.

Keeping the whole thing up-to-date

So, as you're no doubt aware, both NextCloud and OnlyOffice are always being improved and updated. I certainly encourage you to keep your installations up-to-date.

While you'll periodically be alerted that NextCloud apps have available updates (these can be upgraded through the browser interface) updates to the NextCloud and OnlyOffice systems themselves need to be undertaken by upgrading their containers. Luckily it's easy to do (although I strongly urge you to ensure you have a very recent backup of both database and uploaded files - they're the files in /home/data/nextcloud/data and /home/data/onlyoffice/ (note, backups of OnlyOffice are complicated somewhat by the fact that you can't reliably back up running PostgreSQL instance simply by backing up its files - see a solution below):

Updating the container should be as easy as either doing another

docker-compose pull

and then shutting down Docker container via a

docker-compose up -d

which will remove any old containers (this won't remove any data you want to save if you followed the directions above! But remember to do it in the right directory!) and start up the new versions you've just pulled.

Use docker-compose logs -f to watch the logs - you'll likely see useful debugging information in the unlikely event that something goes wrong in the upgrade process.

Backing up NextCloud

To back up your instance on your server, you need two things: a file system backup of your /home/data/nextcloud directory, and database dumps of your database.

There're lots of ways to back up your files (I've recently updated to using a system called Restic to make off-server incremental encrypted backups - I plan to document this in a future howto! - although there're other documented approaches - leave a comment below if you'd like to learn more about my approach!).

Backing up your MariaDB databases is as easy installing automysqlbackups:

sudo apt install automysqlbackups

You'll find daily versioned dumps of your MariaDB database(s) in /var/lib/automysqlbackups on your VM host's filesystem. To run an ad hoc backup (which will replace the previous backup from that day, if there is one) just run

sudo automysqlbackups

Backup OnlyOffice

Along with backing up the files in your /home/data/onlyoffice directory, you'll also want a proper "dump" of your PostgreSQL backup (you can write simple bash scripts to do this regularly, automatically), particularly prior to doing an upgrade (to allow for recovery if something goes badly wrong, which is always possible). You can achieve this by going to

cd /home/docker/onlyoffice

and running this

DATE=`date +%Y%m%d` && FILE=/home/data/onlyoffice/backup/fullbackup-${DATE}.sql && docker-compose exec onlyoffice sudo -u postgres pg_dumpall > ${FILE} && gzip ${FILE}

which will assign the current date to DATE, the relevant filename to FILE, and then put the backup SQL into a dated file called $FILE and compress the result with gzip :)

At some point, I'll modify my normal versioned dated database backup scripts to cater for this solution and make the result available on https://git.oeru.org - in the meantime, you can use the above before you do a backup and manually delete older backups if they start taking up too much space (or, better still, write your own clever script that does it automatically and let me know about it!).