Giving developers production access without revealing secrets

It’s a classic trade off in the devops world:

On the one hand you want to give developers access to production systems so that they can see how their services are running and help debug problems that only occur in production.

On the other hand, these are production services. As such they necessarily have access to production “secrets” like SSL certificates, database passwords, or other data that you don’t want to share with anyone unless you absolutely have to.

So how do you balance these two conflicting priorities?

This is complicated even further by the fact that to have useful access from a developer perspective it needs to be as certain role accounts that their services run under. There’s no point giving them access to an unprivileged account that can’t see what it needs to be able to diagnose issues. But the role account itself necessarily has access to the secrets you want to restrict access to, otherwise it couldn’t provide the services in question.

Using an Apparmor profile to confine access

Here at Canonical we decided to solve this problem by giving developers access to connect to the servers in question via an unprivileged account (their own user account), and then giving those users sudo to the role account in question, but confined by an apparmor profile.

This apparmor profile would have all the access the role account does, but with read-only permissions, and with files known to contain secrets not even readable.

So how do we achieve this?

We’re heavy users of Juju so the natural choice for us was to write a subordinate charm that we could deploy to any servers and services we needed. Juju is a service orchestration tool that focuses on the relationships between services. A Charm encapsulates how a service is defined. Subordinates are services with knowledge of, and access to, their principal services.

Let’s take a look at how you’d deploy our confined-role-account subordinate charm in practice. In our example, we’re going to deploy MediaWiki and MySQL, manually add an unprivileged user account on the MediaWiki server, and then configure the confined-role-account subordinate charm so that our unprivileged user can sudo to the www-data user, but with restricted access.

Example deployment with MediaWiki and MySQL

First of all, configure your juju environment and fire up the bootstrap instance.

Then, deploy your charms into the environment:


juju deploy cs:charms/trusty/mediawiki
juju deploy cs:charms/trusty/mysql
juju add-relation mysql mediawiki:db


Now let’s manually create our unprivileged user. In a production environment we’d likely have a different means of doing this, such as via ldap.


juju ssh mediawiki/0
$ sudo adduser service-user


Now we can confirm that we can login as the account in question and see what sudo permissions it has by default:


juju ssh mediawiki/0
$ sudo su - service-user
$ sudo -l
Sorry, user service-user may not run sudo on juju-env-machine-1.


Now we can deploy the confined-role-account subordinate charm, set our config options for it, and add a relation to the mediawiki charm:


juju deploy cs:~canonical-sysadmins/trusty/confined-role-account
juju set confined-role-account group=service-user role-account=www-data extra-apparmor-rules=” audit deny /etc/mediawiki/AdminSettings.php r,”
juju add-relation mediawiki confined-role-account


What is this doing? It’s saying that anyone in the service-user group can sudo to the www-data role account with the default read-only permissions, but also with /etc/mediawiki/AdminSettings.php not visible to them. This file contains the database password, and in our scenario we want to allow developers access to the mediawiki server as the www-data role account without giving them access to the production database password.

In future we plan to modify the confined-role-account charm so that extra apparmor rules can be passed over the relation with the primary charm. This will allow charms to be opinionated about what secrets they should restrict access to, so no extra configuration needs to be set.

And now let’s confirm how things look:


juju ssh mediawiki/0
$ sudo su - service-user
$ sudo -l
Matching Defaults entries for service-user on juju-canonistack-machine-1:
env_reset, mail_badpass, secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin


User service-user may run the following commands on juju-canonistack-machine-1:


(www-data) NOPASSWD: /usr/local/sbin/apparmor-wrapper-service-user-www-data \"\"
$ sudo -u www-data /usr/local/sbin/apparmor-wrapper-service-user-www-data
$ echo “this is a test” > ~/this-is-a-test.txt
bash: /var/www/this-is-a-test.txt: Permission denied
$ head /etc/mediawiki/LocalSettings.php
$path = array( $IP, "$IP/includes", "$IP/languages" );
set_include_path( implode( PATH_SEPARATOR, $path ) . PATH_SEPARATOR . get_include_path() );

require_once( "$IP/includes/DefaultSettings.php" );

# If PHP's memory limit is very low, some operations may fail.
# ini_set( 'memory_limit', '20M' );

if ( $wgCommandLineMode ) {
$ head /etc/mediawiki/AdminSettings.php
head: cannot open '/etc/mediawiki/AdminSettings.php' for reading: Permission denied

We can also see our command history in /var/log/apparmor-wrapper/www-data-history.log:


echo "this is a test" > ~/this-is-a-test.txt
head /etc/mediawiki/LocalSettings.php
head /etc/mediawiki/AdminSettings.php


And there you have it. How to give developers production access without revealing secrets. So how do you implement this for yourself? You can deploy the charm with the following command:


juju deploy cs:~canonical-sysadmins/trusty/confined-role-account


Then just set any extra apparmor rules you need, and relate to any primary charm you want based on the example above.