Chapter 5. Generating a new cluster certificate

The initial cluster certificate eventually expires. The default setting creates cluster certificates that expire after five years, but a new cluster certificate must be created long before than. The cluster certificate signs node certificates, which individual nodes in the cluster use to authenticate each other. Node certificates' default duration is one year, and a node's certificate cannot remain valid past the expiration date of a cluster certificate that signed it, so a new cluster certificate has to be created after four years, by default.

There are two ways to check the cluster certificate's expiration. The administrative status command shows the expiration from the cluster certificate's public key:

[root@octopus ~]# stasher
Ready, EOF to exit.
> admin
Connected to, node
Maximum 20 objects, 64 Mb aggregate object size, per transaction.
Maximum 10 concurrent subscriptions.
octopus> status

... long output deleted ...

    Cluster certificate: subject `,OU=object repository (c
reated on 2012-03-17 16:12:05 EDT),O=cluster', issuer `,OU
=object repository (created on 2012-03-17 16:12:05 EDT),O=cluster', RSA key 2432
 bits, signed using RSA-SHA256, activated `2012-03-17 20:12:05 UTC', expires `20
17-03-27 20:12:05 UTC'
, SHA-1 fingerprint `1baed521cd8237d90be58665300db65ae5150

... long output deleted ...

octopus> [EOF, CTRL-D]
[root@octopus ~]# 

stashermg shows the same information from the secure directory that holds the cluster's private keys:

[root@octopus ~]# stashermg --clustkey --list /var/stasher/clusters/
Key ID           Cluster name         Activation           Expiration
---------------- -------------------- -------------------  -------------------
1332015126008193  2012-03-17 16:12:05  2017-03-27 16:12:05
[root@octopus ~]# 

With node certificates, creating a node certificate replaces the existing one. Cluster certificates are a little bit different. A new cluster certificate does not replace and invalidate the existing one. The existing certificate must remain valid until it expires, since there may be existing node certificates which are signed by it:

[root@octopus ~]# stashermg --clustkey --generate /var/stasher/clusters/
Generating new key...
[root@octopus ~]# stashermg --clustkey --list /var/stasher/clusters/
Key ID           Cluster name         Activation           Expiration
---------------- -------------------- -------------------  -------------------
1332015126008193  2012-03-17 16:12:05  2017-03-27 16:12:05
1333767749362115  2012-04-06 23:02:28  2017-04-16 23:02:28
[root@octopus ~]# 

This updates only the cluster keys in the secured cluster key directory. The new cluster certificate has to be distributed to the actual cluster repository. If the same machine where the secure cluster key directory gets stored also runs one of the nodes in the cluster:

[root@octopus ~]# stashermg --clustkey --export /var/stasher/clusters/objrepo.ex /var/stasher/nodes/
Connected to /var/stasher/nodes/, checking existing certificates
Updating cluster certificates
[root@octopus ~]# 

In this example, the cluster key directory is /var/stasher/clusters/, and one of the nodes in the cluster runs from /var/stasher/nodes/objrepo.example. Use a temporary directory if no node runs on the same machine, to export the updated certificate list:

[root@octopus ~]# mkdir /tmp/clustkey
[root@octopus ~]# stashermg --clustkey --export /var/stasher/clusters/ /tmp/clustkey
Warning: server not running, certificates are installed locally.
[root@octopus ~]# ls -al /tmp/clustkey
total 52
drwxr-xr-x   2 root root  4096 Apr  6 23:07 .
drwxrwxrwt. 14 root root 36864 Apr  6 23:07 ..
-rw-r--r--   1 root root  8792 Apr  6 23:07 rootcerts.pem

Take rootcerts.pem and copy it to any cluster node directory on some other machine, taking care to set its ownership and permissions appropriately:

[root@octopus ~]# ls -al /var/stasher/nodes/
total 52
drwxr-xr-x 7 daemon daemon 4096 Apr  6 23:13 .
drwxr-xr-x 3 root   root   4096 Apr  6 22:33 ..
-rw------- 1 daemon daemon 6281 Mar 30 20:34 cert.pem
drwxr-xr-x 3 daemon daemon 4096 Mar 17 16:15 data
-rw-r--r-- 1 daemon daemon    0 Mar 17 16:15 .lock
drwxr-xr-x 2 daemon daemon 4096 Apr  6 22:34 logs
drwx------ 2 daemon daemon 4096 Apr  6 22:34 privsocket
-rw-r--r-- 1 daemon daemon  348 Mar 17 16:13 properties
drwxr-xr-x 2 daemon daemon 4096 Apr  6 22:34 pubsocket
-rw-r--r-- 2 daemon daemon 8792 Apr  6 23:15 rootcerts.pem
drwxr-xr-x 2 daemon daemon 4096 Apr  6 23:06 tmp
[root@octopus ~]# stashermg --certreload /var/stasher/nodes/
Certificate reloaded
[root@octopus ~]# 

In either case, the administrative status command should now list both cluster certificates. Once the old certificate expires, it should be removed from the cluster. See stashermg's man page for more details. The updated cluster certificate list, without the expired certificate, gets pushed into the cluster using the same process as adding a new one.