Git

[[TOC]]

TPA uses Git in several places in its infra. Several services are managed via repos hosted in GitLab, but some services are managed by repos stored directly in the target systems, such as Puppet, LDAP, DNS, TLS, and probably others.

Commit signature verification

In order to resist tampering attempts such as GitLab compromise, some key repositories are configured to verify commit signatures before accepting ref updates. For that, TPA uses sequoia-git to authenticate operations against certificates and permissions stored in a centralized OpenPGP policy file. See TPA-RFC-90: Signed commits for the initial proposal.

Terminology

Throughout this section, we use the term "certificate" to refer to OpenPGP Transferable Public Keys (see section 11.1 of RFC 4880).

sequoia-git basics

In order to authenticate changes in a Git repository, sequoia-git uses two pieces of information:

  • an OpenPGP policy file, containing authorized certificates and a list of permissions for each certificate, and
  • a "trust-root", which is the ID of a commit that is considered trusted.

With these, sequoia-git goes through commit by commit checking whether the signature is valid and authorized to perform operations.

By default, sequoia-git uses the openpgp-policy.toml file in the root of the repo being checked, but a path to an external policy file can be passed instead. In TPA, we do the former on the client side and the latter on the server side, as we'll see in the next section.

The TPA setup

In TPA we use one OpenPGP policy file to authenticate changes for all our repositories, namely the [openpgp-policy.toml][] file in the root of the Puppet repository. Using one centralized file allows for updating certificates and permissions in only one place and have it deployed to the relevant places.

For authenticating changes on the server-side:

  • the TPA OpenPGP policy file is deployed to /etc/openpgp-policy/policies/tpa.toml,
  • trust-roots for the Puppet repos (stored in hiera data for the puppetserver role in the Puppet repo) are deployed to /etc/openpgp-policy/gitconfig/${REPO}.conf, and
  • per-repo Git hooks use the above info to authenticate changes.

On the client-side:

  • we use the TPA OpenPGP policy file in the root of the Puppet repo,
  • trust-roots are stored in the [.mrconfig][] file in tpo/tpa/repos> and set as Git configs in the relevant repos by mr update (see [doc on repos.git][]), and
  • per-repo Git hooks use the above info to authenticate changes.

Note: When the trust-root for a repository changes, it needs to be updated in the hiera data for the puppetserver role and/or the [.mrconfig][] file, depending on whether it's supposed to be authenticated on server and/or client side.

Authentication in the Puppet Server

The Puppet repositories stored in the Puppet server are configured with hooks to verify authentication of the incoming commits before performing ref updates.

Puppet deploys in the Puppet server:

  • the TPA OpenPGP policy file ([openpgp-policy.toml][]) to /etc/openpgp-policy/policies/tpa.toml,
  • global Git configuration containing per-repo policy file and trust-root configs to /etc/openpgp-policy/gitoconfig/, and
  • Git update-hooks to the Puppet repositories that only allow ref updates if authentication is valid

See the [profile::openpgp_policy][] Puppet profile for the implementation.

With this, ref updates in the Puppet Git repos are only performed if all commits since the trust-root are signed with authorized certificates contained in the installed TPA OpenPGP policy file.

Certificate updates

While a certificate is still valid and has the [sign_commit][] capability, it's allowed to update any certificate contained in the openpgp-policy.toml file.

To update one or more certificates, first make sure you have up-to-date versions in your local store. One way to do that is by using sq to import the certificate from Tor's Web Key Directory:

sq network wkd search <ADDRESS>

Then use sq-git to update the OpenPGP policy file with certificates from your local store:

sq-git policy sync --disable-keyservers

Note that, if you don't use --disable-keyservers, expired subkeys may end up being included by a sync, and you may think that there are updates to the key when there really aren't. So it's better to just do as suggested above.

You can also edit the openpgp-policy.toml file manually and perform the needed changes.

Note that, because we use a centralized OpenPGP policy file, when permissions are removed for a certificate, we may need to update the trust-root, otherwise old commits may fail to be authenticated against the new policy file.

Expired certificates

If a certificate expires before it's been updated in the openpgp-policy.toml file, changes signed by that certificate will not be accepted, and you'll need to (1) ask another sysadmin with a valid certificate to perform the needed changes and (2) wait for or force deployment of the new file in the server.

See the above section for instructions on how to update the OpenPGP policy file.

Manual override

There may be extreme situations in which you need to override the authentication check, for example if your certificate expired and you're the only sysadmin in duty. In these cases, you can manually remove/update the corresponding Git hooks in the server and push the needed changes. If you do this, make sure to:

  • update the trust root both in the hiera data for the puppetserver role and in tpo/tpa/repos>.
  • instruct the other sysadmins to pull tpo/tpa/repos> and run mr update, so their local Git configs for trust-roots is automatically updated. If you don't do that, their local checks will start failing when they pull commits that can't be authenticated.

Other repositories

Even though we initially deployed this mechanism to Pupper repositories only, the current implementation of the OpenPGP policy profile allows for configuration of the same setup for arbitrary repositories, which can be configured via hiera. See the hiera data for the puppetserver role for an example.

Setting trust-roots is mandatory, while policy files are optional. If no policy file is explicitly set, the Git hook will perform the authentication checks against the policy file in the root of the repository itself.