Manual web-based OpenStack configuration

Connection quirks

Connecting to safespring is not particularly trivial without all of the required information.

Of course, to login you'll need access credentials. Those are given by another member of TPA only on an as-needed basis.

In order to connect, go to the dashboard for the swedish cluster and then choose the Safespring login option. The Domain field should have the value users and for the rest, enter your own credentials.

Create a security group

The default security group doesn't seem to properly allow inbound traffic, so a new security group called anything was created. A few notes:

  • there's no menu to select IPv6 vs IPv4, just type ::/0 for IPv6 and 0.0.0.0 for IPv4
  • changes take effect immediately
  • an instance can be moved between security groups on the fly and even have multiple groups
  • we just keep a simple "all open" configuration and rely on host-level firewalls to do their jobs
  • safespring network configuration hints

Create an instance

  1. go to Compute -> Instances -> Launch instance
  2. pick the FQDN as the "instance name"
  3. click "Next" to get to the "Sources" type
  4. click "Yes" below "Create new volume", set it to the desired size (e.g. 30G)
  5. choose "Yes" to "Delete volume on instance delete"
  6. below "available", type "debian" to look for a Debian image, there should be a debian-11 image, click on the arrow to move it up to "Allocated"
  7. click "Next" to get to the "Flavour" tab
  8. pick an instance "flavour", for instance b2.c1r4 has 2 cores, 4GB of RAM and no built-in disk (which is why we create a volume above, we could also have used existing flavor size if we needed a larger disk
  9. click "Next" to go to the "Networks" tab
  10. click on the arrow on the sunet.se-public line
  11. go to the "Security groups" tab and pick "anything"
  12. add the anarcat key pair, create a new one for you if missing, but add it too
  13. click "launch instance"

Then the job will be dispatched and the instance created, which should be very fast (in the order of a few seconds, certainly less than a minute). Console logs show up in the "Log" tab after you click on the instance, and should contain the SSH host keys in their output.

From there follow the normal new-machine procedure. Once that is done, you also need to do a little bit of cleanup:

  1. remove the debian user:

    deluser debian

  2. reconfigure the interfaces(5) file to add the proper IPv6 address, it should look something like this:

    auto lo
iface lo inet loopback

auto ens3
iface ens3 inet static
    address 89.45.235.46/28
    gateway 89.45.235.33

iface ens3 inet6 static
    address 2001:6b0:5a:4021::37d/64
    accept_ra 1

  3. purge the cloud-init package:

    apt purge cloud-init

Resizing an instance

Normally, resizing an instance can be done through the normal OpenStack menus and APIs, but we don't actually have the permissions to do so ourselves in their web interface.

File a ticket with their support (@safespring.com) and ask them which "flavor" to switch to. That should be visible in the OpenStack UI, however, follow this path:

  1. go in the VM listing
  2. click on the VM dropdown menu (we want to resize collector-02)
  3. pick "resize instance"

You should then see a menu of "flavors" to choose from.

OpenStack API

We were granted access to Linaro's OpenStack cluster. The following instructions were originally written to create virtual machines in that cluster, but were adapted to also work on OSUOSL's clusters.

We provide command line instructions below because they are easier to document, but an equivalent configuration can be performed through the web interface as well.

Preparation

You first need a adminrc.sh file with the right configuration and credentials.

In general, the credentials it can be downloaded in API access page of an OpenStack web UI (/project/api_access/) by clicking on the Download OpenStack RC File. We call the downloaded file the adminrc.sh file, but it can be named anything, as long as it's source in your shell for the following commands.

Here are platform-specific instructions:

  • the credentials for Linaro were extracted from ticket 453 and the password prompted is the login.linaro.org SSO password stored in tor-passwords.git (note: the domain is linaro).
  • the OSUOSL password is in tor-passwords.git

Then you need to install some OpenStack clients:

apt install openstack-clients

Yes, that installs 74 packages, no kidding.

Add your SSH key to the server:

openstack keypair create --public-key=~/.ssh/id_rsa.pub anarcat

If your key is stored in GnuPG:

openstack keypair create --public-key=<(gpg --export-ssh-key anarcat@debian.org) anarcat

You will probably need to edit the default security group (or create a new one) to allow ingress traffic as well. For example, this will create an "allow all" ingress rule on IPv4:

openstack security group rule create default

During this entire process, it's useful to take a look at the effect of the various steps through the web interface.

Launching an instance

This procedure will create a new VM in the OpenStack cluster. Make sure you first source the adminrc.sh script you found in the previous step.

  1. list the known flavors and images:

    openstack flavor list
openstack image list

    let's say we deploy a uk.nano flavor with debian-10-openstack-arm64 image.

  2. create the server (known as an "instance" in the GUI):

    openstack server create --key-name=anarcat --security-group=default --image=debian-10-openstack-arm64 --flavor=uk.nano build-arm-10.torproject.org

    In the above:

    • --keypair=anarcat refers to the keypair created in the preparation
    • --security-group is taken from openstack security group list output, which typically has a default one. in previous installs, we setup a security group through the web interface possibly to allow the floating IP routing (unclear)
    • --image and --flavor were picked from the previous step

  3. you can see the status of the process with:

    openstack server list

  4. inspect the server console log to fetch the SSH public keys:

    openstack console log show build-arm-10.torproject.org | sed '0,/-----BEGIN SSH HOST KEY KEYS-----/d;/-----END SSH HOST KEY KEYS-----/,$d;s/^/213.146.141.28 /' >> ~/.ssh/known_hosts

    Note: the above doesn't actually work. In my tests (on OSUOSL) the keys do show up in the web console, but not in the above command. Use this command to load the web console:

    openstack console url show build-arm-10.torproject.org

  5. the VM should be up by now, and you should be able to SSH in:

    openstack server ssh -l debian build-arm-10.torproject.org

    You unfortunately have to blindly TOFU (Trust On First Use) the SSH server's public key because it's not visible in the API or web interface. The debian user has sudo access.

Note that the above might fail on OSUOSL's OpenStack cluster sometimes. The symptom is that the host would be named "unassigned-hostname" (visible in the console) and SSH login would be impossible. Sometimes, the console would also display this message:

no authorized SSH keys fingerprints found for user debian

This is cloud-init failing to fetch the configuration from the metadata service. This is an upstream issue with OSUOSL, file an issue with them (aarch64-hosting-request@osuosl.org), documenting the problem. Our previous ticket for this was `[support.osuosl.org

31901]` and was resolved upstream by restarting the metadata service.

Floating IP configuration

The above may fail in some OpenStack clusters that allocate RFC1918 private IP addresses to new instances. In those case, you need to allocate a floating IP and route it to the instance.

  1. create a floating IP

    openstack floating ip create ext-net

    The IP address will be shown in the output:

    | floating_ip_address | 213.146.141.28 |

    The network name (ext-net above) can be found in the network list:

    openstack network list

  2. link the router in the private network if not already done:

    openstack router add subnet router-tor 7452852a-8b5c-43f6-97f1-72b1248b2638

    The subnet UUID comes from the Subnet column in the output of openstack network list for the "internal network" (the one that is not ext-net.

  3. map the floating IP address to the server:

    openstack server add floating ip build-arm-10.torproject.org 213.146.141.28

Renumbering a server

To renumber a server in Openstack, you need to first create a port, associate it with the server, remove the old port, and renumber IP elsewhere.

Those steps were followed for ns5:

  1. Make sure you have access to the server through the web console first.

  2. add the new port:

    openstack port create --network sunet.se-public ns5.torproject.org

  3. assign it in the right security group:

    openstack port set --security-group anything ns5.torproject.org

  4. attach the port to the instance:

    openstack server add port ns5.torproject.org ns5.torproject.org

  5. remove the old port from the instance:

    openstack server remove port ns5.torproject.org dcae4137-03cd-47ae-9b58-de49fb8eecea

  6. in the console, change the IP in /etc/network/interfaces...

  7. up the new interface:

    ifup -a

  8. renumber the instance, see the ganeti.renumber-instance fabric job for tips, typically it involves grepping around in all git repositories and changing LDAP

References