Automated ISO/IMG builds and tests on Jenkins

:warning: This service will change during [[policy/tpa-rfc-73-tails-infra-merge-roadmap]] and this page should be updated when that happens.

[[TOC]]

Access

jenkins.lizard

• SSH onion service: fw4ntog3yqvtsjielgqhzqiync5p4gjz4t7hg2wp6glym23jwc6vedqd.onion
• SSH fingerprints:
• SHA256:EtL9m3hZGBPvu/iqrtwa4P/J86nE1at9LwymH66d1JI (ECDSA)
• SHA256:HEvr0mTfY4TU781SOb0xAqGa52lHPl00tI0mxH5okyE (ED25519)
• SHA256:sgH1SYzajDrChpu26h24W1C8l+IKYV2PsAxzSGxemGk (RSA)

isoworker2.dragon

• SSH onion service: xdgjizxtbx2nnpxvlv63wedp7ruiycxxd2onivt4u4xyfuhwgz33ikyd.onion
• SSH fingerprints:
• SHA256:/YUC5h2NM9mMIv8CDgvQff4F1lCcrJEH3eKzSFOMwDA (ECDSA)
• SHA256:dqHiHIPpgIkraIW8FjNsRxwH8is++/UOA8d8rGcwJd0 (ED25519)
• SHA256:FOOQTcVtParu2Tr9LT6i9pkXAOPZMLjO/HMmD7G3cQw (RSA)

isoworker3.dragon

• SSH onion service: 3iyrk76brjx3syp2d7etwgsnd7geeikoaowhrfdwn2gk3zax4xavigqd.onion
• SSH fingerprints:
• SHA256:IStnb3Nmi8Bg8KVjCFFdt1MHopbddEZmo5jxeEuYZf8 (ECDSA)
• SHA256:wZ3WQOc75f0WuCnoLtXREKuNLheLkAKRnQM2k5xd+X4 (ED25519)
• SHA256:yJuhwnDwq3pzyxJVQP1U3eXlSUb4xceORopDQhs3vMU (RSA)

isoworker4.dragon

• SSH onion service: 5wpwcpsoeunziylis45ax4zvr7dnwtini6y2id4ixktmlfbjdz4izkyd.onion
• SSH fingerprints:
• SHA256:IQ2cd5t7D4PigIlbDb50B33OvoWwavKOEmGbnz7gQZ0 (ECDSA)
• SHA256:bxqMdon5kYpu1/vsw8kYpSIXdsYh6rnDzPmP3j25+W4 (ED25519)
• SHA256:UYwTZVSqYOU1dpruXfZTs/AO7I7jYPFROd20Z5PeXWc (RSA)

isoworker5.dragon

• SSH onion service: dimld3wfazcimopetopeikrngqc6gzgxn5ozladbr5bwsoqtfj7fzaqd.onion
• SSH fingerprints:
• SHA256:7tF5FYunYVoFWwzDcShOPKrYqzSbKo56BWQjR++xXrw (ECDSA)
• SHA256:mT5q/FLyvm24FmRKCGafwaoEaJORYCjZu/3N0Q10X+o (ED25519)
• SHA256:7CPhM2zZZhTerlCyYLyDEnTltPv9nK7rAmVCRbg64Qg (RSA)

isoworker6.iguana

• SSH onion service: katl2wa6urkwtpkhr4y6z5vit2c3in6hhmvz4lsx3qw22p5px7ae4syd.onion
• SSH fingerprints:
• SHA256:KTzB+DufG+NISjYL35BjY4cF3ArPMl2iIm/+9pBO0LE (ECDSA)
• SHA256:t0OboSv/JFmViKKmv8oiFbpURMZdilNK3/LQ99pAQaM (ED25519)
• SHA256:UHdM8EZ8ZTAxbutXfZYQQLNrxItpmNAKEChreC/bl+o (RSA)

isoworker7.iguana

• SSH onion service: ooqzyfecxzbfram766hkcaf6u4pe4weaonbstz355phsiej4flus7myd.onion
• SSH fingerprints:
• SHA256:7HmibtchW6iu9+LS7f5bumONrzMIj1toSaYaPRq3FwU root@isoworker7 (ECDSA)
• SHA256:VSvGkrpw49ywmHrHtEOgHnpFVkUvlfBoxFswY3JeMpk root@isoworker7 (ED25519)
• SHA256:k+TeoXoEeF3yIFLHorHOKp9MlJWAjkojS3spbToW5/U root@isoworker7 (RSA)

isoworker8.iguana

• SSH onion service: j6slikp4fck5drnkkshjhtwvxbywuf5bruivteojt3b52conum6dskqd.onion
• SSH fingerprints:
• SHA256:cSLhbY3CSi6h5kQyseuAR64d0EPn0JE3o6rwfIXJqgQ root@isoworker8 (ECDSA)
• SHA256:iZT9WstFjoX93yphLNS062Vll5KjIQF6Y2FQbc1/prw root@isoworker8 (ED25519)
• SHA256:dzaWqWYO/4HERtFx2xBhv9S1Jnzv1GjGfHegusEK4X0 root@isoworker8 (RSA)

Configuration

Orchestrator

• jenkins class
• tails::jenkins::master class
• a few Jenkins plugins installed with jenkins::plugin
• YAML jobs configuration lives in a dedicated Git repository; Jenkins Job Builder uses it to configure Jenkins
• Manual configuration (not handled by Puppet):
  • In the Jenkins web interface:
  • Security → Agents → TCP port for inbound agents → Fixed: 42585
  • System → # of executors: 8 (actually, set to the same number of configured agents)
  • System → Git plugin → Global Config user.name Value: jenkins
  • System → Git plugin → Global Config user.email Value: sysadmins@tails.net
  • System → Priority Sorter → Only Admins can edit job priorities: checked
  • Job Priorities → Add 2 job groups:
    • Description: Top priority
    • Jobs to include: Jobs marked for inclusion
    • Job Group Name: 1
    • Priority: 1
    • Description: Test suite
    • Jobs to include: Jobs marked for inclusion
    • Job Name: 2
    • Priority: 2
  • Create one node for each agent, in Nodes → New node:
    • Node name: use the hostname of the agent (eg. "isoworker6.iguana")
    • Number of executors: 1
    • Remote root directory: /var/lib/jenkins
    • Usage: Use this node as much as possible
    • Launch method: Launch agent by connecting it to the controller
    • Disable WorkDir: checked
    • Internal data directory: remoting
    • Availability: Keep this agent online as much as possible
    • Preference of Node: choose a preference depending on the node specs
  • In the Jenkins VM:
  • For backups: Make sure there exists an SSH key for root and it's public part is configured in profile::tails::backupserver::backupagents for stone.tails.net (or the current backup server).
  • Document the onion server address and SSH fingerprints for the VM.
  • The configuration for the build_IUKs job is only stored in /var/lib/jenkins and nowhere else.
  • Create 4 different "Views":
  • RM:
    • Use a regular expression to include jobs into the view
    • Regular expression: ^(build_IUKs|(reproducibly_)?(test|build)_Tails_ISO_(devel|stable|testing|feature-trixie|experimental|feature-tor-nightly-master)(-force-all-tests)?)
  • Tails Build:
    • Use a regular expression to include jobs into the view
    • Regular expression: build_Tails_ISO_.*
  • Tails Build Reproducibility:
    • Use a regular expression to include jobs into the view
    • Regular expression: reproducibly_build_.*
  • Tails Test Suite:
    • Use a regular expression to include jobs into the view
    • Regular expression: test_Tails_ISO_.*

Agents

• tails::profile::jenkins::isoworker, tails::iso_builder, tails::jenkins::slave, tails::jenkins::slave::iso_builder, tails::jenkins::slave::iso_tester, and tails::tester classes

Web server

• tails::jenkins::reverse_proxy class

Upgrades

Upgrade policy

Here are some guidelines to triage security vulnerabilities in Jenkins and the plugins we have installed:

1. Protecting our infra from folks who have access to Jenkins

→ Upgrading quarterly is sufficient.

1. Protecting our infra from attacks against folks who have access to Jenkins

For example, XSS that could lead a legitimate user to perform unintended actions with Jenkins credentials (i.e. root in practice).

→ We should stay on top of security advisories and react more quickly than "in less than 3 months".

1. Protecting our infra from other 3rd-parties that affect Jenkins' security

For example, say some Jenkins plugin, that connects to remote services, has a TLS certificate checking bug. This could potentially allow a MitM to run arbitrary code with Jenkins orchestrator or workers permissions, i.e. root.

→ We should stay on top of security advisories and react more quickly than "in less than 3 months".

Upgrade procedure

• Preparation:
• [ ] Go through the changelog, paying attention to changes on how agents connect to controller, config changes that may need update, important changes in plugins, etc.
• Deployment:
• [ ] Take note of currently running builds before starting the upgrades.
• [ ] Deploy Jenkins upgrade to latest version available using Puppet.
• [ ] Generate a list of up-to-date plugins by running [this Groovy script](jenkins/1.generate-plugins-list.groovy] in the Jenkins Script Console. Make sure to update the initial list containing actively used plugins if there were changes.
• [ ] Generate updated Puppet code for tails::jenkins::master using [[this Python3 script|contribute/working_together/roles/sysadmins/Jenkins/2.generate-puppet-code.py]] and the output of the above script.
• [ ] Deploy plugin upgrades using the code generated above.
• [ ] Restart all agents.
• [ ] Manually run the Update jobs script (may be needed so XML is valid with current Jenkins): sudo -u jenkins /usr/local/sbin/deploy_jenkins_jobs update
• Wrap up:
• [ ] Go through warnings in Jenkins interface.
• [ ] Manually remove unneeded plugins from /var/lib/jenkins/plugins.
• [ ] Restart builds that were interrupted by Jenkins restart.
• [ ] Update the Jenkins upgrade steps documentation in case there were changes.
• [ ] Schedule next update.

Agent to controller connections

These are the steps a Jenkins agent does when connecting to the controller:

1. Fetch connection info from http://jenkins.dragon:8080 (see the tails::jenkins::slave Puppet class).
2. Receive the connection URL https://jenkins.tails.net ("Jenkins URL", manually configured in Configure System).
3. Resolve jenkins.tails.net to 192.168.122.1 (because of libvirt config).
4. Connect using HTTPS to jenkins.tails.net:443.
5. Learn about port 42585 (fixed "TCP port for inbound agents", manually configured in Configure Global Security).
6. Finally, connect using HTTP to jenkins.tails.net:42585.

Generating jobs

We generate automatically a set of Jenkins jobs for branches that are active in the Tails main Git repository.

The first brick extracts the list of active branches and output the needed information:

• config/chroot_local-includes/usr/lib/python3/dist-packages/tailslib/git.py
• config/chroot_local-includes/usr/lib/python3/dist-packages/tailslib/jenkins.py

This list is parsed by the generate_tails_iso_jobs script run by a cronjob and deployed by our puppet-tails tails::jenkins::iso_jobs_generator manifest.

This script output YAML files compatible with jenkins-job-builder. It creates one project for each active branch, which in turn uses several JJB job templates to create jobs for each branch:

• build_Tails_ISO_*
• reproducibly_build_Tails_ISO_*
• test_Tails_ISO_*

This changes are pushed to our jenkins-jobs git repo by the cronjob, and thanks to their automatic deployment in our tails::jenkins::master and tails::gitolite::hooks::jenkins_jobs manifests in our puppet-tails repo, these new changes are applied to our Jenkins instance.

Passing parameters through jobs

We pass information from build job to follow-up jobs (reproducibility testing, test suite) via two means:

• the Parameterized Trigger plugin, whenever it's sufficient
• the EnvInject plugin, for more complex cases:
• In the build job, a script collects the needed information and writes it to a file that's saved as a build artifact.
• This file is used by the build job itself, to setup the variables it needs (currently only $NOTIFY_TO).
• Follow-up jobs imported this file in the workspace along with the build artifacts, then use an EnvInject pre-build step to load it and set up variables accordingly.

Builds

See [[jenkins/automated-builds-in-jenkins]].

Tests

See [[jenkins/automated-tests-in-jenkins]].