rt.torproject.org is an installation of Request Tracker used for support. Users (of the Tor software, not of the TPA infrastructure) write emails, support assistants use web interface.

Note that support requests for the infrastructure should not go to RT and instead be directed at our usual support channels.

[[TOC]]

How-to

Creating a queue

On the RT web interface:

  1. authenticate to https://rt.torproject.org/
  2. head to the Queue creation form (Admin -> Queues -> Create)
  3. pick a Queue Name, set the Reply Address to QUEUENAME@rt.torproject.org and leave the Comment Address blank
  4. hit the Create button
  5. grant a group access to the queue, in the Group rights tab (create a group if necessary) - you want to grant the following to the group
    • all "General rights"
    • in "Rights for staff":
    • Delete tickets (DeleteTicket)
    • Forward messages outside of RT (ForwardMessage)
    • Modify ticket owner on owned tickets (ReassignTicket)
    • Modify tickets (ModifyTicket)
    • Own tickets (OwnTicket)
    • Sign up as a ticket or queue AdminCc (WatchAsAdminCc)
    • Take tickets (TakeTicket)
    • View exact outgoing email messages and their recipients (ShowOutgoingEmail)
    • View ticket private (commentary) That is, everything but:
    • Add custom field values only at object creation time (SetInitialCustomField)
    • Modify custom field values (ModifyCustomField)
    • Steal tickets (StealTicket)
  6. if the queue is public (and it most likely is), grant the following to the Everyone, Privileged, and Unprivileged groups:
    • Create tickets (CreateTicket)
    • Reply to tickets (ReplyToTicket)

In Puppet:

  1. add the queue to the profile::rt::queues list in the hiera/roles/rt.yaml file
  2. add an entry in the main mail server virtual file (currently tor-puppet/modules/postfix/files/virtual) like:
    QUEUENAME@torproject.org         QUEUENAME@rt.torproject.org

TODO: the above should be automated. Ideally, QUEUENAME@rt.torproject.org should be an alias that automatically sends the message to the relevant QUEUENAME. That way, RT admins can create Queues without requiring the intervention of a sysadmin.

Using the commandline client

RT has a neat little commandline client that can be used to operate on tickets. To install it, in Debian:

sudo apt install rt4-clients

Then add this to your ~/.rtrc:

server https://rt.torproject.org/

If your local UNIX username is different than your user on RT, you'll also need:

user anarcat

Then just run, say:

rt ls

... which will prompt you for your RT password and list the open tickets! This will, for example, move the tickets 1 and 2 to the Spam queue:

rt edit set queue=Spam 1 2

This will mark all tickets older than 3 weeks as deleted in the roots queue:

rt ls -i -q roots "Status=new and LastUpdated < '3 weeks ago'" | parallel --progress --pipe -N50 -j1  -v --halt 1 rt edit - set status=deleted

See also rt help for more information.

This page describes the role of the help desk coordinator. This role is currently handled by Colin "Phoul" Childs.

Maintenance

For maintenance, the service can be shut down by stopping the mail server:

sudo service postfix stop

Then uncomment lines related to authentication in /etc/apache2/sites-staging/rt.torproject.org, then update Apache by:

sudo apache2-vhost-update rt.torproject.org

Once the maintenance is down, comment the lines again in /etc/apache2/sites-staging/rt.torproject.org and update the config again:

sudo apache2-vhost-update rt.torproject.org

Don't forget to restart the mail server:

sudo service postfix start

Support Tasks

The support help desk coordinator handles the following tasks:

  • Listowner of the support-team-private mailing list.
  • Administrator for the Request Tracker installation at https://rt.torproject.org.
  • Keeping the list of known issues at https://help.torproject.org/ up to date.
  • Sending monthly reports on the tor-reports mailing list.
  • Make the life of support assistants as good as it can be.
  • Be the contact point for other parts of the project regarding help desk matters.
  • Lead discussions about non-technical aspects of help requests to conclusions.
  • Maintain the support-tools Git repository.
  • Keep an eye on the calendar for the 'help' queue.

Create accounts for webchat / stats

  • Login to the VM "moschatum"
  • Navigate to /srv/support.torproject.org/pups
  • Run sudo -u support python manage.py createuser username password
  • Open a Trac ticket for a new account on moschatum's Prosody installation (same username as pups)
  • Send credentials for pups / prosody to support assistant

Manage the private mailing list

Administration of the private mailing list is done through Mailman web interface.

Create the monthly report

To create the monthly report chart, one should use the script rude.torproject.org:/srv/rtstuff/support-tools/monthly-report/monthly_stats.py.

Also, each month data need to be added for the quarterly reports for the business graph and for the time graph.

Data for the business graph is generated by monthly_stats. Data for the response time graph is generated by running rude.torproject.org:/srv/rtstuff/support-tools/response-time/response_time.py.

Read/only access to the RT database

Member of the rtfolks group can have read-only access to the RT database. The password can be found in /srv/rtstuff/db-info.

To connect to the database, one can use:

psql "host=drobovi.torproject.org sslmode=require user=rtreader dbname=rt"

Number of tickets per week

    SELECT COUNT(tickets.id),
           CONCAT_WS(' ', DATE_PART('year', tickets.created),
                          TO_CHAR(date_part('week', tickets.created), '99')) AS d
     FROM tickets
     JOIN queues ON (tickets.queue = queues.id)
    WHERE queues.name LIKE 'help%'
    GROUP BY d
    ORDER BY d;

Extract the most frequently used articles

Replace the dates.

   SELECT COUNT(tickets.id) as usage, articles.name as article
     FROM queues, tickets, links, articles
    WHERE queues.name = 'help'
      AND tickets.queue = queues.id
      AND tickets.lastupdated >= '2014-02-01'
      AND tickets.created < '2014-03-01'
      AND links.type = 'RefersTo'
      AND links.base = CONCAT('fsck.com-rt://torproject.org/ticket/', tickets.id)
      AND articles.id = TO_NUMBER(SUBSTRING(links.target from '[0-9]+$'), '9999999')
    GROUP BY articles.id
    ORDER BY usage DESC;

Graphs of activity for the past month

Using Gnuplot:

set terminal pngcairo enhanced size 600,400
set style fill solid 1.0 border
set border linewidth 1.0
set bmargin at screen 0.28
set tmargin at screen 0.9
set key at screen 0.9,screen 0.95
set xtics rotate
set yrange [0:]
set output "month.png"
plot "<                                                                                                      \
  echo \"SELECT COUNT(tickets.id),                                                                           \
                TO_CHAR(tickets.created, 'YYYY-MM-DD') AS d                                                  \
     FROM tickets                                                                                            \
     JOIN queues ON (tickets.queue = queues.id)                                                              \
    WHERE queues.name LIKE 'help%'                                                                           \
      AND tickets.created >= TO_DATE(TO_CHAR(NOW() - INTERVAL '1 MONTH', 'YYYY-MM-01'), 'YYYY-MM-DD')        \
      AND tickets.created <  TO_DATE(TO_CHAR(NOW(), 'YYYY-MM-01'), 'YYYY-MM-DD')                             \
    GROUP BY d                                                                                               \
    ORDER BY d;\" |                                                                                          \
  ssh rude.torproject.org psql \\\"host=drobovi.torproject.org sslmode=require user=rtreader dbname=rt\\\" | \
  sed 's/|//'                                                                                                \
" using 1:xtic(2) with boxes title "new tickets"

Get the most recent version of each RT articles

SELECT classes.name AS class,
       articles.name AS title,
       CASE WHEN objectcustomfieldvalues.content != '' THEN objectcustomfieldvalues.content
            ELSE objectcustomfieldvalues.largecontent
       END AS content,
       objectcustomfieldvalues.lastupdated,
       articles.id
  FROM classes, articles, objectcustomfieldvalues
 WHERE articles.class = classes.id
   AND objectcustomfieldvalues.objecttype = 'RT::Article'
   AND objectcustomfieldvalues.objectid = articles.id
   AND objectcustomfieldvalues.id = (
           SELECT objectcustomfieldvalues.id
             FROM objectcustomfieldvalues
            WHERE objectcustomfieldvalues.objectid = articles.id
              AND objectcustomfieldvalues.disabled = 0
            ORDER BY objectcustomfieldvalues.lastupdated DESC
            LIMIT 1)
 ORDER BY classes.id, articles.id;

Creating a new RT user

When someone needs to access RT in order to review and answer tickets, they need to have an account in RT. We're currently using RT's builtin user base for access management (e.g. accounts are not linked to LDAP).

RT tends to create accounts for emails that it sees passing in responses to tickets (or ticket creations), so most likely if the person has already interacted with RT in some way, they already have a user. The user might not show up in the list on the page Admin > Users > Select, but you can find them by searching by email address. If a user already exists, you simply need to:

  • modify it to tick the Let this user be granted rights(Privileged) option in their account
  • add them as member to the appropriate groups (see with RT service admins and team lead)

In the unlikely case of a person not having an account at all, here's how to do it from scratch:

  • As an administrator, head over to Admin > Users > Create
  • In the Identity section, fill in the Username, Email and Real Name fields.
  • For Real Name, you can use the same as we have in the person's LDAP account, if they have one. Or it can just be the same value as the username
  • In the Access Control section, tick the Let this user be granted rights(Privileged) option.
  • Click on Create at the bottom
  • Check in with RT service admins and team lead to identify which groups the account should be a member of and add the account as member of those groups.

Granting access to a support help desk coordinator

The support help desk coordinator needs the following assets to perform their duties:

  • Administration password for the support-team-private mailing list.
  • Being owner in the support-team-private mailing list configuration.
  • Commit access to help wiki Git repository.
  • Shell access to rude.torproject.org.
  • LDAP account member of the rtfolks group.
  • LDAP account member of the support group.
  • root password for Request Tracker.
  • Being owner of the “Tor Support” component in Trac.

New RT admin

This task is typically done by TPA, but can technically be done by any RT admin.

  1. find the RT admin password in hosts-extra-info in the TPA password manager and login as root OR login as your normal RT admin user

  2. create an account member of rt-admin

Pager playbook

Ticket creation failed / No permission to create tickets in the queue

If you receive an email like this:

From: rt@rt.torproject.org Subject: Ticket creation failed: [ORIGINAL SUBJECT] To: root@rude.torproject.org Date: Tue, 05 Jan 2021 01:01:21 +0000

No permission to create tickets in the queue 'help'

[ORIGINAL EMAIL]

Or like this:

Date: Fri, 14 Feb 2025 12:20:30 +0000 From: rt@rt.torproject.org To: root@rude.torproject.org Subject: Failed attempt to create a ticket by email, from EMAIL

EMAIL attempted to create a ticket via email in the queue giving; you might need to grant 'Everyone' the CreateTicket right.

In this case, it means a RT admin disabled the user in the web interface, presumably to block off a repeated spammer. The bounce is harmless, but noise can be reduced by adding the sender to the denylist in the profile::rspamd::denylist array in data/common/mail.yaml.

See Also issue 33314 for more information.

Reference

Installation

Request Tracker is installed from the Debian package request-tracker4.

Configuration lives in /etc/request-tracker4/RT_SiteConfig.d/ and is not managed in Puppet (yet).

Upgrades

RT upgrades typically require a migration to complete successfully. Those are typically done with the rt-setup-database-5 --action upgrade command, but there are specifics that depend on the version. See the /usr/share/doc/request-tracker5/NEWS.Debian.gz for instructions:

zless /usr/share/doc/request-tracker5/NEWS.Debian.gz

For example, the trixie upgrade suggested multiple such commands:

root@rude:~# zgrep rt-setup /usr/share/doc/request-tracker5/NEWS.Debian.gz
  rt-setup-database-5 --action upgrade --upgrade-from 5.0.5 --upgrade-to 5.0.6
  rt-setup-database-5 --action upgrade --upgrade-from 5.0.4 --upgrade-to 5.0.5
  rt-setup-database-5 --action upgrade --upgrade-from 5.0.3 --upgrade-to 5.0.4
  rt-setup-database-5 --action upgrade --upgrade-from 4.4.6 --upgrade-to 5.0.3

The latter in there was the bullseye to bookworm upgrade, so it's irrelevant, but the previous ones can be squashed together in a single command:

rt-setup-database-5 --action upgrade --dba rtuser --upgrade-from 5.0.3 --upgrade-to 5.0.6

The password that gets prompted for is in /etc/request-tracker5/RT_SiteConfig.d/20-database.pm.

Consulting the NEWS.Debian file is nevertheless mandatory to ensure we don't miss anything.

Logs

RT sends its logs to syslog tagged with RT. To view them:

# journald -t RT

The log level may be adjusted via /etc/request-tracker4/RT_SiteConfig.d/60-logging.pm.

Retention of the RT logs sent to syslog is controlled by the retention of journald (by default up to 10% of the root filesystem), and syslog-ng / logrotate (30 days).

The configured log level of warning does not regularly log PII but may on occasion log IP and email addresses when an application error occurs.

Auto-reply to new requesters

When an unknown email address sends an email to the support, it will be automatically replied to warn users about the data retention policy.

A global Scrip is responsible for this. It will be default use the global template named “Initial reply”. It is written in English. In each queue except help, a template named exactly “Initial reply” is defined in order to localize the message.

Expiration of old tickets

Tickets (and affiliated users) get erased from the RT database after 100 days. This is done by the expire-old-tickets script. The script is run everyday at 06:02 UTC through a cronjob run as user colin.

Encrypted SQL dumps of the data removed from the database will be written to /srv/rtstuff/shredded and must be put away regularly.

Dump of RT templates

RT articles are dumped into text files and then pushed to the rt-articles Git repository. An email is sent each time there's a new commit, so collective reviews can happen by the rest of the support team.

The machinery is spread through several scripts. The one run on rude is dump_rt_articles, and it will run everyday through a cronjob as user colin.

Issues

There is no issue tracker specifically for this project, File or search for issues in the team issue tracker with the ~RT label.

Spammers blocklist

In order to help deal with repeat spam senders, in tpo/tpa/team#40425 a script was deployed to scan all recent tickets in the spam queue and add any senders that appear more than once to an MTA blocklist.

The script is located at /usr/local/sbin/rt-spam-blocklist and runs hourly via root's crontab. The blocklist itself containing the banned senders is located at /etc/postfix/rt-spam-blocklist, and is configured as a header_checks table for Postfix.

While senders are added automatically to the blocklist, they can only be removed manually. Before removing and entry from the list, ensure tickets from this sender are also deleted or moved out of the RT spam queue, otherwise they will be re-added.

DMARC filter

In order to prevent trivial sender address spoofing, incoming mail is filtered through OpenDMARC. This adds an Authentication-Results header containing the DMARC result, which is then analysed by the Verify DMARC scrip.

If the result is dmarc=fail then the message's queue is changed to spam, a comment is added to the ticket and a message is logged to the system logs.

If the Authentication-Results header is missing, such as when a ticket is created through the web interface, the check is skipped altogether.

Discussion

Spam filter training design

RT is designed to be trained for spam filtering. RT users put spam in the "Spam" queue and then a set of scripts run in the background to train spamassassin, based on a mail archive that procmail keeps of every incoming mail.

This runs as a cronjob in the rtmailarchive user, which looks like this:

/srv/rtstuff/support-tools/train-spam-filters/train_spam_filters && bin/spam-learn && find Maildir/.spam.learned Maildir/.xham.learned -type f -delete

The train_spam_filters script basically does this:

  1. for each mail in the Maildir/.help* archive
  2. find its Message-Id header
  3. load the equivalent message from RT:
    • if it is in the Spam queue, marked as "Rejected", it is spam.
    • if it is in a help-* queue, marked as "Resolved", it is ham.
  4. move the email in the right directory mail folder (.spam.learn, .xham.learn) depending on status
  5. if the file is more than 100 days old, delete it.

Then the rest of the cron job continues. spam-learn is this shell script:

#!/bin/bash

dbpath="/var/cache/spampd"

learn() {
    local what="$1"; shift;
    local whence="$1"; shift;
    local whereto="$1"; shift;

    (
        cd "$whence"
        find -type f | \
          while read f; do
            sudo -u spampd -H sa-learn --dbpath "$dbpath" --"$what" < "$f"
            mv "$f" "$whereto/$f"
        done
    )
}

set -e

learn spam /srv/rtmailarchive/Maildir/.spam.learn /srv/rtmailarchive/Maildir/.spam.learned
learn ham /srv/rtmailarchive/Maildir/.xham.learn /srv/rtmailarchive/Maildir/.xham.learned

# vim:set et:
# vim:set ts=4:
# vim:set shiftwidth=4:

which, basically, calls sa-learn on each individual email in the folder, moving it to .spam.learned or .xham.learned when done.

Then, interestingly, those emails are destroyed. It's unclear why that is not done in the spam-learn step directly.

Possible improvements

The above design has a few problems:

  1. it assumes "ham" queues are named "help-*" - but there are other queues in the system
  2. it might be slow: if there are lots of emails to process, it will do an SQL query for each and a move, and not all at once
  3. it is split over multiple shell scripts, not versioned

I would recommend the following:

  1. reverse the logic of the queue checks: instead of checking for folders and queues named help-*, check if the folders or queues are not named spam* or xham*
  2. batch jobs: use a generator to yield Message-Id, then pick a certain number of emails and batch-send them to psql and the rename
  3. do all operations at once: look in psql, move the files in the learning folder, and train, possibly in parallel, but at least all in the same script
  4. sa-learn can read from a folder now, so there's no need for that wrapper shell script in any case
  5. commit the script to version control and, even better, puppet

We could also add a CAPTCHA and look at the RT::Extension::ReportSpam...

Alternatives