This procedure documents various benchmarking procedures in use inside TPA.

[[TOC]]

HTTP load testing

Those procedures were quickly established to compare various caching software as part of the cache service setup.

Common procedure

  1. punch a hole in the firewall to allow the test server to access tested server, in case it is not public yet

    iptables -I INPUT -s 78.47.61.104 -j ACCEPT
ip6tables -I INPUT -s 2a01:4f8:c010:25ff::1 -j ACCEPT

  2. point the test site (e.g. blog.torproject.org) to the tested server on the test server, in /etc/hosts:

    116.202.120.172 blog.torproject.org
2a01:4f8:fff0:4f:266:37ff:fe26:d6e1 blog.torproject.org

  3. disable Puppet on the test server:

    puppet agent --disable 'benchmarking requires /etc/hosts override'

  4. launch the benchmark on the test server

Siege

Siege configuration sample:

verbose = false
fullurl = true
concurrent = 100
time = 2M
url = http://www.example.com/
delay = 1
internet = false
benchmark = true

Might require this, which might work only with varnish:

proxy-host = 209.44.112.101
proxy-port = 80

Alternative is to hack /etc/hosts.

apachebench

Classic commandline:

ab2 -n 1000 -c 100 -X cache01.torproject.org https://example.com/

-X also doesn't work with ATS, modify /etc/hosts instead.

bombardier

We tested bombardier as an alternative to go-wrk in previous benchmarks. The goal of using go-wrk was that it supported HTTP/2 (while wrk didn't), but go-wrk had performance issues, so we went with the next best (and similar) thing.

Unfortunately, the bombardier package in Debian is not the HTTP benchmarking tool but a commandline game. It's still possible to install it in Debian with:

export GOPATH=$HOME/go
apt install golang
go get -v github.com/codesenberg/bombardier

Then running the benchmark is as simple as:

./go/bin/bombardier --duration=2m --latencies https://blog.torproject.org/

wrk

Note that wrk works similarly to bombardier, sampled above, and has the advantage of being already packaged in Debian. Simple cheat sheet:

sudo apt install wrk
echo "10.0.0.0 target.example.com" >> /etc/hosts
wrk --latency -c 100 --duration 2m https://target.example.com/

The main disadvantage is that it doesn't (seem to) support HTTP/2 or similarly advanced protocols.

Other tools

Siege has trouble going above ~100 concurrent clients because of its design (and ulimit) limitations. Its interactive features are also limited, here's a set of interesting alternatives:

Project Lang Proto Features Notes Debian
ali golang HTTP/2 real-time graph, duration, mouse support unsearchable name no
bombardier golang HTTP/2 better performance than siege in my 2017 tests RFP
boom Python HTTP/2 duration rewrite of apachebench, unsearchable name no
drill Rust scriptable, delay, stats, dynamic inspired by JMeter and friends no
go-wrk golang no duration rewrite of wrk, performance issues in my 2017 tests no
hey golang rewrite of apachebench, similar to boom, unsearchable name yes
Jmeter Java interactive, session replay yes
k6.io JMeter rewrite with "cloud" SaaS no
Locust distributed, interactive behavior yes
oha Rust TUI inspired by hey no
Tsung Erlang multi distributed yes
wrk C multithreaded, epoll, Lua scriptable yes

Note that the Proto(col) and Features columns are not exhaustive: a tool might support (say) HTTPS, HTTP/2, or HTTP/3 even if it doesn't explicitly mention it, although it's unlikely.

It should be noted that very few (if any) benchmarking tools seem to support HTTP/3 (or even QUIC) at this point. Even HTTP/2 support is spotty: for example, while bombardier supports HTTP/2, it only does so with the slower net/http library at the time of writing (2021). It's unclear how many (if any) other projects to support HTTP/2 as well.

More tools, unreviewed: