The APUs are neat little devices from PC Engines. We use them as jump hosts and, generally, low-power servers where we need them.

This documentation was written with a APU3D4, some details may vary with other models.

[[TOC]]

Tutorial

How to

Console access

The APU comes with a DB-9 serial port. You can connect to that port using, typically, a null modem cable and a serial-to-USB adapter. Once properly connected, the device will show up as /dev/ttyUSB0 on Linux. You can connect to it with GNU screen with:

screen /dev/ttyUSB0 115200

... or with plain cu(1):

cu -l /dev/ttyUSB0 -s 115200

If you fail to connect, PC Engines actually has minimalist but good documentation on the serial port.

BIOS

When booting, you should be able to see the APU's BIOS on the serial console. It looks something like this after a few seconds:

PCEngines apu3
coreboot build 20170302
4080 MB ECC DRAM

SeaBIOS (version rel-1.10.0.1)

Press F10 key now for boot menu

The boot menu then looks something like that:

Select boot device:

1. USB MSC Drive Kingston DataTraveler 3.0 
2. SD card SD04G 3796MiB
3. ata0-0: SATA SSD ATA-9 Hard-Disk (111 GiBytes)
4. Payload [memtest]
5. Payload [setup]

Hitting 4 puts you in a Memtest86 memory test (below). The setup screen looks like this:

### PC Engines apu2 setup v4.0.4 ###
Boot order - type letter to move device to top.

  a USB 1 / USB 2 SS and HS 
  b SDCARD 
  c mSATA 
  d SATA 
  e iPXE (disabled)


  r Restore boot order defaults
  n Network/PXE boot - Currently Disabled
  t Serial console - Currently Enabled
  l Serial console redirection - Currently Enabled
  u USB boot - Currently Enabled
  o UART C - Currently Disabled
  p UART D - Currently Disabled
  x Exit setup without save
  s Save configuration and exit

i.e. it basically allows you to change the boot order, enable network booting, disable USB booting, disable the serial console (probably ill-advised), and mess with the other UART ports.

The network boot actually drops you in iPXE which is nice (version 1.0.0+ (f8e167) from 2016) as it allows you to bootstrap one rescue host with another (see the installation section below).

Memory test

The boot menu (F10 then 4) provides a built-in memory test which runs Memtest86 5.01+ and looks something like this:

Memtest86+ 5.01 coreboot 001| AMD GX-412TC SOC                               
CLK: 998.3MHz  (X64 Mode)   | Pass  6% ##
L1 Cache:   32K  15126 MB/s | Test 67% ##########################              
L2 Cache: 2048K   5016 MB/s | Test #5  [Moving inversions, 8 bit pattern]     
L3 Cache:  None             | Testing: 2048M - 3584M   1536M of 4079M
Memory  : 4079M   1524 MB/s | Pattern:   dfdfdfdf           | Time:   0:03:49
------------------------------------------------------------------------------
Core#: 0 (SMP: Disabled)  |  CPU Temp  | RAM: 666 MHz (DDR3-1333) - BCLK: 100
State: - Running...       |    48 C    | Timings: CAS 9-9-10-24 @ 64-bit Mode
Cores:  1 Active /  1 Total (Run: All) | Pass:       0        Errors:      0  
------------------------------------------------------------------------------












                                PC Engines APU3
(ESC)exit  (c)configuration  (SP)scroll_lock  (CR)scroll_unlock (l)refresh

Pager playbook

Disaster recovery

Reference

Installation

The current APUs were ordered directly from the PC Engines shop, specifically the USD section. The build was:

    2 apu3d4   144.00 USD 288.00  HTS 8471.5000     TW Weight    470g
      APU.3D4 system board 4GB

    2 case1d2redu 10.70 USD  21.40  HTS 8473.3000     CN Weight    502g
      Enclosure 3 LAN, red, USB

    2 ac12vus2 4.40 USD   8.80  HTS 8504.4000     KH Weight    266g
      AC adapter 12V US plug for IT equipment

    2 msata120c 15.50 USD  31.00  HTS 8523.5100     CN Weight     14g
      SSD M-Sata 120GB TLC

    2 sd4b     6.90 USD  13.80  HTS 8523.5100     TW Weight      4g
      SD card 4GB pSLC Phison

    2 assy2    7.50 USD  15.00  HTS 8471.5000     CH Weight    120g
      assembly + box

Shipping TBD !!!       USD   0.00    Weight   1376g
VAT                    USD   0.00

Total                  USD 378.00

Note how the price is for two complete models. The devices shipped promptly; it was basically shipped in 3 days, but customs added an additional day of delay over the weekend, which led to a 6 days (4 business days) shipping time.

One of the machine was connected over serial (see above) and booted with a GRML "96" (64 and 32 bit) image over USB. Booting GRML from USB is tricky, however, because you need to switch from 115200 to 9600 once grub finishes loading, as GRML still defaults to 9600 baud instead of 115200. It may be possible to tweak the GRUB commandline to change the speed, but since it's in the middle of the kernel commandline and that the serial console editing capabilities are limited, it's actually pretty hard to get there.

The other box was chain-loaded with iPXE from the first box, as a stress-test. This was done by enabling the network boot in the BIOS (F10 to enter the BIOS in the serial console, then 5 to enter setup and n to enable network boot and s to save). Then hit n to enable network boot and choose "iPXE shell" when prompted. Assuming both hosts are connected over their eth1 storage interfaces, you should then do:

iPXE> dhcp net1
iPXE> chain autoexec.ipxe

This will drop you in another DHCP sequence, which will try to configure each interface. You can control-c to skip net0 and then the net1 interface will self-configure and chain-load the kernel and GRML. Because the autoexec.ipxe stores the kernel parameters, it will load the proper serial console settings and doesn't suffer from the 9600 bug mentioned earlier.

From there, SSH was setup and key was added. We had DHCP in the lab so we just reused that IP configuration.

service ssh restart
cat > ~/.ssh/authorized_keys
...

Then the automated installer was fired:

./install -H root@192.168.0.145 \
          --fingerprint 3a:4d:dd:91:79:af:4e:c4:17:e5:c8:d2:d6:b5:92:51   \
          hetzner-robot \
          --fqdn=dal-rescue-01.torproject.org \
          --fai-disk-config=installer/disk-config/dal-rescue \
          --package-list=installer/packages \
          --post-scripts-dir=installer/post-scripts/ \
          --ipv4-address 204.8.99.100 \
          --ipv4-subnet 24 \
          --ipv4-gateway 204.8.99.1

WARNING: the dal-rescue disk configuration is incorrect. The 120GB disk gets partitioned incorrectly, as its RAID-1 partition is bigger than the smaller SD card.

Note that IP configuration was actually performed manually on the node, the above is just an example of the IP address used by the box.

Next, the new-machine procedure was followed.

Finally, the following steps need to be performed to populate /srv:

  • GRML image, note that we won't be using the grml.ipxe file, so:

    apt install debian-keyring &&
wget https://download.grml.org/grml64-small_2022.11.iso &&
wget https://download.grml.org/grml64-small_2022.11.iso.asc &&
gpg --verify --keyring /usr/share/keyrings/debian-keyring.gpg grml64-small_2022.11.iso.asc &&
echo extracting vmlinuz and initrd from ISO... &&
mount grml64-small_2022.11.iso /mnt -o loop &&
cp /mnt/boot/grml64small/* . &&
umount /mnt &&
ln grml64-small_2022.11.iso grml.iso

  • build the iPXE image but without the floppy stuff, basically:

apt install build-essential &&
git clone git://git.ipxe.org/ipxe.git &&
cd ipxe/src &&
mkdir config/local/tpa/ &&
cat > config/local/tpa/general.h <<EOF
#define DOWNLOAD_PROTO_HTTPS    /* Secure Hypertext Transfer Protocol */
#undef NET_PROTO_STP        /* Spanning Tree protocol */
#undef NET_PROTO_LACP       /* Link Aggregation control protocol */
#undef NET_PROTO_EAPOL      /* EAP over LAN protocol */
#undef CRYPTO_80211_WEP /* WEP encryption (deprecated and insecure!) */
#undef CRYPTO_80211_WPA /* WPA Personal, authenticating with passphrase */
#undef CRYPTO_80211_WPA2    /* Add support for stronger WPA cryptography */
#define NSLOOKUP_CMD        /* DNS resolving command */
#define TIME_CMD        /* Time commands */
#define REBOOT_CMD      /* Reboot command */
#define POWEROFF_CMD        /* Power off command */
#define PING_CMD        /* Ping command */
#define IPSTAT_CMD      /* IP statistics commands */
#define NTP_CMD     /* NTP commands */
#define CERT_CMD        /* Certificate management commands */
EOF
make -j4 CONFIG=tpa bin-x86_64-efi/ipxe.efi bin-x86_64-pcbios/undionly.kpxe

  • copy the iPXE files in /srv/tftp:

    cp bin-x86_64-efi/ipxe.efi bin-x86_64-pcbios/undionly.kpxe /srv/tftp/

  • create a /srv/tftp/autoexec.ipxe:

#!ipxe

dhcp
kernel http://172.30.131.1/vmlinuz
initrd http://172.30.131.1/initrd.img
initrd http://172.30.131.1/grml.iso /grml.iso
imgargs vmlinuz initrd=initrd.magic boot=live config fromiso=/grml.iso live-media-path=/live/grml64-small noprompt noquick noswap console=tty0 console=ttyS1,115200n8
boot

Upgrades

SLA

Design and architecture

Services

Storage

Queues

Interfaces

Serial console

The APU should provide a serial console access over the DB-9 serial port, standard 115200 baud. The install is configured to offer the bootloader and a login prompt over the serial console, and a basic BIOS is also available.

LEDs

The APU has no graphical interface (only serial, see above), but there are LEDs in the front that have been configured from Puppet to make systemd light them up in a certain way.

From left to right, when looking at the front panel of the APU (not the one with the power outlets and RJ-45 jacks):

  1. The first LED lights up when the machine boots, and should be on when the LUKS prompt waits. then it briefly turns off when the kernel module loads and almost immediately turns back on when filesystems are mounted (DefaultDependencies=no and After=local-fs.target)
  2. The second LED lights up when systemd has booted and has quieted (After=multi-user.target and Type=idle)
  3. The third LED should blink according to the "activity" trigger which is defined in ledtrig_activity kernel module

Network

The three network ports should be labeled according to which VLAN they are supposed to be configured for, see the Quintex network layout for details on that configuration.

From left to right, when looking at the back panel of the APU (the one with the network ports, after the DB-9 serial port):

  1. eth0 public: public network interface, to be hooked up to the public VLAN, mapped to eth0 in Linux

  2. eth1 storage: private network interface, to be hooked up to the storage VLAN and where DHCP and TFTP is offered, mapped to eth1 in Linux

  3. eth2 OOB: private network interface, to be hooked up to the OOB ("Out Of Band" management) VLAN, to allow operators to access the OOB interfaces of the other servers

Authentication

Implementation

Issues

There is no issue tracker specifically for this project, File or search for issues in the team issue tracker with the label ~Foo.

Maintainer

Users

Upstream

Monitoring and metrics

Tests

Logs

Backups

Other documentation

Discussion

Overview

Security and risk assessment

Technical debt and next steps

Proposed Solution

Other alternatives

APU hardware

We also considered a full 1U case but that seemed really costly. We have also considered a HDD enclosure but that didn't seem necessary either.

APU EOL and alternatives

As of 2023-04-18, the PC Engines website has a stronger EOL page that explicitly states that "The end is near!" and that:

Despite having used considerable quantities of AMD processors and Intel NICs, we don't get adequate design support for new projects. In addition, the x86 silicon currently offered is not very appealing for our niche of passively cooled boards. After about 20 years of WRAP, ALIX and APU, it is time for me to move on to different things.

It therefore seems unlikely that new PC Engines product will be made in the future, and that platform should be considered dead.

In our initial research (tpo/tpa/team#41058) we found two other options (the SolidRun and Turris, below), but since then we've expanded the search and we're keeping a list of alternatives here.

The specification is, must have:

  • small (should fit in a 1U)
  • low power (10-50W max)
  • serial port or keyboard and monitor support
  • at least three network ports
  • 3-4GB storage for system (dal-rescue-02 uses 2.1GB as of this writing)
  • 1-5GB storage for system images (dal-rescue-02 uses 1GB)

Nice to have:

  • faster than the APU3 (AMD GX-412TC SOC 600MHz)
  • rack-mountable
  • coreboot
  • "open hardware"
  • 12-24 network ports (yes, that means it's a switch, and that we don't need an extra OOB switch)

Other possibilities: