QHRD6YOX6RZRDR7NCHNDO7GLCEA6VFM6

Hi all, Following hiro's "Security incident affecting some relay operators" post on Oct 5, here is what we found on our own infrastructure, in case it helps others check theirs. Two of our servers were rooted with the Ebury OpenSSH backdoor (self-reported version 1.8.3), and we use these two names for them throughout: - the management server: our Contabo VPS, which was our WireGuard VPN gateway and SSH jump host and also ran an obfs4 bridge; - Switzerland2: a FranTech VPS running two of our Quetzalcoatl exit relays (29FEFE36A5F66A6C93D24775B9D2364A3831B597 and 8427937D5A39E15699C850F26FED3CD59C379C48). When we examined them, neither server showed the NAT redirect, new users or SSH keys, cron jobs, systemd units or modified tor binary from the advisory's checklist. Both were examined read-only through several independent AI-assisted review passes, cross-checked against logs, process memory and public Tor Metrics data. All times are UTC. One caveat up front: the attacker truncated some logs on the management server, timestomped the files they planted, and turned off sshd's login logging on Switzerland2, and Ebury's backdoor logins leave no login records or shell history at all. Some details are therefore missing, and parts of this timeline are inferred from what was left; we say so where it matters. 1. TIMELINE - Background (management server): SSH password authentication was enabled, and an encrypted backup of our admin workstation was stored on the management server. That backup held our operator SSH private key, our ansible inventory and copies of our relays' offline Ed25519 master keys. - Background (both servers): both trusted the same operator SSH key, and the operator account's password was also its sudo password, sat in cleartext in a helper script on both servers, and was the backup's passphrase. We don't know which of our other relays had that script. - Aug 28, 03:19: the attacker logged in to the management server from 185.225.226.62 (VikHost, AS207560) with the operator account's password on the first try, with no failed attempt from that IP ever recorded, so they almost certainly knew the password in advance. By 03:24 they had root via sudo su. - Aug 28, 03:25-03:43: the attacker installed Ebury on the management server, truncated some logs and installed gcc. They then tried apt inside our WireGuard dashboard (WGDashboard) container, which runs on Alpine Linux and only has Alpine's apk package manager; when apt failed, an SSH client and nmap appeared in the container within minutes (dated by file change times). - Aug 28, 04:33-04:37: the same IP opened two SSH connections to Switzerland2 as the operator account and disconnected before authenticating; Switzerland2 only accepted SSH keys. - Aug 28, 04:38: Switzerland2 accepted a login for the operator account from the management server's IP address (also our VPN exit), using an RSA key that was in no authorized_keys file we found on Switzerland2 (how it was accepted is still open; see section 3). One second later, sudo bash -c 'echo | base64 -d | perl' installed the same Ebury build on Switzerland2. - The management server's logs show no SSH login or sudo at 04:38, and the connection's source port fits the management server's own port range rather than our VPN clients', so the attacker most likely made this hop through the management server's backdoor. - Aug 31 - Sep 6 (management server): root actions with no sudo record behind them installed nmap on the management server itself on Aug 31. On Sep 6, starting while no one was logged in, more such root actions pulled an alpine image, loaded NFS client kernel modules, installed duplicity and decrypted the workstation backup's metadata (whether the backup itself was copied out is open). - Sep 30 (Switzerland2): the attacker restarted sshd from inside the backdoor, after which sshd stopped writing its "Accepted ..." lines, the ones that record the user, key and source address of each login, because its LogLevel was ERROR. The PAM "session opened" lines kept appearing, but they record neither key nor source address. - Sep 30 - Oct 1 (Switzerland2): the attacker installed curl and wireguard-tools outside sudo, loaded the WireGuard kernel module, and started hidden processes named guard, then failover and snf, none of which left binaries on disk (what they did is open). - Oct 1, 15:35-15:43 (Switzerland2): with no SSH login, sudo command or cron job in the logs, the attacker prepended these lines to /etc/tor/torrc, sent Tor SIGHUPs and then restarted it: # ==== OUR_POLICY_BEGIN 2026-10-01 ==== ExitPolicy reject *:25 ExitPolicy reject *:465 ExitPolicy reject *:587 ExitPolicy reject *:110 ExitPolicy reject *:143 ExitPolicy reject *:993 ExitPolicy reject *:995 ExitPolicy reject *:3389 ExitPolicy reject *:135 ExitPolicy reject *:137-139 ExitPolicy reject *:445 ExitPolicy accept *:* # ==== OUR_POLICY_END ==== That file is unused on Switzerland2: the two relays run as Debian tor@ instances that read /etc/tor/instances/ /torrc, and the only unit that reads /etc/tor/torrc (tor@default) was masked. The published policy did not change. - Oct 2, 09:34 (Switzerland2): again with no SSH login, sudo command or cron job in the logs, both instance torrcs were rewritten. Our ansible-generated accept-list (60 accepted port ranges and a final reject *:*) was removed and five lines were appended after "# end of torrc": ExitPolicy reject *:25, *:465, *:587, *:993 and *:995. - Oct 2, 09:34:04: both Switzerland2 relays published the result, Tor's default rejects plus those five ports and then accept *:* (confirmed in the public CollecTor archive). At 09:42 the attacker tidied up: they emptied the OUR_POLICY block in /etc/tor/torrc, appended the same five reject lines to that file, and saved copies of all three files as *.bak20261002. - Our reading of Oct 1-2, a hypothesis the logs can't prove: the attacker assumed Tor reads the default /etc/tor/torrc, saw that reloads and a full restart didn't change the published policy, and came back the day to edit the instance files. The policy they tried first also differed: it blocked plaintext POP3/IMAP and RDP, while the one that went live opened those and blocked the TLS mail ports instead. - Oct 4, 05:46 (Switzerland2): a routine kernel upgrade and reboot killed the hidden tools, but Ebury came straight back. Oct 4, 13:56 (Switzerland2): after the Tor Project's network health team spotted the exit-policy change and contacted us, we confirmed we hadn't made it, asked for the relays to be marked bad and stopped Tor on Switzerland2. The open policy had been live for 52 hours on two Guard+Exit relays moving roughly 3.4 TiB a day. - Oct 5, 08:56-10:26: relays on 55 of our family's hosts were restarted in one scripted sweep, in text-sorted IP order (who ran it is open). On Switzerland2 that restart came from inside the backdoor, the directory authorities rejected both relays' descriptor uploads (HTTP 400), and we stopped Tor again. - Oct 6: Onionoo's family query listed 168 of our relays on 57 IPs, all down and all advertising that same near-open policy. Both servers are now offline and deleted, and all of our relays and our bridge are retired and blocklisted. 2. WHAT WE LEARNED - The root cause was ours: one password reused for SSH login, sudo and the backup passphrase, stored in plaintext on both servers, plus SSH password authentication left enabled on the internet-facing management server. The attacker already had the password when they first connected. - The management server was a single point of failure. From it the attacker reached Switzerland2 from an IP address that blended in with our automation, every device on our WireGuard VPN was in reach, and Ebury also hooks the SSH client on the management server. - This Ebury is one unowned, setuid file, /usr/lib/x86_64-linux-gnu/libkeyutils.so.1.10.2 (63,368 bytes, timestomped), with the libkeyutils.so.1 symlink re-pointed at it. The packaged library is untouched, so package verification (dpkg -V, debsums) won't catch it, and sshd, sudo and PAM matched their package checksums on both servers. - On Debian 13 (Switzerland2) it is loaded by sshd-session through libkrb5, so ldd /usr/sbin/sshd and the sshd listener process look clean; from there it spreads into every process in an SSH session, sudo included. - Infected processes are lied to (a shell can't see the library in its own /proc/self/maps). The credential collector runs as a fake /sbin/rsyslogd (earlier as systemd-udevd or arpd) on abstract socket @event-0YsOG9WSOE, inside ssh.service's cgroup on Debian (Switzerland2) and, after a reboot, inside init.scope on Ubuntu 24.04 (the management server), where sshd-socket-generator starts it at boot. - Backdoor logins bypass PAM, logging and shell history; the implant's config includes UsePAM no, LogLevel QUIET, HISTFILE=/dev/null, AuthorizedKeysFile /proc/self/environ and PermitTunnel yes. The main traces were root actions no logged-in user could have made: apt history entries without "Requested-By:", su sessions opened by uid 0, systemctl calls from the ssh.service cgroup, Tor reloads nobody had asked for, and systemd's "remains running after unit stopped" when sshd restarted. - SSH encrypts passwords in transit, but sshd has to decrypt them to check them, and Ebury runs inside sshd. So it captures every login password exactly as typed, failed attempts included, notes which key logged in from where for key logins, and it captured the password our investigators used from every address they logged in from. - Version 1.8.3 is newer than ESET's public IOCs (last updated May 2024, up to 1.8.2), its socket name isn't on their list, and ESET's detect_ebury.sh would not flag it, although their YARA rule libkeyutils_with_ctor still matches. Each install stamps an 8-byte victim ID into the file, so the implant's hash differs on every server. - The policy flip opened plaintext POP3/IMAP, telnet, IRC, SSH and RDP while closing IMAPS/POP3S, so mail retrieval through the Switzerland2 relays could only happen in cleartext. A hidden process named snf was running on Switzerland2 for most of that window. - It looks like the attacker went well beyond these two servers: every relay Onionoo listed in our family on Oct 6 advertised the same near-open policy, and the Oct 5 restart sweep covered 55 of our hosts. The management server and Switzerland2 shared an operator key and password; we didn't examine our other hosts, so we can't say whether they did too. - Some of our review passes got it wrong. One concluded there was no intrusion (it dismissed the setuid library as normal, trusted a clean dpkg -V and grouped logins by IP instead of key), and others concluded the policy change never reached the published descriptors; the public CollecTor descriptor archive and reads of process memory settled it. - Our "offline" master keys were not offline: copies sat in the workstation backup on the management server, encrypted with the password the attacker already had. - systemctl disable tor is not enough on Debian multi-instance relays, because instance units can be enabled on their own (on Switzerland2 they were, and the attacker added another boot link); mask the tor@ units, and since a live backdoor can undo that, take the affected server offline too. - On Switzerland2, sudo logged the entire base64 installer across 67 "(command continued)" journal records, so we could decode it. On the management server the attacker ran sudo su first, and nothing of the install command was logged. 3. OPEN QUESTIONS - Where did the attacker get the operator password before Aug 28? Not by brute force, and not through the WGDashboard on the management server (its web UI had been failing to start since April); the candidates are another of our relays, the admin workstation, or reuse somewhere we haven't found. - How did Switzerland2's sshd accept a key that was in no authorized_keys file, one second before Ebury was installed on Switzerland2? ~/.ssh and sshd_config.d on Switzerland2 never changed, which leaves a line in the main sshd_config (replaced on Sep 30, so we can't check it) or an earlier backdoor. - What did guard, failover and snf do on Switzerland2? Their binaries died with the Oct 4 reboot, so we don't know whether the newly opened port 22 was used to intercept SSH or whether a WireGuard tunnel was built (udev logged two network-interface events on Sep 30, but no interface or config survived). - Where do the stolen credentials go? ESET's research on earlier versions describes operators pulling them through the backdoor, with an automatic encrypted send over UDP port 53 if nobody collects them for two weeks. On our servers, memory holds an template but no host, and a sandboxed run made no outbound connection, but the implant checks for packet sniffers and strace, so that test doesn't settle it. - Which of our other relays had their exit policy changed, when, and were any non-exits turned into exits? We are pulling the public CollecTor consensus and descriptor history for the whole family, back to May, and will follow up. - How many of the other 56 hosts in our family carry the implant, and was any infected before Aug 28 03:25 (an earlier implant birth time would mean an earlier victim)? - Who ran the Oct 5 restart sweep, and from where: the management server's backdoor, the admin workstation, or relay to relay? The logs on the management server and Switzerland2 can't tell these apart, and we haven't examined the workstation yet. - Did the 5.3 GB workstation backup leave the management server? Its metadata was decrypted on the management server on Sep 6 and NFS client modules were loaded that night, but nothing in the logs shows a transfer. - Was there a NAT redirect on Switzerland2 before the Oct 4 reboot? iptables was installed outside sudo on Sep 2 (we don't know by whom), and unsaved rules would not survive a reboot. - Is this the same actor as the other cases behind hiro's notice (NAT redirects, a fake uppercase CRON)? If you have seen libkeyutils.so.1.10.2, @event-0YsOG9WSOE, 185.225.226.62 or the SSH keys listed below, please tell us or bad-relays@. 4. BEST PRACTICES - Never put passwords in scripts, and never reuse one across SSH login, sudo and backups. Use key-only automation with narrowly scoped NOPASSWD sudo rules. - Use key-only SSH everywhere, especially on jump and management servers, and reach them only through a VPN or an IP allow-list. Check the effective config with sshd -T, because sshd takes the first value it finds (on Switzerland2, a PermitRootLogin yes left near the top by the VPS image silently beat the PermitRootLogin no our setup script appended). - Treat your management/VPN server as the crown jewels: nothing web-facing on it, no key backups on it, and no admin tooling in its containers. The WGDashboard 4.3.0.1 on our management server was exposed to the internet (its web UI happened to be broken since April), has an unauthenticated file-read bug (CVE-2026-44343, fixed in 4.3.2) and logs every sign-in password in cleartext. - Keep OfflineMasterKey material genuinely offline. If a machine that ever held it is compromised, retire those identities and roll a new family key and FamilyId. - Check for this Ebury from a rescue system or an offline disk image if you can. On a live server, log in with a key (never a password) and use ESET's clean-shell trick, H=1 LD_PRELOAD=":" LD_DEBUG=":" bash; a setuid libkeyutils file, a libkeyutils.so.1 symlink to a file no package owns, an @event- abstract socket, or a non-sshd process lingering in the ssh.service cgroup is a strong sign of infection: find / -xdev -name 'libkeyutils*' -perm -4000 2>/dev/null dpkg -S "$(readlink -f /usr/lib/x86_64-linux-gnu/libkeyutils.so.1)" (rpm -qf on RPM systems) grep -E '@event-|@/dev/(event|stats)-' /proc/net/unix systemctl status ssh | sed -n '/CGroup/,$p' (on Ubuntu 24.04, check init.scope too) - your journal for apt transactions with no "Requested-By:" (unattended-upgrades also lacks it), su sessions opened by uid 0, ssh.service "remains running after unit stopped" or unusually large "Consumed ... CPU time" lines (Switzerland2's showed 41 minutes and 281 MB), "Reload requested from client ... (unit ssh.service)", and Tor reloads nobody on your team caused. On Ubuntu, the install also leaves apparmor="DENIED" operation="bind" profile="rsyslogd" addr="@event-..." in the kernel log. - Look for sudo running "base64 -d | perl" (long commands are split across "(command continued)" records) or "echo 3 > /proc/sys/vm/drop_caches", a missing /etc/falco/rules.d/ebury.yaml if you run Falco, and logins from 185.225.226.62, and review accepted logins by key fingerprint, not just source IP. Keys: RSA SHA256:dvACokMYcXJ2om4wuS+TvdtjrAgPK7rC7ZQHDgvBHzk (used to log in to Switzerland2) and the two embedded in the implant, RSA SHA256:KlICz/B59619CpuQdGzgmmvzWBfTMUJ77EwihTYobuo and ED25519 SHA256:OfLY0uF9SwOW9xVjiMXZGDfk6Atmz6CbHy4epknLm8Y. - Monitor your relays from outside, as the Tor Project's network health team did for us. Alert when the exit policy, family or info in your published descriptors (Onionoo or CollecTor) differs from what you deployed, since files on a compromised server can be rewritten. - Ship logs off your servers, keep NTP on, alert if sshd's LogLevel drops below INFO, and watch for unowned or setuid files in library directories (AIDE or similar). - Never type a password into a server you suspect, because Ebury will capture it. Investigate with keys from a clean machine or from an offline image, and capture memory before rebooting or wiping. - If you find it, take the relays down and mask the tor units, snapshot the disk, and rebuild from scratch rather than cleaning in place. Then rotate every SSH key, password and WireGuard key, generate new relay identities, and report to [email protected]. Thanks to the Tor Project's network health team for catching the change and acting quickly. We're happy to more detail, including the de-fanged installer, with anyone investigating a similar case. Zimmer Quetzalcoatl relays

attachment.html (text/html — 18.9 KB)

10 9 participants