Encrypted traffic interception on Hetzner and Linode targeting the largest Russian XMPP (Jabber) messaging service (2023)

TL;DR: we have discovered XMPP (Jabber) instant messaging protocol encrypted TLS connection wiretapping (Man-in-the-Middle attack) of jabber.ru (aka xmpp.ru) service’s servers on Hetzner and Linode hosting providers in Germany. The attacker has issued several new TLS certificates using Let’s Encrypt service which were used to hijack encrypted STARTTLS connections on port 5222 using transparent MiTM proxy. The attack was discovered due to expiration of one of the MiTM certificates, which haven’t been reissued. There are no indications of the server breach or spoofing attacks on the network segment, quite the contrary: the traffic redirection has been configured on the hosting provider network. The wiretapping may have lasted for up to 6 months overall (90 days confirmed). We believe this is lawful interception Hetzner and Linode were forced to setup.

Last update: 03 Nov 2023

oxpa , experienced UNIX administrator who is in charge for the oldest Russian XMPP service jabber.ru established in year 2000, was puzzled by “Certificate has expired” message upon connecting to the service on 16 October 2023. All the certificates seemed to be up-to-date on the server, but the connection to port 5222 (XMPP STARTTLS) was presenting an outdated certificate to the client.

During all kinds of software, network and configuration checks with the help of other professionals and ejabberd software author, it was discovered that:

The affected machines are dedicated server on Hetzner and two virtual servers on Linode, all hosted in Germany.

Traffic dump on port 5223, all the data intact as it should be.

Traffic dump on port 5222, the connection is hijacked on application level (L7), the server receives replaced ClientHello message from the client.

It was apparent that the connections on port 5222 are under Man-in-the-Middle attack which intercepts encrypted communications.

At first we had 2 hypothesis to check: whether the servers have been hacked and whether the traffic is altered due to some kind of local network (neighbor) spoofing.

For the server, we tested:

But nothing stood out. We noticed the only uncommon record in the kernel log on Hetzner dedicated machine, it lost the physical network link for 19 seconds on 18 July 2023:

In addition, Linode machines are only used as a tunnel to Hetzner server as an alternative route, and are running only bare-bones basic services like SSH and NTP. Yet we still see unaltered ServerHello packet from the Hetzner server in Hetzner↔Linode tunnel on Linode machine, which is alerted upon sending it to the client over Linode network interface. In other words, the encrypted connection interception of the same kind happens both on Hetzner dedicated and Linode VPS machines.

Note kernel memory dump Linode VPS are running Linode-compiled kernel which is booted by the hypervisor and is not present in the VM image. The source code of the kernel is not distributed by Linode. Luckily, there’s embedded build configuration in /proc/config.gz , and the kernel is built with full symbols map CONFIG_KALLSYMS_ALL=y , which could be grabbed from /proc/kallsyms and is sufficient for dump analysis.

From the network perspective, we tested:

No sights of any hijacking by the network segment neighbors on either servers, nothing out of ordinary in the network configuration. While it’s still not uncommon to see insufficient spoofing protection on physical networks for dedicated servers, it’s hard to believe that such sophisticated stealth spoofing is possible on a virtual infrastructure.

On the Linode machines there aren’t any neighbors at all except the router. Affected Linode machines are unable to ARP-discover or ping any adjacent IP addresses in the same network segment which are perfectly functioning from the outside network. "That’s weird", I thought.

The unaffected (third-party) Linode VM can ping its neighbors. It is connected to the Cisco systems router:

However the affected VPS are connected to some unidentified MAC manufacturer:

It became clear that the affected Linode VM have uncommon network setup compared to regular Linode instances, possibly have been isolated into separate VLAN.

After checking crt.sh certificate transparency database , rogue certificates have been discovered which were not issued by any of jabber.ru servers.

There are two genuine certificates used in XMPP service: the one issued for xmpp.ru, *.xmpp.ru and the other one is for jabber.ru, *.jabber.ru .

The maliciously-issued certificates are slightly different from the regular ones for these domains: either the wildcard Subject Alternative Name is missing or a single certificate is issued for both jabber.ru, xmpp.ru . Moreover, MiTM configuration on xmpp.ru domain (which points to Linode servers) was slightly misconfigured: it serves only xmpp.ru certificate, yet the original server is configured to serve both jabber.ru and xmpp.ru certificates depending on requested XMPP domain.

List of rogue certificates:

18 July 2023 issuing time is the same when Hetzner server has lost network link for several seconds.

We have a confirmation from the external network scanner that Linode servers started to serve 04:b7:85… certificate on port 5222 since at least 21 July 2023. Unfortunately, this scanner doesn’t process Hetzner ranges.

We decided to run more network tests of Linode VPS from outside of the machine.

All adjacent Linode hosts are reachable by hop 13 from my laptop’s internet connection.

# lft -d 22 172.104.226.23 Tracing ............*T TTL LFT trace to 172-104-226-23.ip.linodeusercontent.com (172.104.226.23):22/tcp 1 _gateway (192.168.100.1) 31.2ms … 9 a23-210-52-59.deploy.static.akamaitechnologies.com (23.210.52.59) 67.0ms 10 10.210.32.1 62.1ms ** [neglected] no packets received from TTL 11 12 10.210.3.93 67.5ms 13 [target open] 172-104-226-23.ip.linodeusercontent.com (172.104.226.23):22 67.4ms As well as hijacked port 5222 on dawn.jabber.ru Linode server:

# lft -d 5222 dawn.jabber.ru Tracing ............**.T TTL LFT trace to dawn.jabber.ru (172.104.226.29):5222/tcp 1 _gateway (192.168.100.1) 29.3ms … 9 a23-210-52-57.deploy.static.akamaitechnologies.com (23.210.52.57) 64.9ms 10 10.210.32.2 64.7ms ** [neglected] no packets received from TTL 11 12 10.210.1.235 60.7ms 13 [target open] dawn.jabber.ru (172.104.226.29):5222 60.2ms However all other non-hijacked ports, such as SSH port 22, is reachable only via additional non-disclosed hop and now reachable only on hop 14:

# lft -d 22 dawn.jabber.ru Tracing ............?*?**.?T TTL LFT trace to dawn.jabber.ru (172.104.226.29):22/tcp 1 _gateway (192.168.100.1) 28.8ms … 9 a23-210-52-57.deploy.static.akamaitechnologies.com (23.210.52.57) 65.4ms 10 10.210.32.2 66.2ms ** [neglected] no packets received from TTL 11 12 10.210.1.235 61.7ms ** [neglected] no packets received from TTL 13 14 [target open] dawn.jabber.ru (172.104.226.29):22 61.1ms This means that the IP address of Linode VM have been placed behind additional machine which handles TCP port 5222 by itself and routes other ports to the real destination.

From inside the VPS it’s not possible to establish new connections using source port 5222. The router does not for the packets originating from this source port, as well as doesn’t send any ICMP error or Time-to-Live Exceeded for TTL=0 or TTL=1 outgoing packets.

Using testssl.sh SSL/TLS testing script, it was discovered that the intercepting proxy accepts connections with anonymous ciphers on both Linode (xmpp.ru) and Hetzner (jabber.ru). This is a rare (mis)configuration which could be used as one of the XMPP MiTM detection fingerprint. Regular SSL/TLS libraries are usually compiled without anonymous ciphers support, you won’t be able to configure your service to use anonymous ciphers even if you explicitly allow it in configuration file most of the time, and it is confirmed to be not configured on the original server.

We have tested 20 other popular XMPP services for the same MiTM method with the same testssl.sh command and discovered no anomalies (no support for Anonymous NULL Ciphers).

All jabber.ru and xmpp.ru communications between these dates should be assumed compromised. Given the nature of the interception, the attacker have been able to execute any action as if it is executed from the authorized account, without knowing the account password. This means that the attacker could download account's roster, lifetime unencrypted server-side message history, send new messages or alter them in real time.

End-to-end encrypted communications, such as OMEMO, OTR or PGP, are protected from the interception only if both parties have validated the encryption keys. The users are asked to check their accounts for new unauthorized OMEMO and PGP keys in their PEP storage, and change passwords.

We tend to assume this is lawful interception Hetzner and Linode were forced to setup based on German police request. Another possible, although much more unlikely scenario is an intrusion on the internal networks of both Hetzner and Linode targeting specifically jabber.ru — much harder to believe but not entirely impossible.

As of 20 Oct 2023, we’re still waiting for the adequate from Hetzner and Linode to our inquiries.

There are several indications which could be used to discover the attack from day 1:

More recommendations from ACME developer Hugo Landau .

We haven't received any proper from both Hetzner and Linode regarding this incident since 20 Oct 2023. The inquiries were sent as a dedicated tickets with all the evidence information, written in a clear and easy for comprehension manner to every support level staff, with the following wording (snippet):

The only received by Hetzner is: