A Cautionary Tale About Data Breach Claims Verification And Carhartt

You're not going to believe this, but turns out you can't always take criminals at their word. Actually, I'll walk that back a bit as it may not even be the cybercrime guys who got this wrong, but it all starts here:

🚨Cyber Alert ‼️ 🇺🇸US - 𝗖𝗮𝗿𝗵𝗮𝗿𝘁𝘁 ShinyHunters hacking group claims to have compromised Carhartt and allegedly stolen more than 50 GB of compressed data containing millions of customer records, employee information, customer metadata, loyalty-related information, and… pic.twitter.com/BvzsEQS54l

My usual Have I Been Pwned (HIBP) workflow after a breach emerges is to run our open source Email Address Extractor tool over the corpus of data, get a count of the unique addresses then drop it into the processing pipeline. Having a count helps prioritise: it's roughly the same effort to process a breach with 10M addresses in it as it is 100k addresses, but we obviously reach 100x more people with the larger corpus so that goes first. So, following the release of the Carhartt data above, I ran the tool and got 24,876,077 addresses. That's a biggy, let's dive in.

One of the first things I do when analysing a breach is look for news and industry commentary. Once you get through all the results for class actions ( ugh ), you usually find various other services and news sites reporting on the incident. One such example (which I won't name, because they're wrong) cited a very similar number to my initial pass, which at face value, reinforced my thinking. But here's where it all takes a turn because, inevitably, this is where the other analysts stopped...

I've been using OpenClaw pretty extensively this year to help with a number of tasks, including breach analysis. Here's how I normally begin (timestamps are AEST and yes, I called it "PwnedClaw"):

Ok, cool, nothing too revolutionary there. Let's start analysing the data, and the first thing I do is begin with the report of our open source Email Address Extractor tool, which lists every file an address was found in, and then how many unique ones are in there:

Think of it as using the AI to distil large volumes of information down into something much more workable. In this case, I gave it four of the files in question and let the analysis run:

I'll save you from all the to-and-fro of my questions and PwnedClaw's responses, here's where it gets interesting and where AI plays a genuinely valuable role:

Huh, yeah, that feels odd. Let's drill down further, and just a quick caveat: I'm highly confident that all the data I'm to show is fabricated and that I'm not inadvertently putting someone else's PII on display here.

Oh yeah, that's definitely junk. It was enough for me to start trawling through the data myself, old-school, like I used to, well, before earlier this year! And that's where I found data like this:

It's the same sort of .edu pattern but not on a .edu TLD. First name, last name, random domain name. And "Benin"?! I'll save you Googling it like I had to, it's a small country in West Africa and if I'm guessing, not a major Carhartt market.

And the more I probed, the more it all unravelled:

Annoying AI sentence structure aside ("that's not [one thing], that's [this other thing]"), this is just gold. And PwnedClaw gives ShinyHunters a bit of a free pass too: these guys just grabbed and published everything without realising there was a bunch of synthetic data in there. But there's one more smoking gun:

It would have been so easy just to dump this all into HIBP, put out social posts with a big headline number and be done with it, but it'd also be completely wrong. And here's what's really worrying watching other people do exactly this: headline numbers get misrepresented, victim organisations are unfairly reported on (news headlines with the HIBP number often follow quickly), I run the risk of attracting the ire of the company for inaccurate reporting, and the whole thing just becomes a mess.

So, was Carhartt actually breached, and did ShinyHunters fabricate the data?

Almost certainly "yes" and then, almost certainly "no", and both positions are supported by ShinyHunters form. These guys are approaching 100 breaches sitting on their Tor site (all of which I've reviewed), and I'm yet to see a single one with data they've fabricated. These guys have a track record of successfully gaining access to large orgs and dumping data that genuinely came from the claimed origin. Not to trivialise their criminal extortion campaigns, but misrepresenting the scope of their activities would be bad for business in the same way that not following through on threats and dumping data when they don't get payment would be. This presents an alternate theory PwnedClaw has already touched on: Carhartt simply had a bunch of test data sitting in their Databricks instance.

Going back to the output of our tool, here are the largest files which we now know have junk data in them:

There were 600 of these files, then another 1,200 files with similar data in this common path:

Once excluded, the unique address count plummeted to just 13,306,258 results. That's a 47% drop from the 24M+ number we started at... and it's still too high.

I always eyeball the distinct list of email addresses after they're exported by our tooling. Call me old-fashioned, but I still reckon there are things I'll see with my experienced human eyes that the AI will miss (at least, for now). And it missed these:

I've previously removed these manually by regex'ing them out myself, but let's not repeat laborious tasks that can be automated away:

Continuing the eyeballing work, another pattern emerges:

That's enough of human eyes, what else have we got?

Not quite the 24M+ headline any more, that's almost half the addresses in the original corpus discarded because they don't represent actual people. Whilst nothing empirically bad would have happened if I'd loaded them all, the incident would have looked twice as bad as what it actually is (based on the raw numbers, at least), and when I later put out a like this, the percentage of pre-existing addresses would have read way too low:

But let's do one more AI-assisted sanity check, and I'm tending to ask this on each and every breach I process now:

Of course, anything AI says needs to be validated and in the cases above, that had already been done. I've included that transcript here anyway just to illustrate the due diligence involved in properly validating data, not just picking it up and chucking it in with a massive headline figure. But it does massively accelerate activities that were previously very manual:

And so on and so forth. I don't need to labour the point with more examples, but let me come back around to the premise of this post: take headline numbers with a grain of salt unless you're confident in the processes of those making the claims. I'm fond of saying that whenever there are misleading statements made breaches, "the truth is in the data", but you have to be willing to go and seek that truth out, even if it requires a significant investment in time.