Looking Under Every Log
They say nothing on the internet is ever truly gone, and Truffle Security has recently proven the adage to be true. They have recovered and scanned 600,000 PyPI package releases with TruffleHog, discovering 190 unique secrets, including a GitHub Personal Access Token (PAT) with admin access to the Apache and Astronomer organizations. The discovery began with finding a GitHub PAT in a package from April 2023. It grants admin access to the Apache and Astronomer GitHub organizations, and was still active, along with some – thankfully – now invalid AWS tokens. They disclosed both findings to the appropriate hosts, who revoked them within hours.
So how does this happen, and why? Truffle tested the process to understand exactly what’s happening when a package is deleted. What they found has been dubbed ‘ghost persistence’. When a user deletes a package, the registry’s database removes the pointer to the website and API, and a 404 error code will pop up when anyone tries to download it from PyPI.. The file itself, however, stays where it was: on the Python Package Index official object storage server. Which means that the secrets it holds do too. PyPI’s policy on hard deletion is reserved only for when Personally Identifiable Information (PII) is leaked. If one has the full URL and tarball of a package, it can still be found.
That said, gaining that information runs along the lines of ‘you have to know it’s there’; it’s not something one might stumble across organically. Generally speaking, if one tried to download a deleted package and saw the error code, they’d assume that package is gone. Still, it is a vulnerability in developer ecosystems. Truffle says devs should not consider deletion a security control, and that once a secret touches a public ecosystem, it should be considered compromised.
The URL is only part of the equation, though. Object storage is a hash-based directory structure. Leveraging PyPI’s metadata to reconstruct the path is possible. PyPI maintains a public BigQuery dataset containing metadata for every package ever uploaded, one assumes for uniqueness in naming and filing. Querying that dataset can be used to discover the full URL of every package contained in it. Filtering through a python script for availability narrows the field to the ones that have been ‘deleted’, which still evidently contain their active tokens, keys and/or secrets.
And what kinds of secrets are we talking about? Truffle breaks down the variety they found into several categories, nearly half of which were present in every version and some contained multiples. These included Discord Webhooks, Telegram bot tokens, Infura, AWS and Slack Webhooks. All of which would be valuable resources for malicious infiltration, exploitation and selling on the dark web. And these are just the most common secrets found, it’s not an exhaustive list.
This is not a new issue. The oldest one was from 2018, containing a still live Telegram token, while the newest was from January of this year, containing a still valid Postgres credentials. Truffle’s article has some tips for checking if secrets are being leaked, and how to remediate the issue. Being aware that this issue exists, however, is probably the biggest key to preventing it from being abused. Rotating credentials on a regular basis would make these secrets more inaccessible, or at least invalid in terms of exploiting them. Short of that, revocation of these tokens as soon as they hit a public ecosystem is the most reliable way to keep secrets secret.
Posted, 6/3/26
